Custom DSI Reader

DSI Readers are the primary way to convert outside data to metadata that DSI can ingest, and they must must include 2 methods, __init__, and add_rows.

Loading Custom Reader into DSI

Before explaining the structure of Readers, it is important to note there are two ways to load your Reader, externally and internally.

  • If your Reader is intended for use within your own code base and not added to DSI’s modules or for public use, you can load it externally. Doing so allows you to store your Reader separately from DSI yet compatible with all versions of DSI.

    • With the Core.Terminal.add_external_python_module method, you can make your Reader temporarily accessible to DSI in a workflow and load normally.

    • This example can be better seen at Example 7: External Readers/Writers where you can try loading an external TextFile reader class.

  • If you want your Reader loadable internally with the rest of the provided implementations (in dsi/plugins), it must be registered in the VALID_READERS class variable of Terminal in dsi/core.py. If this is done correctly, your Reader will be loadable by the load_module method of Terminal.

Initializer: __init__(self) -> None:

__init__ is where you can include all of your initialization logic, and specify the parameters needed for a given application.

Example __init__:

def __init__(self, filenames) -> None:
  # see "plugins" to determine which superclass your Reader should extend
  super().__init__()

  # allow users to read multiple files at once, or just one file at a time
  if isinstance(filenames, str):
      self.filenames = [filenames]
  else:
      self.filenames = filenames

  # data structure to load data into that is compatible with DSI
  self.data_dict = OrderedDict()

Add Rows: add_rows(self) -> None

add_rows is responsible for appending to the internal DSI metadata abstraction. This function should ensure the data that is loaded is in the form of an OrderedDict (the internal DSI data structure).

After converting all data to be in an Ordered Dictionary, users must call set_schema_2() to assign the data to the internal DSI abstaction layer. You can pass data through set_schema_2(self, collection) -> None by using the collection variable, assuming your data is an OrderedDict.

If you have multiple tables of data loaded at once, you can create a nested OrderedDict. In this case, each table’s data is still an OrderedDict and is now a value in a larger OrderedDict whose keys are each table’s name. Ex: OrderedDict( table1: OrderedDict(), table2: OrderedDict() )

add_rows example:

def add_rows(self) -> None:

  # data is stored as an OrderedDict so use set_schema2
  my_data = OrderedDict()
  my_data["jack"] = 10
  my_data["joey"] = 20
  my_data["amy"] = 30

  self.set_schema_2(my_data)

Contributing Your Reader

If your Reader is helpful and acceptable for public use, you should consider making a pull request (PR) into DSI.

Please note that any accepted PRs into DSI should satisfy the following:

  • Passes all tests in dsi/plugins/tests

  • Has no pylama errors/warnings (see dsi/.githooks)

Examples

Full Reader examples in-code, can be found in dsi/plugins/file_reader.py. Csv is an especially simple example to view for loading one table. YAML1 and TOML1 are more complex examples with loading multiple tables of data with units