Federation for DSI
==================

This allows users to pull databases from many different locations and provide a centralized view of the databases. To run, from the DSI folder:

To run::

    python tools/federated/federate_datasets.py tools/federated/input.yaml

where ``input.yaml`` is a config file.


Config file
-----------

``input.yaml`` is the config file that contains the paths from which data can be pulled::

    repo_paths:
        - "~/remote_sources.csv"
        - "local_dsi_sources.csv"
    workspace_folder: "dsi_databases_01"
    download_limit: 10485760  # 10 MB

- ``repo_paths``: repo_paths: points to CSV files where the user can specify DSI repos. The paths should be relative to the config file or absolute paths
- ``workspace_folder``: where the remote federated datasets will be stored; those local to your computer will not be moved
- ``download_limit``: after this file limit, the user will be asked to confirm


Data Sources
------------

An example of database sources is as follows::

    location_type,location,path,type,submitter_name,submitter_email,timsestamp
    local,local,tools/federated/database/ocean_11_datasets.db,data,pascal grosset,pascalgrosset@lanl.gov,2026-2-10--16:40:00s
    local,local,/home/pascalgrosset/data/artimis/fracture/3d/aleks/fracture_aleks.sqlite,data,pascal grosset,pascalgrosset@lanl.gov,2026-2-10--16:40:00s
    HPC,ch-fe.lanl.gov,/lustre/scratch5/pascalgrosset/test_db/nif.db,data,pascal grosset,pascalgrosset@lanl.gov,2026-3-10--16:38:00
    url,url,https://www.timestored.com/data/sample/sakila.db,data,unknown,unknown,2026-3-10--16:38:00

**Note:** only ``location_type``, ``location``, and ``path`` are required

- ``location_type``: the currently supported values are:

  - ``local``: refers to your local computer
  - ``HPC``: refers to a supercomputer
  - ``url``: a file on the web
  - ``github``: a file on a GitHub repository

- ``location``: on HPC systems, indicates the name of the cluster the data is on. For the others, it is the same as ``location_type``
- ``path``: the path of the dataset


Other Notes
-----------

- Local repositories will not be downloaded; there will just be a reference to them
- A file called ``dsi_database_list.json`` will be created that shows all the paths of the downloaded files