Sync

The DSI Sync middleware defines the Sync concepts. An instantiated Sync supports data movement capabilities between local and remote locations and captures metadata documentation

The purpose of Sync is to provide file metadata documentation and data movement capabilities when moving data to/from local and remote locations. The purpose of data documentation is to capture and archive metadata (i.e. location of local file structure, their access permissions, file sizes, and creation/access/modification dates), and track their movement to the remote location for future access.

The functions, index, copy, and move serve as mechanisms to index data for documentation, and copy/move data by creating a DSI database using Terminal. The function get serves as a mechanism to retrieve data from remote locations by downloading an existing DSI database that contains the pointers to the target data.

class dsi.sync.Sync(project_name, isVerbose=False, no_parent=False, skip_index=False, **kwargs)

A class defined to assist in data management activities for DSI

Sync is where data movement functions such as copy (to remote location) and sync (local filesystem with remote) exist.

__init__(project_name, isVerbose=False, no_parent=False, skip_index=False, **kwargs)

change_group(local_loc, user_group)

Change group permissions for data and db. Only works for OS with Unix (not Windows)

change_permissions(local_loc)

Change read permissions for data and db. Only works for OS with Unix (not Windows)

copy(tool='copy')

Helper function to perform the data copy over using a preferred API

dircrawl(filepath, verbose=False)

Crawls the root ‘filepath’ directory and returns files

filepath: source filepath to be crawled

return: returns crawled file-list

execute_cmd(cmd, cmd_name, timer=False)

Internal helper for Sync to call executable actions

gen_uuid(st)

Generates a unique file identifier using the os.stat data object as the input

get(config_file: str, workspace_folder: str)

Helper function that searches remote location-based input config file, and retrieves metadata that contains DSI databases

gufi_query_index(gufi_prefix, gufi_index, db_path, dsi_table_name, dsi_columns, gufi_columns, collection_name, custom_query=None, isVerbose=False)

Helper function to query GUFI for metadata and join on the dsi table by uuid

gufi_prefix: the directory where GUFI is installed

gufi_index: the directory where GUFI indexes are

db_path: the full path of the dsi database

dsi_table_name: the DSI table name that has the UUID for each file as a column

dsi_columns: the DSI table columns that should be included in the join with GUFI

gufi_columns: the GUFI columns that should be included in the join with DSI

collection_name: the name that identifies the collection that the DSI database belongs to

custom_query: an sql string to query the dsi and gufi index

index(local_loc, remote_loc)

Helper function to gather filesystem information, local and remote locations to create a filesystem entry in a new or existing database