Sync

The DSI Sync middleware defines the Sync concepts. An instantiated Sync supports data movement capabilities between local and remote locations and captures metadata documentation

The purpose of Sync is to provide file metadata documentation and data movement capabilities when moving data to/from local and remote locations. The purpose of data documentation is to capture and archive metadata (i.e. location of local file structure, their access permissions, file sizes, and creation/access/modification dates), and track their movement to the remote location for future access.

The functions, index, copy, and move serve as mechanisms to index data for documentation, and copy/move data by creating a DSI database using Terminal. The function get serves as a mechanism to retrieve data from remote locations by downloading an existing DSI database that contains the pointers to the target data.

class dsi.sync.Sync(project_name=None, isVerbose=False, no_parent=False, **kwargs)

A class defined to assist in data management activities for DSI

Sync is where data movement functions such as copy (to remote location) and sync (local filesystem with remote) exist.

__init__(project_name=None, isVerbose=False, no_parent=False, **kwargs)

change_group(local_loc, user_group)

Change group permissions for data and db. Only works for OS with Unix (not Windows)

change_permissions(local_loc)

Change read permissions for data and db. Only works for OS with Unix (not Windows)

copy(tool='copy')

Helper function to perform the data copy over using a preferred API

dircrawl(filepath, verbose=False)

Crawls the root ‘filepath’ directory and returns files

filepath: source filepath to be crawled

return: returns crawled file-list

execute_cmd(cmd, cmd_name, timer=False)

Internal helper for Sync to call executable actions

gen_uuid(st)

Generates a unique file identifier using the os.stat data object as the input

get(input_yaml=None, workspace_folder=None)

Helper function that searches remote location based input yaml file, and retrieves metadata that contains DSI databases

index(local_loc, remote_loc)

Helper function to gather filesystem information, local and remote locations to create a filesystem entry in a new or existing database