pyDRESCALk package

Submodules

pyDRESCALk.config module

pyDRESCALk.config.init(arg)

Global variables declaration here. The variables declared within this function in this file are shared across other files and functions during import.

pyDRESCALk.data_generator module

class pyDRESCALk.data_generator.data_generator(args)

Bases: object

Generates synthetic data in distributed manner where each MPI process generates a chunk from the data parallelly. The W matrix is generated with gaussian distribution whereas the H matrix is random.

Parameters

• args (class) -- Class which comprises following attributes

• fpath (str) -- Directory path of file to be stored

• p_r (int) -- Count of row processor in the cartesian grid

• p_c (int) -- Count of column processor in the cartesian grid

• m (int) -- row dimension of the data

• n (int) -- Column dimension of the data

• k (int) -- Feature count

create_folder_dir(fpath)

Create a folder if doesn't exist

determine_block_index_range_asymm()

Determines the start and end indices for the Data block for each rank

determine_block_shape_asymm()

Determines the shape for the Data block for each rank

dist_fromfunction(func, shape, pgrid, *args, unravel_index=<function unravel_index>, **kwargs)

produces X_{i,j} = func(i,j) in a distributed manner, so that each processor has an array_split section of X according to the grid.

fit()

generates and save factors

gauss_matrix_generator(n, k, axis=0)

Construct a matrix of dimensions n by k where the ith column is a Gaussian kernel corresponding to approximately N(i*n/k, 0.01*n^2)

Parameters

• n (int) -- the ambient space dimension

• k (int) -- the latent space diemnsion

Returns

W -- A matrix with Gaussian kernel columns of size n x k.

Return type

ndarray

generate_factors_data()

Generates the chunk of factors W,H and data X for each MPI process

random_matrix_generator(k, n, seed)

Generator for random matric with given seed

unravel_column()

finds the column rank for 2d grid

unravel_row()

finds the row rank for 2d grid

pyDRESCALk.data_generator.parser()

Reads the input arguments from the user and parses the parameters to the data generator module.

pyDRESCALk.data_io module

class pyDRESCALk.data_io.data_read(args)

Bases: object

Class for reading data.

Parameters

• args (class) -- Class which comprises following attributes

• fpath (str) -- Directory path of file to be read

• pgrid (tuple) -- Cartesian grid configuration

• ftype (str) -- Type of data to read(mat/npy/csv/folder)

• fname (str) -- Name of the file to read

• comm (object) -- comm object for distributed read

data_partition()

This function divides the input matrix into chunks as specified by grid configuration.

Return n array of shape (nrows_i, ncols_i) where i is the index of each chunk. Sum_i^n ( nrows_i * ncols_i ) = arr.size

If arr is a 2D array, the returned array should look like n subblocks with each subblock preserving the "physical" layout of arr.

read()

Data read function

read_dat()

Function for reading the data and split into chunks to be reach by each MPI rank

read_file_csv()

CSV data read function

read_file_mat()

mat file read function

read_file_npy()

Numpy data read function

read_file_pickle()

save_data_to_file(fpath)

This function saves the splitted data to numpy array indexed with chunk number

class pyDRESCALk.data_io.data_write(args)

Bases: object

Class for writing data/results.

Parameters

• args (class) -- class which comprises following attributes

• results_path (str) -- Directory path of file to write

• pgrid (tuple) -- Cartesian grid configuration

• ftype (str) -- Type of data to read(mat/npy/csv/folder)

• comm (object) -- comm object for distributed read

create_folder_dir(fpath)

Create directory if not present

save_cluster_results(params)

Save cluster results to a h5 file with rank 0

save_factors(factors, reg=False)

Save the W and H factors for each MPI process

class pyDRESCALk.data_io.read_factors(factors_path, pgrid)

Bases: object

Class for reading saved factors.

Parameters

• factors_path (str) -- Directory path of factors to read from

• pgrid (tuple) -- Cartesian grid configuration

custom_read_npy(fpath)

Read numpy files

load_factors()

Load the final stacked factors for visualization

read_factor(fpath)

Read factors as chunks and stack them

class pyDRESCALk.data_io.split_files_save(data, pgrid, fpath)

Bases: object

Rank 0 based data read, split and save

save_data_to_file()

Function to save the chunks into numpy files

split_files()

Compute the index range for each block and partition the data as per the chunk

pyDRESCALk.dist_clustering module

class pyDRESCALk.dist_clustering.custom_clustering(Wall, Hall, params)

Bases: object

Greedy algorithm to approximate a quadratic assignment problem to cluster vectors. Given p groups of k vectors, construct k clusters, each cluster containing a single vector from each of the p groups. This clustering approximation uses cos distances and mean centroids.

Parameters

• A_all (ndarray) -- Order three tensor of shape m by k by p, where m is the ambient dimension of the vectors, k is the number of vectors in each group, and p is the number of groups of vectors.

• R_all (ndarray) -- Order three tensor of shape n by k by p, where n is the ambient dimension of the vectors, k is the number of vectors in each group, and p is the number of groups of vectors.

• params (class) -- Class object with communication parameters which comprises of grid information (p_r,p_c) , commincator (comm) and epsilon (eps).

change_order(tens)

change the order of features

dist_custom_clustering(centroids=None, vb=0)

Performs the distributed custom clustering

Parameters

• centroids (ndarray, optional) -- The m by k initialization of the centroids of the clusters. None corresponds to using the first slice, A_all[:,:,0], as the initial centroids. Defaults to None.

• vb (bool, optional) -- Verbose to display intermediate results

Returns

• centroids (ndarray) -- The m by k centroids of the clusters

• A_all (ndarray) -- Clustered organization of the vectors A_all

• R_all (ndarray) -- Clustered organization of the vectors R_all

• permute_order (list) -- Indices of the permuted features

dist_feature_ordering(centroids, W_sub)

return the features in proper order

dist_silhouettes()

Computes the cosine distances silhouettes of a distributed clustering of vectors.

Returns

sils -- The k by p array of silhouettes where sils[i,j] is the silhouette measure for the vector A_all[:,i,j]

Return type

ndarray

fit()

Calls the sub routines to perform distributed custom clustering and compute silhouettes

Returns

• centroids (ndarray) -- The m by k centroids of the clusters

• CentStd (ndarray) -- Absolute deviation of the features from the centroid

• A_all (ndarray) -- Clustered organization of the vectors A_all

• R_all (ndarray) -- Clustered organization of the vectors R_all

• S_avg (ndarray) -- mean Silhouette score

• permute_order (list) -- Indices of the permuted features

greedy_lsa(A)

Return the permutation order

mad(data, flag=1, axis=-1)

Compute the median/mean absolute deviation

normalize_by_A()

Normalize the factors A and R

pyDRESCALk.dist_comm module

class pyDRESCALk.dist_comm.MPI_comm(comm, p_r, p_c)

Bases: object

Initialization of MPI communicator to construct the cartesian topology and sub communicators

Parameters

• comm (object) -- MPI communicator object

• p_r (int) -- row processors count

• p_c (int) -- column processors count

Free()

Frees the sub communicators

cart_1d_column()

Constructs a cartesian column communicator through construction of a sub communicator across columns

Returns

cartesian1d_column -- Sub Communicator object

Return type

object

cart_1d_row()

Constructs a cartesian row communicator through construction of a sub communicator across rows

Returns

cartesian1d_row -- Sub Communicator object

Return type

object

pyDRESCALk.dist_rescal module

class pyDRESCALk.dist_rescal.rescal_algorithms_2D(X_ijk, A_i, A_j, R_ijk, params=None)

Bases: object

Performs the distributed RESCAL operation along 2D cartesian grid

Parameters

• X_ijk (ndarray) -- Distributed Data

• A_ij (ndarray) -- Distributed factor A

• R_ijk (ndarray) -- Distributed factor R

• params (class) -- Class which comprises following attributes

• params.comm1 (object) -- Global Communicator

• params.comm (object) -- Modified communicator object

• params.k (int) -- Rank for decomposition

• params.m (int) -- Global dimensions m

• params.n (int) -- Global dimensions n

• params.p_r (int) -- Cartesian grid row count

• params.p_c (int) -- Cartesian grid column count

• params.row_comm (object) -- Sub communicator along row

• params.col_comm (object) -- Sub communicator along columns

• params.W_update (bool) -- flag to set W update True/False

• params.norm (str) -- NMF norm to be minimized

• params.method (str) -- NMF optimization method

• params.eps (float) -- Epsilon value

Fro_MU_update(A_update=True)

Frobenius norm based multiplicative update of A and R parameter Function computes updated A and R parameter for each mpi rank

Parameters

self (object) --

Returns

• self.A_i (ndarray)

• self.R_ijk (ndarray)

column_broadcast(A)

Performs all reduce along column sub communicator

column_mm(A, B)

Distributed matrix multiplication along column of matrix

Computes the matrix multiplication of matrix A and B along column sub communicator .. math:: AB

Parameters

• A (ndarray) --

• B (ndarray) --

Returns

AB_glob

Return type

ndarray

column_reduce(A)

Performs all reduce along column sub communicator

element_op(A, B, operation)

Performs Element operations between A and B

global_gram(A)

Distributed gram computation

Computes the global gram operation of matrix A .. math:: A^TA

Parameters

A (ndarray) --

Returns

A_TA_glob

Return type

ndarray

gram_mul(A)

Computes the gram operation of matrix A

matrix_mul(A, B)

Computes the matrix multiplication of matrix A and B

row_broadcast(A)

Performs broadcast along row sub communicator

row_mm(A, B)

Distributed matrix multiplication along row of matrix

Computes the matrix multiplication of matrix A and B along row sub communicator .. math:: AB

Parameters

• A (ndarray) --

• B (ndarray) --

Returns

AB_glob

Return type

ndarray

row_reduce(A)

Performs all reduce along row sub communicator

update()

Performs 1 step Update for factors W and H based on NMF method and corresponding norm minimization

Returns

• W_ij (ndarray) -- The m/p X k distributed factor W

• H_ij (ndarray) -- The k X n/p distributed factor H

pyDRESCALk.main module

pyDRESCALk.main.parser_pyRescal(parser)

pyDRESCALk.main.parser_pyRescalk(parser)

pyDRESCALk.plot_results module

pyDRESCALk.plot_results.box_plot(dat, respath)

Plots the boxplot from the given data and saves the results

pyDRESCALk.plot_results.plot_W(W)

Reads a factor and plots into subplots for each component

pyDRESCALk.plot_results.plot_err(err)

Plots the relative error for NMF decomposition as a function of number of iterations

pyDRESCALk.plot_results.plot_results(startProcess, endProcess, stepProcess, RECON, SILL_AVG, SILL_MIN, out_put, name)

Plots the relative error and Silhouette results for estimation of k

pyDRESCALk.plot_results.plot_results_paper(startProcess, endProcess, stepProcess, RECON, SILL_AVG, SILL_MIN, out_put, name, k=-1)

pyDRESCALk.plot_results.plot_timing_stats(fpath, respath)

Plots the timing stats for the MPI operation. fpath: Stats data path respath: Path to save graph

pyDRESCALk.plot_results.read_plot_factors(factors_path, pgrid)

Reads the factors W and H and Plots them

pyDRESCALk.plot_results.timing_stats(fpath)

Reads the timing stats dictionary from the stored file and parses the data.

pyDRESCALk.pyDRESCAL module

class pyDRESCALk.pyDRESCAL.pyDRESCAL(X_ijk, factors=None, save_factors=False, params=None)

Bases: object

Performs the distributed NMF decomposition of given matrix X into factors W and H

Parameters

• A_ij (ndarray) -- Distributed Data

• factors (tuple) -- Distributed factors W and H

• params (class) -- Class which comprises following attributes

• params.init (str) -- NMF initialization(rand/nnsvd)

• params.comm1 (object) -- Global Communicator

• params.comm (object) -- Modified communicator object

• params.k (int) -- Rank for decomposition

• params.m (int) -- Global dimensions m

• params.n (int) -- Global dimensions n

• params.p_r (int) -- Cartesian grid row count

• params.p_c (int) -- Cartesian grid column count

• params.row_comm (object) -- Sub communicator along row

• params.col_comm (object) -- Sub communicator along columns

• params.W_update (bool) -- flag to set W update True/False

• params.norm (str) -- NMF norm to be minimized

• params.method (str) -- NMF optimization method

• params.eps (float) -- Epsilon value

• params.verbose (bool) -- Flag to enable/disable display results

• params.save_factors (bool) -- Flag to enable/disable saving computed factors

compute_global_dim()

Computes global dimensions m and n from given chunk sizes for any grid configuration

dist_norm(X, proc=-1, norm='fro', axis=None)

Computes the distributed norm

fit()

Calls the sub routines to perform distributed NMF decomposition with initialization for a given norm minimization and update method

Returns

• W_i (ndarray) -- Factor W of shape m/p_r * k

• H_j (ndarray) -- Factor H of shape k * n/p_c

• recon_err (float) -- Reconstruction error for NMF decomposition

init_factors()

Initializes Rescal factors with rand/nnsvd method

normalize_features(Wall, Wall1, Hall)

Normalizes features Wall and Hall

relative_err()

Computes the relative error for NMF decomposition

pyDRESCALk.pyDRESCALk module

class pyDRESCALk.pyDRESCALk.pyDRESCALk(X_ijk, factors=None, params=None)

Bases: object

Performs the distributed RESCAL decomposition with custom clustering for estimating hidden factors k

Parameters

• A_ij (ndarray) -- Distributed Data

• factors (tuple) -- Distributed factors W and H

• params (class) -- Class which comprises following attributes

• params.init (str) -- RESCAL initialization(rand/nnsvd)

• params.comm1 (object) -- Global Communicator

• params.comm (object) -- Modified communicator object

• params.k (int) -- Rank for decomposition

• params.m (int) -- Global dimensions m

• params.n (int) -- Global dimensions n

• params.p_r (int) -- Cartesian grid row count

• params.p_c (int) -- Cartesian grid column count

• params.row_comm (object) -- Sub communicator along row

• params.col_comm (object) -- Sub communicator along columns

• params.A_update (bool) -- flag to set W update True/False

• params.norm (str) -- RESCAL norm to be minimized

• params.method (str) -- RESCAL optimization method

• params.eps (float) -- Epsilon value

• params.verbose (bool) -- Flag to enable/disable display results

• params.save_factors (bool) -- Flag to enable/disable saving computed factors

• params.perturbations (int) -- Number of Perturbations for clustering

• params.noise_var (float) -- Set noise variance for perturbing the data

• params.sill_thr (float) -- Set the sillhouette threshold for estimating K with p-test

• params.start_k (int) -- Starting range for Feature search K

• params.end_k (int) -- Ending range for Feature search K

fit()

Calls the sub routines to perform distributed RESCAL decomposition and then custom clustering to estimate k

Returns

nopt -- Estimated value of latent features

Return type

int

pyrescalk_per_k()

Performs RESCAL decomposition and clustering for each k to estimate silhouette statistics

class pyDRESCALk.pyDRESCALk.sample(data, noise_var, method, params, seed=None)

Bases: object

Generates perturbed version of data based on sampling distribution.

Parameters

• data (ndarray, sparse matrix) -- Array of which to find a perturbation.

• noise_var (float) -- The perturbation amount.

• method (str) -- Method for sampling (uniform/poisson)

• seed (float) -- Set seed for random data generation

fit()

Calls the sub routines to perform resampling on data

Returns

X_per -- Perturbed version of data

Return type

ndarry

poisson()

Resamples each element of a matrix from a Poisson distribution with the mean set by that element. Y_{i,j} = Poisson(X_{i,j}

randM()

Multiplies each element of X by a uniform random number in (1-epsilon, 1+epsilon).

pyDRESCALk.utils module

class pyDRESCALk.utils.comm_timing

Bases: object

Decorator class for computing timing for MPI operations. The class uses the global variables flag and time initialized in config file and updates them for each call dynamically.

Parameters

• flag (bool) -- if Set true, enables the decorator to compute the timings.

• time (dict) -- Dictionary to store timing for each function calls

class pyDRESCALk.utils.count_flops

Bases: object

class pyDRESCALk.utils.count_memory

Bases: object

class pyDRESCALk.utils.data_operations(data)

Bases: object

Performs various operations on the data

Parameters

data (ndarray) -- Data to operate on

commonFactors(intList)

cutZero(thresh=1e-08)

Prunes zero columns from the data

desampleT(factor, axis=0)

matSplit(name, p_r, p_c, format='npy')

primeFactors(n)

recZero(indexList)

remove_bad_factors(Wall, Hall, ErrTol, features_k)

class pyDRESCALk.utils.determine_block_params(comm, pgrid, shape)

Bases: object

Computes the parameters for each chunk to be read by MPI process

Parameters

• comm (object) -- MPI communicator object

• pgrid (tuple) -- Cartesian grid configuration

• shape (tuple) -- Data shape

determine_block_index_range_asymm()

Determines the start and end indices for the Data block for each rank

determine_block_shape_asymm()

Determines the shape for the Data block for each rank

pyDRESCALk.utils.norm(X, comm, norm=2, axis=None, p=-1)

Compute the data norm

Parameters

• X (ndarray) -- Data to operate on

• comm (object) -- MPI communicator object

• norm (int) -- type of norm to be computed

• axis (int) -- axis of array for the norm to be computed along

• p (int) -- Processor count

Returns

norm -- Norm of the given data X

Return type

float

class pyDRESCALk.utils.parse

Bases: object

Define a class parse which is used for adding attributes

pyDRESCALk.utils.str2bool(v)

Returns instance of string parameter to bool type

class pyDRESCALk.utils.transform_H_index(grid)

Bases: object

Collected H factors after MPI operation aren't aligned. This operation performs careful reordering of H factors such that the collected factors are aligned

rankidx2blkidx()

This is to transform the column index to rank index for H

transform_H_idx(rank)

This is to transform H based on new index

pyDRESCALk.utils.var_init(clas, var, default)

Checks if class attribute is present and if not, intializes the attribute with given default value

pyDRESCALk.version module

Module contents