adp.ingest.sources.smb.SMBEntitySource

class adp.ingest.sources.smb.SMBEntitySource(entity: EntityType, ingest: Ingest)

Represents a source to a Windows Fileshare

__init__(entity: EntityType, ingest: Ingest)

Methods

__init__(entity, ingest)
all_files_in_staging([abfs_path])	all_files_in_staging retrieves all files in the staging directory for the given entity
download_files(files)
files_matching_patterns(input_paths, pattern)	Retrieve all files corresponding to the pattern string
filter_all_files(files, glob_pattern)	filter_all_files matching files with patterns.
get_data(run_metadata)	Retrieves the data for the staging sources
remove_temp_files()	Removes the temporary files
scan_files([scan_subfolders])
unzip_files()	Unzips all files in the staging folder
upload_to_staging()	upload_to_staging downloads files from the SMB drive and uploads it to the staging folder

Attributes

path	Retrieves the path for the staging folder
source

filter_all_files(files: List[str], glob_pattern: str) → List[Path]

filter_all_files matching files with patterns. Test whether FILENAME matches PATTERN. Patterns are Unix shell style

Parameters:

• files (List[str]) – a list of files converted in a string

• glob_pattern (str) – a list from those elements of the iterable NAMES that match PAT.

Returns:

files str: pattern

Return type:

str

upload_to_staging()

upload_to_staging downloads files from the SMB drive and uploads it to the staging folder

The lock is used for the SMBClient.register_session which registers a GLOBAL smb session. With a lock we make sure that only a single thread modifies the smb session. The with statement will cause the lock to be released in any case (also after an exception)