adp.ingest.sources.azure_files.AzureFilesEntitySource

class adp.ingest.sources.azure_files.AzureFilesEntitySource(entity: EntityType, ingest: Ingest)

Represents a source to a Azure Files via the REST API

__init__(entity: EntityType, ingest: Ingest)

Methods

__init__(entity, ingest)
all_files_in_staging([abfs_path])	all_files_in_staging retrieves all files in the staging directory for the given entity
download_files(files)	download_files function reads file from Azure File and downloads the corresponding files from the list of paths
files_matching_patterns(input_paths, pattern)	Retrieve all files corresponding to the pattern string
get_data(run_metadata)	Retrieves the data for the staging sources
remove_temp_files()	Removes the temporary files
scan_files(path[, glob_pattern])	scan_files function scans all files in the folder and retrieves its path,
unzip_files()	Unzips all files in the staging folder
upload_to_staging()	upload_to_staging downloads files from Azure Files and uploads it to the staging folder

Attributes

path	Retrieves the path for the staging folder
source

download_files(files: List[str])

download_files function reads file from Azure File and downloads the corresponding files from the list of paths

Parameters:

files (List[str]) – path of files

Yields:

file name and file as – file_name, file_bytes

scan_files(path: str, glob_pattern: str = '*') → List[str]

scan_files function scans all files in the folder and retrieves its path,

Returns:

returns a list of paths and file names

Return type:

List[str]

upload_to_staging()

upload_to_staging downloads files from Azure Files and uploads it to the staging folder