Index

_

__init__() (adp.core.adls.ADLSClient method) (adp.core.backup.BackupClient method) (adp.core.backup.BackupStatus method) (adp.core.backup.BackupType method) (adp.core.backup.CustomCredential method) (adp.core.dbsql.DBSQLClient method) (adp.core.loganalytics.LogAnalyticsClient method) (adp.core.settings.ADLSSettings method) (adp.core.settings.BackupSettings method) (adp.core.settings.CoreSettings method) (adp.core.settings.DeliverySettings method) (adp.core.settings.EnvironmentEnum method) (adp.core.settings.IngestSettings method) (adp.core.settings.KeyVaultSettings method) (adp.core.settings.LayersEnum method) (adp.core.settings.LogAnalyticsSettings method) (adp.core.settings.Settings method) (adp.core.settings.SparkSettings method) (adp.delivery.dataclasses.Column method) (adp.delivery.dataclasses.DatabricksJob method) (adp.delivery.dataclasses.DatabricksNotebookSettings method) (adp.delivery.dataclasses.Expectation method) (adp.delivery.dataclasses.JobType method) (adp.delivery.dataclasses.JobWorkflow method) (adp.delivery.dataclasses.OnExpectationViolationAction method) (adp.delivery.dataclasses.OnTypeMismatch method) (adp.delivery.dataclasses.Relationship method) (adp.delivery.dataclasses.SDPTable method) (adp.delivery.dataclasses.Table method) (adp.ingest.cdc.TransformOptions method) (adp.ingest.dataclasses.AppendTypeEnum method) (adp.ingest.dataclasses.AzureDevOpsEntity method) (adp.ingest.dataclasses.AzureDevOpsSource method) (adp.ingest.dataclasses.AzureDevOpsSourceConfig method) (adp.ingest.dataclasses.AzureFilesEntity method) (adp.ingest.dataclasses.AzureFileSourceConfig method) (adp.ingest.dataclasses.AzureFilesSource method) (adp.ingest.dataclasses.DataType method) (adp.ingest.dataclasses.DefaultDataTypeEnum method) (adp.ingest.dataclasses.Entity method) (adp.ingest.dataclasses.EntitySettings method) (adp.ingest.dataclasses.GBOCheckResult method) (adp.ingest.dataclasses.GBOCheckType method) (adp.ingest.dataclasses.GBOColumnMetadata method) (adp.ingest.dataclasses.GBOEntity method) (adp.ingest.dataclasses.GBOPair method) (adp.ingest.dataclasses.GBOSource method) (adp.ingest.dataclasses.GBOSourceConfig method) (adp.ingest.dataclasses.GBOTableMetadata method) (adp.ingest.dataclasses.Ingest method) (adp.ingest.dataclasses.IngestRunMetadata method) (adp.ingest.dataclasses.IngestSettings method) (adp.ingest.dataclasses.Masking method) (adp.ingest.dataclasses.MergeTypeEnum method) (adp.ingest.dataclasses.MSSQLAuthType method) (adp.ingest.dataclasses.MSSQLEntity method) (adp.ingest.dataclasses.MSSQLSource method) (adp.ingest.dataclasses.MSSQLSourceConfig method) (adp.ingest.dataclasses.PersistStorageLevelEnum method) (adp.ingest.dataclasses.SilverTypeEnum method) (adp.ingest.dataclasses.SMBEntity method) (adp.ingest.dataclasses.SMBSource method) (adp.ingest.dataclasses.SMBSourceConfig method) (adp.ingest.dataclasses.Source method) (adp.ingest.dataclasses.SourceConfig method) (adp.ingest.dataclasses.StagingEntity method) (adp.ingest.dataclasses.StagingSource method) (adp.ingest.dataclasses.StagingSourceConfig method) (adp.ingest.dataclasses.TypeEscalationModeEnum method) (adp.ingest.dataclasses.Variable method) (adp.ingest.dataclasses.VariablesPerEnvironment method) (adp.ingest.schema.StructTypeGraph method) (adp.ingest.sources.azdevops.AzDevOpsEntitySource method) (adp.ingest.sources.azure_files.AzureFilesEntitySource method) (adp.ingest.sources.EntitySource method) (adp.ingest.sources.gbo.GBOEntitySource method) (adp.ingest.sources.mssql.MSSQLEntitySource method) (adp.ingest.sources.smb.SMBEntitySource method) (adp.ingest.sources.staging.StagingEntitySource method) (adp.maintain.dataclasses.BackupRule method) (adp.maintain.dataclasses.Env method) (adp.maintain.dataclasses.Job method) (adp.maintain.dataclasses.Maintain method) (adp.maintain.dataclasses.OptimizeRule method) (adp.maintain.dataclasses.RetentionRule method) (adp.maintain.dataclasses.RuleType method) (adp.maintain.dataclasses.VacuumRule method) (adp.maintain.dataclasses.Variable method) (adp.maintain.datasource.DataSource method)

A

abfs_to_uc() (in module adp.delivery.rollback) add_to_timeline() (in module adp.ingest.cdc) AdditionalColumnsError ADLSClient (class in adp.core.adls) ADLSSettings (class in adp.core.settings) adp.core module adp.core.adls module adp.core.authentication module adp.core.backup module adp.core.dbsql module adp.core.exceptions module adp.core.log module adp.core.loganalytics module adp.core.retry module adp.core.secrets module adp.core.settings module adp.core.spark module adp.delivery module adp.delivery.data_io module adp.delivery.dataclasses module adp.delivery.exceptions module adp.delivery.gold module adp.delivery.jobs module adp.delivery.rollback module adp.delivery.transformation module adp.ingest module adp.ingest.cdc module adp.ingest.dataclasses module adp.ingest.datatypes module

adp.ingest.exceptions module adp.ingest.ingest module adp.ingest.schema module adp.ingest.sources module adp.ingest.sources.azdevops module adp.ingest.sources.azure_files module adp.ingest.sources.gbo module adp.ingest.sources.mssql module adp.ingest.sources.smb module adp.ingest.sources.staging module adp.ingest.transform module adp.ingest.udf module adp.maintain module adp.maintain.dataclasses module adp.maintain.datasource module adp.maintain.exceptions module adp.maintain.maintain module adp.maintain.parser module adp.maintain.rules module all_files_in_staging() (adp.ingest.sources.staging.StagingEntitySource method) append() (in module adp.ingest.transform) append_timestamp_data() (in module adp.ingest.cdc) append_unknown_record() (in module adp.delivery.gold) AppendTypeEnum (class in adp.ingest.dataclasses) apply() (adp.ingest.schema.StructTypeGraph method) apply_maskings() (adp.ingest.schema.StructTypeGraph method) AuthenticationError AzDevOpsEntitySource (class in adp.ingest.sources.azdevops) AzureDevOpsEntity (class in adp.ingest.dataclasses) AzureDevOpsSource (class in adp.ingest.dataclasses) AzureDevOpsSourceConfig (class in adp.ingest.dataclasses) AzureFilesEntity (class in adp.ingest.dataclasses) AzureFilesEntitySource (class in adp.ingest.sources.azure_files) AzureFileSourceConfig (class in adp.ingest.dataclasses) AzureFilesSource (class in adp.ingest.dataclasses)

B

BackupClient (class in adp.core.backup) BackupError BackupRule (class in adp.maintain.dataclasses) BackupSettings (class in adp.core.settings)

BackupStatus (class in adp.core.backup) BackupType (class in adp.core.backup) bronze() (in module adp.ingest.ingest) build_from_appsettings() (adp.core.adls.ADLSClient class method) (adp.core.backup.BackupClient class method)

C

CannotModifyDataTypeForHistory cast_dataframes() (in module adp.ingest.datatypes) CDCException clean_column() (in module adp.ingest.transform) Column (class in adp.delivery.dataclasses) column_aware_difference() (in module adp.ingest.transform) ColumnNotInDataFrameError ColumnNotInDataframeException ColumnNotNullableError ColumnTypeMismatchError

connection_string (adp.ingest.dataclasses.MSSQLSourceConfig property) CoreSettings (class in adp.core.settings) create_backup() (adp.core.backup.BackupClient method) create_ingest_from_string() (in module adp.ingest.dataclasses) create_initial_timestamp_df() (in module adp.ingest.cdc) create_snapshots() (in module adp.ingest.cdc) create_sparksession() (in module adp.core.spark) create_timeline() (in module adp.ingest.cdc) custom_query() (adp.ingest.sources.mssql.MSSQLEntitySource method) CustomCredential (class in adp.core.backup)

D

database_name (adp.delivery.dataclasses.SDPTable property) DatabricksJob (class in adp.delivery.dataclasses) DatabricksNotebookSettings (class in adp.delivery.dataclasses) DataSource (class in adp.maintain.datasource) DataType (class in adp.ingest.dataclasses) DataTypeNotValid, [1] date_now() (in module adp.delivery.gold) DBSQLClient (class in adp.core.dbsql) decorator() (in module adp.core.retry) default_headers (adp.core.backup.BackupClient property)

DefaultDataTypeEnum (class in adp.ingest.dataclasses) delete_null_keys() (adp.ingest.schema.StructTypeGraph method) delivery_load() (in module adp.delivery) delivery_run() (in module adp.delivery) delivery_write() (in module adp.delivery) DeliverySettings (class in adp.core.settings) DeltaFileNotFoundException DeltaTableNotFoundException download_file() (adp.core.adls.ADLSClient method) download_files() (adp.ingest.sources.azdevops.AzDevOpsEntitySource method) (adp.ingest.sources.azure_files.AzureFilesEntitySource method)

E

Entity (class in adp.ingest.dataclasses) entity_to_entity_source() (in module adp.ingest.sources) EntitySettings (class in adp.ingest.dataclasses) EntitySource (class in adp.ingest.sources) env (adp.core.settings.Settings property)

Env (class in adp.maintain.dataclasses) EnvironmentEnum (class in adp.core.settings) escalate_datatypes() (in module adp.ingest.datatypes) exclude() (adp.ingest.schema.StructTypeGraph method) exclude_jobs_by_name() (in module adp.delivery.jobs) Expectation (class in adp.delivery.dataclasses)

F

files_matching_patterns() (adp.ingest.sources.staging.StagingEntitySource method) filter_all_files() (adp.ingest.sources.smb.SMBEntitySource method) fix() (adp.ingest.dataclasses.GBOColumnMetadata method) flatten_df() (in module adp.ingest.schema) FolderNotFoundException

ForeignKeyNotFound format_string_as_valid_python_boolean_value() (in module adp.ingest.udf) format_string_as_valid_python_boolean_value_udf() (in module adp.ingest.udf) format_string_as_valid_python_numeric_value() (in module adp.ingest.udf) format_string_as_valid_python_numeric_value_udf() (in module adp.ingest.udf) from_string() (in module adp.ingest.ingest)

G

GBOCheckException GBOCheckResult (class in adp.ingest.dataclasses) GBOCheckType (class in adp.ingest.dataclasses) GBOColumnMetadata (class in adp.ingest.dataclasses) GBOEntity (class in adp.ingest.dataclasses) GBOEntitySource (class in adp.ingest.sources.gbo) GBOIngestError GBOPair (class in adp.ingest.dataclasses) GBOSource (class in adp.ingest.dataclasses) GBOSourceConfig (class in adp.ingest.dataclasses) GBOTableMetadata (class in adp.ingest.dataclasses) get_access_token() (in module adp.core.authentication) get_all_directories_with_extension() (adp.core.adls.ADLSClient method) get_backup() (adp.core.backup.BackupClient method) get_backups() (adp.core.backup.BackupClient method) get_client() (in module adp.core.authentication) get_data() (adp.ingest.sources.gbo.GBOEntitySource method) (adp.ingest.sources.mssql.MSSQLEntitySource method) (adp.ingest.sources.staging.StagingEntitySource method)

get_data_rdd() (adp.ingest.sources.gbo.GBOEntitySource method) get_dataframe() (adp.maintain.datasource.DataSource method) get_datasources() (in module adp.maintain.datasource) get_escalation_target() (in module adp.ingest.datatypes) get_file_pairs() (adp.ingest.sources.gbo.GBOEntitySource method) get_node_by_name() (adp.ingest.schema.StructTypeGraph method) get_secret() (in module adp.core.secrets) get_status() (adp.core.backup.BackupClient method) get_statusmessage() (adp.core.backup.BackupClient method) get_timestamp_based_on_file_name() (in module adp.ingest.udf) get_timestamp_based_on_file_name_udf() (in module adp.ingest.udf) get_timestamp_based_on_path() (in module adp.ingest.udf) get_timestamp_based_on_path_udf() (in module adp.ingest.udf) get_timestamp_data() (in module adp.ingest.cdc) get_timestamp_df() (in module adp.ingest.cdc) get_token() (adp.core.backup.CustomCredential method) get_version_to_rollback_to() (in module adp.delivery.rollback)

H

hashkey() (in module adp.delivery.gold)

I

IncorrectColumnTypeException Ingest (class in adp.ingest.dataclasses) IngestionError IngestRunMetadata (class in adp.ingest.dataclasses)

IngestSettings (class in adp.core.settings) (class in adp.ingest.dataclasses) inject_spark_read_options() (adp.ingest.dataclasses.MSSQLSourceConfig method) InvalidRuleImplementationException is_dir() (adp.core.adls.ADLSClient method)

J

Job (class in adp.maintain.dataclasses)

JobType (class in adp.delivery.dataclasses) JobWorkflow (class in adp.delivery.dataclasses)

K

KeyVaultSettings (class in adp.core.settings)

L

LayersEnum (class in adp.core.settings) list_dir() (adp.core.adls.ADLSClient method) load() (in module adp.delivery.transformation)

load_table() (in module adp.delivery.data_io) LogAnalyticsClient (class in adp.core.loganalytics) LogAnalyticsSettings (class in adp.core.settings)

M

Maintain (class in adp.maintain.dataclasses) maintain_from_string() (in module adp.maintain.maintain) Masking (class in adp.ingest.dataclasses) max_depth() (adp.ingest.schema.StructTypeGraph method) merge() (in module adp.ingest.transform) merge_dictionaries() (in module adp.ingest.dataclasses) merge_root_with_entities() (in module adp.ingest.dataclasses) merge_settings() (in module adp.core.settings) (in module adp.ingest.dataclasses) MergeTypeEnum (class in adp.ingest.dataclasses) MissingColumnsError model_config (adp.core.settings.ADLSSettings attribute) (adp.core.settings.BackupSettings attribute) (adp.core.settings.CoreSettings attribute) (adp.core.settings.DeliverySettings attribute) (adp.core.settings.IngestSettings attribute) (adp.core.settings.KeyVaultSettings attribute) (adp.core.settings.LogAnalyticsSettings attribute) (adp.core.settings.Settings attribute) (adp.core.settings.SparkSettings attribute) (adp.delivery.dataclasses.Column attribute) (adp.delivery.dataclasses.DatabricksJob attribute) (adp.delivery.dataclasses.DatabricksNotebookSettings attribute) (adp.delivery.dataclasses.Expectation attribute) (adp.delivery.dataclasses.JobWorkflow attribute) (adp.delivery.dataclasses.Relationship attribute) (adp.delivery.dataclasses.SDPTable attribute) (adp.delivery.dataclasses.Table attribute) (adp.ingest.dataclasses.AzureDevOpsEntity attribute) (adp.ingest.dataclasses.AzureDevOpsSource attribute) (adp.ingest.dataclasses.AzureDevOpsSourceConfig attribute) (adp.ingest.dataclasses.AzureFilesEntity attribute) (adp.ingest.dataclasses.AzureFileSourceConfig attribute) (adp.ingest.dataclasses.AzureFilesSource attribute) (adp.ingest.dataclasses.DataType attribute) (adp.ingest.dataclasses.Entity attribute) (adp.ingest.dataclasses.EntitySettings attribute) (adp.ingest.dataclasses.GBOCheckResult attribute) (adp.ingest.dataclasses.GBOColumnMetadata attribute) (adp.ingest.dataclasses.GBOEntity attribute) (adp.ingest.dataclasses.GBOSource attribute) (adp.ingest.dataclasses.GBOSourceConfig attribute) (adp.ingest.dataclasses.GBOTableMetadata attribute) (adp.ingest.dataclasses.Ingest attribute) (adp.ingest.dataclasses.IngestRunMetadata attribute) (adp.ingest.dataclasses.IngestSettings attribute) (adp.ingest.dataclasses.Masking attribute) (adp.ingest.dataclasses.MSSQLEntity attribute) (adp.ingest.dataclasses.MSSQLSource attribute) (adp.ingest.dataclasses.MSSQLSourceConfig attribute) (adp.ingest.dataclasses.SMBEntity attribute) (adp.ingest.dataclasses.SMBSource attribute) (adp.ingest.dataclasses.SMBSourceConfig attribute) (adp.ingest.dataclasses.Source attribute) (adp.ingest.dataclasses.SourceConfig attribute) (adp.ingest.dataclasses.StagingEntity attribute) (adp.ingest.dataclasses.StagingSource attribute) (adp.ingest.dataclasses.StagingSourceConfig attribute) (adp.ingest.dataclasses.Variable attribute) (adp.ingest.dataclasses.VariablesPerEnvironment attribute) (adp.maintain.dataclasses.BackupRule attribute) (adp.maintain.dataclasses.Env attribute) (adp.maintain.dataclasses.Job attribute) (adp.maintain.dataclasses.Maintain attribute) (adp.maintain.dataclasses.OptimizeRule attribute) (adp.maintain.dataclasses.RetentionRule attribute) (adp.maintain.dataclasses.VacuumRule attribute) (adp.maintain.dataclasses.Variable attribute)

module adp.core adp.core.adls adp.core.authentication adp.core.backup adp.core.dbsql adp.core.exceptions adp.core.log adp.core.loganalytics adp.core.retry adp.core.secrets adp.core.settings adp.core.spark adp.delivery adp.delivery.data_io adp.delivery.dataclasses adp.delivery.exceptions adp.delivery.gold adp.delivery.jobs adp.delivery.rollback adp.delivery.transformation adp.ingest adp.ingest.cdc adp.ingest.dataclasses adp.ingest.datatypes adp.ingest.exceptions adp.ingest.ingest adp.ingest.schema adp.ingest.sources adp.ingest.sources.azdevops adp.ingest.sources.azure_files adp.ingest.sources.gbo adp.ingest.sources.mssql adp.ingest.sources.smb adp.ingest.sources.staging adp.ingest.transform adp.ingest.udf adp.maintain adp.maintain.dataclasses adp.maintain.datasource adp.maintain.exceptions adp.maintain.maintain adp.maintain.parser adp.maintain.rules monitor_backup() (adp.core.backup.BackupClient method) MSSQLAuthType (class in adp.ingest.dataclasses) MSSQLEntity (class in adp.ingest.dataclasses) MSSQLEntitySource (class in adp.ingest.sources.mssql) MSSQLSource (class in adp.ingest.dataclasses) MSSQLSourceConfig (class in adp.ingest.dataclasses)

N

NoFilesFoundError

NotebookException

O

OnExpectationViolationAction (class in adp.delivery.dataclasses) OnTypeMismatch (class in adp.delivery.dataclasses)

OptimizeRule (class in adp.maintain.dataclasses) overwrite() (adp.maintain.datasource.DataSource method)

P

path (adp.delivery.dataclasses.SDPTable property) (adp.ingest.sources.staging.StagingEntitySource property) PersistStorageLevelEnum (class in adp.ingest.dataclasses) plot() (adp.ingest.schema.StructTypeGraph method) post_data() (adp.core.loganalytics.LogAnalyticsClient method) PowerBIDatatypeNotFoundError preprocess() (in module adp.ingest.ingest) PreviousRunDidNotFinishCorrectly printSchema() (adp.ingest.schema.StructTypeGraph method) privacy_binary() (in module adp.ingest.udf) privacy_binary_udf() (in module adp.ingest.udf) privacy_boolean() (in module adp.ingest.udf) privacy_boolean_udf() (in module adp.ingest.udf) privacy_date() (in module adp.ingest.udf)

privacy_date_udf() (in module adp.ingest.udf) privacy_double() (in module adp.ingest.udf) privacy_double_udf() (in module adp.ingest.udf) privacy_hide_email() (in module adp.ingest.udf) privacy_hide_email_udf() (in module adp.ingest.udf) privacy_int() (in module adp.ingest.udf) privacy_int_udf() (in module adp.ingest.udf) privacy_long() (in module adp.ingest.udf) privacy_long_udf() (in module adp.ingest.udf) privacy_string() (in module adp.ingest.udf) privacy_string_sha1() (in module adp.ingest.udf) privacy_string_sha1_udf() (in module adp.ingest.udf) privacy_string_udf() (in module adp.ingest.udf) privacy_timestamp() (in module adp.ingest.udf) privacy_timestamp_udf() (in module adp.ingest.udf)

Q

query_timeline() (in module adp.ingest.cdc)

R

RecreateNonNullableColumnError register_udfs() (in module adp.delivery.gold) Relationship (class in adp.delivery.dataclasses) remove_temp_files() (adp.ingest.sources.staging.StagingEntitySource method) replace_variables_and_secrets() (in module adp.ingest.dataclasses) RetentionRule (class in adp.maintain.dataclasses) retry() (in module adp.core.retry) retry_call() (in module adp.core.retry) rollback_job() (in module adp.delivery.rollback) rollback_table() (in module adp.delivery.rollback) RollbackTableNotFoundError

RuleType (class in adp.maintain.dataclasses) run_all_jobs() (in module adp.delivery.jobs) run_backup_rule() (in module adp.maintain.rules) run_databricks_job() (in module adp.delivery.jobs) run_ingest() (in module adp.ingest.ingest) run_job() (in module adp.maintain.maintain) run_maintain() (in module adp.maintain.maintain) run_optimize_rule() (in module adp.maintain.rules) run_retention_rule() (in module adp.maintain.rules) run_vacuum_rule() (in module adp.maintain.rules) RunTableNotFoundError

S

scan_files() (adp.ingest.sources.azure_files.AzureFilesEntitySource method) SDPTable (class in adp.delivery.dataclasses) select() (adp.ingest.schema.StructTypeGraph method) server (adp.ingest.dataclasses.SMBSourceConfig property) set_timestamp_data() (in module adp.ingest.cdc) Settings (class in adp.core.settings) setup_logging_for_databricks() (in module adp.core.log) silver() (in module adp.ingest.ingest) SilverTypeEnum (class in adp.ingest.dataclasses) SMBEntity (class in adp.ingest.dataclasses)

SMBEntitySource (class in adp.ingest.sources.smb) SMBSource (class in adp.ingest.dataclasses) SMBSourceConfig (class in adp.ingest.dataclasses) Source (class in adp.ingest.dataclasses) SourceConfig (class in adp.ingest.dataclasses) SparkSettings (class in adp.core.settings) StagingEntity (class in adp.ingest.dataclasses) StagingEntitySource (class in adp.ingest.sources.staging) StagingSource (class in adp.ingest.dataclasses) StagingSourceConfig (class in adp.ingest.dataclasses) StructTypeGraph (class in adp.ingest.schema)

T

Table (class in adp.delivery.dataclasses) TableNotAvailableError

test_path() (adp.core.adls.ADLSClient method) TransformOptions (class in adp.ingest.cdc) TypeEscalationModeEnum (class in adp.ingest.dataclasses)

U

udf_date() (in module adp.delivery.gold) udf_hashkey() (in module adp.delivery.gold) unity_catalog_identifier (adp.delivery.dataclasses.SDPTable property) unity_catalog_identifier_for_load (adp.delivery.dataclasses.SDPTable property)

unzip_files() (adp.ingest.sources.staging.StagingEntitySource method) upload_to_staging() (adp.ingest.sources.azdevops.AzDevOpsEntitySource method) (adp.ingest.sources.azure_files.AzureFilesEntitySource method) (adp.ingest.sources.smb.SMBEntitySource method) (adp.ingest.sources.staging.StagingEntitySource method)

V

VacuumRule (class in adp.maintain.dataclasses) validate_auth_protocol() (adp.ingest.dataclasses.SMBSourceConfig class method) ValidationError

Variable (class in adp.ingest.dataclasses) (class in adp.maintain.dataclasses) VariablesPerEnvironment (class in adp.ingest.dataclasses) verify_keys() (adp.ingest.schema.StructTypeGraph method)

W

with_datatypes_escalated() (in module adp.ingest.transform) write() (in module adp.delivery.transformation)

write_table() (in module adp.delivery.data_io) write_tables() (in module adp.delivery.data_io)