adp.ingest.transform.clean_column

adp.ingest.transform.clean_column(col_name: str, replace_dots: bool = False, lowercase: bool = True) → str

clean_column to make it 100% compatible with PySpark

Replaces incompatible special characters with empty strings and spaces with underscores.

It uses the following dictionary:

char_to_replace = {
    " ":'_',
    ",":'',
    ";":'',
    "{":'',
    "}":'',
    "(":'',
    ")":'',
    "\\n":'',
    "\\t":'',
    "=":''
    }

Example

>>> clean_column('hi, there')
'hi_there'