Features

List of features that may come in handy.

Time Filter

If you need to filter data by date this feature’s for you. Each rule can define time interval for selection of suitable rows with parameter time_filter. It expects name of the column, optionally also length of the interval in days (with default 30). It is possible to name multiple time-based columns, which will be joined with AND.

Format

{
    "time_filter": [
        {"column": "a", "days": 10},
        {"column": "b", "days": 24},
    ],
}

# this will select data where column `a` is between running time and running time minus 10 days AND
# column `b` column is between running time and running time minus 24 days
# then it will do the check against those filtered data only

Miscellaneous

• (deprecated) Backward-compatible time filter format defined with column name only. This uses the default 30 days value.

{
    "time_filter": "a"
}

• separate_time_filters parameter.
  • Creates a matrix of rules and filters - i.e. provided rule is applied with each separate_time_filters value separately - acts as a generator of rules. In case both separate_time_filters and time_filter are defined, only separate_time_filters would be considered.

{
    "name": NOT_NULL,
    "columns": ["a", "b", "c"],
    "separate_time_filters": [{"column": "c"}, {"column": "d"}]
}
# Checks NOT_NULL while filtering by column c and writes the result,
# then repeats the same check while filtering by column d and writes the result as a separate value.

Context

Each run has its own context, mostly used for templating the final sql. This is its context:

{
    "table_fullname" : "public.my_cool_table",
    "task_ts": "passed by client or datetime.now()"
}

Quality Check Result

Result will be storred to result_table in your database. Its model is defined in QualityCheck. Table name will be prefixed with quality_check, in this example case the resulting table would be dq.quality_check_my_table.

Tip

You can suppress this behaviour by passing result_table={"schema_name": "dq", "table_name": "this_is_my_name", "use_prefix": False} to ContessaRunner.