Add support for inbuilt quality metrics with relative thresholds as per ODCS 3.1.

[HenrikSpiegel]

OIDC 3.1 describe quality metrics for both absolute (rows) and relative (percent) thresholds:

https://bitol-io.github.io/open-data-contract-standard/v3.1.0/data-quality/#null-values:~:text=Example%20with%20percent%3A

I'll be happy to open a PR if you are taking contributions.

[jochenchrist]

Yes, that would be nice!

[HenrikSpiegel]

Two questions:

1. I don't appear to be able to create a branch on the repository - do I need to be added as a contributor or what is your flow?

2. Suggested approach:

I believe there are a couple of approaches to implement the unit.

A.1:
Update datacontract-cli/datacontract/engines/data_contract_checks.py::to_sodacl_threshold to return the threshold including % in case of unit: percent.

Detect the % in the various check_property_* functions and determine metric function based on that.

A.2:
Explicitly determine the quality.unit and store in threshold_unit variable which can then be passed into the check_property_*
functions.

for both 1.1 and 1.2 the logic for selecting the SODA metric ie. invalid_count vs invalid_percent would sit in the check_property_invalid_values function.

B
Alternalively create specific check_property_* function for absolute and relative and move the switching logic from the check_property_* functions and up into the check_quality_list function.

---

I think 1.1 is the cleanest implementation at the cost of being implicit.

[simonharrer]

Flow would be that you fork the repo, and then create a PR from a branch in your fork.

[struesda]

Thanks @HenrikSpiegel ! - Will this also support importing these 3.1 Metrics from the Excel template?

[HenrikSpiegel]

Thanks @HenrikSpiegel ! - Will this also support importing these 3.1 Metrics from the Excel template?

I have initially only update the data-contract checks I am not familiar with the Excel template.