Support transformations for the OpenLineage Column Level Lineage dataset facet.

Why it is useful?

Transformations are an integral part in any data flow representation. They comprise logic describing data, functions and process information that are vital for deriving lineage on an attribute.

A field of an output in Column Level Lineage Dataset Facet can contain transformation field. This includes transformations to understand which input columns are used to produce which output columns and in what way. It helps portray the nature of relation between the input and the output columns.

[https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/]

Manta currently does not parse this information for lineage. Excluding it, Manta does not gather key information for a full end-to-end data lineage from source to target, crucial for understanding impact, troubleshooting, and governance within data platforms

Who would benefit from it ?

Any solution built on Manta using OpenLineage.

How it should work ?

Transformation in OpenLineage is described by 4 fields: type, subtype, description and masking. And each of these 4 fields can have different values. Type can be Direct or Indirect. And depending on the type, the subtype can have different values.

Manta should be able to parse both Direct and Indirect types of transformations.

It should be able to parse logic in the transformation field, aggregations, joins, filters, conditions, be able to read SQL clauses; that is essential to show the changes a field undergoes to get to the output.