Support transformations for the OpenLineage Column Level Lineage dataset facet.

This item is different from your original comment

Transforming if the value is IDENTITY, non-transforming if the value is TRANSFORMATION or AGGREGATION.

The ask is for the reverse -

1. If DIRECT and IDENTITY, the TRANSFORMATION_TAG is Non Transforming 
2. If DIRECT and either of TRANSFORMATION or AGGREGATION, the TRANSFORMATION_TAG is Transforming.

Hi Sawelita,

Thank you, that makes sense. Unless I missed something, the only additional item is:

"If the Description in the OpenLineage payload is present, the existing Transformation Logic attribute in Manta should be populated by the Description field."

I added this to the spec (see in the comment above).

Please let me know if you have any additional comments.

Best, Jakub

Hello, 

1. Transformation Type - The proposed representation for this is in line with expected behavior.
2. Masking 
   1. No action if value is missing or set to false is in line with expectation.
   2. If present, the expectation is to have similar behavior when representing transformations on columns. Display  TT label indicator against the column. When user hovers over it, the dialog appears showing TRANSFORMATION_DESC and TRANSFORMATION_TAG. On clicking the column, the top left popup includes Transformation Type and Transformation Logic , alongside the proposed attribute Masking showing True. The TRANSFORMATION_DESC and Transformation Logic  should take the value from the description field of the payload. For example below, it should print HASH  "transformations": [
                                  {
                                      "type": "DIRECT",
                                      "subtype": "IDENTITY",
                                      "description": "HASH",
                                      "masking": true
                                  },
                              ]
3. Subtype
   1. No action if value is missing is expected
   2. If value is present,  display the TT label indicator
   3. If DIRECT and IDENTITY, the TRANSFORMATION_TAG is Non Transforming 
   4. If DIRECT and either of TRANSFORMATION or AGGREGATION, the TRANSFORMATION_TAG is Transforming. The TRANSFORMATION_DESC, Transformation Logic, Expression should be derived from the description.
   5. For all of the INDIRECT subtypes, the TRANSFORMATION_DESC, Transformation Logic, Expression should be derived from the description. An example below with description - "transformations": [
                                  {
                                      "type": "INDIRECT",
                                      "subtype": "JOIN",
                                      "description": "ON (DISCOUNTS.CUSTOMERS_ID=CUST.ID)",
                                  },
                              ] 

Hi Sewalita,

we reviewed this requirement and propose the following specification for the improvement. Can you confirm this is in line with your expectations and will support your uses?

Thank you,
Jakub

The Manta OpenLineage scanner should parse and use the additional information about the transformations described in [https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/] in the following manner:

• • Transformation Type

    • If present, create the following lineage edge type in Manta

      • a direct lineage edge for the DIRECT value in the OpenLineage payload

      • and an indirect (filter) lineage edge for the INDIRECT value in the OpenLineage payload

    • If this information is not present, create a direct lineage edge (current behaviour)

  • Subtype

    • If the value is present, create a custom attribute (Transformation Type) with this value. This attribute will be shown in the lineage UI and show the exact value provided in the OpenLineage payload.

    • In addition, if the value is any of the 3 allowed values for DIRECT transformations, populate Manta's transforming/non-transforming attribute/tag based on this value. Non-transforming if the value is IDENTITY, transforming if the value is TRANSFORMATION or AGGREGATION.

    • No action if the value is missing - no attribute will be set

  • Description

    • If the Description in the OpenLineage payload is present, the existing Transformation Logic attribute in Manta should be populated by the Description field.

  • Masking

    • If the value is present and is true ("masking": true in the OpenLineage event), create a custom attribute (Masking) with value true

    • No action if the value is missing or set to false