Automated Reference Data Cleanup and Versioning Maintenance

Problem Statement: Currently, IKC creates a new version for each updated dataset, leading to the accumulation of older versions. There is no built-in mechanism to automatically purge older versions or retain only a specific number of recent versions.

Proposed Solution: Implement an automated data cleanup and versioning mechanism for IKC-RDM (Reference Data Management) , allowing for the configuration and retention of older versions.

Function Requirements:

• Version Retention Policy:

  • Define a retention policy that specifies the number of versions to keep for each dataset.

  • Implement a mechanism to automatically purge older versions that exceed the retention limit.

• Version Tagging:

  • Determine whether IKC tags the entire dataset as a new version or only the changed data.

  • If IKC tags only the changed data, implement a CDC-like process to track changes and tag accordingly.

• Data Cleanup:

  • Provide a mechanism to purge older versions of data, either automatically based on the retention policy or manually.

• Configuration Management:

  • Allow for the configuration of version retention policies and data cleanup settings for each dataset.

  • Maintain these configurations in source control to ensure consistency and traceability.

Benefits of Automated Data Cleanup and Versioning:

• Storage Optimization: Reduce storage usage by purging older versions of data that are no longer needed.

• Performance Improvement: Improve query performance by reducing the number of versions to search through.

• Data Governance: Implement a data governance framework to manage data lifecycle and retention policies.

• Compliance: Ensure compliance with data retention and privacy regulations.