Proposal for a Proactive Platform Monitoring and Health Check System

We are currently encountering significant issues during our backup executions. We face database inconsistencies and version mismatches in the cpbr scripts, among other errors. Because these problems are only discovered during the backup process itself, it causes major delays—stalling story implementation for weeks instead of days.

To alleviate the current frustration, we need to surface these issues as early as possible. Relying solely on pod health is insufficient, as end-users often lack visibility into how services are interconnected.

1) Our Requirements: We need a platform-level monitoring system that tracks service health and triggers alerts in real-time. This would allow us to react immediately rather than waiting for a failure. The check intervals should be fully adjustable.

2) Proposed Solution: We envision a centralized health check dashboard displaying all platform-wide checks and attached tasks. 

For example:

Platform - Backup: 

• Version checks (comparing cpbr versions with platform versions) 
• database sync checks (Postgres state).

DataStage - Compile Status: 

• Sample compiles to verify if a compile is possible.

DataStage - System Checks: 

• status of runtime (are the pods running and helthy)
• compute pods (are the compute pods running and healthy).

Project Health: (iterates all projects)

• Connection Checks - Tests all Connections defined in a project
• Runtime Checks - Test if the attache runtimes are running and available
• ...

By creating a registry where every product or service logs its available health checks, we can easily provide a mix of default and custom validations. This would allow us to discover and resolve issues the moment they appear.

This also would put the customer in the situation to see how processes are connected in the platform, what is currently not possible.

3) Key Features:

Default Health Checks: The most common health checks should be provided by the service teams (for example datastage team provides their checks. they know best what is needed so that the product works correctly)

Adjustable Check Intervals: Flexibility to choose how often monitoring runs.

Centralized Health Dashboard: Where alle checks are visible, can be controlled and arranged.

Service Registry: A database where services register their own health endpoints for easy integration.

Automatic chain information collection: Today the customer has to choose what information he needs to connect and provide. With this chain defained, you could just create a button on every line in the dashboard to collect all information for a failed service process.

Shifting to this proactive model is essential for catching problems as they arise and would allow for much smoother alerting and remediation.