Expose runtime observability metrics for custom foundation models on watsonx.ai SaaS

Who would benefit from it?

Straker plans to run Tiri, its own Gemma 3-based model, as a custom foundation model on watsonx.ai SaaS for SwiftBridge AI v2, Straker's IR translation solution.

This capability would also benefit other partners and enterprise customers operating custom foundation models in production under SLAs.

Why is it useful?

A deployed custom foundation model currently exposes no runtime telemetry to its owner. The deployments management API returns lifecycle state only and provides no throughput, latency, utilization, queue, or error data. The serving process's metrics endpoint is not reachable, and watsonx.ai Runtime is not listed as a supported service in IBM Cloud Monitoring. As a result, there is no platform-native path to per-deployment observability for custom foundation models, and direct model performance monitoring is not possible.

The deployments management API also identifies a deployment by its UUID within a space, while inference requests identify it by its serving name. These two identities represent the same deployment but are resolvable in only one direction, further complicating the correlation of operational data.

To compensate, Straker aggregates request counts, token throughput, latency, and availability through its own gateway, using token usage returned in inference responses. Straker also derives deployment availability by polling the deployment state.

This approach works, but it measures the platform externally rather than providing direct visibility into the deployed model runtime. Partners should not need to build and maintain their own observability layer to operate production workloads under SLAs.

In its self-hosted proof of concept, Straker built a Prometheus dashboard using vLLM runtime metrics. However, the same runtime metrics are not accessible to deployment owners on watsonx.ai SaaS.

How should it work?

watsonx.ai SaaS should provide per-deployment metrics for custom foundation model deployments, including:

• Request volume

• Input and output token throughput

• Latency percentiles

• Deployment availability

• GPU utilization

The metrics should be exposed through the watsonx.ai API and console and/or by enabling watsonx.ai Runtime custom deployments to integrate with IBM Cloud Monitoring or Instana so that standard monitoring tools can collect them.

The capability should provide a secure and supported path for deployment owners to monitor their own custom model deployments without requiring access to IBM-managed internal components.

Standard monitoring integration, including compatibility with Prometheus-based monitoring where feasible, would allow Straker and other customers to incorporate watsonx.ai custom model deployments into their existing dashboards, alerts, and SLA monitoring processes.

Related validation context

Hosting Tiri on watsonx.ai SaaS has already been verified as technically feasible through Case TS022356513 and the related internal GitHub issue View internal GitHub issue 61501. Official confirmation of Transformers v5 support remains open and is being handled separately from this observability request.