Add preventive monitoring and automated maintenance for Tiny Milvus etcd backend in watsonx Orchestrate on-prem

In watsonx Orchestrate on-prem, Tiny Milvus uses a dedicated etcd backend with a fixed backend quota of around 2 GB. When this quota is reached, etcd raises a NOSPACE alarm and Milvus can no longer start correctly.

In our environment, this resulted in the Milvus standalone pod entering CrashLoopBackOff with the following error:

etcdserver: mvcc: database space exceeded
panic: etcdserver: mvcc: database space exceeded

The recovery was possible by manually taking an etcd snapshot, running etcd defragmentation, disarming the NOSPACE alarm, and restarting Milvus. However, this is a reactive action after the service is already unavailable.

We would like IBM to provide a supported preventive mechanism for this situation.

Current behavior

When the dedicated Milvus etcd backend reaches its quota:

• etcd enters NOSPACE state,
• writes are blocked,
• Milvus RootCoord cannot initialize its session,
• the Milvus standalone pod crashes,
• the pod enters CrashLoopBackOff,
• RAG / knowledge base capabilities depending on Milvus become unavailable.

There does not seem to be an automatic preventive mechanism, such as auto-compaction, scheduled defragmentation, or an operator-level alert before the issue becomes blocking.

Requested enhancement

Please provide an IBM-supported mechanism to prevent or detect this issue before Milvus becomes unavailable.

This could include one or more of the following:

1. Automatic etcd compaction for the dedicated Tiny Milvus etcd instance.
2. Automatic or scheduled etcd defragmentation managed by the operator.
3. A supported configuration option to enable / tune this maintenance behavior.
4. Built-in monitoring of the Milvus etcd backend size and quota usage.
5. Clear alerts when the backend size reaches warning thresholds, for example 70%, 80%, and 90%.
6. A visible operator condition or CPD / Software Hub health status when Milvus etcd is close to NOSPACE.
7. A documented and supported script / command to check Milvus etcd backend usage.
8. Clear guidance on when customers should move from Tiny Milvus to another supported Milvus architecture for production-like RAG usage.

Business value

Milvus is a key component for vector storage and RAG scenarios in watsonx Orchestrate.

Without preventive monitoring, the first visible symptom is the Milvus pod entering CrashLoopBackOff, which directly impacts knowledge base ingestion, RAG validation, and agents depending on Milvus.

Thx