IBM Knowledge Catalog uses RabbitMQ as part of its internal asynchronous processing architecture. In large Metadata Enrichment environments, messaging availability and consumption rates can become critical dependencies for the overall stability of IKC.
Bradesco has experienced recurring operational incidents involving the RabbitMQ layer in its IKC environment.
One significant incident involved RabbitMQ HA pods entering CrashLoopBackOff after their persistent volumes became full. Analysis identified disk_almost_full / ENOSPC conditions and problems associated with RabbitMQ quorum/WAL storage. Recovery required infrastructure-level intervention, including recovery/recreation of RabbitMQ storage and restart/recovery activities involving dependent WKC components.
As a mitigation, RabbitMQ PVC capacity was increased from 10 GiB to 20 GiB. Under normal/smaller Metadata Enrichment workloads, actual utilization subsequently remained below approximately 1 GiB and messages were being produced and consumed normally.
However, investigation of the period in which RabbitMQ storage had previously grown identified an important dependency: catalog-api pods had been evicted after reaching their ephemeral-storage limits. The ephemeral-storage allocation was increased from 1 GiB to 3 GiB, which temporarily mitigated the issue, but eviction was observed again.
The working technical hypothesis was that if services such as catalog-api responsible for consuming RabbitMQ messages become unavailable or are evicted, message consumption can stop or become slower than message production. Messages can then accumulate in RabbitMQ, eventually consuming the available persistent storage and potentially causing RabbitMQ itself to become unavailable.
This creates the possibility of a cascading failure:
Consumer degradation/eviction → message backlog → RabbitMQ storage growth → PVC exhaustion → RabbitMQ instability → broader IKC/Metadata Enrichment impact.
The requested enhancement is therefore to improve the overall resilience of the IKC messaging architecture, rather than simply increasing RabbitMQ PVC sizes.
The product should evaluate and implement capabilities such as:
- Automatic detection of abnormal queue/backlog growth.
- Built-in alerts based on queue depth, message age, consumer availability and storage utilization.
- Clear visibility of producers, consumers and queue health through IKC administration/monitoring.
- Protection against unbounded queue growth.
- Configurable message retention/TTL where technically appropriate.
- Dead-letter handling for messages that cannot be processed.
- Backpressure/throttling mechanisms when consumers cannot keep up with producers.
- Improved automatic recovery when consumers restart or are evicted.
- Graceful handling of temporary consumer outages without allowing messaging storage to exhaust.
- Automated or supported recovery procedures when RabbitMQ storage approaches critical thresholds.
- Appropriate sizing recommendations based on Metadata Enrichment workload/concurrency.
- Health checks capable of identifying messaging degradation before it impacts user workloads.
- Review of dependencies between RabbitMQ and components such as catalog-api to prevent cascading failures.
Additionally, IBM Product Engineering should reassess whether the current RabbitMQ-based implementation remains the most appropriate architecture for large-scale IKC workloads.
If architectural limitations prevent the required level of scalability and fault isolation, IBM should consider alternative or complementary messaging technologies/architectures. The requested outcome is not specifically the replacement of RabbitMQ, but a messaging layer that can sustain enterprise-scale workloads and recover from downstream consumer failures without causing broader IKC outages.
