Improve Error Handling and User Feedback for Context Window Exceedance and vLLM Failures

Description

Currently, when users attempt to submit requests that exceed the model’s maximum context length or encounter vLLM-related errors, the system behaves in two problematic ways:

UI Behavior:

The UI accepts input that exceeds the supported context window size.

This results in wasted time and backend resource consumption since the request is only rejected deep in the pipeline.

Backend Behavior (504 errors):

Users frequently see generic 504 Gateway Timeout errors when vLLM rejects input (e.g., too many tokens).

The real cause (e.g., ValueError: This model's maximum context length is 128000 tokens...) is only visible in backend logs.

Proposed Enhancement

UI validation: Proactively check request size against model context window before submission. If the request is too large, the UI should block submission and display a clear error (e.g., “Your request exceeds the maximum context length of 128k tokens. Please shorten your input.”).

Meaningful error surfacing: For backend errors (e.g., token length, unsupported parameters, or invalid inputs), return concise, actionable error messages to the user instead of generic 504 errors.

Example of improved error message

“Your request exceeds the maximum token limit of 128,000 (submitted: 844,373). Please shorten your input.”

Business Impact

Improves client experience: Users immediately understand and resolve issues without needing IBM support.

Reduces support burden: Fewer L1/L2 tickets for basic error triage.

Accelerates adoption: Faster troubleshooting builds user trust in watsonx/CP4D.

Enhances transparency: Meets enterprise expectations for clarity and reliability.

Competitive differentiation: Aligns IBM with industry peers who already surface meaningful error feedback.