Increase AutoAI Data Volume and Resource Limits

Current behavior:

CP4D PoC AutoAI Limits

• Practical dataset size limit of ~1 GB(SaaS) when uploading data via the AutoAI UI.
• Upload appears to be memory‑bound: data is loaded directly into the service pod’s memory.
• Users are blocked from running realistic workloads that require larger training datasets.

Resource Behavior (Suspected)

• In other AutoML products/environments, the pod memory requirement is a multiple of the data size (e.g., 6× data size).
• For CP4D AutoAI, the exact memory‑to‑data ratio is unknown (1:1, 1:4, 1:6, etc.).
• There is no clear documentation on:
  • How AutoAI derives max allowed dataset size from pod memory.
  • What limits are configurable vs hard‑coded.

Existing Spectrum AutoAI Environment

• Current AutoAI/AutoML platform (Spectrum environment, on‑prem) supports up to 5 GB per user for training datasets.
• If raw data exceeds 5 GB, they sample down to ≤5 GB as standard practice.
• This 5 GB cap is widely adopted among data scientists and considered the baseline requirement.

Workaround given by IBM (for now)in CP4D This is functional but adds friction and is not a scalable UX for typical users.

• Users can:
  • Run AutoAI on a smaller sample (<1 GB).
  • Save the experiment as a notebook.
  • Use a larger notebook environment to rerun the same pipeline on a larger dataset.

Requested Enhancement

Increase Supported Dataset Size in AutoAI

• Enable AutoAI on CP4D to support datasets up to 5 GB per experiment, aligning with the Spectrum environment.
• Maintain the ability to sample down to 5 GB when raw data is larger.

Document and Expose Resource Model

• Clearly document how AutoAI uses pod memory to determine:
  • Maximum dataset size per run.
  • Any internal multipliers (e.g., 4×, 6× data size).
• Expose or make configurable:
  • Pod memory settings for AutoAI.
  • Any relevant resource limits/quotas tied to dataset size.

Safe Configuration Bounds

• Define supported min/max values for:
  • Pod memory allocations.
  • Corresponding dataset size limits.
• Provide guidance on what can be safely changed, and what would risk instability (“what will make it break”).