Add the option to pick the document language and support table and image extraction

This proposal addresses a critical need for expanding the document ingestion capabilities of our product, which is particularly relevant to our clients with non-English data.

Currently, our UDI feature is limited to an English-only OCR model. This restriction prevents us from effectively processing documents in other languages, which impacts a significant portion of our user base.

The requested feature is an OCR language selection field. Similar to the functionality in the watsonx.ai Text Extraction API, this would allow users to specify the language list for the OCR model. This simple addition would immediately enable robust, multilingual document ingestion into Milvus.

Furthermore, many clients require more sophisticated data extraction methods to maximize the value of documents for their RAG solutions. Specifically, they need:

Table Extraction: The ability to accurately identify and extract structured data contained within tables.

Image Verbalization: The capability to describe and extract meaning from images embedded in the documents.

Currently, clients must develop and maintain custom pipelines using watsonx.ai APIs to integrate these advanced features.

All of these features are already in WDU and the product can leverage this functionality from it.

The optimal solution would be to consolidate these features into a single, configurable end-to-end ingestion pipeline within our product. This centralized approach, which includes the language selection option, table extraction, and image verbalization, would significantly improve the client experience, accelerate deployment, and enhance the quality of data fed into their RAG systems.