Request for Improved Orthographic Variation Handling in Japanese through Morphological Analysis in Watson Discovery

When using Watson Discovery for Japanese natural language searches, we are experiencing issues due to variations in okurigana (Japanese verb endings) and spelling differences, which prevent certain search results from appearing as expected. For example, the following variations of essentially the same terms are treated differently based on their orthographic differences:

「打ち合わせ」「打合せ」「打ち合せ」
「引越し」「引っ越し」「引越」
「組合せ」「組み合わせ」「組み合せ」
https://ibm.box.com/s/j9v6fk90igzlll80cm5v7jtwjbyplqgd

Although these variations can be managed by registering them as synonyms, there are operational constraints such as:
(1) The significant amount of manual effort required to prepare and maintain these synonym lists
(2) The 5,000-term limit on synonyms

In light of the above, we would like to request that Watson Discovery perform morphological analysis and text normalization internally so that these spelling differences can be unified into a standard form. Implementing such an internal processing mechanism would allow us to absorb these variations without manual synonym registration, improving search accuracy and reducing maintenance costs.

Needed By

Quarter

Post comment

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

Request for Improved Orthographic Variation Handling in Japanese through Morphological Analysis in Watson Discovery