Skip to Main Content
IBM Data and AI Ideas Portal for Customers


This portal is to open public enhancement requests against products and services offered by the IBM Data & AI organization. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:


Search existing ideas

Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,


Post your ideas

Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea


Specific links you will want to bookmark for future use

Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.

IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.

ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.

IBM Employees should enter Ideas at https://ideas.ibm.com


Status Not under consideration
Workspace Watson Discovery
Created by Guest
Created on Jun 15, 2018

Transferring ownership of Annotation Sets

Problem

When annotation sets are created by WKS and assigned to the selected (by the Project Manager) human annotators (HA), the ownership of the annotation sets cannot be changed. This means that if a HA is temporarily or permanently unavailable to finish the assigned (via annotation set) annotation work, the work cannot be transfered to another HA and it cannot be done.

Consequences:

  • When the HA is temporarily unavailable to finish the work, the entire Annotation Task (a group of one or more annotation sets, whose progress is tracked as a whole) is blocked and the entire annotation team is slowed down.
  • When the HA becomes permanently unavailable, the entire Annotation Task is in limbo. The workaround is to redo a lot of work in a new Annotation Task - not an acceptable solution. 

Several options (ideas)

Allow transferring of Annotation Set ownership.

Several ideas:

  1. Allow Project Manager to re-assign annotation sets - relatively safe (controlled) way to transfer ownership, but still may create issues with the fidelity of the annotation overlaps, as these overlaps were created by WKS with the original HAs in "mind."
    • PROS: Relatively easy and safe to implement
    • CONS: Side effects are expected; addresses a specific issue, but does not improve much the overall process.
  2. Dynamically adjusted hidden annotation sets - this would be radical change of the annotation workflow and UX, but it may be a promising one:
    • Annotation sets can be maintained internally by the system, only for the purpose of ensuring the overlapping document annotations, but no need to be explicitly managed by Proj Mgr and assigned to specific HAs, etc. In this case, the workflow could be something like:
      1. Project Manager uploads the training documents and organizes them in Document Sets (Btw, the management of Document Sets also needs improvement.)
      2. The Project Manager creates Annotation Tasks and adds Document Sets to them
      3. The HAs signup for (pull, not push approach) one of more available (unfinished) Annotation Tasks
      4. For each HA in an Annotation Task, WKS automatically creates a pool of documents, making sure that the Project Manager's setting for annotation overlap are met
      5. Each HA is presented with their own pool of documents for annotation (that is the automatically generated hidden Annotation Set)
      6. As HAs work at different speed, and join and leave the Annotation Task (the main unit of work progress that is tracked by the Project Manager), and as Project Manager adds/removes documents from the Annotation Tasks (yes, in real life this happens and should be supported by WKS), WKS dynamically adjusts the pool of documents for each HA, to ensure proper annotation overlap and to maximize the annotation speed based on the different speed of the HAs.
      7. Ta-da!
    • PROS: Dynamic optimization of the annotation process by WKS should yield best overall results (hypothesis to be validated yet)
    • CONS: Radical change - longer and more costly to execute; higher risk (compared to 1) as this is real innovation.