Improve INDOUBT thread processing in two-phase commit scenario

Recently we had a pretty severe production outage on an 8-way data sharing group. After resolution of the root cause, WAS and Db2 z were quite slow in recovery. Following the crash, there were dozens of indoubt threads and lots of retained locks as well. The indoubt thread handling did not complete as expected, resulting in retained locks continuing to be held and indoubt threads still persistent until WAS transaction logs were wiped clean and the WAS servers restarted. The threads in question were involved in two-phase commit (2PC) processing, with WAS as the commit coordinator and Db2 as a participant.

The proposal is to improve the way indoubt thread processing is handled for 2PC transactions such that if a coordinator (Db2) completes its portion of the 2PC work, it should be resolved as far as Db2 is concerned. In our case, we performed RECOVER INDOUBT ACTION(COMMIT) commands and all the threads showed as COMMITED-H. Following that, we ran RESET INDOUBT to clear the indoubt thread information from Db2. However, when WAS had an intact transaction log, we were not able to return Db2 to "normal" status due to persistent retained locks and indoubt threads. We belive that even though the work was committed heuristically, Db2 was still waiting for confirmation to do so from the commit coordinator, and that confirmation was never coming.

We're basically looking for Db2 (as a participant in 2PC) to have the ability to manage its portion of the global transaction in a recovery scenario like this. rather than having to wait on the commit coordinator.