IBM Data and AI Ideas Portal for Customers


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:

Post your ideas

Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea

Help IBM prioritize your ideas and requests

The IBM team may need your help to refine the ideas so they may ask for more information or feedback. The product management team will then decide if they can begin working on your idea. If they can start during the next development cycle, they will put the idea on the priority list. Each team at IBM works on a different schedule, where some ideas can be implemented right away, others may be placed on a different schedule.

Receive notification on the decision

Some ideas can be implemented at IBM, while others may not fit within the development plans for the product. In either case, the team will let you know as soon as possible. In some cases, we may be able to find alternatives for ideas which cannot be implemented in a reasonable time.

Additional Information

To view our roadmaps: http://ibm.biz/Data-and-AI-Roadmaps

Reminder: This is not the place to submit defects or support needs, please use normal support channel for these cases

IBM Employees:

The correct URL for entering your ideas is: https://hybridcloudunit-internal.ideas.aha.io


Status Delivered
Workspace Spectrum Symphony
Components Version 7.2.0.2
Created by Guest
Created on Jul 15, 2021

SD bypass patch

IBM has suggested to implement SD Bypass patch in order to resolve ongoing issue in Citi grid LNPRD cluster. Details of issue is below in form of investigations:

We have analyzed the data we collected yesterday and have found the delay in authentication process is caused mostly in OS syscalls. Below is snippets from both LND and SW grids.


1. strace
23933 20:04:20.437828 write(3, "2021-06-21 20:04:20.437 GMT DEBUG [23585:139771402884864] sd.ssmManager.SdSsmManager - SsmManager::sdkConnect(): Entered:...for app name: symping7.2\n", 150) = 150 <0.000014>
-> SD thread PID=23933 logs sdkConnect message prior to start authentication and authorization processes of the job submission user.

23933 20:04:20.444712 poll([{fd=26, events=POLLIN}], 1, 15000
23933 20:04:20.580480 ) = 1 ([{fd=26, revents=POLLIN}]) <0.135755>
-> a delay occurred during poll() syscall. Waiting on file descriptor 26.

23933 20:04:20.584180 write(3, "2021-06-21 20:04:20.584 GMT DEBUG [23585:139771402884864] servers.common.resmngr.VEMResManagerBase - VEMResManagerBase::checkRbacUserPermissionInternal() : cache kit key=Admin#/SymTesting/Symping72#SOAM_APP_LOGIN cache size=489\n", 228) = 228 <0.000022>
-> logged by SD as soon as authentication and authorization process is done.

2. lsof
sd 23585 symadmp 26u IPv4 28674654 0t0 TCP gridmstsw30p.nam.nsroot.net:47690->gmwgtdcpsp07p.nam.nsroot.net:44443 (ESTABLISHED)
-> checking SD's lsof output, 26u file descriptor is TCP socket between SDSiteminder. This indicates SD was waiting for data arrival at FD=26 from Siteminder port 44443. (POLLIN event waits until data is ready at given FD.)


1. strace
25048 19:18:56.989822 write(3, "2021-06-21 19:18:56.989 GMT DEBUG [24709:139991709902592] sd.ssmManager.SdSsmManager - SsmManager::sdkConnect(): Entered:...for app name: symping7.2\n", 150) = 150 <0.000016>
-> SD thread PID=25048 logs sdkConnect message prior to start authentication and authorization processes of the job submission user.

25048 19:18:56.989893 futex(0x34bf760, FUTEX_WAKE_PRIVATE, 1) = 1 <0.000012>
25048 19:18:56.989940 futex(0x3bf4750, FUTEX_WAIT_PRIVATE, 2, NULL
25047 19:18:57.645953 futex(0x34bf760, FUTEX_WAKE_PRIVATE, 1) = 1 <0.000012>
25047 19:18:57.646001 futex(0x3bf4750, FUTEX_WAKE_PRIVATE, 1
25048 19:18:57.646037 ) = 0 <0.656088>
25048 19:18:57.650643 poll([{fd=26, events=POLLIN}], 1, 15000
25048 19:18:57.777387 ) = 1 ([{fd=26, revents=POLLIN}]) <0.126732>
25048 19:18:57.780711 poll([{fd=25, events=POLLIN}], 1, 15000
25048 19:18:57.894638 ) = 1 ([{fd=25, revents=POLLIN}]) <0.113916>
25048 19:18:57.897883 poll([{fd=26, events=POLLIN}], 1, 15000
25048 19:18:58.024320 ) = 1 ([{fd=26, revents=POLLIN}]) <0.126426>
-> delay occurred during 3 poll() sequential syscalls. SD waiting on file descriptor 26 and 25.

25048 19:18:58.027782 write(3, "2021-06-21 19:18:58.027 GMT DEBUG [24709:139991709902592] servers.common.resmngr.VEMResManagerBase - VEMResManagerBase::checkRbacUserPermissionInternal() : cache kit key=Admin#/SymTesting/Symping72#SOAM_APP_LOGIN cache size=167\n", 228
-> logged by SD as soon as authentication and authorization process is done.

2. lsof
sd 24709 symadmp 25u IPv4 136235231 0t0 TCP gridmstln30p.eur.nsroot.net:53580->gmwmwdcpsp07p.nam.nsroot.net:44443 (ESTABLISHED)
sd 24709 symadmp 26u IPv4 133881256 0t0 TCP gridmstln30p.eur.nsroot.net:40578->gmwgtdcpsp07p.nam.nsroot.net:44443 (ESTABLISHED)
-> checking SD's lsof output, those FDs are TCP sockets between SDSiteminder. It contacted 2 different siteminder server.


• We observe delays are related to siteminder authentications.
• In LND grid case, during authentication phase SD thread had to contact two different siteminder servers. Please turn off second siteminder server (we believe it is gmwmwdcpsp07p based on comparison with SW case), as it may help avoid attempt sending the user query to the second siteminder.
• Or check with Siteminder team if two siteminder servers are in synced in terms of user database. (i.e. if user A is not query-able in siteminder A, it may reach out to B)
• Siteminder plugin code in symphony is owned in Citi, suggest to investigate plugin code to remediate the delays.

  • Guest
    Aug 16, 2021

    .The required enhancement is available on IBM Fix Center: sym-7.2.0.2-build600609-citi