DB2 Pacemaker upgrade break cluster when Red Hat Satellite is not accessible

Hello,

My idea is to add some check if Red Hat satellite is visible into db2installPCMK because if it's not, update will break cluster and DB2 will be unusable until visibility is fixed and db2installPCMK is executed again. As workaround you should before every PM update run some dnf test command (e.g. dnf whatprovides ksh). I've opened PMR for it, but I've got recommendation to open Idea as well.

Below you can see my findings:

My goal was to do manual update of pacemaker from 2.0.5.1 to 2.1.2.4 by db2installPCMK -i , which I run successfully on one node of cluster but failed on second due to lost visibility to satellite.

2. PM stop

[root@db2lop4 ~]# systemctl stop pacemaker
[root@db2lop4 ~]# systemctl stop corosync
[root@db2lop4 ~]# systemctl stop corosync-qdevice

3. Version check

[root@db2lop4 pcmk]# ./db2ckpcmk -v media
2.1.2.4
[root@db2lop4 pcmk]# ./db2ckpcmk -v install
2.0.5.1
[root@db2lop4 pcmk]#

4. update from installation media

[root@db2lop4 pcmk]# ./db2installPCMK -i
Installing "Pacemaker"

Errors during downloading metadata for repository 'kt-bootstrap':

 - Status code: 404 for https://xxxxx.xxxx.xxx.net:8443/bootstrap/8/ppc64le/repodata/repomd.xml (IP: xx.xx.xx.xx)

Error: Failed to download metadata for repo 'kt-bootstrap': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Failure

DBI1065E Program db2installPCMK terminated prematurely.

Explanation:

An error has occurred during the execution and that has caused this program to terminate prematurely.

User response:

Correct the problem and try the command again.

[root@db2lop4 pcmk]#

5. db2installPCMK log file

[root@db2lop4 tmp]# cat db2installPCMK.log.3538264
Pacemaker 2.0.5-10.db2pcmk.el8
Written by Andrew Beekhof
Pacemaker version in the media: 2.1.2.4\n
Corosync version in the media: 3.1.6.2\n
2.0.5.1
3.0.4.2
Updating Subscription Management repositories.
Subscription-manager dependencies for katello-c 23 kB/s | 1.5 kB 00:00
[root@db2lop4 tmp]#

I understand, that problem is on OS side, when PM has problem to reach satellite, because also commands like dnf are failing with same error. However, this update failure shot down db2 instance, which cannot be manually started again, same as PM cluster:

[root@db2lop4 ~]# systemctl start pacemaker
[root@db2lop4 ~]# crm status
Could not connect to the CIB: Transport endpoint is not connected
crm_mon: Error: cluster is not available on this node
ERROR: status: crm_mon (rc=102):
[root@db2lop4 ~]#
[db2prod@db2lop4 ~]$ db2start
08/08/2023 11:30:35 0 0 SQL1042C An unexpected system error occurred.
SQL1032N No start database manager command was issued. SQLSTATE=57019
[db2prod@db2lop4 ~]$

2023-08-08-11.30.35.164361+180 I261066691E455 LEVEL: Event
PID : 2221528 TID : 140736009840240PROC : db2star2
INSTANCE: db2prod NODE : 000
HOSTNAME: db2lop4.tieto.com
FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:7783
MESSAGE : ZRC=0xFFFFF973=-1677
SQL1677N DB2START or DB2STOP processing failed due to a DB2 cluster services error.

Even when I disable cluster via:

crm cluster disable

db2 instance still has:

 Cluster manager = PACEMAKER

so manual start is not working. I believe drop of entire cluster configuration will help, but I don't want to do it.