Example Scenarios
We try to provide in-depth comprehensible example scenarios here. Please make sure that you have a deep understanding of how GSCC, SPORD (if used), IBM Storage Protect and db2 HADR are working and how to configure them before trying to follow these instructions.
Installation Scenarios
GSCC IP Cluster
- In this example, the following input is used:
Two Linux nodes with SLES 11SP3: lxnode1, lxnode2
Each Node has two additional internal disks besides OS disk: /dev/fioa, /dev/fiob
Two ISP instances are created: TSM1, TSM2
Step 1: LVM Configuration
For each ISP instance a separate volume group is created on each Linux Node. The advantage is that ISP instances are not sharing any disks. The main disadvantage is that both ISP instances cannot grow beyond the capacity of that disk. An alternative setup would be to create a common volume group for both ISP instances and share the disk. This would allow the instances to grow differently.
In this example the Fusion I/O Flash Card’s two partitions are dedicated to one instance.
First login to the Linux hosts as root. Create physical volumes, volume groups and logical volumes on both hosts. In this example all configurations regarding naming are the same on both host. The volume group name could be different on both hosts, if this is required. Be aware that if you plan to move the volume group to a SAN later, that even the logical volumes and filesystem mount points should be different on both hosts. In this IP only cluster this would make the configuration more complex.
Example LVM configuration:
lvmdiskscan
/dev/sda1 [ 2.01 GiB]
/dev/root [ 117.99 GiB]
/dev/fioa [ 1248.00 GiB]
/dev/fiob [ 1248.00 GiB]
4 disks
1 partition
0 LVM physical volume whole disks
0 LVM physical volumes
pvcreate /dev/fioa
pvcreate /dev/fiob
vgcreate TSM1 /dev/fioa
vgcreate TSM2 /dev/fiob
lvcreate -L 128G -n tsm1db1 TSM1
lvcreate -L 128G -n tsm1db2 TSM1
lvcreate -L 128G -n tsm1db3 TSM1
lvcreate -L 128G -n tsm1db4 TSM1
lvcreate -L 128G -n tsm1act TSM1
lvcreate -L 128G -n tsm1arc TSM1
lvcreate -L 8G -n tsm1cfg TSM1
lvcreate -L 128G -n tsm2db1 TSM2
lvcreate -L 128G -n tsm2db2 TSM2
lvcreate -L 128G -n tsm2db3 TSM2
lvcreate -L 128G -n tsm2db4 TSM2
lvcreate -L 128G -n tsm2act TSM2
lvcreate -L 128G -n tsm2arc TSM2
lvcreate -L 8G -n tsm2cfg TSM12
lxnode1:/ # vgs
VG #PV #LV #SN Attr VSize VFree
TSM1 1 7 0 wz--n- 1248.00g 0
TSM2 1 7 0 wz--n- 1248.00g 0
lxnode1:/ # lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
tsm1act TSM1 -wi-ao--- 128.00g
tsm1arc TSM1 -wi-ao--- 128.00g
tsm1cfg TSM1 -wi-ao--- 8.00g
tsm1db1 TSM1 -wi-ao--- 128.00g
tsm1db2 TSM1 -wi-ao--- 128.00g
tsm1db3 TSM1 -wi-ao--- 128.00g
tsm1db4 TSM1 -wi-ao--- 128.00g
tsm2act TSM1 -wi-ao--- 128.00g
tsm2arc TSM1 -wi-ao--- 128.00g
tsm2cfg TSM1 -wi-ao--- 8.00g
tsm2db1 TSM1 -wi-ao--- 128.00g
tsm2db2 TSM1 -wi-ao--- 128.00g
tsm2db3 TSM1 -wi-ao--- 128.00g
tsm2db4 TSM1 -wi-ao--- 128.00g
In SLES 11 it might be necessary to activate LVM at system start. The warning about “boot.device-mapper” can be ignored:
chkconfig --level 235 boot.lvm on
Step 2: Database filesystem
Also on both hosts the filesystems are configured including mount points and fstab entries.
The commands for creating filesystems could look like this:
mkfs.ext3 /dev/TSM1/tsm1db1
mkfs.ext3 /dev/TSM1/tsm1db2
mkfs.ext3 /dev/TSM1/tsm1db3
mkfs.ext3 /dev/TSM1/tsm1db4
mkfs.ext3 /dev/TSM1/tsm1act
mkfs.ext3 /dev/TSM1/tsm1arc
mkfs.ext3 /dev/TSM1/tsm1cfg
mkfs.ext3 /dev/TSM2/tsm2db1
mkfs.ext3 /dev/TSM2/tsm2db2
mkfs.ext3 /dev/TSM2/tsm2db3
mkfs.ext3 /dev/TSM2/tsm2db4
mkfs.ext3 /dev/TSM2/tsm2act
mkfs.ext3 /dev/TSM2/tsm2arc
mkfs.ext3 /dev/TSM2/tsm2cfg
tune2fs -c 0 -i 0 /dev/TSM1/tsm1db1
tune2fs -c 0 -i 0 /dev/TSM1/tsm1db2
tune2fs -c 0 -i 0 /dev/TSM1/tsm1db3
tune2fs -c 0 -i 0 /dev/TSM1/tsm1db4
tune2fs -c 0 -i 0 /dev/TSM1/tsm1act
tune2fs -c 0 -i 0 /dev/TSM1/tsm1arc
tune2fs -c 0 -i 0 /dev/TSM1/tsm1cfg
tune2fs -c 0 -i 0 /dev/TSM2/tsm2db1
tune2fs -c 0 -i 0 /dev/TSM2/tsm2db2
tune2fs -c 0 -i 0 /dev/TSM2/tsm2db3
tune2fs -c 0 -i 0 /dev/TSM2/tsm2db4
tune2fs -c 0 -i 0 /dev/TSM2/tsm2act
tune2fs -c 0 -i 0 /dev/TSM2/tsm2arc
tune2fs -c 0 -i 0 /dev/TSM2/tsm2cfg
mkdir -p /tsm/TSM1/db1
mkdir -p /tsm/TSM1/db2
mkdir -p /tsm/TSM1/db3
mkdir -p /tsm/TSM1/db4
mkdir -p /tsm/TSM1/act
mkdir -p /tsm/TSM1/arc
mkdir -p /tsm/TSM1/cfg
mkdir -p /tsm/TSM2/db1
mkdir -p /tsm/TSM2/db2
mkdir -p /tsm/TSM2/db3
mkdir -p /tsm/TSM2/db4
mkdir -p /tsm/TSM2/act
mkdir -p /tsm/TSM2/arc
mkdir -p /tsm/TSM2/cfg
mount /dev/TSM1/tsm1db1 /tsm/TSM1/db1
mount /dev/TSM1/tsm1db2 /tsm/TSM1/db2
mount /dev/TSM1/tsm1db3 /tsm/TSM1/db3
mount /dev/TSM1/tsm1db4 /tsm/TSM1/db4
mount /dev/TSM1/tsm1arc /tsm/TSM1/arc
mount /dev/TSM1/tsm1act /tsm/TSM1/act
mount /dev/TSM1/tsm1cfg /tsm/TSM1/cfg
mount /dev/TSM2/tsm2db1 /tsm/TSM2/db1
mount /dev/TSM2/tsm2db2 /tsm/TSM2/db2
mount /dev/TSM2/tsm2db3 /tsm/TSM2/db3
mount /dev/TSM2/tsm2db4 /tsm/TSM2/db4
mount /dev/TSM2/tsm2arc /tsm/TSM2/arc
mount /dev/TSM2/tsm2act /tsm/TSM2/act
mount /dev/TSM2/tsm2cfg /tsm/TSM2/cfg
cat /etc/fstab
/dev/mapper/TSM1-tsm1db1 /tsm/TSM1/db1 ext3 defaults 0 0
/dev/mapper/TSM1-tsm1db2 /tsm/TSM1/db2 ext3 defaults 0 0
/dev/mapper/TSM1-tsm1db3 /tsm/TSM1/db3 ext3 defaults 0 0
/dev/mapper/TSM1-tsm1db4 /tsm/TSM1/db4 ext3 defaults 0 0
/dev/mapper/TSM1-tsm1arc /tsm/TSM1/arc ext3 defaults 0 0
/dev/mapper/TSM1-tsm1act /tsm/TSM1/act ext3 defaults 0 0
/dev/mapper/TSM1-tsm1cfg /tsm/TSM1/cfg ext3 defaults 0 0
/dev/mapper/TSM2-tsm2db1 /tsm/TSM2/db1 ext3 defaults 0 0
/dev/mapper/TSM2-tsm2db2 /tsm/TSM2/db2 ext3 defaults 0 0
/dev/mapper/TSM2-tsm2db3 /tsm/TSM2/db3 ext3 defaults 0 0
/dev/mapper/TSM2-tsm2db4 /tsm/TSM2/db4 ext3 defaults 0 0
/dev/mapper/TSM2-tsm2arc /tsm/TSM2/arc ext3 defaults 0 0
/dev/mapper/TSM2-tsm2act /tsm/TSM2/act ext3 defaults 0 0
/dev/mapper/TSM2-tsm2cfg /tsm/TSM2/cfg ext3 defaults 0 0
Step 3: ISP Installation
Now TSM can be installed. This covers only the main step. Consult TSM documentation for detailed description of TSM installation. On both Linux hosts the TSM instance users and groups need to be created. The user names are the same as the TSM instance. The user and group ids are chosen to be the same as the TSM instances TCP ports:
groupadd -g 1601 tsm1
useradd -d /tsm/TSM1/cfg -g tsm1 -s /bin/bash -u 1601 tsm1
groupadd -g 1602 tsm2
useradd -d /tsm/TSM2/cfg -g tsm2 -s /bin/bash -u 1602 tsm2
These users need to be tuned in the Linux limits configuration file to allow more open files. (RHEL: /etc/security/limits.d/90-nproc.conf):
grep tsm /etc/security/limits.conf
tsm1 hard nofile 65536
tsm2 hard nofile 65536
Then the previously created filesystem’s ownership has to be changed:
cd /tsm
chown -R tsm1:tsm1 TSM1
chown -R tsm2:tsm2 TSM2
Now the latest ISP client will be installed:
rpm -ivh ./gskcrypt64-8.0.14.43.linux.x86_64.rpm
rpm -ivh ./gskssl64-8.0.14.43.linux.x86_64.rpm
rpm -ivh ./TIVsm-API64.x86_64.rpm
rpm -ivh ./TIVsm-BA.x86_64.rpm
The ISP server is installed using a previously response file in this case:
./install.sh -s -input TSMresponse.xml -acceptLicense
For the ISP database backup the following lines are added to the dsm.sys ISP client configuration file:
* TSM DB2
SERVERNAME TSMDBMGR_TSM1
TCPPORT 1601
TCPSERVERADDRESS 127.0.0.1
NODENAME $$_TSMDBMGR_$$
PASSWORDDIR /tsm/TSM1/cfg
ERRORLOGNAME /tsm/TSM1/cfg/tsmdbmgr.log
SERVERNAME TSMDBMGR_TSM2
TCPPORT 1602
TCPSERVERADDRESS 127.0.0.1
* PASSWORDACCESS generate
NODENAME $$_TSMDBMGR_$$
PASSWORDDIR /tsm/TSM2/cfg
ERRORLOGNAME /tsm/TSM2/cfg/tsmdbmgr.log
Be aware that with ISP/TSM 7 it is no longer necessary to save a password. Furthermore the API configuration is in a dedicated path. In this example a link is created to the standard dsm.sys file:
cd /opt/tivoli/tsm/server/bin/dbbkapi
ln -s /opt/tivoli/tsm/client/ba/bin/dsm.sys .
Step 4: ISP First Steps
In this step the DB2 database instances are created on both hosts. Most steps need to be performed for the primary and the standby DB2 instance. Only when ISP is started in foreground after the initial format in order to register users the steps are only needed for the primary instance:
/opt/tivoli/tsm/db2/instance/db2icrt -a server -s ese -u tsm1 tsm1
/opt/tivoli/tsm/db2/instance/db2icrt -a server -s ese -u tsm2 tsm2
The users’ profiles should look like this:
su - tsm1
cat .profile
export LANG=en_US
export LC_ALL=en_US
if [ -f /tsm/TSM1/cfg/sqllib/db2profile ]; then
. /tsm/TSM1/cfg/sqllib/db2profile
fi
The db2 user profile (tsm/TSM1/cfg/sqllib/userprofile) will contain this:
LD_LIBRARY_PATH=/opt/tivoli/tsm/server/bin/dbbkapi:/usr/local/ibm/gsk8_64/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
DSMI_CONFIG=/tsm/TSM1/cfg/tsmdbmgr.opt
DSMI_DIR=/opt/tivoli/tsm/server/bin/dbbkapi
DSMI_LOG=/tsm/TSM1/cfg
export DSMI_CONFIG DSMI_DIR DSMI_LOG
Before the ISP instance will be formatted these preparation tasks are performed. The dsmserv.opt file needs to be adjusted to customer requirements:
lxnode1:/tsm/TSM1/cfg # ln –s /opt/tivoli/tsm/server/bin/dsmserv dsmserv_TSM1
lxnode1:/tsm/TSM1/cfg # cat tsmdbmgr.opt
SERVERNAME TSMDBMGR_TSM1
lxnode1:/tsm/TSM1/cfg # cat dsmserv.opt
*
* TSM1 - dsmserv.opt
*
COMMmethod TCPIP
TCPPORT 1601
* SSLTCPPORT 1611
TCPADMINPORT 1621
SSLTCPADMINPORT 1631
ADMINONCLIENTPORT NO
TCPWINDOWSIZE 128
TCPBUFSIZE 64
TCPNODELAY YES
DIRECTIO NO
MAXSessions 200
COMMTimeout 3600
IDLETIMEOUT 240
RESOURCETIMEOUT 60
EXPInterval 0
EXPQuiet yes
TXNGroupmax 8192
MOVEBATCHSIZE 1000
MOVESizethresh 32768
DNSLOOKUP YES
LANGUAGE en_US
SEARCHMPQUEUE
REPORTRETRIEVE YES
ADMINCOMMTIMEOUT 3600
ADMINIDLETIMEOUT 600
AliasHalt stoptsm
SANDISCOVERY OFF
SANDISCOVERYTIMEOUT 30
* REORGBEGINTIME 14:30
* REORGDURATION 4
ALLOWREORGINDEX YES
ALLOWREORGTABLE YES
* DISABLEREORGTable
* DISABLEREORGIndex BF_AGGREGATED_BITFILES,BF_BITFILE_EXTENTS,BACKUP_OBJECTS,ARCHIVE_OBJECTS
* DISABLEREORGCleanupindex
* KEEPALIVE YES
DBDIAGLOGSIZE 128
BACKUPINITIATIONROOT OFF
VOLUMEHistory /tsm/TSM1/cfg/volhist.out
DEVCONFig /tsm/TSM1/cfg/devconf.out
As the ISP instance user set these parameters:
su – tsm1
db2start
db2 update dbm cfg using dftdbpath /tsm/TSM1/cfg
db2set -i tsm1 DB2CODEPAGE=819
Finally the ISP instance will be formatted. This step is also required on the planned standby instance for this ISP server:
su – tsm1
./dsmserv_TSM1 format dbdir=/tsm/TSM1/db1,/tsm/TSM1/db2,/tsm/TSM1/db3,/tsm/TSM1/db4 activelogsize=102400 activelogdir=/tsm/TSM1/act archlogdir=/tsm/TSM1/arc
Now the first configuration steps are done for ISP. This step is only needed for the primary database. The instance is started in foreground. Here also the GSCC user is created (gscc):
./dsmserv_TSM1
set servername tsm1
set serverpassword tsm1
reg lic file=/opt/tivoli/tsm/server/bin/tsmee.lic
reg admin admin admin passexp=0
grant auth admin class=system
reg admin gscc gscc passexp=0
grant auth gscc class=system
DEFINE DEVCLASS TSMDBB devt=file maxcap=10g dir=/tsm/TSM1/ISI2/sf/tsmdbb
set dbrecovery TSMDBB
stoptsm
After the ISP instance was stopped it can be restarted in the background. It should be verified if a database backup is working as expected before continuing with the next step.
/tsm/TSM1/cfg/dsmserv_TSM1 -i ${lockpath} –quiet >/dev/null 2>&1 &
Step 5: HADR Configuration
The TCP ports used for HADR are added to the /etc/services file:
echo DB2_tsm1_HADR 60100/tcp >> /etc/services
echo DB2_tsm2_HADR 60200/tcp >> /etc/services
The next task is to configure HADR on all databases. This is an example for TSM1 including both cluster sides.
This is on the primary side. After the configuration a database restart is required in order to activate the settings. Before the settings the initial db2 database backup is performed:
db2 connect to tsmdb1
db2 GRANT USAGE ON WORKLOAD SYSDEFAULTUSERWORKLOAD TO PUBLIC
db2 backup database TSMDB1 online to /tsm/TSM1/ISILON01/sf/db2backup COMPRESS include logs
db2 update db cfg for TSMDB1 using LOGINDEXBUILD yes
db2 update db cfg for TSMDB1 using HADR_LOCAL_HOST lxhost1
db2 update db cfg for TSMDB1 using HADR_LOCAL_SVC 60100
db2 update db cfg for TSMDB1 using HADR_REMOTE_HOST lxhost2
db2 update db cfg for TSMDB1 using HADR_REMOTE_SVC 60100
db2 update db cfg for TSMDB1 using HADR_REMOTE_INST tsm1
db2 update db cfg for TSMDB1 using HADR_SYNCMODE NEARSYNC
db2 update db cfg for TSMDB1 using HADR_PEER_WINDOW 300
On the standby database the configuration also needs to be done. First we restore from the db2 backup to sync the databases:
db2 restore db TSMDB1 from /tsm/TSM1/ISILON01/sf/db2backup on /tsm/TSM1/db1,/tsm/TSM1/db2,/tsm/TSM1/db3,/tsm/TSM1/db4 into TSMDB1 logtarget /tsm/TSM1/arc replace history file
When the restore has been successful, we can set HADR and finally activate the database:
db2start
db2 update db cfg for TSMDB1 using LOGINDEXBUILD yes
db2 update db cfg for TSMDB1 using HADR_LOCAL_HOST 53.51.130.153
db2 update db cfg for TSMDB1 using HADR_LOCAL_SVC 60100
db2 update db cfg for TSMDB1 using HADR_REMOTE_HOST 53.51.130.151
db2 update db cfg for TSMDB1 using HADR_REMOTE_SVC 60100
db2 update db cfg for TSMDB1 using HADR_REMOTE_INST tsm1
db2 update db cfg for TSMDB1 using HADR_SYNCMODE NEARSYNC
db2 update db cfg for TSMDB1 using HADR_PEER_WINDOW 300
db2start
db2 start hadr on db tsmdb1 as standby
On the primary side HADR is also activated. This is done while ISP is running:
db2 start hadr on db tsmdb1 as primary
db2pd –hadr –db tsmdb1
Database Member 0 -- Database TSMDB1 -- Active -- Up 1 days 01:01:40 -- Date 2015-03-23-01.13.43.718655
HADR_ROLE = PRIMARY
REPLAY_TYPE = PHYSICAL
HADR_SYNCMODE = NEARSYNC
STANDBY_ID = 1
LOG_STREAM_ID = 0
HADR_STATE = PEER
HADR_FLAGS =
PRIMARY_MEMBER_HOST = lxnode1
PRIMARY_INSTANCE = tsm1
PRIMARY_MEMBER = 0
STANDBY_MEMBER_HOST = lxnode2
STANDBY_INSTANCE = tsm1
STANDBY_MEMBER = 0
HADR_CONNECT_STATUS = CONNECTED
HADR_CONNECT_STATUS_TIME = 03/22/2015 00:32:04.317345 (1426980724)
HEARTBEAT_INTERVAL(seconds) = 30
HADR_TIMEOUT(seconds) = 120
TIME_SINCE_LAST_RECV(seconds) = 3
PEER_WAIT_LIMIT(seconds) = 0
LOG_HADR_WAIT_CUR(seconds) = 0.000
LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.001156
LOG_HADR_WAIT_ACCUMULATED(seconds) = 333.423
LOG_HADR_WAIT_COUNT = 222865
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 16384
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 87380
PRIMARY_LOG_FILE,PAGE,POS = S0000041.LOG, 28504, 22057976497
STANDBY_LOG_FILE,PAGE,POS = S0000041.LOG, 28504, 22057976497
HADR_LOG_GAP(bytes) = 0
STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000041.LOG, 28504, 22057976497
STANDBY_RECV_REPLAY_GAP(bytes) = 0
PRIMARY_LOG_TIME = 03/23/2015 01:13:39.000000 (1427069619)
STANDBY_LOG_TIME = 03/23/2015 01:13:39.000000 (1427069619)
STANDBY_REPLAY_LOG_TIME = 03/23/2015 01:13:39.000000 (1427069619)
STANDBY_RECV_BUF_SIZE(pages) = 4096
STANDBY_RECV_BUF_PERCENT = 0
STANDBY_SPOOL_LIMIT(pages) = 3145728
STANDBY_SPOOL_PERCENT = 100
PEER_WINDOW(seconds) = 300
PEER_WINDOW_END = 03/23/2015 01:18:39.000000 (1427069919)
READS_ON_STANDBY_ENABLED = N
The following files should be transferred from the primary configuration directory to the standby side:
cd /tsm/TSM1/cfg
tar -cvf cert_tsm1.tar ./cert*
./cert.arm
./cert.crl
./cert.kdb
./cert.rdb
./cert.sth
./cert256.arm
tar -cvf cfg_tsm1.tar dsmserv.opt NODELOCK dsmkeydb.* volhist.out devconf.out dsmserv.dbid
dsmserv.opt
NODELOCK
dsmkeydb.sth
dsmkeydb.kdb
volhist.out
devconf.out
dsmserv.dbid
scp *.tar lxnode2:/tsm/TSM1/cfg
Password:
cert_tsm1.tar 100% 140KB 140.0KB/s 00:00
cfg_tsm1.tar 100% 90KB 90.0KB/s 00:00
On the other host these files are unpacked:
ssh lxnode2
cd /tsm/TSM1/cfg
tar -xvf cert_tsm1.tar
./cert.arm
./cert.crl
./cert.kdb
./cert.rdb
./cert.sth
./cert256.arm
tar -xvf cfg_tsm1.tar
NODELOCK
dsmkeydb.sth
dsmkeydb.kdb
volhist.out
devconf.out
dsmserv.dbid
Step 6: GSCC Installation
On both Linux hosts the software is installed first:
rpm -ihv gscc-3.8.0.409-0.x86_64.rpm
Set a link for ksh93:
ln –s /bin/ksh /bin/ksh93
ln -s /usr/bin/ksh /bin/ksh93att
The GSCC configuration files are placed in /etc/gscc. A typical configuration could look like this: The files will be created in the next steps. They have to be created on both Linux nodes. Be aware that the MEMBER.def files are not identical on the nodes as they contain information about the remote node. The GSCC member names are the same as the ISP instance names:
lax04:/etc/gscc # ls -la
total 84
drwxr-xr-x 4 root root 4096 Mar 22 00:04 .
drwxr-xr-x 109 root root 12288 Mar 19 07:23 ..
-rw------- 1 root root 13 Feb 21 17:04 .gscckey
-rw-r--r-- 1 root root 4005 Feb 18 16:59 README
-rw-r--r-- 1 root root 3671 Mar 21 23:44 TSM1.def
-rw-r--r-- 1 root root 296 Feb 21 16:41 TSM1.tsmlocalfs
-rw-r--r-- 1 root root 3674 Feb 23 12:37 TSM2.def
-rw-r--r-- 1 root root 296 Feb 23 12:38 TSM1.tsmlocalfs
-rw-r--r-- 1 root root 188 Feb 23 12:37 cl.def
-rw-r--r-- 1 root root 1461 Mar 21 23:37 gscc.conf
-rw-r--r-- 1 root root 10 Feb 18 17:00 gscc.version
-rw-r--r-- 1 root root 15 Feb 21 16:05 heartbeat
-rw-r--r-- 1 root root 10 Feb 21 16:03 localhost
-rw-r--r-- 1 root root 143 Mar 8 20:25 logrot.conf
-rw-r--r-- 1 root root 40 Feb 21 16:00 passweb
-rw-r--r-- 1 root root 40 Feb 21 16:00 alertscript
drwxr-xr-x 2 root root 4096 Feb 23 12:59 samples
drwxr-xr-x 2 root root 4096 Feb 21 00:04 ui
-rw-r--r-- 1 root root 215 Feb 21 17:07 web.def
The GSCC members are defined in cl.def. Be aware, that the second column is used for the ISP/TSM instance name, also referred as member team name (grouping all members together, which support the same ISP instance):
cat /etc/gscc/cl.def
#
# Cluster Object Definitions
#
# (C) General Storage, 2018
#
#
# <Member_Name> <Member_Team> <Member Description>
#
TSM1 TSM1 Production
TSM2 TSM2 Production
The central configuration file:
cat /etc/gscc/gscc.conf
# Activate Auto register
AutoRegister Yes
#Lifetime 1200
#Loclifetime 600
FileLogLevel Debug
#FileLogLevel Info
#FileLogLevel Notice
#FileLogLevel Warning
#FileLogLevel Error
Environment LC_ALL=C
Environment LANG=C
Environment StatusReadTimeout=300
Environment StatusFailureThreshold=5
Environment RemoteFailureThreshold=2
Environment TSMKillCounter=40
Environment ForceDuration=600
#Environment ForceDuration=0
Environment ErrCountMax=1
#Environment TSMAltPort=0
#Environment TSMAltPort=3940
Environment sshPath=/usr/bin/ssh
Environment hadrCOM=intern
Environment SSLPATH=/usr/bin
For each GSCC member a local filesystem configuration file is required (this is only needed for GSCCAD):
cat /etc/gscc/TSM1.tsmlocalfs
/dev/mapper/TSM-tsm1cfg /tsm/TSM1/cfg
/dev/mapper/TSM-tsm1db1 /tsm/TSM1/db1
/dev/mapper/TSM-tsm1db2 /tsm/TSM1/db2
/dev/mapper/TSM-tsm1db3 /tsm/TSM1/db3
/dev/mapper/TSM-tsm1db4 /tsm/TSM1/db4
/dev/mapper/TSM-tsm1arc /tsm/TSM1/arc
/dev/mapper/TSM-tsm1act /tsm/TSM1/act
Each member is configured with a definition file “MEMBER.def”. The file is required on both Linux nodes. In contrast to the previous configuration files, the files on the hosts have different entries for “peer_ip”. Otherwise they should be the same as in this example the Linux hosts are configured a like. There are two LACP interfaces in this example (bond0, bond1). The service ip is the virtual IP alias which is not configured on the physical interface. The value is the same for Primary and Standby. The only difference is the “peer_ip” which points to the remote node. The “home_system_id” is the preferred host for this member. There it needs to be the same.
You can get the value by using the “gshostid” tool:
lxnode1:/ #gshostid
lxnode1 33359782
lxnode1:/ #cat /etc/gscc/TSM1.def
#
# (C) General Storage
#
#
# TSM1.def
# this file is needed for each Cluster Member !
# default path: $HOME/.gscc/sample.def
#
# use ":" for any blank in var-values
#
# <var_name> <value> <discription>
expert_domain edUNIX_T6_HADR Expert Domain
am_tsm_server_path /tsm/TSM1/cfg Path to TSM Server
service_interface bond0,bond1 Comma separated TSM Server Network Interface(s)
service_ip_address 192.168.178.111,192.168.177.111 Comma separated TSM Server IP Addresse(s)
service_netmask 255.255.255.0,255.255.255.0 Comma separated TSM Server Netmask(s)
service_broadcast 192.168.178.255,192.168.177.255 Comma separated TSM Server Netmask(s)
log_file /var/gscc/logs/im.tsm1.log Activity Log File
am_log_file /var/gscc/logs/tsm1.log Action method log file
tsm_auth_string dsmadmc:-se=tsm1_local:SECURED TSM Authentication String
peer_tsm_auth_string dsmadmc:-se=tsm1:SECURED TSM Authentication String - remote removed
forces_command - Reset server verification key
home_system_id 33359782 System ID
member_type prod Member Type: prod | stby
suff_pings 1 Number of Clients that have to be reached
client_ip_address_file /etc/gscc/clipup File for Client IPs
heartbeat /etc/gscc/heartbeat Heartbeat IP Address
peer_ip 192.168.178.102 IP Address of Peer TSM Server
peer_port 3939 IP Port of Peer TSM Server
peer_member_name TSM1 Peer Member ID
local_member_name TSM1 Member ID
oserip_suff_pings 1 Pings to test against Shared Router
tsm_check_string dsmadmc:-se=tsm1_local:SECURED:q:status TSM Check String
otsm_check_string dsmadmc:-se=tsm1_remote:SECURED:q:status Other TSM Check String
tsm_startup_tolerance 600 Startup Time in Seconds for dsmserv
errorlistfile /var/gscc/logs/tsm1.errorlist.out Error List File
pagecommand /etc/gscc/alert Command to page operator
pageinterval 600 Time between paging the Operator on error
trigger auto How Operator triggers Failed status (key: auto)
dbb_rules hadr Full/incremental dbb interval and frequency
#dbb_rules extern Full/incremental dbb interval and frequency
tsm_fulldevc tsmdbb TSM device class for full database backups
tsm_incrdevc tsmdbb TSM device clss for incremental database backups
hadr_local_if bond0 HADR
hadr_local_ip 192.168.178.101 HADR
hadr_local_mask 255.255.255.0 HADR
hadr_local_cast 192.168.177.102 HADR
lxnode2:/ #cat /etc/gscc/TSM1.def
#
# (C) General Storage
#
#
# TSM1.def
# this file is needed for each Cluster Member !
# default path: $HOME/.gscc/sample.def
#
# use ":" for any blank in var-values
#
# <var_name> <value> <discription>
expert_domain edUNIX_T6_HADR Expert Domain
am_tsm_server_path /tsm/TSM1/cfg Path to TSM Server
service_interface bond0,bond1 Comma separated TSM Server Network Interface(s)
service_ip_address 192.168.178.111,192.168.177.111 Comma separated TSM Server IP Addresse(s)
service_netmask 255.255.255.0,255.255.255.0 Comma separated TSM Server Netmask(s)
service_broadcast 192.168.178.255,192.168.177.255 Comma separated TSM Server Netmask(s)
log_file /var/gscc/logs/im.tsm1.log Activity Log File
am_log_file /var/gscc/logs/tsm1.log Action method log file
tsm_auth_string dsmadmc:-se=tsm1_local:SECURED TSM Authentication String
peer_tsm_auth_string dsmadmc:-se=tsm1:SECURED TSM Authentication String - remote removed
forces_command - Reset server verification key
home_system_id 33359782 System ID
member_type stby Member Type: prod | stby
suff_pings 1 Number of Clients that have to be reached
client_ip_address_file /etc/gscc/clipup File for Client IPs
heartbeat /etc/gscc/heartbeat Heartbeat IP Address
peer_ip 192.168.178.101 IP Address of Peer TSM Server
peer_port 3939 IP Port of Peer TSM Server
peer_member_name TSM1 Peer Member ID
local_member_name TSM1 Member ID
oserip_suff_pings 1 Pings to test against Shared Router
tsm_check_string dsmadmc:-se=tsm1_local:SECURED:q:status TSM Check String
otsm_check_string dsmadmc:-se=tsm1_remote:SECURED:q:status Other TSM Check String
tsm_startup_tolerance 600 Startup Time in Seconds for dsmserv
errorlistfile /var/gscc/logs/tsm1.errorlist.out Error List File
pagecommand /etc/gscc/alert Command to page operator
pageinterval 600 Time between paging the Operator on error
trigger auto How Operator triggers Failed status (key: auto)
dbb_rules hadr Full/incremental dbb interval and frequency
#dbb_rules extern Full/incremental dbb interval and frequency
tsm_fulldevc tsmdbb TSM device class for full database backups
tsm_incrdevc tsmdbb TSM device clss for incremental database backups
hadr_local_if bond0 HADR
hadr_local_ip 192.168.178.101 HADR
hadr_local_mask 255.255.255.0 HADR
hadr_local_cast 192.168.177.102 HADR
The heartbeat file contains the remote IP addresses:
cat /etc/gscc/heartbeat
192.168.178.102
192.168.177.102
192.168.176.102
The clipup file contains usually also the heartbeat IP addresses and one more address for a router, gateway or other available IP address, which will be the quorum tiebreaker:
cat /etc/gscc/clipup
192.168.178.102
192.168.177.102
192.168.176.102
192.168.178.1
Also the log rotation can be configured now. You can add the log rotation later to the crontab:
cat /etc/gscc/logrot.conf
/var/gscc/logs/gscc.log 1000000 5
/var/gscc/logs/im.tsm1.log 200000 5
/var/gscc/logs/TSM1.sync 200000 5
/var/gscc/logs/im.tsm2.log 200000 5
/var/gscc/logs/TSM2.sync 200000 5
crontab –l|grep logrot
0 16 * * * /opt/generalstorage/gscc/bin/logrot >/var/gscc/logs/logrot.out 2>&1
echo ‘0 16 * * * /opt/generalstorage/gscc/bin/logrot >/var/gscc/logs/logrot.out 2>&1’ > /etc/cron.d/gscc_logrot.cron
Now add GSCC binaries to the PATH by creating /etc/profile.d/gscc.sh with this line and set the PATH afterwards:
cat /etc/profile.d/gscc.sh
PATH=${PATH}:/opt/generalstorage/gscc/bin
. /etc/profile.d/gscc.sh
GSCC should also be added to the services. This example is SLES11, so the System V init is used. For SLES12 or RHEL 7 see the system configuration below:
cd /etc/init.d
ln -s /opt/generalstorage/gscc/bin/rc.gscc gscc
chkconfig gscc 35
chkconfig --list gscc
gscc 0:off 1:off 2:off 3:on 4:off 5:on 6:off
Usage:
service gscc {start|stop|restart|status}
If you used GSCCAD with SSL access, this additional service is required:
ln -s /opt/generalstorage/gscc/bin/rc.gsuid gsuid
chkconfig gsuid 35
chkconfig --list gsuid
gsuid 0:off 1:off 2:off 3:on 4:off 5:on 6:off
In case SLES12 / RHEL 7 is installed, the following steps are required instead:
cp /opt/generalstorage/gscc/bin/gscc.service /etc/systemd/system
systemctl enable gscc.service
cp /opt/generalstorage/gscc/bin/gsuid.service /etc/systemd/system
systemctl enable gsuid.service
Set “Operator” as the start status for all members:
lxnode1:/ # setoper
INFO: Member TSM1 set to Operator at GSCC startup
INFO: Member TSM2 set to Operator at GSCC startup
The next step is to save the ISP server administrator password for GSCC:
lxnode1:/ # settsmadm
General Storage Cluster Controller for TSM - Version 3.8.0.1
TSM Admin Registration – SECURED
Please enter TSM user id: gscc
Please enter TSM password:
Please enter GSCC key:
Please enter GSCC key again:
INFO: key verification OK.
INFO: Encrypting and saving input.
INFO: Completed.
GSCC can be started now. This is SLES 11 again, see below for SLES12 and RHEL 7:
lxnodes1:/etc/gscc # service gscc start
Starting gscc daemons: done
If GSCCAD is used with SSL, the gsuid daemon needs to be started aswell:
lxnodes1:/etc/gscc # service gsuid start
Starting gsuid daemons: done
In SLES 12 and RHEL the processes are started like this:
lxnodes1:/etc/gscc # systemctl start gscc
lxnodes1:/etc/gscc # systemctl start gsuid
You can verify the startup status in /var/gscc/logs/gscc.log. If there were no obvious errors, GSCC can be verified by stopping and starting the TSM instances in “gsccadm”. This is an example how the output could look like:
gsccadm
General Storage Cluster Controller for TSM - Version 3.8.0.409
Command Line Administrative Interface
(C) Copyright General Storage 2015
Please enter user id: admin
Please enter password:
GSCC CLI-Version: 1988
gscc lxnode1>q status
150323080635 TSM1 Operator
150323080635 TSM2 Operator
gscc lxnode1>q tsm
TSM1 running_p_peer
TSM2 db2sysc_s_peer
gscc lxnode1>lxnode2:q tsm
lxnode2 returns:
TSM1 db2sysc_s_peer
TSM2 running_p_peer
SPORD - IP Cluster
- In this example, we install SPORD and GSCC in the following environment:
Two Linux systems with RHEL9.5: rhelgscc-5.gs.com (192.168.178.138) & rhelgscc-6.gs.com (192.168.178.142)
A single ISP Instance will be installed: ISP01
rhelgscc-5.gs.com will be our Primary system during the installation
rhelgscc-6.gs.com will be our Standby system during the installation
Additionally, we have an NFS share mounted on both systems as /NFS/ which is used to store and access the initial database backups as well as configuration files needed on both sites. The software package for SPORD and the used container image are also accessible via the /NFS-share.
For each installation step, the headline will either say Primary, Standby or Both, meaning that those steps should be done either on the designated primary cluster system, the standby system, or both of them.
Step 1: Installing and checking Prerequisites (Both)
Make sure that the container application of your choice is installed, in our case, podman:
[root@rhelgscc-5 ~]# rpm -qa|grep podman
podman-5.2.2-13.el9_5.x86_64
cockpit-podman-93.1-1.el9_5.noarch
Check for the spord package and the container image:
[root@rhelgscc-5 ~]# ll /NFS/SOFTWARE/SPORD/spord-3.8.9.1-0.x86_64.rpm
-rwxrwx---+ 1 root root 2496626 Feb 26 10:05 /NFS/SOFTWARE/SPORD/spord-3.8.9.1-0.x86_64.rpm
[root@rhelgscc-5 ~]# ll /NFS/SOFTWARE/SPORD/IMAGES/ubi8SP8.1.25-GSCC-DSMISI.tar
-rwxrwx---+ 1 root root 2534952448 Feb 26 09:51 /NFS/SOFTWARE/SPORD/IMAGES/ubi8SP8.1.25-GSCC-DSMISI.tar
Install SPORD:
[root@rhelgscc-5 ~]# yum localinstall -y /NFS/SOFTWARE/SPORD/spord-3.8.9.1-0.x86_64.rpm
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 12 kB/s | 4.1 kB 00:00
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 3.7 kB/s | 4.5 kB 00:01
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 36 MB/s | 50 MB 00:01
Last metadata expiration check: 0:00:01 ago on Wed 26 Feb 2025 10:07:12 AM CET.
Dependencies resolved.
==============================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================
Installing:
spord x86_64 3.8.9.1-0 @commandline 2.4 M
Installing dependencies:
ksh x86_64 3:1.0.6-4.el9_5 rhel-9-for-x86_64-appstream-rpms 883 k
perl x86_64 4:5.32.1-481.el9 rhel-9-for-x86_64-appstream-rpms 12 k
...
...
Verifying : ksh-3:1.0.6-4.el9_5.x86_64 161/162
Verifying : spord-3.8.9.1-0.x86_64 162/162
Installed products updated.
...
sombok-2.4.0-16.el9.x86_64
spord-3.8.9.1-0.x86_64 systemtap-sdt-devel-5.1-4.el9_5.x86_64
Complete!
Load the container image to the local image library:
[root@rhelgscc-5 ~]# spordadm loadimage /NFS/SOFTWARE/SPORD/IMAGES/ubi8SP8.1.25-GSCC-DSMISI.tar
[INFO] Loading container image /NFS/SOFTWARE/SPORD/IMAGES/ubi8SP8.1.25-GSCC-DSMISI.tar...
Getting image source signatures
Copying blob 148a3ed2f70e done |
Copying blob c0e1d129bedc done |
Copying config d9b3c7b8d0 done |
Writing manifest to image destination
Loaded image: localhost/ubi8:SP8.1.25-GSCC-DSMISI
Step 2: Configure SPORD (Both)
Create a SPORD configuration file or copy from /etc/spord/samples/spord.conf and edit:
[root@rhelgscc-6 ~]# cat /etc/spord/spord.conf
#######################################################
# General Storage Software GmbH
# SPORD - Spectrum Protect On Resilient Docker
#
# spord.conf
#######################################################
SPORDNAME ISP01
SPORDHOME /SPORD/HOME/ISP01
DOCKERIMAGE localhost/ubi8:SP8.1.25-GSCC-DSMISI
ADDMAPPING /SPORD/ISP/ISP01,/SPORD/GSCC/ISP01/etcgscc:/etc/gscc,/SPORD/GSCC/ISP01/vargscc:/var/gscc,/NFS/dbbackups
DKDFILE ISP01.dkd
ADDOPTION --pids-limit -1
Create all SPORD related directories:
[root@rhelgscc-5 ~]# mkdir -p /SPORD/HOME/ISP01
[root@rhelgscc-5 ~]# mkdir -p /SPORD/GSCC/ISP01/etcgscc
[root@rhelgscc-5 ~]# mkdir -p /SPORD/GSCC/ISP01/vargscc/logs
[root@rhelgscc-5 ~]# mkdir -p /NFS/dbbackups/
[root@rhelgscc-5 ~]# mkdir -p /SPORD/ISP/ISP01
Create the container configuration file ISP01.dkd in the specified SPORDHOME directory (e.g. /SPORD/HOME/ISP01) or copy from /etc/spord/samples/TSM1.dkd and edit accordingly:
[root@rhelgscc-5 samples]# vi /SPORD/HOME/ISP01/ISP01.dkd
# General Storage ISP Server Image for Docker Configuration File
user isp01
userid 1601
group isp01
groupid 1601
home /SPORD/ISP/ISP01/config
dbdir /SPORD/ISP/ISP01/db1,/SPORD/ISP/ISP01/db2,/SPORD/ISP/ISP01/db3,/SPORD/ISP/ISP01/db4
actlog /SPORD/ISP/ISP01/actlog
arclog /SPORD/ISP/ISP01/arclog
actlogsize 131072
instdir /SPORD/ISP/ISP01/config
licdir /SPORD/HOME/ISP01
tcpport 1601
Add the user, group and directories as defined in the configuration files:
[root@rhelgscc-5 samples]# useradd -d /SPORD/ISP/ISP01/config -m -u 1601 isp01
Add the directories for the IBM Storage Protect Instance as defined in the docker configuration file:
[root@rhelgscc-5 samples]# mkdir -p /SPORD/ISP/ISP01/db1 /SPORD/ISP/ISP01/db2 /SPORD/ISP/ISP01/db3 /SPORD/ISP/ISP01/db4 /SPORD/ISP/ISP01/actlog /SPORD/ISP/ISP01/arclog
Change the ownership for those directories to the instance owner:
[root@rhelgscc-5 samples]# chown -R isp01:isp01 /SPORD/ISP/ISP01/*
Upload the Storage Protect License file (dsmreg.sl) and the DB2 License file (db2*.lic, exact name depends on the package) obtainable from your licensed ISP download package. The location of these files differ from version to version, in our case they were located as following:
SP_8.1.25_LIN86_SERSTG_AGT_ML.bin\repository\native\com.tivoli.dsm.license.linux.x86_64_8.1.25.20241205_0440.zip\server\bin\dsmreg.sl
SP_8.1.25_LIN86_SERSTG_AGT_ML.bin\repository\native\com.tivoli.dsm.server.db2.linux.x86_64_11.5.9000.7534.zip\ese\db2\license\db2awse_o.lic
Both files should be uploaded to /SPORD/HOME/ISP01 (licdir directory in ISP01.dkd)
Caution
The db2*.lic file needs to be renamed to ‘db2ese.lic’, otherwise SPORD will not recognize it correctly!
Step 3: Initial Container creation (Both)
SPORD is now ready for the initial Container start. You can either preview the docker command if you want to review the configuration prior to starting or directly run the container by providing the -execute parameter to spordadm <Instance> create:
[root@rhelgscc-6 ISP01]# spordadm ISP01 create
[INFO] Command PREVIEW for "ISP01" SPORD ISP creation.
[INFO] SPORD: /opt/generalstorage/spord/bin/spord create -sp=ISP01
Wed Feb 26 11:44:58 2025
##################################################################
# General Storage SPORD - (c) 2022 General Storage Software GmbH #
##################################################################
INFO: Reading configuration file /etc/spord/spord.conf.
INFO: Receiving configuration for ISP01!
------------------------------
SPORDNAME: ISP01
SPORDBIN:
SPCONTBIN:
SPORDCONF: /etc/spord/spord.conf
SPORDHOME: /SPORD/HOME/ISP01
GSCCVERSION:
CUSTOMFILE:
DOCKIMG: localhost/ubi8:SP8.1.25-GSCC-DSMISI
ADDMAPPING: /SPORD/ISP/ISP01,/SPORD/GSCC/ISP01/etcgscc:/etc/gscc,/SPORD/GSCC/ISP01/vargscc:/var/gscc,/NFS/dbbackups,/SPORD/STG/ISP01:/SPORD/STG/IS
ADDOPTION: --pids-limit -1
MAINCOMMAND: create
ARGUMENTS:
------------------------------
INFO: Using docker image localhost/ubi8:SP8.1.25-GSCC-DSMISI!
CREATE localhost/ubi8:SP8.1.25-GSCC-DSMISI
DOCKER COMMAND(NEW): podman run -it --rm --privileged --net=host --ipc=host --name=ISP01 --pids-limit -1 -v /etc/localtime:/etc/localtime -v /dev:/dev/ISP01 -v /opt/generalstorage/spord/bin:/opt/generalstorage/spord/bin -v /SPORD/ISP/ISP01:/SPORD/ISP/ISP01 -v /SPORD/GSCC/ISP01/etcgscc:/etc/gscc -v /c -v /NFS/dbbackups:/NFS/dbbackups -v /SPORD/STG/ISP01:/SPORD/STG/ISP01:rshared -v /etc/lvm/lvm.conf:/etc/lvm/lvm.conf localhost/ubi8:SP8.1.25-GSCC-Din/SPserver create /SPORD/HOME/ISP01/ISP01.dkd
INFO: Preview Mode. Command not executed.
If you did not specify -execute, as above, you can review if the output is correct. If everything is fine, issue the command again using the -execute parameter:
[root@rhelgscc-6 ISP01]# spordadm ISP01 create -execute
[INFO] Creating SPORD ISP instance "ISP01".
[INFO] SPORD: /opt/generalstorage/spord/bin/spord create -sp=ISP01 -execute
Wed Feb 26 11:46:15 2025
##################################################################
# General Storage SPORD - (c) 2022 General Storage Software GmbH #
##################################################################
INFO: Reading configuration file /etc/spord/spord.conf.
INFO: Receiving configuration for ISP01!
------------------------------
SPORDNAME: ISP01
SPORDBIN:
SPCONTBIN:
SPORDCONF: /etc/spord/spord.conf
SPORDHOME: /SPORD/HOME/ISP01
GSCCVERSION:
CUSTOMFILE:
DOCKIMG: localhost/ubi8:SP8.1.25-GSCC-DSMISI
ADDMAPPING: /SPORD/ISP/ISP01,/SPORD/GSCC/ISP01/etcgscc:/etc/gscc,/SPORD/GSCC/ISP01/vargscc:/var/gscc,/NFS/dbbackups,/SPORD/STG/ISP01:/SPORD/STG/ISP01:rshared,/etc/lvm/lvm.conf
ADDOPTION: --pids-limit -1
MAINCOMMAND: create
ARGUMENTS:
------------------------------
INFO: Using docker image localhost/ubi8:SP8.1.25-GSCC-DSMISI!
CREATE localhost/ubi8:SP8.1.25-GSCC-DSMISI
DOCKER COMMAND(NEW): podman run -it --rm --privileged --net=host --ipc=host --name=ISP01 --pids-limit -1 -v /etc/localtime:/etc/localtime -v /dev:/dev -v /SPORD/HOME/ISP01:/SPORD/HOME/ISP01 -v /opt/generalstorage/spord/bin:/opt/generalstorage/spord/bin -v /SPORD/ISP/ISP01:/SPORD/ISP/ISP01 -v /SPORD/GSCC/ISP01/etcgscc:/etc/gscc -v /SPORD/GSCC/ISP01/vargscc:/var/gscc -v /NFS/dbbackups:/NFS/dbbackups -v /SPORD/STG/ISP01:/SPORD/STG/ISP01:rshared -v /etc/lvm/lvm.conf:/etc/lvm/lvm.conf localhost/ubi8:SP8.1.25-GSCC-DSMISI /opt/generalstorage/spord/bin/SPserver create /SPORD/HOME/ISP01/ISP01.dkd
/opt/generalstorage/spord/bin/SPserver
SYSTEM PREPARATION
INFO: Starting SPdb2init: /opt/generalstorage/spord/bin/SPdb2init
INFO: Running /opt/generalstorage/spord/bin/SPdb2init create /SPORD/HOME/ISP01/ISP01.dkd
DETECTING ENVIRONMENT
PROCESSING DKDFILE /SPORD/HOME/ISP01/ISP01.dkd..............
ISP User: isp01
ISP UserID: 1601
ISP Group: isp01
ISP GroupID: 1601
ISP User Home Directory: /SPORD/ISP/ISP01/config
DB Directory Location(s): /SPORD/ISP/ISP01/db1,/SPORD/ISP/ISP01/db2,/SPORD/ISP/ISP01/db3,/SPORD/ISP/ISP01/db4
Active Log Directory: /SPORD/ISP/ISP01/actlog
Archive Log Directory: /SPORD/ISP/ISP01/arclog
Active Log Size: activelogsize=4096
ISP DB2 Instance Directory: /SPORD/ISP/ISP01/config
DB2 License Directory: /SPORD/HOME/ISP01
TCP Port: 1601
ISP LICENSE FILE REINSTALLED
DB2 PREPARATION
CREATING GROUP..............
CREATING USER..............
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
CREATING USER PROFILE..............
CREATING NEW INSTANCE (FORMAT)!
INFO: Checking Direcotories now...
WARNING: No dsmserv.opt found, creating base option file!
PREPARING dsmserv format command: /SPORD/ISP/ISP01/config/dsmserv.format
PREPARING DATABASE MANAGER..............
isp01
CREATING DATABASE INSTANCE..............
DBI1446I The db2icrt command is running.
DB2 installation is being initialized.
Total number of tasks to be performed: 4
Total estimated time for all tasks to be performed: 309 second(s)
Task #1 start
Description: Setting default global profile registry variables
Estimated time 1 second(s)
Task #1 end
Task #2 start
Description: Initializing instance list
Estimated time 5 second(s)
Task #2 end
Task #3 start
Description: Configuring DB2 instances
Estimated time 300 second(s)
Task #3 end
Task #4 start
Description: Updating global profile registry
Estimated time 3 second(s)
Task #4 end
The execution completed successfully.
For more information see the DB2 installation log at "/tmp/db2icrt.log.170".
DBI1070I Program db2icrt completed successfully.
DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed
successfully.
DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed
successfully.
SETTING DATABASE INSTANCE ENVIRONMENT..............
APPLYING DB2 LICENSE..............
LIC1402I License added successfully.
LIC1426I This product is now licensed for use as outlined in your License Agreement. USE OF THE PRODUCT CONSTITUTES ACCEPTANCE OF THE TERMS OF THE IBM LICENSE AGREEMENT, LOCATED IN THE FOLLOWING DIRECTORY: "/opt/tivoli/tsm/db2/license/en_US.iso88591"
PREPARING DB2 CLIENT FOR DBB..............
INFO: Environment prepared for /SPORD/HOME/ISP01/ISP01.dkd
INFO: Preparing DBB for SPserver
INFO: Formatting ISP01 from /SPORD/HOME/ISP01/ISP01.dkd
su -c "cd /SPORD/ISP/ISP01/config;dsmserv format dbdir=/SPORD/ISP/ISP01/db1,/SPORD/ISP/ISP01/db2,/SPORD/ISP/ISP01/db3,/SPORD/ISP/ISP01/db4 activelogsize=4096 activelogdirectory=/SPORD/ISP/ISP01/actlog archlogdirectory=/SPORD/ISP/ISP01/arclog " - isp01
ANR7800I DSMSERV generated at 04:36:59 on Dec 5 2024.
IBM Storage Protect for Linux/x86_64
Version 8, Release 1, Level 25.000
Licensed Materials - Property of IBM
(C) Copyright IBM Corporation 1990, 2024.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.
ANR7801I Subsystem process ID is 12409.
ANR0900I Processing options file /SPORD/ISP/ISP01/config/dsmserv.opt.
ANR0010W Unable to open message catalog for language C.utf8. The default language message catalog will be used.
ANR7814I Using instance directory /SPORD/ISP/ISP01/config.
ANR3339I Default Label in key data base is TSM Server SelfSigned SHA Key.
ANR4726I The ICC support module has been loaded.
ANR0152I Database manager successfully started.
ANR2976I Offline DB backup for database TSMDB1 started.
ANR2974I Offline DB backup for database TSMDB1 completed successfully.
ANR0992I Server's database formatting complete.
ANR0369I Stopping the database manager because of a server shutdown.
INFO: Format successful!
INFO: Starting dsmserv with runfile /SPORD/ISP/ISP01/config/dsmserv.runfile
ANR7800I DSMSERV generated at 04:36:59 on Dec 5 2024.
IBM Storage Protect for Linux/x86_64
Version 8, Release 1, Level 25.000
Licensed Materials - Property of IBM
(C) Copyright IBM Corporation 1990, 2024.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.
ANR7801I Subsystem process ID is 12884.
ANR0900I Processing options file /SPORD/ISP/ISP01/config/dsmserv.opt.
ANR0010W Unable to open message catalog for language C.utf8. The default language message catalog will be used.
ANR7814I Using instance directory /SPORD/ISP/ISP01/config.
ANR3339I Default Label in key data base is TSM Server SelfSigned SHA Key.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0302W Server monitoring has been disabled by option.
ANR0152I Database manager successfully started.
ANR1628I The database manager is using port 51601 for server connections.
ANR2277W The server master encryption key was not found. A new master encryption key will be created.
ANR1636W The server machine GUID changed: old value (), new value (80.4b.23.38.3f.c1.ef.11.91.d9.aa.10.82.ed.f2.57).
ANR2100I Activity log process has started.
ANR4726I The NAS-NDMP support module has been loaded.
ANR1794W IBM Storage Protect SAN discovery is disabled by options.
ANR2200I Storage pool BACKUPPOOL defined (device class DISK).
ANR2200I Storage pool ARCHIVEPOOL defined (device class DISK).
ANR2200I Storage pool SPACEMGPOOL defined (device class DISK).
ANR2560I Schedule manager started.
ANR0993I Server initialization complete.
ANR0916I IBM Storage Protect distributed by International Business Machines is now ready for use.
ANR1434W No files have been identified for automatically storing device configuration information.
ANR1912I Stopping the activity log because of a server shutdown.
ANR0369I Stopping the database manager because of a server shutdown.
INFO: dsmserv runfile completed!
total 76
drwxr-xr-x. 7 isp01 isp01 4096 Feb 26 11:48 .
drwxr-xr-x. 9 root root 92 Feb 26 10:50 ..
-rw-r--r--. 1 isp01 isp01 18 Feb 15 2024 .bash_logout
-rw-r--r--. 1 isp01 isp01 141 Feb 15 2024 .bash_profile
-rw-r--r--. 1 isp01 isp01 660 Feb 26 11:46 .bashrc
drwxrwxr-x. 3 isp01 isp01 19 Feb 26 11:46 .config
-rw-r-----. 1 isp01 isp01 0 Feb 26 11:47 .dsmserv.ilock
drwxrwxr-x. 3 isp01 isp01 17 Feb 26 11:46 .ibm
-rw-r--r--. 1 isp01 isp01 172 Oct 15 11:38 .kshrc
drwxrwxr-x. 3 isp01 isp01 19 Feb 26 11:46 .local
-rw-r--r--. 1 isp01 isp01 395 Feb 26 11:46 .profile
-rw-------. 1 isp01 isp01 80 Feb 26 11:46 cert.crl
-rw-------. 1 isp01 isp01 5080 Feb 26 11:46 cert.kdb
-rw-------. 1 isp01 isp01 80 Feb 26 11:46 cert.rdb
-rw-------. 1 isp01 isp01 193 Feb 26 11:46 cert.sth
-rw-rw-r--. 1 isp01 isp01 1257 Feb 26 11:46 cert256.arm
-rw-r-----. 1 isp01 isp01 0 Feb 26 11:46 dsmffdc.log
-rw-------. 1 isp01 isp01 3358 Feb 26 11:47 dsmkeydb.kdb
-rw-------. 1 isp01 isp01 193 Feb 26 11:47 dsmkeydb.sth
-rw-r-----. 1 isp01 isp01 27 Feb 26 11:47 dsmserv.dbid
-rw-r--r--. 1 root root 253 Feb 26 11:46 dsmserv.format
-rw-rw-r--. 1 isp01 isp01 170 Feb 26 11:47 dsmserv.opt
-rw-r--r--. 1 root root 0 Feb 26 11:47 dsmserv.runfile
lrwxrwxrwx. 1 isp01 isp01 34 Feb 26 11:48 dsmserv_ISP01 -> /opt/tivoli/tsm/server/bin/dsmserv
drwxrwxr-x. 3 isp01 isp01 22 Feb 26 11:46 isp01
drwxrwsr-t. 24 isp01 isp01 4096 Feb 26 11:47 sqllib
-rw-r--r--. 1 root root 26 Feb 26 11:46 tsmdbmgr.opt
INFO: Entering Container Command Line
SPORD@rhelgscc-6.gs.com #
Step 4: Initial ISP configuration (Primary)
You are now inside the newly created container and should start the ISP Instance and create an initial ISP system class administrator in order to continue:
SPORD@rhelgscc-5.gs.com # su - isp01
Last login: Wed Feb 26 11:49:14 CET 2025 on pts/4
[isp01@rhelgscc-5 ~]$ ls -l
total 52
-rw-------. 1 isp01 isp01 80 Feb 26 11:47 cert.crl
-rw-------. 1 isp01 isp01 5080 Feb 26 11:47 cert.kdb
-rw-------. 1 isp01 isp01 80 Feb 26 11:47 cert.rdb
-rw-------. 1 isp01 isp01 193 Feb 26 11:47 cert.sth
-rw-rw-r--. 1 isp01 isp01 1257 Feb 26 11:47 cert256.arm
-rw-r-----. 1 isp01 isp01 0 Feb 26 11:47 dsmffdc.log
-rw-------. 1 isp01 isp01 3358 Feb 26 11:48 dsmkeydb.kdb
-rw-------. 1 isp01 isp01 193 Feb 26 11:48 dsmkeydb.sth
-rw-r-----. 1 isp01 isp01 27 Feb 26 11:48 dsmserv.dbid
-rw-r--r--. 1 root root 253 Feb 26 11:47 dsmserv.format
-rw-rw-r--. 1 isp01 isp01 170 Feb 26 11:48 dsmserv.opt
-rw-r--r--. 1 root root 0 Feb 26 11:48 dsmserv.runfile
lrwxrwxrwx. 1 isp01 isp01 34 Feb 26 11:49 dsmserv_ISP01 -> /opt/tivoli/tsm/server/bin/dsmserv
drwxrwxr-x. 3 isp01 isp01 22 Feb 26 11:47 isp01
drwxrwsr-t. 24 isp01 isp01 4096 Feb 26 11:48 sqllib
-rw-r--r--. 1 root root 26 Feb 26 11:47 tsmdbmgr.opt
[isp01@rhelgscc-5 ~]$ ./dsmserv_ISP01
ANR7800I DSMSERV generated at 04:36:59 on Dec 5 2024.
IBM Storage Protect for Linux/x86_64
Version 8, Release 1, Level 25.000
Licensed Materials - Property of IBM
(C) Copyright IBM Corporation 1990, 2024.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.
ANR7801I Subsystem process ID is 13478.
ANR0900I Processing options file /SPORD/ISP/ISP01/config/dsmserv.opt.
ANR0010W Unable to open message catalog for language C.utf8. The default language message catalog will be used.
ANR7814I Using instance directory /SPORD/ISP/ISP01/config.
ANR3339I Default Label in key data base is TSM Server SelfSigned SHA Key.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0152I Database manager successfully started.
ANR1628I The database manager is using port 51601 for server connections.
ANR1635I The server machine GUID, 80.4b.23.38.3f.c1.ef.11.91.d9.aa.10.82.ed.f2.57, has initialized.
ANR2100I Activity log process has started.
ANR4726I The NAS-NDMP support module has been loaded.
ANR1794W IBM Storage Protect SAN discovery is disabled by options.
ANR2803I License manager started.
ANR0984I Process 1 for AUDIT LICENSE started in the BACKGROUND at 11:51:07.
ANR2820I Automatic license audit started as process 1.
ANR8598I Outbound SSL Services were loaded.
ANR8230I TCP/IP Version 6 driver ready for connection with clients on port 1601.
ANR8200I TCP/IP Version 4 driver ready for connection with clients on port 1601.
ANR2560I Schedule manager started.
ANR2825I License audit process 1 completed successfully - 0 nodes audited.
ANR0985I Process 1 for AUDIT LICENSE running in the BACKGROUND completed with completion state SUCCESS at 11:51:08.
ANR0281I Servermon successfully started during initialization, using process 13510.
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW ALLOC
ANR2017I Administrator SERVER_CONSOLE issued command: QUERY PROCESS
ANR0944E QUERY PROCESS: No active processes found.
ANR2017I Administrator SERVER_CONSOLE issued command: INSTRUMENTATION END
ANR2017I Administrator SERVER_CONSOLE issued command: select colname,'' from syscat.columns where tabname='DBSPACE' order by COLNO -comma
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW LOCKS ONLYWAITERS=YES
ANR0993I Server initialization complete.
ANR0916I IBM Storage Protect distributed by International Business Machines is now ready for use.
IBM Storage Protect:SERVER1>
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW REPLICATION
ANR2841W Server is NOT IN COMPLIANCE with license terms.
ANR4502W No files have been defined for automatically storing sequential volume history information.
ANR2017I Administrator SERVER_CONSOLE issued command: QUERY REPLICATION * STATUS=RUNNING FORMAT=DETAIL
ANR2679E QUERY REPLICATION: All of the nodes provided are invalid.
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW LOCKS
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW CLOUDREADCACHE
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM DBSPACE -comma
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW DBCONN
ANR2017I Administrator SERVER_CONSOLE issued command: select colname,'' from syscat.columns where tabname='LOG' order by COLNO -comma
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM LOG -comma
ANR2017I Administrator SERVER_CONSOLE issued command: select colname,'' from syscat.columns where tabname='DEVCLASSES' order by COLNO -comma
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM DEVCLASSES -comma
ANR2017I Administrator SERVER_CONSOLE issued command: select colname,'' from syscat.columns where tabname='DEVCLASSES_DIR' order by COLNO -comma
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM DEVCLASSES_DIR -comma
ANR2034E SELECT: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: select colname,'' from syscat.columns where tabname='STGPOOLS' order by COLNO -comma
ANR2034E QUERY STGPOOLDIRECTORY: No match found using this criteria.
ANR2679E QUERY REPLICATION: All of the nodes provided are invalid.
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW DEDUPTHREAD
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW BANNER
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW RESQUEUE
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW TXNTABLE LOCKDETAIL=NO
ANR2034E QUERY MOUNT: No match found using this criteria.
ANR2034E QUERY SESSION: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW SESSION FORMAT=DETAILED
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW THREADS
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM STGPOOLS -comma
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW JVM
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW PRODCONS
ANR2034E QUERY CLOUDREADCACHE: No match found using this criteria.
ANR0213E QUERY JOB: No jobs found.
ANR2017I Administrator SERVER_CONSOLE issued command: select colname,'' from syscat.columns where tabname='STGPOOL_DIRS' order by COLNO -comma
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM STGPOOL_DIRS -comma
ANR2034E QUERY STGPOOLDIRECTORY: No match found using this criteria.
ANR2034E SELECT: No match found using this criteria.
ANR0984I Process 2 for EXPIRE INVENTORY (Automatic) started in the BACKGROUND at 11:51:17.
ANR0811I Inventory client file expiration started as process 2.
ANR0167I Inventory file expiration process 2 processed for 0 minutes.
ANR0812I Inventory expiration process 2 is completed: processed 0 nodes, examined 0 objects, retained 0 objects, deleted 0 backup objects, 0 archive
objects, 0 database backup volumes, and 0 recovery plan files. 0 objects were retried, 0 errors were detected, and 0 objects were skipped. 0
retention bitfiles were deleted from retention pools.
ANR0985I Process 2 for EXPIRE INVENTORY (Automatic) running in the BACKGROUND completed with completion state SUCCESS at 11:51:17.
ANR2017I Administrator SERVER_CONSOLE issued command: QUERY SYSTEM
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW BANNERS
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW PRERELMSG
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW VERSIONHISTORY
ANR2017I Administrator SERVER_CONSOLE issued command: QUERY STATUS
ANR0944E QUERY PROCESS: No active processes found.
ANR2034E QUERY SESSION: No match found using this criteria.
ANR2034E QUERY VOLHISTORY: No match found using this criteria.
ANR2034E QUERY VOLUME: No match found using this criteria.
ANR2034E QUERY CONNECTION: No match found using this criteria.
ANR2034E QUERY STGPOOLDIRECTORY: No match found using this criteria.
ANR2034E QUERY SCHEDULE: No match found using this criteria.
ANR2034E QUERY ASSOCIATION: No match found using this criteria.
ANR2034E QUERY LIBRARY: No match found using this criteria.
ANR2034E QUERY DATAMOVER: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW DEDUPDELETEINFO
ANR2034E QUERY DIRSPACE: No match found using this criteria.
ANR1999I QUERY REPLRULE completed successfully.
ANR2034E QUERY STGRULE: No match found using this criteria.
ANR2034E QUERY SERVER: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: select node_name,NODETYPE,platform_name,CLIENT_OS_NAME,CLIENT_OS_LEVEL,REG_TIME,LASTACC_TIME,C-
ONCAT(CLIENT_VERSION,CONCAT(CLIENT_RELEASE,CONCAT(CLIENT_LEVEL,CLIENT_SUBLEVEL))) AS CLIENT_VRMF,DEDUPLICATION,SESSION_INITIATION,COMPRESSION,COLLOC-
GROUP_NAME FROM NODESVIEW ORDER BY NODE_NAME
ANR2034E SELECT: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: select platform_name,count(*) as "NODECOUNT" from nodesview group by platform_name
ANR2034E SELECT: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: select stgpool_name,devclass_name,count(*) as "VOLUMES" from volumes group by stgpool_name,dev-
class_name
ANR2034E QUERY VOLUME: No match found using this criteria.
ANR2034E SELECT: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: select volume_name, stgpool_name, last_write_date, last_read_date from volumes where
(access='DESTROYED' or access='UNAVAILABLE' or error_state='YES')
ANR2034E QUERY VOLUME: No match found using this criteria.
ANR2034E SELECT: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: select avg(cast (pct_reclaim as smallint)) as "RECLAIM PCT",stgpool_name as "STGPOOL" from
volumes where pct_reclaim>1 group by stgpool_name
ANR2034E QUERY VOLUME: No match found using this criteria.
ANR2034E SELECT: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: select stgpool_name, type, state, count as CONTAINER_COUNT, sum(CAST (free_space_mb AS BIGINT)
) as FREE_SPACE_MB, sum(CAST(total_space_mb AS BIGINT)) as TOTAL_SPACE_MB, sum(CAST(cloud_space_mb AS BIGINT)) as CLOUD_SPACE_MB, sum(CAST(used_spac-
e_mb AS BIGINT)) as USED_SPACE_MB from CONTAINERSUMMARYVIEW group by (stgpool_name,type,state)
ANR2034E SELECT: No match found using this criteria.
ANR2034E QUERY RETRULE: No match found using this criteria.
ANR2034E QUERY RETSET: No match found using this criteria.
ANR2034E QUERY LIBRARY: No match found using this criteria.
ANR2034E QUERY DRIVE: No match found using this criteria.
ANR1794W IBM Storage Protect SAN discovery is disabled by options.
ANR2034E QUERY SAN: No match found using this criteria.
ANR2034E QUERY SERVER: No match found using this criteria.
ANR2034E QUERY SCRIPT: No match found using this criteria.
ANR2034E QUERY PATH: No match found using this criteria.
ANR2034E QUERY SCHEDULE: No match found using this criteria.
ANR2034E QUERY SCHEDULE: No match found using this criteria.
ANR2034E QUERY STGRULE: No match found using this criteria.
ANR2034E QUERY CONNECTION: No match found using this criteria.
ANR2034E QUERY REPLFAILURES: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW SDPOOL
ANR2017I Administrator SERVER_CONSOLE issued command: SHOW SDDIRECTORY
ANR1999I QUERY REPLRULE completed successfully.
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM STGRULES
ANR2034E SELECT: No match found using this criteria.
ANR2017I Administrator SERVER_CONSOLE issued command: SELECT * FROM SUBRULES
ANR2034E SELECT: No match found using this criteria.
ANR2034E QUERY RETRULE: No match found using this criteria.
Since the server is running in foreground, it can be hard to see whether it is ready for use yet. Check for these lines:
ANR0993I Server initialization complete.
ANR0916I IBM Storage Protect distributed by International Business Machines is now ready for use.
IBM Storage Protect:SERVER1>
If these are displayed, ISP01 is up and awaits commands. First, we set the minimum password length to 8:
IBM Storage Protect:SERVER1>
set minpwlength 8
ANR2017I Administrator SERVER_CONSOLE issued command: SET MINPWLENGTH 8
ANR2138I Minimum password length set to 8
Then we register 2 administrators, one for use with dsmadmc and another one that will be used by GSCC. Both need system privileges:
IBM Storage Protect:SERVER1>
reg admin isp01 isp01isp01isp01
ANR2017I Administrator SERVER_CONSOLE issued command: REGISTER ADMIN isp01 ?***?
ANR2068I Administrator ISP01 registered.
ANR4200I The password for administrator ISP01 is now case sensitive.
IBM Storage Protect:SERVER1>
grant auth isp01 cl=sys
ANR2017I Administrator SERVER_CONSOLE issued command: GRANT AUTHORITY isp01 cl=sys
ANR2076I System privilege granted to administrator ISP01.
IBM Storage Protect:SERVER1>
reg admin gscc gsccgsccgscc
ANR2017I Administrator SERVER_CONSOLE issued command: REGISTER ADMIN gscc ?***?
ANR2068I Administrator GSCC registered.
ANR4200I The password for administrator GSCC is now case sensitive.
IBM Storage Protect:SERVER1>
grant auth gscc cl=sys
ANR2017I Administrator SERVER_CONSOLE issued command: GRANT AUTHORITY gscc cl=sys
ANR2076I System privilege granted to administrator GSCC
Now we set the session security for the newly created admins to transitional to be able to create client certificates automatically later:
IBM Storage Protect:SERVER1>
update admin gscc sessionsec=trans
ANR2017I Administrator SERVER_CONSOLE issued command: UPDATE ADMIN gscc sessionsec=trans
ANR4200I The password for administrator GSCC is now case insensitive.
ANR2071I Administrator GSCC updated.
IBM Storage Protect:SERVER1>
update admin isp01 sessionsec=trans
ANR2017I Administrator SERVER_CONSOLE issued command: UPDATE ADMIN isp01 sessionsec=trans
ANR4200I The password for administrator ISP01 is now case insensitive.
ANR2071I Administrator ISP01 updated.
Now that we finished all necessary configuration for the ISP instance, we stop it:
IBM Storage Protect:SERVER1>
halt
ANR2017I Administrator SERVER_CONSOLE issued command: HALT
ANR1912I Stopping the activity log because of a server shutdown.
ANR0369I Stopping the database manager because of a server shutdown.
ANR0991I Server shutdown complete.
Step 5: Additional configuration files (Both)
Logout (twice for the primary container), check if the containers have been stopped correctly:
[isp01@rhelgscc-5 ~]$ logout
SPORD@rhelgscc-5.gs.com # exit
[root@rhelgscc-5 ISP01]# spordadm
[INFO] No member specified. Showing running containers:
CONTAINER ID IMAGE STATUS NAMES
Now prepare the dsm.sys and dsm.opt files in the SPORD home directory. SPORD will link them to the correct paths inside the container when starting.
Caution
These files are different for Primary and Standby since they define local and remote access to the ISP instance!
Primary dsm.sys (rhelgscc-5):
[root@rhelgscc-5 Server]# cat /SPORD/HOME/ISP01/dsm.sys
SERVERNAME isp01
TCPPORT 1601
TCPSERVERADDRESS 192.168.178.140
NFSTIMEOUT 10
SERVERNAME isp01_local
TCPPORT 1601
TCPSERVERADDRESS 127.0.0.1
NFSTIMEOUT 10
SERVERNAME isp01_remote
TCPPORT 1601
TCPSERVERADDRESS 192.168.178.142
NFSTIMEOUT 10
SERVERNAME TSMDBMGR_ISP01
COMMMethod SHAREdmem
SHMPort 16001
NODENAME $$_TSMDBMGR_$$
PASSWORDDIR /SPORD/ISP/ISP01/config
ERRORLOGNAME /SPORD/ISP/ISP01/config/tsmdbmgr.log
NFSTIMEOUT 10
Standby dsm.sys (rhelgscc-6):
[root@rhelgscc-6 ~]# cat /SPORD/HOME/ISP01/dsm.sys
SERVERNAME isp01
TCPPORT 1601
TCPSERVERADDRESS 192.168.178.140
NFSTIMEOUT 10
SERVERNAME isp01_local
TCPPORT 1601
TCPSERVERADDRESS 127.0.0.1
NFSTIMEOUT 10
SERVERNAME isp01_remote
TCPPORT 1601
TCPSERVERADDRESS 192.168.178.138
NFSTIMEOUT 10
SERVERNAME TSMDBMGR_ISP01
COMMMethod SHAREdmem
SHMPort 16001
NODENAME $$_TSMDBMGR_$$
PASSWORDDIR /SPORD/ISP/ISP01/config
ERRORLOGNAME /SPORD/ISP/ISP01/config/tsmdbmgr.log
NFSTIMEOUT 10
Both systems have the same dsm.opt:
[root@rhelgscc-5 Server]# cat /SPORD/HOME/ISP01/dsm.opt
SERVERNAME isp01
Now we add information for our shared memory connection as well as our designated volume history and device configuration files to the ‘dsmserv.opt’ file:
[root@rhelgscc-5 Server]# cat /SPORD/ISP/ISP01/config/dsmserv.opt
COMMMETHOD TCPIP
TCPPORT 1601
COMMMETHOD SHAREDMEM
SHMPORT 16001
ACTIVELOGSize 4096
ACTIVELOGDirectory /SPORD/ISP/ISP01/actlog
ARCHLOGDirectory /SPORD/ISP/ISP01/arclog
VOLUMEHistory /SPORD/ISP/ISP01/config/volhist.out
DEVCONFig /SPORD/ISP/ISP01/config/devconf.out
ALIASHALT stoptsm
Note
For using GSCC to stop ISP gracefully later on, please at least add the line: ALIASHALT stoptsm to dsmserv.opt
Step 6: Set up GSCC configuration (Both)
Now we set up our GSCC configuration according to the GSCC part of this documentation here: Step 6: GSCC Installation. In our environment, these files are in our /etc/gscc folder:
[root@rhelgscc-5 etcgscc]# cd /SPORD/GSCC/ISP01/etcgscc/
[root@rhelgscc-5 etcgscc]# ll
total 56
-rw-r--r--. 1 root root 125 Feb 26 12:26 cl.def
-rw-r--r--. 1 root root 1199 Feb 28 16:28 gscc.conf
-rw-r--r--. 1 root root 18 Feb 26 12:26 gsuidcustom.conf
-rw-r--r--. 1 root root 26 Feb 26 12:28 ISP01.clipup
-rw-r--r--. 1 root root 2985 Feb 28 16:23 ISP01.def
-rw-r--r--. 1 root root 16 Feb 26 12:29 ISP01.heartbeat
-rw-r--r--. 1 root root 387 Feb 26 12:30 ISP01.ips
-rw-r--r--. 1 root root 36 Feb 26 12:32 ISP01.tsmlocalfs
-rw-r--r--. 1 root root 18 Feb 27 11:44 passweb
-rw-r--r--. 1 root root 4005 Feb 26 12:26 README
Lets have a look at the contents of these configuration files:
Caution
The contents for the GSCC configuration are different on the Primary and Standby host. Please edit these according to the aforementionend Step 6: GSCC Installation.
cl.def:
[root@rhelgscc-5 etcgscc]# cat cl.def
#
# Cluster Object Definitions
#
# (C) General Storage, 2025
#
#
# <Object_Name> <Object Description>
ISP01 ISP01 TSM Server
gscc.conf:
[root@rhelgscc-5 etcgscc]# cat gscc.conf
#
# (C) General Storage, 2025
#
# Cluster Controller Configuration File
#
# Activate Auto register
AutoRegister Yes
#Lifetime 1200
#Loclifetime 600
#FileLogLevel Debug
#FileLogLevel Info
#FileLogLevel Notice
#FileLogLevel Warning
FileLogLevel Error
SysLogLevel Error
Environment LC_ALL=C
Environment LANG=C
Environment StatusReadTimeout=180
Environment StatusFailureThreshold=10
Environment RemoteFailureThreshold=2
port 3939
Environment TSMKillCounter=40
Environment ErrCountMax=1
Environment sshPath=/usr/bin/ssh
Environment SSLPATH=/usr/bin
Environment sshConnectTimeout=10
Environment scpConnectTimeout=10
Environment DiskForceDuration=300
Environment hadrCOM=intern
Environment Docker=on
Environment GratuitousARP=yes
Environment DockerMon=on
Environment DockerMonPort=3930
Environment ISPConsole=on
gsuidcustom.conf:
[root@rhelgscc-5 etcgscc]# cat gsuidcustom.conf
Listen 3941 https
ISP01.def:
[root@rhelgscc-5 etcgscc]# cat ISP01.def
#
# (C) General Storage
#
#
# ISP01.def
# this file is needed for each Cluster Member !
# default path: $HOME/.gscc/sample.def
#
# use ":" for any blank in var-values
#
# <var_name> <value> <description>
expert_domain edUNIX_SP8_HADR_STG Expert Domain
am_tsm_server_path /SPORD/ISP/ISP01/config Path to TSM Server
service_interface extern Comma separated TSM Server Network Interface(s)
service_ip_address extern Comma separated TSM Server IP Addresse(s)
service_netmask extern Comma separated TSM Server Netmask(s)
service_broadcast extern Comma separated TSM Server Broadcast(s)
log_file /var/gscc/logs/im.isp01.log Activity Log File
am_log_file /var/gscc/logs/isp01.log Action method log file
tsm_auth_string dsmadmc:-se=isp01_local:SECURED TSM local Authentication String
peer_tsm_auth_string dsmadmc:-se=isp01:SECURED TSM Authentication String
forces_command - Reset server verification key
home_system_id rhelgscc-5 System ID
member_type prod Member Type: prod | stby
suff_pings 1 Number of Clients that have to be reached
client_ip_address_file /etc/gscc/ISP01.clipup File for Client IPs
heartbeat /etc/gscc/ISP01.heartbeat Heartbeat IP Address
peer_ip 192.168.178.142 Peer System IP
peer_port 3939 IP Port of Peer TSM Server
peer_member_name ISP01 Peer Member ID
local_member_name ISP01 Member ID
oserip_suff_pings 1 Pings to test against Shared Router
tsm_check_string dsmadmc:-se=isp01_local:SECURED:q:status TSM Check String
otsm_check_string dsmadmc:-se=isp01_remote:SECURED:q:status Other TSM Check String
tsm_startup_tolerance 600 Startup Time in Seconds for dsmserv
errorlistfile /var/gscc/logs/isp01.errorlist.out Error List File
pagecommand gsalert Command to page operator
pageinterval 3600 Time between paging the Operator on error
trigger auto How Operator triggers Failed status (key: auto)
dbb_rules hadr Full/incremental dbb interval and frequency or hadr
tsm_fulldevc tsmdbb TSM device class for full database backups
tsm_incrdevc tsmdbb TSM device clss for incremental database backups
hadr_local_ip 192.168.178.138 HADR
ISP01.clipup:
[root@rhelgscc-5 etcgscc]# cat ISP01.clipup
192.168.178.142
127.0.0.1
ISP01.heartbeat:
[root@rhelgscc-5 etcgscc]# cat ISP01.heartbeat
192.168.178.142
ISP01.ips:
[root@rhelgscc-5 etcgscc]# cat ISP01.ips
# Use this file MEMBER.ips to define IP configuration outside MEMBER.def
# In MEMBER.def the IP parameters needs to be set to "extern"
# service_interface extern
# service_ip_address extern
# service_netmask extern
# service_broadcast extern
# MEMBER.ips format
# Will be filled with "INSTANCE_NET"-Variables
ens33 192.168.178.140 255.255.255.0 192.168.178.255
ISP01.tsmlocalfs:
[root@rhelgscc-5 etcgscc]# cat ISP01.tsmlocalfs
/dev/mapper/rhel_rhelgscc--5-root /
After providing the configuration on both systems, we start the containers:
[root@rhelgscc-6 ~]# spordadm 1 start
[INFO] Starting "ISP01" container. Logs are written to "/SPORD/HOME/ISP01/init_ISP01.log"
[INFO] Showing running containers:
CONTAINER ID IMAGE STATUS NAMES
aa6bbbf59dbe localhost/ubi8:SP8.1.25-GSCC-DSMISI Up 2 seconds gscc-ISP01
Check the init log (optionally):
[root@rhelgscc-6 ~]# tail -f /SPORD/HOME/ISP01/init_ISP01.log
...
Starting GSCC processes:
INFO: First 'start' after 'create' detected, setting GSCC to Operator
INFO: Member ISP01 set to Operator at GSCC startup
INFO: Writing Semaphore /SPORD/HOME/ISP01/SPgscc.lock
INFO: Checking for addtional scripts to be executed.
INFO: No Script folder found.
GSCC Container initialization completed!
Now we generate a key for encrypting our GSCC ISP user with setgsccadm:
[gscc-ISP01 root@rhelgscc-5 gscc]$ setgsccadm
General Storage Cluster Controller for TSM - 3.8.8.9
GSCC Registration
Please enter GSCC internal key:
Please enter GSCC internal key again:
INFO: key verification OK.
INFO: Saving input.
INFO: Completed.
Then we store the credentials of said GSCC ISP user with settsmadm and encrypt it with the key we specified earlier:
[gscc-ISP01 root@rhelgscc-5 gscc]$ settsmadm
General Storage Cluster Controller for TSM - Version 3.8.8.9
TSM Admin Registration - SECURED
Please enter TSM user id: gscc
Please enter TSM password:
Please enter GSCC key:
Please enter GSCC key again:
INFO: key verification OK.
INFO: Encrypting and saving input.
INFO: Completed.
For GSCC itself, we should also add a user different from the default admin user:
[gscc-ISP01 root@rhelgscc-5 gscc]$ gsccadm
General Storage Cluster Controller for ISP - Version 3.8.8.9
Command Line Administrative Interface
(C) Copyright General Storage 2024
Please enter user id: admin
Please enter password:
GSCC Version(CLI Revision): 3.8.8.9(3295)
gscc rhelgscc-5>update admin add isp isp
INFO: User isp added!
(Optionally) we should delete the default admin User:
gscc rhelgscc-5>update admin del admin
INFO: User admin deleted !
gscc rhelgscc-5>quit
Step 7: Check GSCC connection and setup HADR configuration (Both)
Next we configure our instances to be in sync by utilizing the db2 HADR functions.
Step 7a: Set up HADR configuration and initial db2 backup (Primary)
Now we start our ISP Instance via GSCC:
[gscc-ISP01 root@rhelgscc-5 gscc]$ gsccadm
General Storage Cluster Controller for ISP - Version 3.8.8.9
Command Line Administrative Interface
(C) Copyright General Storage 2024
Please enter user id: isp
Please enter password:
GSCC Version(CLI Revision): 3.8.8.9(3295)
gscc rhelgscc-5>oper starttsm ISP01
INFO: Command starttsm for ISP01 listed !
(Optionally) Check the startup-logs for errors:
[root@rhelgscc-5 etcgscc]# tail -F /SPORD/GSCC/ISP01/vargscc/logs/isp01.log
250227115601 rhelgscc-5.gs.com ISP01 am_starttsm(16855) Method started.
250227115601 rhelgscc-5.gs.com ISP01 am_starttsm(16855) INFO: Preparing ISP server startup...
250227115601 rhelgscc-5.gs.com ISP01 am_starttsm(16855) INFO: Verifying HADR status.
250227115601 rhelgscc-5.gs.com ISP01 am_starttsm(16855) INFO: Processing peer Member:
...
250227115717 rhelgscc-5.gs.com ISP01 am_starttsm(16855) ISP: ANR0993I Server initialization complete.
250227115718 rhelgscc-5.gs.com ISP01 am_starttsm(16855) ISP: ANR0916I IBM Storage Protect distributed by International Business Machines is
250227115719 rhelgscc-5.gs.com ISP01 am_starttsm(16855) ISP: now ready to use.
250227115719 rhelgscc-5.gs.com ISP01 am_starttsm(16855) INFO: ISP server ready.
250227115719 rhelgscc-5.gs.com ISP01 am_starttsm(16855) INFO: Terminating initial ISP Console Logging.
250227115719 rhelgscc-5.gs.com ISP01 am_starttsm(16855) INFO: ISP server running as process 17425.
250227115719 rhelgscc-5.gs.com ISP01 am_starttsm(16855) Method completed.
250227115842 rhelgscc-5.gs.com ISP01 am_addalias(18262) Method started.
250227115842 rhelgscc-5.gs.com ISP01 am_addalias(18262) INFO: Adding IP alias addresses.
250227115842 rhelgscc-5.gs.com ISP01 am_addalias(18262) INFO: ip addr add dev ens33 local 192.168.178.140/255.255.255.0 broadcast 192.168.178.255 label ens33:ISP01
250227115842 rhelgscc-5.gs.com ISP01 am_addalias(18262) INFO: 192.168.178.140 added successfully!
250227115842 rhelgscc-5.gs.com ISP01 am_addalias(18262) INFO: Sending Gratuitous ARP: /usr/sbin/arping -U 192.168.178.140 -I ens33 -c 3
250227115844 rhelgscc-5.gs.com ISP01 am_addalias(18262) ARPING 192.168.178.140 from 192.168.178.140 ens33
250227115844 rhelgscc-5.gs.com ISP01 am_addalias(18262) Sent 3 probes (3 broadcast(s))
250227115844 rhelgscc-5.gs.com ISP01 am_addalias(18262) Received 0 response(s)
250227115844 rhelgscc-5.gs.com ISP01 am_addalias(18262) INFO: Adding IP addresses successful.
250227115844 rhelgscc-5.gs.com ISP01 am_addalias(18262) Method completed.
Check if GSCC can communicate with the ISP instance correctly with the stored admin, using the checktsmadm method:
gscc rhelgscc-5>gscmd ISP01 checktsmadm
Admin command line ok.
This command should automatically generate the client certificates (dsmcert.*) in the containers /opt/tivoli/tsm/client/ba/bin directory. Copy these files to the SPORD home directory (make sure to be inside the container):
[gscc-ISP01 root@rhelgscc-5 ~]$ cp /opt/tivoli/tsm/client/ba/bin/dsmcert.* /SPORD/HOME/ISP01/
Now we set up the DB2 HADR configuration for the primary system. For that, we change to the instance owner:
[gscc-ISP01 root@rhelgscc-5 ~]$ su - isp01
Last login: Thu Feb 27 12:20:48 CET 2025 on pts/0
Then we connect to the ISP db2 instance:
[isp01@rhelgscc-5 ~]$ db2 connect to tsmdb1
Database Connection Information
Database server = DB2/LINUXX8664 11.5.9.0
SQL authorization ID = ISP01
Local database alias = TSMDB1
And then we update all necessary parameters according to our setup:
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using LOGINDEXBUILD yes
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using HADR_LOCAL_HOST 192.168.178.138
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, the database
must be shutdown and reactivated before the configuration parameter changes
become effective.
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using HADR_LOCAL_SVC 60100
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, the database
must be shutdown and reactivated before the configuration parameter changes
become effective.
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using HADR_REMOTE_HOST 192.168.178.142
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, the database
must be shutdown and reactivated before the configuration parameter changes
become effective.
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using HADR_REMOTE_SVC 60100
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, the database
must be shutdown and reactivated before the configuration parameter changes
become effective.
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using HADR_REMOTE_INST isp01
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, the database
must be shutdown and reactivated before the configuration parameter changes
become effective.
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using HADR_SYNCMODE NEARSYNC
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, the database
must be shutdown and reactivated before the configuration parameter changes
become effective.
[isp01@rhelgscc-5 ~]$ db2 update db cfg for TSMDB1 using HADR_PEER_WINDOW 300
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, the database
must be shutdown and reactivated before the configuration parameter changes
become effective.
After setting all parameters for the database configuration, we back up the db to our network share:
[isp01@rhelgscc-5 ~]$ db2 backup db TSMDB1 online to /NFS/dbbackups/ COMPRESS INCLUDE LOGS
Backup successful. The timestamp for this backup image is : 20250227123442
Step 7b: Set up HADR configuration (Standby)
First, we switch to the instance owner inside the container and recover the backup which was done on the Primary site:
[isp01@rhelgscc-6 ~]$ db2 start dbm
DB20000I The START DATABASE MANAGER command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 restore db TSMDB1 from /NFS/dbbackups/ on /SPORD/ISP/ISP01/db1,/SPORD/ISP/ISP01/db2,/SPORD/ISP/ISP01/db3,/SPORD/ISP/ISP01/db4 into TSMDB1 logtarget /SPORD/ISP/ISP01/arclog/ replace history file
SQL2523W Warning! Restoring to an existing database that is different from
the database on the backup image, but have matching names. The target database
will be overwritten by the backup version. The Roll-forward recovery logs
associated with the target database will be deleted.
Do you want to continue ? (y/n) y
DB20000I The RESTORE DATABASE command completed successfully.
Then we update the db2 HADR configuration accordingly:
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using LOGINDEXBUILD yes
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using HADR_LOCAL_HOST 192.168.178.142
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using HADR_LOCAL_SVC 60100
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using HADR_REMOTE_HOST 192.168.178.138
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using HADR_REMOTE_SVC 60100
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using HADR_REMOTE_INST isp01
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using HADR_SYNCMODE NEARSYNC
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
[isp01@rhelgscc-6 ~]$ db2 update db cfg for TSMDB1 using HADR_PEER_WINDOW 300
DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
(Optionally) Check the HADR configuration by issuing db2 get db cfg for TSMDB1 |grep HADR:
[isp01@rhelgscc-6 ~]$ db2 get db cfg for TSMDB1 |grep HADR
HADR database role = STANDARD
HADR local host name (HADR_LOCAL_HOST) = 192.168.178.142
HADR local service name (HADR_LOCAL_SVC) = 60100
HADR remote host name (HADR_REMOTE_HOST) = 192.168.178.138
HADR remote service name (HADR_REMOTE_SVC) = 60100
HADR instance name of remote server (HADR_REMOTE_INST) = isp01
HADR timeout value (HADR_TIMEOUT) = 120
HADR target list (HADR_TARGET_LIST) =
HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC
HADR spool log data limit (4KB) (HADR_SPOOL_LIMIT) = AUTOMATIC(0)
HADR log replay delay (seconds) (HADR_REPLAY_DELAY) = 0
HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 300
HADR SSL certificate label (HADR_SSL_LABEL) =
HADR SSL Hostname Validation (HADR_SSL_HOST_VAL) = OFF
Now we start db2 as Standby:
[isp01@rhelgscc-6 ~]$ db2 start hadr on db TSMDB1 as standby
DB20000I The START HADR ON DATABASE command completed successfully.
Step 7c: Start HADR and check connection (Primary)
If not already, connect to the container and switch to the instance owner, then:
[isp01@rhelgscc-5 ~]$ db2 start hadr on db tsmdb1 as primary
DB20000I The START HADR ON DATABASE command completed successfully.
Now we check the connection with:
[isp01@rhelgscc-5 ~]$ db2pd -hadr -db tsmdb1
Database Member 0 -- Database TSMDB1 -- Active -- Up 0 days 00:32:30 -- Date 2025-02-27-12.53.31.101260
HADR_ROLE = PRIMARY
REPLAY_TYPE = PHYSICAL
HADR_SYNCMODE = NEARSYNC
STANDBY_ID = 1
LOG_STREAM_ID = 0
HADR_STATE = PEER
HADR_FLAGS = TCP_PROTOCOL
PRIMARY_MEMBER_HOST = 192.168.178.138
PRIMARY_INSTANCE = isp01
PRIMARY_MEMBER = 0
STANDBY_MEMBER_HOST = 192.168.178.142
STANDBY_INSTANCE = isp01
STANDBY_MEMBER = 0
HADR_CONNECT_STATUS = CONNECTED
HADR_CONNECT_STATUS_TIME = 02/27/2025 12:52:01.670943 (1740657121)
HEARTBEAT_INTERVAL(seconds) = 5
HEARTBEAT_MISSED = 0
HEARTBEAT_EXPECTED = 18
HADR_TIMEOUT(seconds) = 120
TIME_SINCE_LAST_RECV(seconds) = 5
PEER_WAIT_LIMIT(seconds) = 0
LOG_HADR_WAIT_CUR(seconds) = 0.000
LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.000000
LOG_HADR_WAIT_ACCUMULATED(seconds) = 0.000
LOG_HADR_WAIT_COUNT = 0
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 87040
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 131072
PRIMARY_LOG_FILE,PAGE,POS = S0000007.LOG, 2368, 3794235009
STANDBY_LOG_FILE,PAGE,POS = S0000007.LOG, 2368, 3794235009
HADR_LOG_GAP(bytes) = 0
STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000007.LOG, 2368, 3794235009
STANDBY_RECV_REPLAY_GAP(bytes) = 0
PRIMARY_LOG_TIME = 02/27/2025 12:51:48.000000 (1740657108)
STANDBY_LOG_TIME = 02/27/2025 12:51:48.000000 (1740657108)
STANDBY_REPLAY_LOG_TIME = 02/27/2025 12:51:48.000000 (1740657108)
STANDBY_RECV_BUF_SIZE(pages) = 4096
STANDBY_RECV_BUF_PERCENT = 0
STANDBY_SPOOL_LIMIT(pages) = 1048576
STANDBY_SPOOL_PERCENT = 0
STANDBY_ERROR_TIME = NULL
PEER_WINDOW(seconds) = 300
PEER_WINDOW_END = 02/27/2025 12:58:29.000000 (1740657509)
READS_ON_STANDBY_ENABLED = N
HADR_LAST_TAKEOVER_TIME = NULL
Step 8: Register ISP and configure initial file sync for GSCC (Both)
Next, we want to set up GSCC file sync to make sure that changes to important ISP Server files (dsmserv.opt, dsmserv.dbid, etc.) are synced among both cluster systems. Also, the newly created ISP instance needs to be registered.
..note:
Configure your ISP instance dsmserv.opt to specify VOLUMEHISTORY and DEVCONF, if not already done.
Step 8a: Setting up ISP for backup and register ISP (Primary)
We connect to the started ISP instance with our system administrator and configure everything necessary:
Protect: SERVER1>def devcl dbbackup devt=file maxcap=50G dir=/NFS/dbbackups
ANR2203I Device class DBBACKUP defined.
Protect: SERVER1>set dbrecovery dbbackup password=XXX
ANR2417W SET DBRECOVERY: The database backup password or the PROTECTKEYS setting has changed.
ANR2782I SET DBRECOVERY completed successfully and device class for automatic DB backup is set to DBBACKUP.
Then we start our initial ISP backup:
Protect: SERVER1>backup db t=f devcl=dbbackup
ANR2280I Full database backup started as process 2.
ANS8003I Process number 2 started.
To register ISP, we issue the following commands:
Protect: SERVER1>reg lic file=tsmee.lic
ANR2852I Current license information:
ANR2853I New license information:
ANR2828I Server is licensed to support IBM Storage Protect Basic Edition.
ANR2828I Server is licensed to support IBM Storage Protect Extended Edition.
Protect: SERVER1>audit lic
ANR2817I AUDIT LICENSES: License audit started as process 3.
ANS8003I Process number 3 started.
Protect: SERVER1>q lic
Last License Audit: 02/27/25 13:17:46
Number of Data Protection for Oracle in use: 0
Number of Data Protection for Oracle in try buy mode: 0
Number of Data Protection for Microsoft SQL in use: 0
Number of Data Protection for Microsoft SQL in try buy mode: 0
Number of Data Protection for Microsoft Exchange in use: 0
Number of Data Protection for Microsoft Exchange in try buy mode: 0
Number of TDP for Lotus Notes in use: 0
Number of TDP for Lotus Notes in try buy mode: 0
Number of Data Protection for Lotus Domino in use: 0
Number of Data Protection for Lotus Domino in try buy mode: 0
Number of TDP for Informix in use: 0
Number of TDP for Informix in try buy mode: 0
Number of TDP for SAP R/3 in use: 0
Number of TDP for SAP R/3 in try buy mode: 0
Number of TDP for ESS in use: 0
Number of TDP for ESS in try buy mode: 0
Number of TDP for ESS R/3 in use: 0
Number of TDP for ESS R/3 in try buy mode: 0
Number of TDP for EMC Symmetrix in use: 0
Number of TDP for EMC Symmetrix in try buy mode: 0
Number of TDP for EMC Symmetrix R/3 in use: 0
Number of TDP for EMC Symmetrix R/3 in try buy mode: 0
Number of TDP for WAS in use: 0
Number of TDP for WAS in try buy mode: 0
Is IBM Storage Protect for Data Retention in use ?: No
Is IBM Storage Protect for Data Retention licensed ?: No
Is IBM Storage Protect Basic Edition in use: Yes
Is IBM Storage Protect Basic Edition licensed: Yes
Is IBM Storage Protect Extended Edition in use: No
Is IBM Storage Protect Extended Edition licensed: Yes
Server License Compliance: Valid
For the initial sync, we need to copy all neccessary files from the primary systems config directory to our shared /NFS directory:
[gscc-ISP01 root@rhelgscc-5 /]$ cd /SPORD/ISP/ISP01/config/
[gscc-ISP01 root@rhelgscc-5 config]$ tar -cvf /NFS/dbbackups/serverfiles.tar NODELOCK cert* dsmkeydb.* dsmserv.dbid
NODELOCK
cert.crl
cert.kdb
cert.rdb
cert.sth
cert256.arm
dsmkeydb.kdb
dsmkeydb.sth
dsmserv.dbid
In addition, we copy the dsmcert.* files from the primary systems HOME directory to the /NFS-directory:
[gscc-ISP01 root@rhelgscc-5 ISP01]$ cd /SPORD/HOME/ISP01/
[gscc-ISP01 root@rhelgscc-5 ISP01]$ tar -cvf /NFS/dbbackups/clientcerts.tar dsmcert.*
dsmcert.idx
dsmcert.kdb
dsmcert.sth
Step 8b: Setting up GSCC for file sync (Standby)
For the standby site, we now extract those files to the correct paths.
Server files first:
[gscc-ISP01 root@rhelgscc-6 ISP01]$ cd /SPORD/ISP/ISP01/config/
[gscc-ISP01 root@rhelgscc-6 config]$ tar -xvf /NFS/dbbackups/serverfiles.tar
NODELOCK
cert.crl
cert.kdb
cert.rdb
cert.sth
cert256.arm
dsmkeydb.kdb
dsmkeydb.sth
dsmserv.dbid
And the same for the client certificates:
[gscc-ISP01 root@rhelgscc-6 config]$ tar -xvf /NFS/dbbackups/clientcerts.tar -C /SPORD/HOME/ISP01/
dsmcert.idx
dsmcert.kdb
dsmcert.sth
Now we create a checkpoint file to enable file syncing on the STANDBY system:
[gscc-ISP01 root@rhelgscc-6 /]$ cd /var/gscc/flags/ISP01/
[gscc-ISP01 root@rhelgscc-6 ISP01]$ echo 0 > .ckpt.sync
Step 9: Finishing Setup (Both)
Now we just need to check if everything comes up correctly and join the cluster to finish our configuration.
Step 9a: Stopping ISP (Primary)
Properly stop the running ISP instance with:
[gscc-ISP01 root@rhelgscc-5 ~]$ gsccadm
General Storage Cluster Controller for ISP - Version 3.8.8.9
Command Line Administrative Interface
(C) Copyright General Storage 2024
Please enter user id: isp
Please enter password:
GSCC Version(CLI Revision): 3.8.8.9(3295)
gscc rhelgscc-5>oper stoptsm ISP01
INFO: Command stoptsm for ISP01 listed !
Then wait until the Instance is shut down.
Step 9b: Restarting Containers (Both)
After ISP is stopped, we can exit from both containers and restart them.
Primary:
[gscc-ISP01 root@rhelgscc-5 ~]$ exit
[root@rhelgscc-5 ~]# spordadm 1 stop
[INFO] Command received to stop "gscc-ISP01" container
[INFO] Shutting down the container "gscc-ISP01" (it might take up to 10s to stop the container, check with "spordadm" command)
Standby:
[gscc-ISP01 root@rhelgscc-6 ~]$ exit
[root@rhelgscc-6 ~]# spordadm 1 stop
[INFO] Command received to stop "gscc-ISP01" container
[INFO] Shutting down the container "gscc-ISP01" (it might take up to 10s to stop the container, check with "spordadm" command)
And when they are shut down, we start them again.
Primary:
[root@rhelgscc-5 ~]# spordadm 1 start
[INFO] Starting "ISP01" container. Logs are written to "/SPORD/HOME/ISP01/init_ISP01.log"
[INFO] Showing running containers:
CONTAINER ID IMAGE STATUS NAMES
8523d3a5e8e6 localhost/ubi8:SP8.1.25-GSCC-DSMISI Up 2 seconds gscc-ISP01
Standby:
[root@rhelgscc-6 ~]# spordadm 1 start
[INFO] Starting "ISP01" container. Logs are written to "/SPORD/HOME/ISP01/init_ISP01.log"
[INFO] Showing running containers:
CONTAINER ID IMAGE STATUS NAMES
aa1bf33d46ee localhost/ubi8:SP8.1.25-GSCC-DSMISI Up 2 seconds gscc-ISP01
We check for errors at the initialization of the container:
[root@rhelgscc-6 ~]# cat /SPORD/HOME/ISP01/init_ISP01.log
Thu Feb 27 13:31:09 2025
##################################################################
# General Storage SPORD - (c) 2022 General Storage Software GmbH #
##################################################################
INFO: Reading configuration file /etc/spord/spord.conf.
INFO: Receiving configuration for ISP01!
------------------------------
SPORDNAME: ISP01
...
GSCC Container initialization completed!
Step 9c: Starting ISP and Joining (Both)
When everything looks good on both sides, we connect to GSCC on our PRIMARY system to start the ISP instance once again:
[root@rhelgscc-5 ~]# spordadm 1 gsccadm
General Storage Cluster Controller for ISP - Version 3.8.8.9
Command Line Administrative Interface
(C) Copyright General Storage 2024
Please enter user id: isp
Please enter password:
GSCC Version(CLI Revision): 3.8.8.9(3295)
gscc rhelgscc-5>oper starttsm ISP01
INFO: Command starttsm for ISP01 listed !
And finally we join the cluster on both systems. Primary:
gscc rhelgscc-5>oper join ISP01
INFO: Command join for ISP01 listed !
Standby:
gscc rhelgscc-6>oper join ISP01
INFO: Command join for ISP01 listed !
…and check the status. Primary:
gscc rhelgscc-5>q status
250227133809 ISP01 Up
Standby:
gscc rhelgscc-6>q status
250227133910 ISP01 Available
Update - SPORD & GSCC
In this example, the following environment is used:
Two Linux systems with SLES 12SP5: sles-01 (192.168.178.67) & sles-02 (192.168.178.68)
A single ISP instance is managed: ISP01
Container image used is: SP819.100-GSCC-DSMISI, containing GSCC 3.8.5.4.
SPORD installed version is: 3.8.5.4
SPORD Update is 3.8.7.0
New container image to use is: SP8.1.12.100-GSCC-DSMISI
Additionally, we want to provide GSCC version 3.8.7.0, which is newer than the version in the container image.
Step 1: Update SPORD (Both)
We need to update the SPORD application on both systems:
sles-01:/home/spordupdate # zypper in spord-3.8.7.0-28.x86_64.rpm
Refreshing service 'Containers_Module_12_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_12_SP5_x86_64'.
Refreshing service 'SUSE_Package_Hub_12_SP5_x86_64'.
Loading repository data...
Reading installed packages...
Resolving package dependencies...
The following package is going to be upgraded:
spord
The following package has no support information from it's vendor:
spord
1 package to upgrade.
Overall download size: 31.1 KiB. Already cached: 0 B. After the operation, additional 15.6 KiB will be used.
Continue? [y/n/...? shows all options] (y): y
Retrieving package spord-3.8.7.0-28.x86_64 (1/1), 31.1 KiB ( 96.5 KiB unpacked)
spord-3.8.7.0-28.x86_64.rpm:
Package is not signed!
spord-3.8.7.0-28.x86_64 (Plain RPM files cache): Signature verification failed [6-File is unsigned]
Abort, retry, ignore? [a/r/i] (a): i
Checking for file conflicts: .....................................................................................................................................................................................................[done]
(1/1) Installing: spord-3.8.7.0-28.x86_64 ........................................................................................................................................................................................[done]
Step 2: Deploy new container image (Both)
Now we decompress the new container image:
sles-01:/home/spordupdate # gzip -dc SP8.1.12.100-GSCC-DSMISI.tar.gz| tar -xvf -
SP8.1.12.100-GSCC-DSMISI
Next, we need to load the image into our container library:
sles-01:/home/spordupdate # docker load -i SP8.1.12.100-GSCC-DSMISI
44cd80fcd113: Loading layer [==================================================>] 4.353GB/4.353GB
Loaded image: dockercentos74:SP8.1.12.100-GSCC-DSMISI
Check, if the image is in our library:
sles-01:/home/spordupdate # docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
dockercentos74 SP8.1.12.100-GSCC-DSMISI 77b771e2615a 5 months ago 4.52GB
dockercentos74 SP819.100-GSCC-DSMISI eebafa1bbf35 21 months ago 7.12GB
Step 3: Update SPORD configuration (Both)
First, we set both instances to operator mode with GSCCAD or GSCCADM to prevent GSCC trying to start ISP after the container startup. Then we can stop the running containers:
sles-01:/home/spordupdate # spordadm 1 stop
Command received to stop "gscc-ISP01" container
Shuting down the container "gscc-ISP01" (it might take up to 10s to stop the container, check with "docker ps")
Then, in spord.conf, we specify our newly added image as DOCKERIMAGE:
sles-01:/home/spordupdate # cat /etc/spord/spord.conf
#######################################################
# General Storage Software GmbH
# SPORD - Spectrum Protect On Resilient Docker
#
# spord.conf
#######################################################
SPORDNAME ISP01
SPORDHOME /home/SPORD/ISP01
DOCKERIMAGE dockercentos74:SP8.1.12.100-GSCC-DSMISI
ADDMAPPING /home/stgpools/ISP01,/home/SPORD/ISP01/etcgscc:/etc/gscc,/home/SPORD/ISP01/vargscc:/var/gscc
DKDFILE ISP01.dkd
ADDOPTION --pids-limit 6000
Caution
Since we are performing an ISP update from 8.1.9 to 8.1.12, the used DB2 Version used has changed from 11.1 to 11.5. We need to provide a new db2ese.lic license file to be able to start properly.
Step 4: Prepare GSCC RPM for update (Both)
Since we want to use an updated GSCC version inside the container, we have to prepare the rpm in our SPORDHOME directory:
sles-01:/home/SPORD/ISP01 # pwd
/home/SPORD/ISP01
sles-01:/home/SPORD/ISP01 # mkdir RPM
sles-01:/home/SPORD/ISP01 # cp /home/spordupdate/gscc-3.8.7.0-0.x86_64.rpm ./RPM/
Note
Every RPM inside the SPORDHOME/RPM directory will be installed after starting up the container. So if you need additional RPMs inside the SPORD containers, deploy the needed packages there.
Step 5: Upgrade DB2 instances (Both)
Now we can start our containers with spordadm 1 start and have to upgrade the db2 instance tsmdb1, since we have a new Db2 version in our container.
Starting with the standby site:
sles-02:/home/SPORD/ISP01 # spordadm ISP01 start
Starting the container ...
"docker ps" output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
737b7bc3340f dockercentos74:SP819.100-GSCC-DSMISI "/opt/generalstorage…" 2 seconds ago Up 1 second gscc-ISP01
sles-02:/home/SPORD/ISP01 # spordadm ISP01
You're inside 'gscc-ISP01' container
[gscc-ISP01 root@sles-02 /]$ su - isp01
Last login: Mon Jan 3 10:31:21 CET 2022 on pts/0
[gscc-ISP01 isp01@sles-02 ~]$ db2start
11/26/2021 14:15:59 0 0 SQL1063N DB2START processing was successful.
SQL1063N DB2START processing was successful.
[gscc-ISP01 isp01@sles-02 ~]$ db2 upgrade db tsmdb1
SQL1103W The UPGRADE DATABASE command was completed successfully.
Caution
Don’t try to do anything else on the standby db after issuing the db2 upgrade db tsmdb1 command, since it is now in a state waiting for the primary db to be upgraded and reinitiate the HADR connection.
Now do the same for the primary site:
sles-01:/home/SPORD/ISP01 # spordadm ISP01 start
Starting the container ...
"docker ps" output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
737b7bc3340f dockercentos74:SP819.100-GSCC-DSMISI "/opt/generalstorage…" 2 seconds ago Up 1 second gscc-ISP01
sles-01:/home/SPORD/ISP01 # spordadm ISP01
You're inside 'gscc-ISP01' container
[gscc-ISP01 root@sles-01 /]$ su - isp01
Last login: Mon Jan 3 10:31:21 CET 2022 on pts/0
[gscc-ISP01 isp01@sles-01 ~]$ db2start
11/26/2021 14:15:59 0 0 SQL1063N DB2START processing was successful.
SQL1063N DB2START processing was successful.
[gscc-ISP01 isp01@sles-01 ~]$ db2 upgrade db tsmdb1
SQL1103W The UPGRADE DATABASE command was completed successfully.
Note
For more information regarding Db2 upgrades with HADR, check the IBM documentation
Now you’re ready to start GSCC and the ISP instances again!
ISP Instance Update
This scenario is an example of how to update the IBM Spectrum Protect Server Version in a GSCC environment without SPORD.
- In this example, the following environment is used:
Two linux hosts (doc01&doc02)
One ISP instance (ISP11)
Installed ISP Version is 8.1.10.100
Target ISP Version is 8.1.13.000
For each installation step, the headline will either say “Primary”, “Standby” or “Both”, meaning that those steps should be done either on the designated primary cluster system, the standby system, or both of them. In our case, doc01 is the primary system and doc02 is the standby system.
Step 1: Get systems ready (Both)
- To start the update, make sure that the following is done:
Recent DB backup in case of an error
The Cluster is in operator mode
GSCCAD shows that both sites are in operator mode:
Then we can stop the ISP instance via GSCCAD:
Now both instances are shut down:
We need to do a backup of the userprofile file inside the ISP db2 instance directory, since it will be deleted during the software update:
[isp11@doc02 sqllib]$ cp userprofile /tmp/ISP11_userprofile.bak
[isp11@doc02 sqllib]$
[isp11@doc01 sqllib]$ cp userprofile /tmp/ISP11_userprofile.bak
[isp11@doc01 sqllib]$
Step 2: Updating IBM Spectrum Protect (Both)
Please extract the application by executing the .bin-File. In our case SP_8.1.13_LIN86_SERSTG_AGT_ML.bin.
Afterwards, you should see an install.sh file, which then can be executed in console mode:
[root@doc02 ISP8113000]# ./install.sh -c
Preprocessing the input.
Loading repositories...
Preparing and resolving the selected packages...
First of all, we install a new IBM Installation Manager Version:
=====> IBM Installation Manager> Install
Select packages to install:
1. [X] IBM® Installation Manager 1.9.1.6
O. Check for Other Versions, Fixes, and Extensions
N. Next, C. Cancel
-----> [N]
Validating package prerequisites...
=====> IBM Installation Manager> Install> Licenses
Read the following license agreements carefully.
View a license agreement by entering the number:
1. IBM Installation Manager - License Agreement
Options:
A. [ ] I accept the terms in the license agreement
D. [ ] I do not accept the terms in the license agreement
B. Back, C. Cancel
-----> [C] A
=====> IBM Installation Manager> Install> Licenses
Read the following license agreements carefully.
View a license agreement by entering the number:
1. IBM Installation Manager - License Agreement
Options:
A. [X] I accept the terms in the license agreement
D. [ ] I do not accept the terms in the license agreement
B. Back, N. Next, C. Cancel
-----> [N]
=====> IBM Installation Manager> Install> Licenses> Summary
Target Location:
Package Group Name : IBM Installation Manager
Installation Directory : /opt/IBM/InstallationManager/eclipse
Packages to be installed:
IBM® Installation Manager 1.9.1.6
Options:
G. Generate an Installation Response File
B. Back, I. Install, C. Cancel
-----> [I]
25% 50% 75% 100%
------------------|------------------|------------------|------------------|
............................................................................
=====> IBM Installation Manager> Install> Licenses> Summary> Completion
The install completed successfully.
Options:
R. Restart Installation Manager
-----> [R]
Preprocessing the input.
After completing this, the Installation Manager restarts. Now we choose to Update software:
=====> IBM Installation Manager
Select:
1. Install - Install software packages
2. Update - Find and install updates and fixes to installed software packages
3. Modify - Change installed software packages
4. Roll Back - Revert to an earlier version of installed software packages
5. Uninstall - Remove installed software packages
Other Options:
L. View Logs
S. View Installation History
V. View Installed Packages
------------------------
P. Preferences
------------------------
A. About IBM Installation Manager
------------------------
X. Exit Installation Manager
-----> 2
Checking repositories...
Loading repositories...
The installation manager finds our IBM Spectrum Protect installation:
=====> IBM Installation Manager> Update
Select a package group to update:
1. [X] IBM Spectrum Protect
Details of package group IBM Spectrum Protect:
Package Group Name : IBM Spectrum Protect
Shared Resources Directory : /opt/IBM/IBMIMShared
Installation Directory : /opt/tivoli/tsm
Translations : English
Architecture : 64-bit
Other Options:
U. Update All
A. Unselect All
N. Next, C. Cancel
-----> [N] U
=====> IBM Installation Manager> Update
Select a package group to update:
1. [X] IBM Spectrum Protect
Details of package group IBM Spectrum Protect:
Package Group Name : IBM Spectrum Protect
Shared Resources Directory : /opt/IBM/IBMIMShared
Installation Directory : /opt/tivoli/tsm
Translations : English
Architecture : 64-bit
Other Options:
A. Unselect All
N. Next, C. Cancel
-----> [N]
Preparing installed packages in the selected package groups...
Validating package group locations...
Searching updates...
Preparing and resolving the selected packages...
Validating the selected updates...
Validating package prerequisites...
.
Depending on your system, there might be validation issues, which should be resolved before updating a productive environment:
=====> IBM Installation Manager> Update> Prerequisites
Validation results:
* [WARNING] IBM Spectrum Protect server 8.1.13.20211119_2036 contains validation warning.
1. WARNING: The command "rpm -qa --queryformat '%{RELEASE}\n' redhat-release* | grep \\." has failed.
2. WARNING: The system does not meet the minimum memory requirement of 12 GB.
3. WARNING: The installation cannot continue if Security-enhanced Linux (SELinux) is enabled and in enforcing mode.
Enter the number of the error or warning message above to view more details.
Options:
R. Recheck status.
B. Back, N. Next, C. Cancel
-----> [N]
=====> IBM Installation Manager> Update> Prerequisites> Custom panels
---- Configuration for IBM Spectrum Protect server 8.1.13.20211119_2036
Caution
The next panel asks whether we want to update the database instance, which we do NOT want right now, because it would break our DB2 HADR connection!
Instance credentials:
--------------------------
------------
Instance name: isp11
Do you want update this instance? Y. Yes N. No
-----> [Y, N]
N
Skipped this instance.
B. Back, N. Next, C. Cancel
-----> [N]
Finally, we execute the update:
=====> IBM Installation Manager> Update> Prerequisites> Custom panels> Summary
Target Location:
Shared Resources Directory : /opt/IBM/IBMIMShared
Update packages:
1-. IBM Spectrum Protect (/opt/tivoli/tsm)
2. IBM Spectrum Protect server 8.1.10.20200818_1643
Options:
G. Generate an Update Response File
B. Back, U. Update, C. Cancel
-----> [U]
25% 50% 75% 100%
------------------|------------------|------------------|------------------|
............................................................................
=====> IBM Installation Manager> Update> Prerequisites> Custom panels>
Summary> Completion
There were problems during the update.
WARNING: Multiple warnings occurred.
V. View Message Details
Options:
F. Finish
-----> [F]
=====> IBM Installation Manager
Select:
1. Install - Install software packages
2. Update - Find and install updates and fixes to installed software packages
3. Modify - Change installed software packages
4. Roll Back - Revert to an earlier version of installed software packages
5. Uninstall - Remove installed software packages
Other Options:
L. View Logs
S. View Installation History
V. View Installed Packages
------------------------
P. Preferences
------------------------
A. About IBM Installation Manager
------------------------
X. Exit Installation Manager
Before exiting, let’s check the installed version:
-----> V
=====> IBM Installation Manager> Installed Packages
View the following installed packages and fixes. Enter the number to see the details of a package group, package, or fix.
1-. IBM Spectrum Protect
2. IBM Spectrum Protect server 8.1.13.20211119_2036
Other Options:
O. OK
-----> [O]
We successfully updated IBM Spectrum Protect server to version 8.1.13.20211119_2036 !
Step 3: Recreating the db instances (Both)
Since we did not choose to update the ISP instance in the installer, we need to recreate it manually:
[root@doc02 ISP8113000]# /opt/tivoli/tsm/db2/instance/db2icrt -a server -u isp11 isp11
DBI1446I The db2icrt command is running.
DB2 installation is being initialized.
Total number of tasks to be performed: 4
Total estimated time for all tasks to be performed: 309 second(s)
Task #1 start
Description: Setting default global profile registry variables
Estimated time 1 second(s)
Task #1 end
Task #2 start
Description: Initializing instance list
Estimated time 5 second(s)
Task #2 end
Task #3 start
Description: Configuring DB2 instances
Estimated time 300 second(s)
Task #3 end
Task #4 start
Description: Updating global profile registry
Estimated time 3 second(s)
Task #4 end
The execution completed successfully.
For more information see the DB2 installation log at "/tmp/db2icrt.log.1756".
DBI1070I Program db2icrt completed successfully.
After creating the instance, we can restore our userprofile files into the db2 instance directory.
In our environment, it looks like this:
[isp11@doc02 sqllib]$ cat userprofile
LD_LIBRARY_PATH=/opt/tivoli/tsm/server/bin/dbbkapi:/usr/local/ibm/gsk8_64/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
DSMI_CONFIG=/ISP/ISP11/cfg/tsmdbmgr.opt
DSMI_DIR=/opt/tivoli/tsm/server/bin/dbbkapi
DSMI_LOG=/ISP/ISP11/cfg
export DSMI_CONFIG DSMI_DIR DSMI_LOG
Now we tell db2 to re-catalog our Spectrum Protect instances’ database:
[isp11@doc02 ~]$ db2 catalog database tsmdb1
DB20000I The CATALOG DATABASE command completed successfully.
DB21056W Directory changes may not be effective until the directory cache is
refreshed.
After logging out and in again as the instance owner, we see that a database upgrade is neccessary:
[isp11@doc02 ~]$ exit
logout
[root@doc02 ISP8113000]# su - isp11
Last login: Thu Jan 13 10:54:59 CET 2022 on pts/1
[isp11@doc02 ~]$ db2start
01/13/2022 10:56:06 0 0 SQL1063N DB2START processing was successful.
SQL1063N DB2START processing was successful.
[isp11@doc02 ~]$ db2 get db cfg for tsmdb1
SQL5035N The connection failed because the database product has been updated
and the database needs to be upgraded SQLSTATE=55001
This will be done in the next step.
Step 4a: Upgrading the instance db (Standby)
In an HADR environment, we always upgrade the standby db first:
[isp11@doc02 sqllib]$ db2 upgrade db tsmdb1
SQL1103W The UPGRADE DATABASE command was completed successfully.
Step 4b: Upgrading the instance db (Primary)
When the standby db is successfully upgrade, we do the same for our primary db:
[isp11@doc01 ~]$ db2 upgrade db tsmdb1
DB20000I The UPGRADE DATABASE command completed successfully.
Step 5: Start the instances again (Both)
The installation is finished now, so we can start our services again. In GSCCAD, our systems will look like this now:
So we click on “Start TSM/SP Services” on our primary site:
High Availability for Storage Disks
This scenario is aimed to show how to configure highly available disks (e.g. SAN, virtualized HA, etc.) to be used as storage for GSCC-clustered IBM Storage Protect instances.
In this example, the following environment is used:
Two Linux systems with RHEL 9.5 : rhelgscc-5.gs.com & rhelgscc-6.gs.com
A single ISP instance is managed: ISP01
SPORD installed version is: 3.8.9.1
A single disk shared via VMware SCSI Bus Sharing across both systems
The ISP01 instance is down and the cluster is in operator mode.
We use RHELs Highly Available LVM configuration to control the disk availability in an Active/Passive configuration.
Note
In a production environment, the shared disk usually should be SAN disks accessible via multipath. For our documented scenario, we use a single path.
Step 1: Prepare the disk
In order to use the disk, we need to create the LVM structure and a filesystem first. This can be done on any system with access to the disk.
First, we create the Physical Volume (PV):
[root@rhelgscc-5 ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.
Note
Like mentioned above, in a productive environment usually this would be a /dev/mapper/…. device
Then, we create a Volume Group (VG) on that PV:
[root@rhelgscc-5 ~]# vgcreate stg_vg /dev/sdb
Volume group "stg_vg" successfully created
And on top, we create a Logical Volume (LV) for this VG:
[root@rhelgscc-5 ~]# lvcreate -l 100%FREE -n stg_lv stg_vg
Logical volume "stg_lv" created.
The LV should be available now, so we create a filesystem:
[root@rhelgscc-5 ~]# mkfs.xfs /dev/stg_vg/stg_lv
meta-data=/dev/stg_vg/stg_lv isize=512 agcount=4, agsize=3276544 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=1 inobtcount=1 nrext64=0
data = bsize=4096 blocks=13106176, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Now, we prepare a path on both systems where the filesystem should be mounted later on, in our case:
[root@rhelgscc-5 ISP01]# mkdir -p /SPORD/STG/ISP01/sf
[root@rhelgscc-6 ISP01]# mkdir -p /SPORD/STG/ISP01/sf
Step 2: Configure LVM
To not try to activate the newly created LV on boot or accidentally, but only when GSCC issues it, we now configure both systems LVM to not consider the “stg_lv” for standard operations.
First, we define the volume_list in /etc/lvm/lvm.conf:
[root@rhelgscc-5 ~]# cat /etc/lvm/lvm.conf |grep "volume_list ="
# volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
volume_list = [ "rhel_rhelgscc-5", "@rhelgscc-5" ]
# auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
# read_only_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
Note
rhel_rhelgscc-5is the name of the locally used volume group, which should always be considered for standard LVM operations (e.g. taken online at boot!).@rhelgscc-5is a tag used to identify the system that has a lock on a VG. We use the hostname for this.
Caution
This can be more than one VG in your environment. Make sure to list all relevant VGs accordingly.
Now we define the locking_type in the global part of our LVM configuration:
[root@rhelgscc-6 ~]# cat /etc/lvm/lvm.conf |grep locking_type -A4 -B4
# Configuration section global.
# Miscellaneous global LVM settings.
global {
locking_type=1
# Configuration option global/umask.
# The file creation mask for any files and directories created.
# Interpreted as octal if the first digit is zero.
To check the settings, we can use the lvmconfig command:
[root@rhelgscc-5 ~]# lvmconfig activation
activation {
volume_list=["rhel_rhelgscc-5","@rhelgscc-5"]
}
[gscc-ISP01 root@rhelgscc-5 /]$ lvmconfig global
global {
locking_type=1
}
Step 3: Persist LVM settings
The LVM settings we did in Step 2 are only valid until the system reboots. To make the changes persistent, we have to rebuild the initramfs on both systems.
First, we take a backup of the existing initramfs:
[root@rhelgscc-6 ~]# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak
Then, we rebuild the initramfs on both systems:
[root@rhelgscc-6 ~]# dracut -f -v
dracut: Executing: /usr/bin/dracut -f -v
dracut: dracut module 'systemd-networkd' will not be installed, because command 'networkctl' could not be found!
dracut: dracut module 'systemd-networkd' will not be installed, because command '/usr/lib/systemd/systemd-networkd' could not be found!
dracut: dracut module 'systemd-networkd' will not be installed, because command '/usr/lib/systemd/systemd-networkd-wait-online' could not be found!
dracut: dracut module 'systemd-resolved' will not be installed, because command 'resolvectl' could not be found!
...
dracut: Mode: real
dracut: Files: 1684
dracut: Linked: 12 files
dracut: Compared: 0 xattrs
dracut: Compared: 400 files
dracut: Saved: 35.49 KiB
dracut: Duration: 0.026664 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/boot/initramfs-5.14.0-503.26.1.el9_5.x86_64.img' ***
dracut: dracut: using auto-determined compression method 'pigz'
dracut: *** Creating initramfs image file '/boot/initramfs-5.14.0-503.26.1.el9_5.x86_64.img' done ***
Danger
This WILL corrupt your boot if Step 2 was not done properly. Make sure your entries to /etc/lvm/lvm.conf are correct!
Now we reboot both systems to validate our configuration.
Step 4: SPORD configuration
Now we need to adjust our /etc/spord/spord.conf configuration to ensure that the following is mapped to the container:
the parent directory of our designated mount point (e.g.
/SPORD/STG/ISP01) with the rshared flag to reproduce mount points of the host inside the container!the
/etc/lvm/lvm.conffilethe
/devdirectory
The additional mappings have to be added to the ADDMAPPING line of the file (on both systems):
[root@rhelgscc-6 ~]# cat /etc/spord/spord.conf
#######################################################
# General Storage Software GmbH
# SPORD - Spectrum Protect On Resilient Docker
#
# spord.conf
#######################################################
SPORDNAME ISP01
SPORDHOME /SPORD/HOME/ISP01
DOCKERIMAGE localhost/ubi8:SP8.1.25-GSCC-DSMISI
ADDMAPPING /SPORD/ISP/ISP01,/SPORD/GSCC/ISP01/etcgscc:/etc/gscc,/SPORD/GSCC/ISP01/vargscc:/var/gscc,/NFS/dbbackups,/SPORD/STG/ISP01:/SPORD/STG/ISP01:rshared,/etc/lvm/lvm.conf,/dev
DKDFILE ISP01.dkd
ADDOPTION --pids-limit -1
Step 5: GSCC configuration
GSCC needs to be aware that it should handle the HA disks, so we configure the correct Expert Domain edUNIX_SP8_HADR_STG in our member.def file on both systems:
[root@rhelgscc-5 ~]# cat /SPORD/GSCC/ISP01/etcgscc/ISP01.def
#
# (C) General Storage
#
#
# ISP01.def
# this file is needed for each Cluster Member !
# default path: $HOME/.gscc/sample.def
#
# use ":" for any blank in var-values
#
# <var_name> <value> <description>
expert_domain edUNIX_SP8_HADR_STG
...
Then we need this additional configuration member.stglvm file to specify which devices should be handled by GSCC:
[root@rhelgscc-5 ~]# cat /SPORD/GSCC/ISP01/etcgscc/ISP01.stglvm
stg_vg /dev/stg_vg/stg_lv /SPORD/STG/ISP01/sf
Note
Make sure that you created the defined mountpoint “/SPORD/STG/ISP01/sf” and to have it writable by the instance owner to use it for ISP later.
To be able to force an activation of a volume group still tagged by another host, we also add the following line to our gscc.conf:
Environment DiskForceDuration=300
Our gscc.conf now looks like this:
[root@rhelgscc-5 ~]# cat /SPORD/GSCC/ISP01/etcgscc/gscc.conf
#
# (C) General Storage, 2025
#
# Cluster Controller Configuration File
#
# Activate Auto register
AutoRegister Yes
#Lifetime 1200
#Loclifetime 600
FileLogLevel Error
SysLogLevel Error
Environment LC_ALL=C
Environment LANG=C
Environment StatusReadTimeout=180
Environment StatusFailureThreshold=10
Environment RemoteFailureThreshold=2
port 3939
Environment TSMKillCounter=40
Environment ForceDuration=600
Environment ErrCountMax=1
Environment sshPath=/usr/bin/ssh
Environment SSLPATH=/usr/bin
Environment sshConnectTimeout=10
Environment scpConnectTimeout=10
Environment DiskForceDuration=300
Environment hadrCOM=intern
Environment Docker=on
Environment GratuitousARP=yes
Environment DockerMon=on
Environment DockerMonPort=3930
Environment ISPConsole=on
Step 6: Validating the configuration
To activate the changes to our mapped devices, we need to restart the containers first (if running).
Then, we can test on rhelgscc-5.gs.com if GSCC can properly handle our newly configured HA device with gsccadm:
gscc rhelgscc-5>oper addstg ISP01
INFO: Command addstg for ISP01 listed !
gscc rhelgscc-5>q res vg
rhel_rhelgscc-5 98.41g 1 3 active
stg_vg <50.00g 1 1 active rhelgscc-5
gscc rhelgscc-5>q res fs
/dev/mapper/rhel_rhelgscc--5-root /etc/gscc
/dev/mapper/rhel_rhelgscc--5-root /usr/share/zoneinfo/Etc/UTC
/dev/mapper/rhel_rhelgscc--5-root /var/gscc
/dev/mapper/rhel_rhelgscc--5-root /etc/lvm/lvm.conf
/dev/mapper/rhel_rhelgscc--5-root /SPORD/HOME/ISP01
/dev/mapper/rhel_rhelgscc--5-root /SPORD/ISP/ISP01
/dev/mapper/rhel_rhelgscc--5-root /SPORD/STG/ISP01
/dev/mapper/rhel_rhelgscc--5-root /opt/generalstorage/spord/bin
/dev/mapper/stg_vg-stg_lv /SPORD/STG/ISP01/sf
We see that the VG has been activated on the system and that the file system is mounted at /SPORD/STG/ISP01/sf.
Now let us test removing it:
gscc rhelgscc-5>oper rmstg ISP01
INFO: Command rmstg for ISP01 listed !
gscc rhelgscc-5>q res fs
/dev/mapper/rhel_rhelgscc--5-root /etc/gscc
/dev/mapper/rhel_rhelgscc--5-root /usr/share/zoneinfo/Etc/UTC
/dev/mapper/rhel_rhelgscc--5-root /var/gscc
/dev/mapper/rhel_rhelgscc--5-root /etc/lvm/lvm.conf
/dev/mapper/rhel_rhelgscc--5-root /SPORD/HOME/ISP01
/dev/mapper/rhel_rhelgscc--5-root /SPORD/ISP/ISP01
/dev/mapper/rhel_rhelgscc--5-root /SPORD/STG/ISP01
/dev/mapper/rhel_rhelgscc--5-root /opt/generalstorage/spord/bin
gscc rhelgscc-5>q res vg
rhel_rhelgscc-5 98.41g 1 3 active
stg_vg <50.00g 1 1
The mount is gone and the VG was taken offline correctly.
Now we test the same for our other cluster system, rhelgscc-6.gs.com:
gscc rhelgscc-6>oper addstg ISP01
INFO: Command addstg for ISP01 listed !
gscc rhelgscc-6>q res vg
rhel 98.41g 1 3 active
stg_vg <50.00g 1 1 active rhelgscc-6
gscc rhelgscc-6>q res fs
/dev/mapper/rhel-root /usr/share/zoneinfo/Etc/UTC
/dev/mapper/rhel-root /var/gscc
/dev/mapper/rhel-root /etc/gscc
/dev/mapper/rhel-root /SPORD/HOME/ISP01
/dev/mapper/rhel-root /SPORD/STG/ISP01
/dev/mapper/rhel-root /etc/lvm/lvm.conf
/dev/mapper/rhel-root /SPORD/ISP/ISP01
/dev/mapper/rhel-root /opt/generalstorage/spord/bin
/dev/mapper/stg_vg-stg_lv /SPORD/STG/ISP01/sf
Again, we see that the VG has been activated on the system and that the file system is mounted at /SPORD/STG/ISP01/sf.
Lastly, we remove the VG again by issuing oper rmstg ISP01 and then we can start the ISP instance and join the cluster to automatically take the VG online and mount the filesystem.
SAN-Cluster Operating
This chapter covers common Operator tasks in a shared disk setup and it provides step by step examples for the different situations. The different activities are performed with GSCCAD, but all of them could also be performed with the command line interface “gsccadm”. The following scenarios are covered. Be aware that some scenarios are valid only for certain Expert Domain configurations.
SAN - Starting Point
The first figure shows a typical starting point for a GSCC cluster setup. It is the base for the most scenarios described here. The examples always look only at a single ISP instance with 4 members in the member team. Some Expert Domains like IP cluster or Shared Disk only cluster work with just two members in a cluster, but this is mentioned accordingly.
GSCC Example: Starting Point
In these examples we are looking at a single ISP instance, TSM11, with in most cases a 4 member configuration. This means shared disk resources and HADR are configured for TSM11.
The next chart is the member and resource overview cart which is used throughout the GSCC scenarios to visualize the state changes in certain situations. Inactive resources are covered to see where they could potentially be activated. In this setup it is important to understand that the primary and standby database has its own group of independent resources (HomeVG), while other resources are shared between them like storage pool and IP addresses. The standby (Available) resources are shown in a purple color while the primary resources (Up) are shown in green. Shared resources are light blue.
TSM11 Resources – Starting Point
SAN - GSCCAD
In case the cluster is in a normal state, the GUI “Sites View”-tab would look like this:
Initial Situation: OK
SAN - gsccadm
In “gsccadm” the same status would be checked as follows:
gscc aix1>q status
121119181345 TSM53P Up
121119181400 TSM53S inactive
121119181245 TSM11P Up
121119181345 TSM11S inactive
You have to be aware that in the CLI the status information is only returned from the local cluster side. Command routing (Hostname:command) can be used to also check the opposite cluster side:
gscc aix1>aix2:q status
aix2 returns:
121119181437 TSM53P inactive
121119181522 TSM53S Available
121119181438 TSM11P inactive
121119181438 TSM11S Available
SAN - ISP Instance Maintenance
One possible ISP administration task is to restart an ISP server instance. When an ISP server is stopped by the administrator and GSCC is active, GSCC would just restart the instance. This might in some cases be enough. In other situations ISP must be stopped, some changes on the system need to be performed and ISP is started again afterwards. This kind of maintenance task requires it to give the control to the administrator or operator and not GSCC automatics. GSCC is not stopped in this case, but the state of the different members is changed to Operator, in which GSCC is not reacting on activities on the ISP resources. In Operator Mode (all member in the member team in state Operator) GSCC also provides wrapper commands to control ISP resources. This section will go through the steps to perform a maintenance task in the Operator Mode.
The following member state chart shows the target status for the members of TSM11’s member team. All members have to be in the Operator state and even if only one member would be in Operator all other member would follow automatically. Even if the state is Operator the actual resources are not changed at all.
Operator Status – TSM11 Member Team
Enable Operator Mode
Operator Mode in GSCC is not a simple switch which is executed by pressing the Operator button. This button is rather a request for that state. This is necessary to allow GSCC to finish up the recent tasks or checks. Usually the Operator state is requested on all members of the member team at the same time. However, GSCC would automatically ensure that all other member follow into Operator state anyway.
To switch into Operator state for ISP instance TSM11 we logon to GSCCAD. In the “Sites View” tab you can chose the members. You can choose more than one member at a time by using shift-mouse or ctrl-mouse on the state field. Be aware that with shift-mouse also the fields between the member states are selected, but this will still work as all other fields are ignored. After having selected the correct members press the “Operator” button at the top.
Activate Operator Mode
A confirmation window will allow to double check, which members you want to switch to Operator state.
Confirmation Request
As we want to change the state for all members for TSM11 our selection was correct and we confirm the request for Operator by clicking on “Enable Operator”.
Operator Status
After some time all member states will change into Operator. You are now in Operator Mode with TSM11. GSCC would not react on any action for that instance. Be aware that TSM53 the other ISP instance in the same cluster is not in Operator Mode, so GSCC would still react here. If we would need to perform a system activity like reboot etc. we would need to switch all member teams in the cluster to Operator. The automatic Operator switch is only performed within a member team, not within a cluster.
5.2.2 Maintenance If the TSM11 member team is in Operator Mode, the ISP instance can be stopped and started without intervention by GSCC. In Operator Mode the administrator has the control over the ISP server. If there is a system maintenance or GSCC maintenance all GSCC member have to be in the Operator Mode. While in Operator Mode you could use the common way to stop and start ISP. However, it is recommended to use the GSCC Operator Commands instead as they take care about all other resources like IP addresses, HADR etc. as well. In this example the GSCC Operator commands are used to stop and start ISP. In GSCCAD “Sites View” you can see the Operator status, but the resources of TSM11 are still online.
Resources ISP1P online
You first select the member where ISP is currently active. Then choose the “stoptsm” button in the “Oper Commands” menu to request the stopping of TSM11.
TSM11P in RZ1 selected
“Stoptsm” selected
GSCCAD should open a window only showing the member of the active ISP instance. Verify that the correct member on the right system is chosen. You can see the ISP process status to double check. The actual command needs to be confirmed here again.
Operator: stoptsm
While on the “Sites View”-tab you can follow the progress of stopping ISP and even the standby process. At the same time you can follow the different steps in more detail in the “Action Logs” view.
ISP Server is stopping – Step 1
ISP Server is stopping – Step 2
The Operator tasks like the stopping of ISP can be checked in the “Tasks” tab.:
Task Overview
Be aware, that in this example we only stop the ISP instance. The HomeVG resource was not deactivated. In order to switch the instance to the other host it would another step be required, which will be covered later. Should the ISP instance be started again, the same procedure needs to be followed. There is also a “Start ISP” command. These are the steps for starting the ISP server with the Operator command.
Steps for “starttsm”
Again the details can be watched in the “Action Logs”-tab. The process status can be also seen in the respective column. In this case the process status icon is first yellow. This means that HADR is still catching up and not yet in sync (in peer as it is called in HADR). GSCC will stay in Operator Mode after these actions. The Operator Mode can be left by using the “join” command for all members in the member team.
Exit Operator Mode
As soon as the maintenance jobs were performed, GSCC automatics can be activated again. This step is called as “Join” in GSCC. Accordingly there is a “Join” command to do so. All members in the member team need to receive the “Join” command. As long as one member is still in Operator the actual join will not be performed. This is required to ensure a save activation. After selecting all members and pressing the “Join” button, all members will switch into the state also called “Join”. During this state GSCC waits for other members to also be in this state. The next step then is “Joining”, where GSCC checks the current status of the ISP resources and would switch to the according global state.
In our example the GSCCAD steps are the following. We first select in the “Sites View”-tab all four members (ctrl-mouse) and then execute the “Join”-command. We need to confirm that we really want to send this command to all 4 members we previously selected.
Operator Command “Join”
Operator Command “Join” needs to be confirmed
Join Commands are listed
In the status display you can see all members are switching into the state “Join”.
GSCC starts “Join”
Now GSCC verifies the situation on the hosts. As we started ISP again after we stopped it, GSCC will detect the running instance. Therefore after the “Join” status that member (TSM11P in RZ1) will go to the “Up” state immediately and start to cycle through the different “Up” local states to ensure that all resources are available. The partner member (TSM11P in RZ2) will switch into “inactive” as the HomeVG resource is on the other host. The standby HomeVG is activated in RZ2, therefore the “Down” state is displayed. As soon as HADR availability is detected the “Available” state will reached.
Temporary status while returning into normal status again
Then the cluster join is completed and the members are in the original states.
Normal Status
SAN - Planned ISP Takeover
If the ISP instance should not be started on the same host again, but on the remote host, a planned takeover can be performed. Depending on the configured expert domain this can look different. This takeover scenario describes a shared disk configuration. With a pure IP based cluster this is not possible and the process will be documented later in this chapter. To all other expert domain which have shared disk resources this description apply. So even in the case where in an uncontrolled situation the HADR failover is the first choice, it is still recommended to use the shared disk takeover in a controlled maintenance situation rather than the HADR takeover. During the planned takeover all resources have to be moved to the other side including ISP primary database, logs, ISP home filesystem, storage pool volumes and IP addresses. The starting point is again the joined cluster for ISP instance TSM11 with the typical states.
Resources – Starting Point
The goal is to move the ISP instance, currently controlled by member TSM11P in RZ1, to the other host, where the partner member TSM11P is inactive. Therefore all resources need to be moved accordingly.
The next figure shows the planned status after the move. There are no resources at all for TSM11 on the first host anymore. HADR would still be in peer, but the logs would be shipped locally to the standby. This is possible as also the IP resource for HADR will be moved together with the HomeVG.
ISP Resources have been moved
SAN - Operator Mode
As you have seen in the example before the first step is always to switch into Operator Mode. In order to document the required steps that procedure is also included here.
The Operator Mode will be requested for the members “TSM11P” and “TSM11S” on both sides:
Request Operator Mode
We should see this picture again:
Operator Mode TSM11
Deactivating ISP
As soon as the member team for ISP instance TSM11 is in Operator Mode, the ISP instance can be stopped and the resources can be deactivated. In GSCC there are two steps necessary.
1. Stop ISP Instance (stopping DB2, HADR standby, remove storage pool disks and service IP addresses) gsccadm-Command: operator stoptsm TSM11P
2. Deactivate ISP HomeVG (remove HADR IP Aliasses) gsccadm-Command: operator rmhomevg TSM11P In contrast to the example before, not only the first step needs to be performed, but also the HomeVG must be removed. In certain situation it might be useful to do this in 2 steps, but for convenience reasons the two steps (stoptsm and rmhomevg) can be performed in a single command: “operator deactivatetsm TSM11P” In this example we actually use this command. So we need to select the member, which currently controls the ISP instance and choose “deactivatetsm”.
ISP Server Member TSM11P selected
“Deactivate” selected
GSCCAD requests again to confirm the planned action. It is important to verify that the correct member was selected. Though GSCCAD does some checking to avoid errors, in certain cases it would just perform the requested action. In this example we see that we selected the member which actually controls the ISP instance (running_p_peer).
Execute “Deactivate” Command
The “deactivatetsm”-command will finally then perform the two steps. But it would check that the first step was successful before starting the next one. So GSCC begins to stop the ISP instance.
First Step of “deactivatetsm”: stoptsm
Then GSCC continues to remove the HomeVG, so as seen in the next chart all resources for TSM11P in RZ1 are offline.
Second Step of “deactivatetsm”: rmhomevg
Activating ISP on Target System
The ISP instance is now completely deactivated on the original system. Even the HomeVG disk resources are offline. While still in Operator Mode the ISP instance can now be activated on the remote system. Again this would require two independent GSCC steps: 1. Activate ISP HomeVG (add HADR IP address) 2. Command: operator addhomevg TSM11P 3. Start ISP Server (start DB2 and HADR standby, add storage pool disks and service IP addresses) Command: operator starttsm TSM11P
In certain situation only the HomeVG should be activated in the first step, before already starting ISP (for example ISP update). Then you would only use the “addhomevg” command. Since we plan to immediately starttsm, the best way is to use the combined command “activatetsm”/”activate” now. GSCC is still in Operator Mode for TSM11. So the command can be executed by selecting the member which should control the ISP instance and click “activate” then.
Member selected
Operator Command “activate”
The member and the command can be verified again, before actual starting it.
“Activate” Command Confirmation
The first step for the “activate”-command is to add the HomeVG (see columns DBFS/DBVG and the detailed action log):
“activate” Command – Step 1: addhomevg
When this was successful, the second step is initiated: starttsm. First the storage pool VG is added and then primary and standby DB2 database are started.
“activate” Command – Step 2: starttsm
Now ISP is started and finally the IP alias is added. HADR primary and standby take their role and start to sync.
ISP Server is starting
When the process column is showing the check sign on green background everything went well. In case this would stay yellow, the sync was probably not successful.
Now the cluster can be activated again by performing the “Join”.
Activate Cluster: Join
Is the ISP instance running on the correct side and this is not a temporary action (like a system maintenance job), it is recommended to join the cluster. So again all members for TSM11 must be selected and the “Join”-command has to be run.
“Join”
All 4 members should switch to the state “Join” then:
“Join” States
GSCC detects, that the ISP instance is running on the second hosts and will go to the “Up”-state directly for that member. As the standby HomeVG wasn’t touched, the “Available”-state is set for the other member on the same system (TSM11S).
Takeover completed successfully
SAN - ISP Instance Restart
There are situations where an ISP instance can crash and the “dsmserv”-process ends. In such a case GSCC will restart the ISP instance immediately. The GSCC state cycle “Up” checks all kind of conditions including the ISP instance availability. GSCC would not leave the “Up” cycle, but rather tries to “hold” it by initiating an ISP instance start. Only if this was not successful, this member will tell the other members, that there is a problem by switching to another global state.
Of course this restart function can also be used to intentionally restart an ISP instance. So instead of taking GSCC into Operator Mode and using the starttsm and stoptsm commands, a single “halt” on the ISP instance, would stop and immediately restart ISP.
It really depends on the actual situation which way to choose for the restart. Very often it is sufficient to just restart the instance immediately (dsmserv.opt changes, hang situation, library manager refresh etc.), but sometimes there are other activities necessary before the starting ISP again (device driver update etc.). Also be aware that in a situation where the ISP instance is causing problems which results in the idea to restart the instance, it might be worth to switch into Operator Mode even if a simple restart might solve the problem. The reason is that the cause of the problem might not be clear or resolved. So even a restart would be slow or hang. A typical example would be a bad fibre channel link which causes all disk activities to be slow or hanging. With GSCC active and now restarting ISP this might result in an unwanted failover.
In this example we use the “halt” alias “stoptsm” in the ISP command line to stop the instance:
tsm: TSM11>stoptsm
ANR2234W This command will halt the server; if the command is issued from a remote client, it may not be possible to restart the
server from the remote location.
Do you wish to proceed? (Yes (Y)/No (N)) y
ANS1017E Session rejected: TCP/IP connection failure
ANS8001I Return code -50.
ANS8064E Communication timeout. Reissue the command.
ANS1017E Session rejected: TCP/IP connection failure
ANS8002I Highest return code was -1.
As explained before the member stays in the global state “Up”, but the ISP process indicator switches to “inactive”. GSCC initiates the start of the ISP. The “Up” state announces to the other members clearly that no actions are needed from their side as long as the active member is still tryoing to resolve the situation locally.
ISP Server ISP1P is stopped
The local state within the “Up”-cycle changes to “Starting_ISP”. Only in a situation where this start does not work as expected due to major problems with this instance, the global state would change from “Up” in order to involve the other members in the member team. However, that scenario is shown in another section. In this example the ISP instance start was successful.
ISP Server TSM11 is started automatically
The ISP instance might not be available immediately as first the DB2 instance needs to be prepared (yellow process indicator). Finally the normal status is reached again.
ISP Server TSM11 is started again
SAN - ISP Shared Disk Failover
A shared disk failover can be necessary in different situations. The manual takeover is one of them and was already described. The crash of a complete system is another example and will be covered in the next section. In this section a situation for shared disk failover is shown where both systems are still available but a restart of the ISP instance on the same system was not possible. This could be caused by software installation problems, DB2 instance problem with local home filesystem or IP addresses which could not be activated. As both systems are still available this is in GSCC referred as a controlled situation. This is important as the reaction can differ depending on the expert domain used. In a controlled situation with all shared disk configurations, the preferred failover mechanism is always the shared disk failover.
Problem with TSM11 in RZ1
The shared disk resource of the productive home member is still online, but ISP is not startable or maybe the IP address could not be activated. So while still in “Up” the member tried all local problem fixes, but did not succeed and therefore switch into the cycle “Down” and starts to clean up the resources. When that is done the stable state “Broken” is reached. This is a signal to the partner member on the remote system.
The remote member starts to activate the HomeVG first, represented by the “Down” state.
Shared disk failover – addhomevg step
As soon as this was done the member continues to the “Up” state, where finally ISP is started. When the problem was a local problem ISP would come up now.
Broken – Failover successful
Shared Disk Failover
In this example we assume that the ISP instance suddenly shuts down due to an error. As in the example before, GSCC first tries to restart the ISP instance locally while still in the “Up” state. In the “Sites View”-tab the process indicator for TSM11P in RZ1 turns to inactive. GSCC initiates the restart attempt.
GSCC attempts to start ISP again
However, the attempt was not successful and a page message was sent (am_pageoper in the action method log).
The restart attempt was not successful
GSCC will switch into the state “Down” and begins to clean up the HomeVG resources. As soon as this was done, the global state will be “Broken”. The partner member reacts on the “Broken”-state and initiates the activation of the HomeVG on the remote system.
Status Broken
The activation starts during the “Down”-state, where the HomeVG is added (addhomevg step). When this was successful, the member will continue to the “Up” state, where the ISP instance is started (starttsm step).
Up state reached
Currently the ISP process indicator is still inactive while preparations are ongoing.
ISP Instance start is being prepared
Now the ISP instance is started and right after that the IP addresses are added.
ISP instance failover completed
Cleanup Error State
In this example the failover to the other host was successful. Now the error situation on the original host has to be analyzed. When the cause of the error was found and it could be resolved, the cluster state has also to be cleaned up. This is done in two steps. First the complete member team is brought into Operator state. Be aware that the the member in state “Broken” would not follow the other members in this case, so please ensure to select all members and request the Operator state. The reason for that is to avoid a situation where the “Broken”-state is left unintended, although the error cause was not really fixed. The second step is to clear the “trigger”, which is the indicator for that error.
Operator Mode requested including “Broken” member
Requesting Operator Mode
As soon as the Operator Mode is enabled, there is a “TRIGGER” button available. The intention of this button is to force a cluster reaction which was set to manual before. In this case there was an automatic reaction and it was needed to manually use it. However, the trigger situation (q trigger) still documents the error situation. By clearing this trigger the correction of the error situation is confirmed. This is done by selecting the member and clicking the “Clear Trigger” button.
ISP Instance Failback
In this section the failback after the shared disk failover is explained. As the steps are basically the same as during the manual failover only the most important screenshots are shown.
Currently we are still in the Operator Mode. So we can directly use Operator commands and deactivate the ISP instance on the remote host.
Failback - deactivate
And then on the home system the activate-command is requested. This is only the confirmation window:
“activate”
Finally the cluster is joined again and the status is back to normal.
Failback finished
SAN - ISP Standby Failover
Since HADR is supported the standby failover has different role on the GSCC cluster. With classic sync where the database is further behind the primary this failover was always only the second alternative as it could mean loss of some backup transactions. That’s why the standby failover always required a manual intervention. With HADR and the possibility of nearsync or sync configuration this has changed. An automatic standby failover is supported. This is especially true for the IP only cluster configuration. However, even in a HADR expert domain with primary shared disk failover the standby failover is a manual operator step.
In this section the operator intervention based standby failover is described. We assume that the shared disk failover to the other host was already performed by GSCC automatically, but the problem could still not be resolved. The ISP instance startup was unsuccessful even on the remote host. If we look to the state charts, we see that the “Broken” state was set for the local primary member.
Productive instance has a problem – VG Failover will be initiated
The cleanup was successful and the “Broken”-state was reached which initiates a state change for the partner member on the remote host. This will be “Down” in the first step for adding the HomeVG. The next figure shows that the partner member afterwards changes to the “Up”-state which attempts to start the ISP instance.
Cluster attempts to activate ISP Server on the remote Server
In this example the start does not work on the remote host either. So the “Up” state cannot be fulfilled. Therefore the next step is to return to “Down”. Depending on the configuration this can lead to another restart attempt or it would stay in the “Down” state.
ISP Server Restart after shared disk failover fails
GSCC will send a paging message to the administrator (am_pageoper) in this situation. Also a trigger was set for this member. Just by confirming the error situation for the member TSM11P in RZ2 the standby will react. This is done by selecting this member in the GUI and using the “TRIGGER”-button. The trigger condition will cause a state change to “Failed”, which is in contrast to “Broken” now giving a signal to the peer member rather than to the partner member to get involved. As soon as the peer member is “Failed” the member for the standby database will go to the state “Up” performing a failover. Depending on the configuration this can be a recovery and commitment of a database which was synced the classic way or it will be the HADR failover.
Operator “TRIGGER” activated standby instance
SAN - System Crash – Shared Disk Failover
This scenario describes the situation where a complete system is unavailable. The description focus on a single ISP instance as GSCC is an application based cluster. However, be aware, that during a system crash the same procedure will be followed for all ISP instances which ran on the failing host and all the more that the surviving instances on the other host will lose their standby database in case of an HADR configuration. Although it is configurable, the common configuration would not perform an automatic failover for the standby instances. There are two different expert domains for the system crash situation. This section will cover the expert domain with shared disk failover as the primary reaction on a system outage, while the next section will cover the HADR failover expert domain. In this example we assume that one system is not reachable at all including heartbeat. In case the system would only be isolated and is not really down, the isolated ISP instances would be stopped, to ensure that the instance is not activated on two hosts which were just split.
System crash
With this expert domain the partner member will react on the unaviable remote system. After initially switching to “Down” to activate the HomeVG the partner member will go “Up” and start the ISP instance.
Failover successful
In this example the AIX system “whv22” is not reachable. This is immediately visible in the GUI in the “machines”-tab, but also in the “Sites View”-tab. Of course the reaction in “machines”-tab can have several reasons as it only means that the communication to that host is lost.
Server not reachable
The “Sites View” will provide further information, as the partner member TSM11P in RZ2 would check all conditions and if necessary leave the “inactive” state in case the failover is required.
Server down
In the next screenshot the local state is already in “inactiveSep”, but still checking the remote status.
inactiveSep
After further checks including heartbeat the member TSM11P on the surviving system is leaving the “inactive” cycle and switches to the global state “Down”. This is the first step to add the HomeVG (addhomevg). When this was successful, the next visible global state will be “Up”. Now the ISP instance is started.
ISP is starting
The failover was successful, now only the standby, which was running on the surviving system needs to be synced into peer. During ISP startup it can happen that the standby loses connection. However, GSCC would restart the standby while in “Available”.
ISP is active
SAN - System Crash – HADR Failover
In an HADR setup with shared disks a server crash can be resolved in two ways. The first option is to attempt to activate the disks of the primary database by force on the remote hosts. This was described in the previous section. Option two is to activate the independent synchronized standby database instead. This option is explained here. Both options have their distinct benefits. The discussion about that is beyond the scope of this section. However, it is important to understand that the option is chosen by the GSCC entity called expert domain. By using a different expert domain it is possible to change the cluster behavior. No other changes in the ISP or GSCC configuration are required.
In GSCC the system crash is considered an uncontrolled situation. This means that certain members are not available and the decision has to be made on local inputs only.
When the remote system is unavailable both member in RZ2 for TSM11 will realize the problem. In contrast to the previous scenario, there is no reaction on the partner member, but on the peer member, which is TSM11S in this case.
Server or LPAR crash
The member will after further checking and ensuring the other system is really unavailable switch to “Up” to begin the ISP startup. Please note that compared to the shared disk failover by the partner member before, there is no “Down” step in between needed as the HomeVG for the standby is already activated here. To ensure that the remote system is really down a heartbeat and a client IP quorum is used. So if the local system is not really down but only isolated, the state model would stop ISP when the client IP quorum is not met. Of course the client IP quorum is also checked before the “Up” state is reached on the standby system. During startup GSCC will detect that a DB2 HADR takeover is required to turn the standby database into a primary. To ensure consistency the automatic takeover will only be started as long as the database are in the peer window. As ISP always restart DB2 itself, HADR will be stopped in this case to ensure ISP can be activated. If the failing system did not take real harm the previous primary can be reintegrated into the HADR cluster as standby later and HADR can be activated on the currently active system. Be aware that the reactivation of HADR currently requires manual intervention. In order to resolve the situation it is recommended to use the Operator Mode.
ISP activated on standby database
IP-Cluster Operating
This chapter covers common Operator tasks especially for clusters with IP only failover configurations. It provides step by step examples for the different situations. The different activities are performed with GSCCAD, but all of them could also be performed with the command line interface “gsccadm”. The following scenarios are covered. All the examples are based on the expert domain “edUNIX_T6_HADR”, which contains the ruleset for the HADR based IP only cluster configuration.
IP - ISP Starting Point
The following chart shows a typical setup for a GSCC HADR cluster. The ISP database and log filesystems are located on internal disks. The ISP storage pools are using NAS disks and physical tape libraries. There are two active ISP instances (TSM86/TSM87) which are synchronized via IP by GSCC and DB2 HADR. The active ISP instance is running on the primary DB and the inactive copy of this database is referred as the standby database. This setup is the starting point for the upcoming scenarios described in this chapter.
IP only cluster setup example
From GSCC perspective the primary and standby databases are controlled by member definitions. In this setup there are two members for every ISP instance, one per host. The next chart shows the member status for the ISP instances in the normal operating mode. In contrast to the shared disk scenarios explained before, the disk resources are always online. GSCC controls which database is active (primary database) and where the ISP process is running and IP addresses are set. In normal operation the other database is available (standby database) and is applying log files shipped from the primary database. The storage pools are in this case also connected via IP (NFS) and are therefore already available on both systems.
Member status – starting point
GSCCAD
In case the cluster in a normal state, the GUI “Sites View”-tab would look like this. You can mouse-over to see the detailed status of the process (PROC), the home VG and filesystem should always be online (DBFS). By clicking the right mouse-botton the view can be expanded to show even network and storage pool status.
GSCCAD – normal status
gsccadm
In the command line interface “gsccadm” the status would look like this. Be aware that gsccadm returns the local status on the host where it was started.
gscc lax04>q status
150430081345 TSM86 Up
150430081400 TSM87 Available
gscc lax04>q tsm
TSM86 running_p_peer
TSM87 db2sysc_s_peer
However, you can use command routing to get the status from the remote host.
gscc lax04>lax05:q status
150430081345 TSM86 Available
150430081400 TSM87 Up
gscc lax04>lax05:q tsm
lax05 returns:
TSM86 db2sysc_s_peer
TSM87 running_p_peer
IP - ISP Instance Maintenance
As the activated cluster will react automatically on certain changes it is important that with maintenance situations such as system reboot, ISP changes or network changes the cluster automatics are temporarily disabled. With GSCC this is done by switching the member’s status to “Operator”, so the cluster is in “Operator-Mode” before starting any maintenance activities. While automatic functions are disabled the “Operator-Mode” supports GSCC logics to start and stop ISP, move ISP instances to the other hosts and more. The GSCC logic will take care of all required steps including DB2 and HADR handling, so it is even recommended to perform these actions by using GSCC operator commands instead of executing these steps manually.
Operator Mode
The operator mode needs to be requested through GSCCAD or gsccadm. All members in a member team (all members controlling one ISP instance) will follow into the “Operator” state as soon as on member was set to that state. In the HADR only configuration there is one member on each system for a ISP instance. In this chart both ISP instances TSM86 and TSM87 are shown with their member status und the resources. In this example we will set both member teams into the Operator status. The resources stay unchanged of course. In the following chapter you can follow some of the typical maintenance tasks which can be performed when GSCC is in Operator Mode.
Operator Mode
It is best practice to select all member status fields in GSCCAD using shift to allow multiple selection. Then the Operator button is used.
Operator Mode request
As with all GSCCAD commands a confirmation window appears with a list of members and its current status. In this case the command is “enable operator”. The command will be executed for all 4 members. As this is a request it might take some time until all members have reached the expected status.
Enable operator for all members
All Members are in Operator State
When all members are in Operator state, the maintenance activities can start. In order to stop, start or takeover ISP the GSCC Operator commands should be used. After the maintenance work is done, the cluster automatics are activated again by using the Operator Command “Join”. During the Join the current status of HADR and ISP is considered and the cluster states reached accordingly. In this example, there were no changes, so the previous status is displayed again.
All Members need to be selected for the “Join” Command
The “Join” Command is confirmed for all Members
All Members are changing to the state “Join”
All Members are back to the normal States again
IP - Planned ISP Takeover
To move a ISP instance to the other server, GSCC needs to be set into the Operator Mode as described before. The GUI would then show this status:
All Members are in “Operator”
To perform a takeover, we use the “Switch SP/TSM Services” function in GSCCAD, which will first stop the ISP instance on the active site, then perform an HADR takeover and finally start the instance on the inactive site.
Select a member of the instance you want to switch, then click “Switch SP/TSM Services”
Now a confirmation is requested that the ISP process is in the “running_p_peer” status and we plan to switch this instance.
Confirming the switch operation.
GSCC now will start to stop the ISP instance. It will also take care of other resources like storage pools, IP addresses and will also ensure that HADR is stopped properly.
The “Local Status” in the “Members” tab shows that we issued the command to switch.
“Local Status” shows Switch_services_Oper, meaning that the switch tasks are about to start.
Check the “Action Logs” tab to follow the current state and ongoing GSCC tasks.
The action logs show that ISP is shutting down on machine “lax04”
When the ISP instance is stopped on lax04, the storage volumes are deactivated and the service IP is disabled.
GSCC has completely shut down the ISP instance and the configured resources.
Now the ISP instance will be started on the other server.
Starting ISP on the other server
During the start, GSCC recognizes that an HADR takeover has to be performed.
An HADR takeover will be executed
After a successful HADR takeover, the storage volumes are activated, ISP is started, and the service IP is configured on lax05.
GSCC starts all resources and the ISP server on lax05
When the switch is done, GSCCAD will show that both members are in peer again, with changed roles.
The switch is completed
IP - ISP Instance Restart
If the Spectrum Protect Server needs to be restarted to, for example, activate a configuration change, we can use the “Restart SP/TSM Services” option in GSCCAD.
First of all, let us enable Operator mode on the instance we want to restart.
Make sure we are in Operator Mode
Now we select the member with the running ISP instance and select the “Restart SP/TSM Services” command.
Selecting “Restart SP/TSM Services” from the members operator command menu.
A confirmation window will appear.
We need to confirm the restart with “Run”.
After confirmation, GSCC will perform a full “stop” procedure, meaning, that the service IP address and storage pool resources will also be deactivated if configured.
GSCC stops all the configured resources in addition to the Spectrum Protect server.
Immediately after the stop finished, the resources and the instance are brought up again.
GSCC proceeds to start the DB, activates the volume group and so on…
If configured, GSCC will show the console output of the starting Spectrum Protect Instance.
ISP Console output is shown in GSCC Action Logs
Finally the systems reach a peer state again after re-enabling the service ip address.
The restart is finished.
IP - Automatic Activation after Reboot
Usually the GSCC Operator commands are used to start and stop ISP. In a situation where both or one server node in the cluster were rebooted, it is also possible to just join the cluster to activate ISP. GSCC will consider the last HADR role in this case, so no takeover is required, but ISP is started on that server node, where it ran the last time. GSCC is in Operator Mode, all resources are offline. Only the LVM filesystems for the database are online, because they are local resources.
GSCC in Operator – Everything is down
All Members selected
Executing “Join”
Confirm the “Join”
All Members are joining
There are no resource online. That’s why all members are switching into the state “Down”.
All Members are “Down”
GSCC verifies the HADR roles and decides a state change to Up (primary) or Available (standby). The states will trigger the activities required to reach the expected status.
Roles verified
All Resources activated again
IP - System Crash – HADR Failover
When a complete system is down, a HADR failover is initiated. As the HADR communication is not available anymore, the failover needs to be forced. The client IP quorum needs to be met to allow the automatic failover to be started. In case the other system was not crashed, but isolated (not meeting the quorum) it will stop all activities and will not restart even if communication is re-established. To ensure consistency GSCC will only perform an HADR “force takeover in peer window only” on the surviving server. This HADR function will prevent the takeover, when before the failure the communication between the HADR nodes was already lost. In case of a split between the sites the primary can no longer ensure a synchronized state (“peer” state in HADR terms) with the standby. Therefore during the peer window no transaction will be For more information on HADR and peer window, see the IBM DB2 HADR documentation.
Server Crash
Before the crash the status is as following.
Normal Status
The physical server “lax04” has crashed. The status in “unknown”.
Server Crash
After checking the heartbeat and the client ip quorum, the decision is taken to takeover the database. This is visible by the member state switching to “Up”. The process state is still standby and needs to be changed.
Cluster Reaction
Checking HADR Status
The log shows the final takeover command using the “peer window only” option to ensure consistency.
Takeover – peer window only
The takeover is successful and the ISP server start is prepared. It is required to stop HADR at this point to continue.
Takeover successful
Finally the ISP server is started. The previous primary cannot be activated anymore at this point of time without manual intervention. If the remote server was only temporarily down, the previous primary can be reintegrated to the active database.
ISP is started
Both Instances running on lax05
DB2 Reintegrate after a Failover
After a system crash and a successful failover to the surviving node, cleanup steps will be required to reach a working HADR cluster status again.
In many cases, the databases on the crashed server were not affected, since the disks survived and only system needs to be restarted. In these situations it is most likely possible to activate the databases as HADR partner again.
In the documentation environment, there are two different cases, since two instances are part of the cluster:
TSM86: The database was the “primary” HADR partner before (ISP was affected and activated on the surviving node)
In this case the previous primary database must be “reintegrated” as the standby to establish redundancy again.
TSM87: The database was the “standby” HADR partner before (ISP was not affected and only the redundancy was lost)
In this case the standby database just needs to be activated again.
Is the system automatically restarted after the crash, GSCC would join the member after restarted and would attempt to perform the required activity. In cases where the node stays down and will be reactivated later, for example after repair actions, the tasks would need to be performed as Operator Commands.
Be aware that there are situations where GSCC and HADR cannot reuse the databases on the crashed server. Even with clean database disks the reintegration or standby activation can fail, especially if the required logs are already missing after database actvities or database backups.
This would require a database resync (db2 backup / db2 restore) to recover redundancy.
Automatic Cleanup
This is the GSCC status after the crash.
GSCCAD showing “unknown” status of the failed site
The crashed server is coming up again. GSCC is joining into the cluster.
The failed site is up again and tries to “join” the cluster
Since no components are currently active both member switch to the “Down” state.
Since the site failed, at first both members will have the status “Down”
The remote members are “Up” and are controlling the ISP instances. Therefore the next state is “Available”, Where GSCC attempts to establish redundancy again.
Both members were able to reach the “Available” status.
In order to establish redundancy for the two ISP instances the two “Available” members need to perform different tasks:
TSM86 was the primary before, therefore reintegration is started (am_hadrhealth)
TSM87 was even before the standby, so the database is only activated (am_hadrhealth)
Action logs showing the performed methods.
The two databases of the previously crashed server were still in a good state. Since the server came up in time the required logs are still available and the reintegration and the standby activiation succeeded. HADR is in both cases in “peer” again.
Everything is in “peer” status again.
Manual Cleanup
In contrast to the previous situation the previously crashed server could not be restarted immediately, so to cleanup the cluster we set it to Operator Mode.
Cluster in operator mode
There are two different cases on the restarted node:
TSM86: The database was the “primary” HADR partner before (ISP was affected and activated on the surviving node)
In this case the previous primary database must be “reintegrated” as the standby to establish redundancy again.
TSM87: The database was the “standby” HADR partner before (ISP was not affected and only the redundancy was lost)
In this case the standby database just needs to be activated again
The two cases need to be performed manually now. The “Members” tab in GSCCAD allows to distinguish the cases by checking the “TSM Status” column.
TSM86: running_nohadr
TSM86: running_p_disc
Case 1: Manual Reinitegration (TSM86)
The primary member with the running ISP instance is selected. The Operator Command “reintegratedb” is in the Utilities menu.
Using the “reintegrate” utility in GSCCAD
The confirmation window appears and describes the required conditions, which will be verified by GSCC before execution.
Confirmation window for reintegration.
In the “Action Logs” tab the reintegration task can be monitored. Depending on the number of required logs, the reintegration might not complete immediately like in this example. In such a case a message will be written to the log asking the Operator to check the HADR status.
Action logs showing the performed methods.
Case 2: Manual Standby Activation (TSM87)
In this case the previous standby member needs to be activated. So the standby member is selected and in the DB2 menu the “StartStandby” command is used.
Starting the standby database
The confirmation window appears showing the inactive DB instance.
Confirmation for “StartStandby”
In the “Action Logs” Tab the command can be monitored. In this example the HADR status switched Shortly after the initiation into “peer”. However, this might take longer depending on the required logs to be processed.
Action Logs showing the starting method