Components
GSCC consists of the following parts:
Software
The GSCC components are delivered in a single installable package for AIX, Linux and Solaris:
AIX: gscc-3.8.0.305.bff
Solaris: GSgscc-sparc-3.8.0.305-0.pkg
Linux: gscc-3.8.0.305-0.x86_64.rpm
# lslpp -l gscc.rte
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
gscc.rte 3.8.0.305 COMMITTED General Storage Cluster
Controller
Path: /etc/objrepos
gscc.rte 3.8.0.305 COMMITTED General Storage Cluster
Controller
# rpm -qa|grep gscc
gscc-3.8.0.305-0
This document is based on GSCC version 3.8. Here is some information about the way versioning is working within GSCC.
Versioning
GSCC versions are split up in 4 parts:
Version.Release.Maintenance.Fix
Part |
Description |
|---|---|
Version |
New architecture, design, and or extended functionality |
Release |
Major changes with additional functionality |
Maintenance |
Collection of major bug fixes |
Fix |
Bug fixes and minor updates |
The different update types are always complete packages.
Updates
It is recommended to keep the software versions consistent across the cluster hosts. In order to perform a GSCC update the software needs to be stopped. First the Operator Mode must be enabled then the processes are stopped and after the update started again. ISP needs not to be stopped during the update procedure. Please check for further update information in the attached README file, before performing the update.
Directory Structure
When GSCC is installed on the hosts, there are a few new paths and directories created.
GSCC is using these three paths for binaries, executables, configuration and logging:
Executables and logic ( /opt/generalstorage/gscc )
GSCC configuration ( /etc/gscc )
logs and files for internal usage ( /var/gscc )
The different componentes are explained here:
Executables and logic
Location:
/opt/generalstorage/gscc
In this path you will find all executables for GSCC including the GSCC logic modules. There is no configuration or adjustment recommended in this area as it will be overwritten during the next update.
These are the subdirectories:
Components |
Location |
|---|---|
GSCC Executables |
/opt/generalstorage/gscc/bin |
GSCC Action Methods |
/opt/generalstorage/gscc/am |
GSCC Helpfiles |
/opt/generalstorage/gscc/doc |
GSCC Logic Module |
/opt/generalstorage/gscc/ed |
GSCC Input Methods |
/opt/generalstorage/gscc/im |
Configuration
Location:
/etc/gscc
GSCC configuration files are only located in this directory. This directory needs to be included in the backup to be able to recreate the GSCC configuration. Most important are the definition files for the members (.def). All cluster members are defined in cl.def. For each of the members listed in cl.def a member definition file needs to exist. The filename of this member definition file is created from the member name and the extension “.def” (<member>.def).
The members listed in cl.def are loaded during GSCC startup according the configuration in gscc.conf. Usually “AutoRegister” is configured, so all cl.def members are registered directly into the cluster.
Further configuration files are used for disk configuration. There are 3 different disk configuration file types.
HADR with local HomeVG (AIX/Linux): <member>.tsmlocalfs
Shared Storage for HomeVG (AIX): <member>.tsmvg
Shared Storage for Storage Pools (AIX): <member>.stgvg
Shared Storage for Storage Pools (Linux): <member>.stglvm
In /etc/gscc you can also define pre and post events for the ISP instances. This is done for all ISP instances with tsm.conf and tsm.cleanup. In case only a certain ISP instance should be addressed by these events, the member can be specified with the filename: <member>.tsm.conf and <member>.tsm.cleanup.
In the subdirectory “sample” are examples for the different GSCC configuration files.
Logs and files for internal usage
Location:
/var/gscc
GSCC is using the /var filesystem or path for all logging and for all internal cluster controls like communication and caching. It is recommended to place it on a different filesystem in case /var is not a separate filesystem so far. The size should be at least 1GB. Logging and control files are in different subdirectories:
/var/gscc/flags/<member>
You will find a dedicated subdirectory for each cluster member in this directory. It is only used for the internal cluster communication and some caching and recreated during a GSCC restart.
/var/gscc/logs
This directory contains all logging and debugging information for GSCC. The central cluster logfile is “gscc.log”. The loglevel for this file can be adjusted in the /etc/gscc/gscc.conf file. The Debug loglevel results in massive amount of data and should only be used when requested by the support. Further logfiles are member log files (input methods and action methods), which are defined in the <member>.def file (log_file, am_log_file). All logs can be rotated by using the GSCC logrot utility. The utility needs to be started by a schedule tool like cron. The logfiles which should be monitored are defined in “/etc/gscc/logrot”:
[root@aix1:/]> cat /etc/gscc/logrot.conf
/var/gscc/logs/gscc.log 2000000 3
/var/gscc/logs/tsm53s.im.log 200000 3
/var/gscc/logs/tsm53p.im.log 200000 3
/var/gscc/logs/tsm53.log 200000 3
/var/gscc/logs/tsm11s.im.log 200000 3
/var/gscc/logs/tsm11p.im.log 200000 3
/var/gscc/logs/tsm11.log 200000 3
Logfiles
Since GSCC does generate a lot of logworthy output, it would be confusing to write all output to a single logfile. Hence, there are different logfiles for the different components.
gscc.log
This log contains information about the loading of expert domains and rulesets during the startup of GSCC. While GSCC is running it will log erros in case a input or action method is causing a problem. In case you activate the Debug loglevel this file will contain also information about the method calls performed.
am_logfile
This is the action method log file. This file contains information from the GSCC action methods like starting or stopping TSM. This log file can be used to track Operator Command activities and is also used in the GSCC GUI for that reason.
log_file
(The actual logname is defined in MEMBER.def) This log file can be activated for debugging (instead of “None” state a filename after “log_file”). This log file will contain all input method results for the specific member. This log can grow very large, so it is recommended to rotate it daily when activated.
gscomd.err
This is the log file for the GSCC communication module, responsible for inter cluster communication and administrative access. This log will contain messages when there is a problem within the module rather than general communication problems which will be logged in the socket client log “gsscli.err”, input method logs and in gscc.log. In a normal cluster situation this file should be empty. Messages have to be verified with GSCC support.
gsscli.err
When there are problems in the socket communication error messages are logged here. However, you might see socket errors logged, which are the result of interrupted communications, which are easier to analyze on method level. Temporary connection errors might be ignored but double checked first: Unable to open TCP/IP connection with 172.100.100.10:3939.
gssrep.err
This log file contains internal command errors. Errors logged here are most likely caused my program errors. Exception can be temporary problems on local disks (/var).
GSCC Daemons
GSCC is running with three daemons with different functions. In AIX these are setup as subsystems, while in Linux they are configured as services. For normal operation all three daemons are stopped and started at the same time. 1. gscomd – Communication Interface (gsccadm, cluster interconnect) 2. gsccd – Cluster Controller (controlling the cluster logic) 3. gsemd – Event Manager (controlling and triggering gsccd)
In AIX the subsystems can be grouped which is done during GSCC setup.
[root@aix1] /tsm > lssrc –g gscc
Subsystem Group PID Status
gscomd gscc 10006 active
gsccd gscc 16314 active
gsemd gscc 19518 active
[root@aix1] /tsm > stopsrc –g gscc
0513-044 The gscomd Subsystem was requested to stop.
0513-044 The gsccd Subsystem was requested to stop.
0513-044 The gsemd Subsystem was requested to stop.
[root@aix1] /tsm > lssrc –g gscc
Subsystem Group PID Status
gscomd gscc inoperative
gsccd gscc inoperative
gsemd gscc inoperative
[root@aix1] /tsm > startsrc –g gscc
0513-059 The gscomd Subsystem has been started. Subsystem PID is 19534.
0513-059 The gsccd Subsystem has been started. Subsystem PID is 21144.
0513-059 The gsemd Subsystem has been started. Subsystem PID is 17564.
The processes in the process list:
[root@aix1] /tsm >
root 15270088 3080318 2 Nov 15 - 4:39 /usr/generalstorage/gscc/bin/gscomd
root 16122046 3080318 0 Nov 15 - 17:11 /usr/generalstorage/gscc/bin/gsccd
root 16318664 3080318 0 Nov 15 - 1:31 /bin/ksh93att /usr/generalstorage/gscc/bin/gsemd
In Linux the daemons are registered with chkconfig as shown here, so that also the daemons are started after a system boot:
# cd /etc/init.d
# ln -s /opt/generalstorage/dsmisi/bin/rc.gscc gscc
# chkconfig --add gscc
# chkconfig --list gscc
gscc 0:off 1:off 2:on 3:on 4:on 5:on 6:off
lax02:/var/gscc/flags # service gscc status
Checking gscc daemons: running
lax02:/var/gscc/flags # service gscc stop
Stopping gscc daemons: done
lax02:/var/gscc/flags # service gscc status
Checking gscc daemons: unused
lax02:/var/gscc/flags # service gscc start
Starting gscc daemons: done
lax02:/var/gscc/flags # service gscc status
Checking gscc daemons: running
In AIX the autostart is configured in /etc/inittab:
gscc:2:once:startsrc –g gscc >/dev/null 2>&1
GSCCADM
“gsccadm” is the GSCC command line interface. It can be started like this using the IP address and the GSCC port as options:
gsccadm <IP/Hostname> <Port>
Usually gsccadm is called wothout any option a access the local GSCC communication daemon with the default port:
>> gsccadm
General Storage Cluster Controller for ISP - Version 3.8.2.4
Command Line Administrative Interface
(C) Copyright General Storage 2016
Please enter user id: admin
Please enter password:
GSCC Version(CLI Revision): 3.8.2.4(2154)
gscc aix1>q status
121119154236 TSM53P Up
121119154236 TSM53S inactive
121119154151 TSM11P Up
121119154322 TSM11S inactive
gscc aix1>aix2:q status
aix2 returns:
121119154417 TSM53P inactive
121119154347 TSM53S Available
121119154417 TSM11P inactive
121119154417 TSM11S Available
The command format is like this
command <subcommand> <parameters>
As shown in the example above it is possible route commands to another host. hostname:command <subcommand> <parameters>
The different possible commands and their subcommands an paramter are explained in Command Reference.
GSCCAD
GSCCAD is the graphical user interface for GSCC. It is a Windows based GUI requiring .NET for installation. This tool covers most of the cluster controls and functions. Only a few special commands for debugging still need the command line. However, GSCCAD even provide the command line for the GSCC hosts. GSCCAD can manage more than a single cluster in one view.
Starting and Connecting
After starting the GUI an user ID and Password is requested. The user management is the same as with “gsccadm”. Therefore user ID and password are identical. However, the user ID and password is only checked after the GUI really contacts the different cluster nodes. User Configuration across the cluster nodes which will be managed with this instance of GSCCAD must be identical.
GSCC Login
If machines (cluster hosts) had been configured before, the standard view will show the different entries. Otherwise this view is empty and machines can be added. The machine information is the only input needed. All the member details are loaded after committing the new machines.
Machines and cluster overview
Status View
There are two possible ways to show the GSCC status in GSCCAD, the “Member View” and the “Site View”. The “Member View”-tab displays more detailed information including a HADR status. The “Site View” is simpler, but it organizes the members by its location. With mouse over some of the more detailed information are also available.
Status “Members”
Status “Sites View”
The taps can be organized differently by moving them around with the mouse or choosing one of the preconfigured views which are available in the setting panel.
Views
Example for different tab setup
User Management
GSCCAD does not use a separate user management, but relies on the existing local gscc users on the cluster hosts. This is an example, how a new user can be added. The user administration is provided in the “Settings” window. It is recommended to add the user on all cluster hosts at the same time as this is the requirement when administrating the cluster hosts with the new user in the same GSCCAD setup.
Add a user
Response
In order to change or delete a user another tab is used:
Delete user
Response
Operator Mode
The Operator Mode is a very important function in the GSCCAD GUI. As already discussed before, most maintenance activities with ISP or GSCC require to keep GSCC inactive. The Operator Mode will be activated by selecting one or more members and then pressing the “Operator”-button. After the maintenance was finished, the “Join” command reactivates the GSCC automatics again. While in Operator Mode there are also commands available to work with the ISP instances and their resources. This is covered in more detail later.
Selecting member and requesting Operator Mode
Operator Mode for first member team (TSM11) set
While in Operator GSCC provides certain commands to work with ISP instances, like in this case starting TSM.
Operator command example
To activate GSCC automatics again after maintenance, you need to select the members of the team to be joined and execute the “Join”-command.
Cluster “Join”
Action Method Log
Whenever Operator commands are executed, it is helpful to control the progress of the activities. This is provided by the “Action Logs” tab which collects all current information from all hosts monitored. Be aware that the messages are mixed in this view when performing several activities in parallel, but the output clearly contains the host and the member in each line.
Action Method Log