Eric J Ostrander's

ClearCase / ClearQuest / Git/Stash "how to" pages.

ClearQuest MultiSite, how do I ...
Mastership & Ownership   |   Misc   |   Packets & Orders   |   Replicas

This page is my own personal work. Anyone can use it for their own edification, but must realize that this material is not supported by me and the site is not affiliated with IBM Rational Software or Atlassian.
Mastership and ownership.
Back to the TOP
Transfer mastership. (chmaster)
Transfer record mastership.
Transfer user/group mastership.
Determine mastership properties of a replica. (describe)
Mastership of users records.
Transfer all masterships (retire a replica).
Misc.
Back to the TOP
Licenses.
Resolve naming conflicts.
Add a user not mastered locally to a group.
Configure a client machine to use multiutil. (cqreg installutil)
List the contents of replica's oplog. (dumpoplog)
Remove old oplog entries. (scruboplog)
Restore a replica from backup. (restorereplica)
Set up an alternate storage class.
Avoid ticket number collisions.
Determine a site's clan name.
Determine the site name.
Packets & Orders
Back to the TOP
List epoch numbers. (lsepoch)
List any packets that might be in a shipping bay. (lspacket)
Retransmit missing packets. (recoverpacket chepoch)
Replicas.
Back to the TOP
Create a database replica. (mkreplica)
Import a database replica. (mkreplica)
Change the properties of a replica. (chreplica)
Remove a replica. (rmreplica)
Synchronize replicas. (syncreplica)
Describe a replica. (describe)
Restore a replica from backup. (restorereplica)
List replicas. (lsreplica)
Determine ID block size and threshold. (lsreplica)
Programmatically determine the local replica's name.
Transfer mastership.
CQMS objects have mastership that can be transferred to another site. Under normal circumstances, only one site can master an object at a time. However, if a site forces a mastership change, in theory two sites can master the same object. However, forcing mastership changes should only be used when it's known that the other site is no longer part of the family.
Note that unless the original site's family name was changed, the family name is "<local>". In any record, see the "ratl_mastership" field value. 
  # mu chmaster -clan clanname -site sitename -family familyname -user admin-logon -password admin-password entity-selector
The entity-selector is something like user:login or record-type:record-id. The only objects that can have their mastership transferred are records, users, groups, and Workspace items (such as a query). Each of the selectors can optionally be specified using a sitename, as in user:login.site-name.
Options:
-cl.an   clan-name All replicas that use the same schema repository belong to the same clan. Default: first clan replicated at this site.
-site   site-name Default: current site.
-fam.ily   family-name All replicas of the same database belong to the same family. No default.
-u.ser   admin-logon The name of a user with appropriate database privileges for the object whose mastership is changing.
-p.assword   admin-password CQ password for -user.
-all   [ -l.ong ] Transfer (give up) mastership of all objects locally mastered. For verbose output, use -long.
-working.master Transfer the working schema repository to another site. Must use MASTR with -family.
-force   old-master-replica Transfer mastership of an object to the specified replica.
-forceall New in CQ 2003.06.00. Forcefully take mastership of everything mastered by obsolete-replica.

Back to the INDEX.
Transfer record mastership.
In CQMS, individual records are mastered at a single site. Mastership can only be transferred by the site that currently masters the record. One only needs to change the value of the ratl_mastership field. However, that field is not normally visible in a form and would need to be placed there if you want to allow users to change mastership of records. If a database has not been replicated, the ratl_mastership field value is simply <local>. If placed on the form, the field should probably be locked down to a specific group of users.
Because tickets created at one site often need to be worked on at a remote site, one can automate the mastership transfer. In this case, the ratl_mastership field does not need to be on the form. See "Automatically transfer record mastership".
Each site can have CQ Web enabled, but only allowing administrators at each to login. An administrator could log into CQ via the web to a remote site and assign their local site mastership. The admin would probably have to wait until the next scheduled sync to see the mastership change. However, I can imagine an Action hook that monitors whether the ratl_mastership field value has changed and do a syncreplica if necessary.
Mastership can forcefully be taken using the CLI chmaster command. To transfer mastership on all records, omit "record-type:record-id" and use the "-all" option. 
  # mu chmaster -clan clan -site site-name -family family-name -user admin-login -password admin-password -force old-replica-name record-type:record-id

Back to the INDEX.
Transfer user/group mastership.
WARNING: The following steps are not written correctly in the CQMS manual that shipped with CQ2001 nor CQ2001A.
In CQMS, individual users/groups are mastered at a single site. Mastership can be given up by a site to another site, or can be forcefully taken by a site.
1) Use the chmaster command to change the mastering site: 
  # mu chmaster -clan clanname -site sitename -family familyname -user admin-logon -password admin-password new-master-replica user:username
2) At the old mastering site, send an update packet to the new mastering site.
3) Once the packet is successfully imported at the new site, use the Designer's User Administration tool at the new site to upgrade the user databases.
4) At the new mastering site, send update packets to all the other sites in the family.

Under rare circumstances, mastership can be forcefully taken by a site. At the new mastering site, use the chmaster command in its force mode.

Back to the INDEX.
Mastership of users records.
Updated: 04/24/13
Version: 7.1.2.
Users can be created at any site, as long as there is a user at the site who has the User Administration privilege. A user can only be added to a group at the site that masters the group regardless of what the user mastership is. However, logins can only be pushed to user databases by the working master site. That is, regardless where and how an account gets created and mastered, the user database can only be upgraded by the working master site.

Back to the INDEX.
Transfer all masterships (retire a replica).
Updated: 05/04/16
Version: 7.1.2.14
To properly retire a replica, see https://www.ibm.com/support/knowledgecenter/SSSH5A_7.1.2/com.ibm.rational.clearquest.ms_admin.doc/topics/t_remove_replica_from_clan.htm
To forcefully take mastership of another replica, see https://www.ibm.com/support/knowledgecenter/SSSH5A_7.1.2/com.ibm.rational.clearquest.ms_admin.doc/topics/t_remove_inop_site_from_clan.htm

Back to the INDEX.
Licenses.
CQMS requires a separate license. Any use of a replicated schema repository or user database uses a license.

Back to the INDEX.
Resolve naming conflicts.
With appropriate privileges, users can create new users, groups and Workspace items (such as queries). However, users at separate sites can create objects with the same name. When this happens and CQMS detects a naming conflict upon import, it will add the name of the originating site to the ratl_keysite field, which is normally null in each stateless record type. To avoid these conflicts, simply designate one site to create new users, groups and Workspace items. Or, adopt a naming convention that prevents naming conflicts.
If a naming conflict occurs, rename one of the records, reset the ratl_keysite field to null for that record and synchronize.
Although you cannot see such changes taking place during synchronization, CQMS provides ways to detect changes to the unique key when accessing stateless records.
1) Create a query designed to detect naming conflicts and then manually rename them.
2) At the working schema site, create a field hook that detects naming conflicts and automatically renames them.
3) At the working schema site, add the ratl_keysite field to the form of each stateless record type.

Back to the INDEX.
Configure a client machine to use multiutil.
A machine is automatically configured to run multiutil commands when the "mu activate" (export) or "mu mkreplica -import" (import) command is run.
Also, a client machine utilizes a schema repository by running the CQ Maintenance Tool. However, if you need to run mu commands from a client machine on a replicated schema repository that is not the one to which the client is currently pointing, use the "cqreg adddbset" on UNIX or the "installutil adddbset" on Windows. They both live in the CQ install area. Replicated dbset names are always of the form: CQMS.clan-name.site-name. 
  # path\installutil adddbset dbset db-vendor server-hostname \\unc-path\database admin-login admin-password connect-options
The connect-options are only necessary for ORACLE, otherwise use "".

Back to the INDEX.
List the contents of replica's oplog.
Use dumpoplog to find out more about a replica's oplog entries. 
  # mu dumpoplog -family family -user admin -password password -at replica
Some useful options:
-l.ong View all columns of the oplog.
-s.hort Limit the output to just the database operation.
-from   oplog-entry-number Use with "-to" to specify a range of oplog entries.
-to   oplog-entry-number Use with "-from" to specify a range of oplog entries.
-since   date-time Limit the listing to those entries new since date-time.
-reverse Reverse the order of the output.

Back to the INDEX.
Remove old oplog entries.
Oplog entries are required until all update packets have been successfully imported at the remote sites. Also, in the future oplog entries can help a remote site recover from failures. For these reasons, it is not recommended that oplog entries be scrubbed. However, to reduce database sizes, oplog entries can be scrubbed periodically. 
  # mu scruboplog -family family -user admin -password password -before { number | date-time }
The dumpoplog command can be used to determine the "-before" number or date-time.

Back to the INDEX.
Set up an alternate storage class.
Unfortunately, as of CQ2001A and unlike CCMS, there is only one CQMS storage class allowed and it must be called "cq_default".

Back to the INDEX.
Avoid ticket number collisions.
Updated: 09/28/11
Version: 7.1.1
Because CQMS allows users at different sites to create tickets, ticket numbering collisions need to be avoided. To accomplish that, CQMS allocates a block of ticket numbers to a site when the initial replica is made. The master replica controls which sites have which blocks of numbers. The numbers are allocated in blocks of 4096 at a time. If a site gets close to using up its numbers, the master replica will assign a new block to that site.
See: https://www-304.ibm.com/support/docview.wss?uid=swg21131355

Back to the INDEX.
Determine a site's clan name.
Updated: 05/04/16
Version: 7.1.2.14
The clan name is the same as the schema repository name in the format: CQMS.schema-repo.site-name. On Windows, look at the ClearQuest Maintenance Tool connections. On Unix, run "cqreg show". The site name and clan names were designated when the schema repository was first activated for replication.

Back to the INDEX.
Determine the site name.
Updated: 05/04/16
Version: 7.1.2.14
The site name is at the end of the format: CQMS.clan.site-name. On Windows, look at the ClearQuest Maintenance Tool connections. On Unix, run "cqreg show". The site name and clan names were designated when the schema repository was first activated for replication.

Back to the INDEX.
List epoch numbers.
The epoch (oplog ID) is the number of operations performed at a site. Since each event is numbered, each site keeps track of which events have been sent and which events need to be sent (created since the last syncreplica). Each site always knows exactly at what epoch number it currently is and what epoch number it has rcvd from a remote replica. However, each site can only know that it has sent changes to a remote replica and not whether those changes were actually implemented. That is, a site's epoch table entries for other sites is only an estimate of what it thinks the other site knows about it.
In the following example, there are 3 sites in this CQMS family: replica-name siteA, siteB & siteC. The output of the lsepoch command looks like: 
  # mu lsepoch -family family-name -user admin -password admin

Oplog IDs for row "siteC" (@ siteC-server)
      siteB=221
      siteA=709
      siteC=653
Oplog IDs for row "siteA" (@ siteA-server)
      siteB=504
      siteA=950
      siteC=653
Oplog IDs for row "siteB" (@ siteB-server)
      siteB=504
      siteA=709
      siteC=653
The output of lsepoch is lines of ascii text. The rows can be mentally arranged in a visual format that better illustrates the information.
  updates originating at siteA updates originating at siteB updates originating at siteC
siteA's own status 950 504 653
siteA's estimate of siteB 709 504 653
siteA's estimate of siteC 950 221 653
In this example we assume the lsepoch was run from siteA. The first row in the table tells us that siteA knows it has generated 950 events, knows about 504 events at siteB and 653 at siteC. The second row tells us that siteA has yet to update siteB with epochs 710-950, assumes that siteB is up-to-date with itself and thinks that siteB is up-to-date with siteC. The third row tells us that siteA has already updated siteC with its events, thinks that siteB does not have all of siteC's events and assumes that siteC is up-to-date with itself.
NOTE: If a replica has been removed from the family, its name will still show up in the lsepoch as replica.deleted. This is by design for troubleshooting purposes and there are currently no options to filter them out.

Back to the INDEX.
List any packets that might be in a shipping bay.
If an automatic syncreplica is invoked each time a new packet arrives in a site's shipping bay, there should never be any packets to list. However, if one is doing manual syncreplicas at a site, the following command will list the full pathnames of any packets in the shipping bay. 
  # mu lspacket
Options:
-s.hort List only the pathname of the packet.
-l.ong Also list the oplog IDs (epoch #s) and creating replica's OID.
packet-name Limit the listing to the named packet. Must appear after any options.

Back to the INDEX.
Retransmit a missing packet.
When an exporting site sends a packet out, it assumes that the packet was successfully integrated at the remote site and updates its own epoch table. There can be many reasons why a packet did not reach its destination, but when the next update arrives you are likely to get "packet was not applied". In this case, the exporting site must retransmit the missing information.
The following command will "roll-back" the epoch table to the designated date-time. The importing admin needs to run dumpoplog to determine the last time there was a successful import. See the recoverpacket man page for acceptable date-time string formats. The recoverpacket command doesn't do a syncreplica. That must be done as a separate step or wait until the next automatic sync. 
  At the exporting end:
  # mu recoverpacket -c "Retransmitting data." -family family-name -user admin -password admin -since date-time replica-name
To manually choose the epoch numbers to which you should "roll-back", the admin at the importing end informs the exporting admin of the last epoch number rcvd. 
At the import end:
  # mu lsepoch -family family-name -user admin -password password
At the export end:
  # mu chepoch -family family-name -user admin -password password import-replica export-replica=epoch-given-by-import-admin
Unfortunately, there is no "-actual" option for chepoch like there is in CCMS.

Back to the INDEX.
Create a database replica.
As of CQ2001A, CQMS does not support SQL Anywhere or Microsoft Access database replication. The export and import sites must use the same database vendor.
Test databases cannot be replicated.
Must kill the CQ Integration Server (cqintsrv) prior to running the first "mkreplica -export". If this is not done, various messages will appear in CC stating that the session is no longer valid. This applies to both Windows and UNIX.
Rational recommends replicating user databases prior to subscribing users to it. If users are already subscribed, they will not be able to login at the remote site. This isn't true if a user is simply "Subscribe to all databases". In the case where users are subscribed to specific databases prior to them being replicated, an admin at the working schema repository site must open each of those user's Subscribe dialogs (User Administration) and then simply say ok. After each of the users is done, the admin must "Upgrade user DB". Once the resubscription is complete, syncreplica with the remote site(s).
All users must be logged out of the database to be replicated or data loss will result. The database will be locked during the mkreplica, which can take up to twice as long as a backup of the same database.
WARNING: While users/groups have inherent mastership regarding modification, users and groups can be created at any site. To avoid naming collisions, simply designate one site to be the only site that can create new users or adopt a naming convention that avoids conflicts.

First, a CQ shipping server storage class called cq_default (mandatory name) must be created in the MS Control Panel (Windows) or shipping.conf (UNIX). Use different directories than the CCMS bays.
Before a db set can actually be replicated, it must be activated. That is, it must be told that it is now part of CQMS. This is when you designate the clan and local site names. All databases using the same schema repository belong to the same clan. The site name is your location, such as phoenix or boston. All databases in the schema repository to be replicated must be upgraded to the same CQ version; v2001A or higher. After a database is activated, its logical name becomes CQMS.clan-name.site-name. This command only needs to run once for a given db set. 
  # mu activate -dbset schema-repo-name -user admin-logon -password admin-password -clan clan-name -site site-name -host hostname
The user must have Super User privileges. The hostname is the machine on which the local Rational Shipping Server resides. 

  # mu mkreplica -export -user admin-logon -password admin-password -family family-name -workdir path -fship remote-hostname:remote-sitename
The user database's schema repository database is automatically replicated as well. The user must have Super User privileges. The family name is the name of the local user database (5 characters). A working directory (workdir) must be specified, such as C:\temp\cq_workdir. It will create and then remove cq_workdir when its done. It cannot already exist. The -fship option tells to export immediately. Specify the remote shipping server's name (hostname). The remote site's name is designated at this time.
NOTE: If the -fship option was used and there were errors in contacting the remote site but no errors about the replication itself, the packet will not be deleted. When the transmission problem is resolved, simply execute "shipping_server -poll".
NOTE: The mkreplica –import command now validates that the replica creation packet was exported from a system running the same operating system code page. If the code pages of the exporter and importer do not match, the new replica is not created.
Other options:
-sh.ip Create the outgoing packet, but don't send it just yet. See shipping_server.
-scl.ass   storage-class Use an alternate storage class.
-pex.pire   date-time Specify a time at which the store-and-forward facility stops trying to send the packet. Default is 14 days. Never expire is 0. See the syncreplica man page for date-time syntax.
-not.ify   email-address The delivery-failure message is sent to this address.
-out   packet-name Use packet-name and place it in a directory of your choosing. Used for shipping by tape or email.
-max.size   size(k, m, or g) An upper limit to the packet size in kilo, mega or giga bytes. Subsequent packets are appended with sequential integers.

Back to the INDEX.
Import a database replica.
You will need to create at least two empty database instances to rcv the import packets; one for the new schema repository and one for each user database to be imported. As of CQ2001A, CQMS does not support SQL Anywhere or Microsoft Access database replication. Do not create CQ databases in those instances, the import will do that part for you. The vendor used for the import database must be the same as the export site. The "lspacket" will tell you what information is needed for the import, but you must know from the originating admin what vendor was used for each replicated database. The schema repository and first user database are in the same packet.
While synchronization packets are removed automitcally after successful import, mkreplica packets are not. They need to be removed manually after the database has been tested ok at the new site.
First, a CQ shipping server storage class called "cq_default" (mandatory name) must be created in the MS Control Panel (Windows) or shipping.conf (UNIX). Use different directories than the CCMS bays. If this is done after the first replication packet was sent from the originating site, the incoming packet will be in the CCMS incoming bay.
NOTE: The mkreplica –import command now validates that the replica creation packet was exported from a system running the same operating system code page. If the code pages of the exporter and importer do not match, the new replica is not created. 
  # mu lspacket -s
  # mu mkreplica -import -site site-name -repository physical-schemadb -vendor vendor-db schemadb-params -database physical-userdb -vendor vendor-db userdb-params packet-name
The site-name must be the same as the remote-sitename given during export. It is listed as "Recipients" in the lspacket output. The db-info is for the schema repository and user databases being imported:
DB2 Database Alias
Oracle SQL*Net Alias
SQL Server Physical Database Name
The vendor-db is one of: DB2, SQL_SERVER, ORACLE. The db-params are parameters needed to log into the vendor database. See the mkreplica man page.
NOTE: If importing a user database long after the schema repository and first user database were created, the new user database being imported must come from the same clan (schema repository) as the others.
WARNING! If the import fails for some reason, you must completely remove the vendor databases before reattempting the import. However, you only need to do that for the specific database that failed import, not all related databases. For example, if the schema repository imported correctly and the user database import failed, it's only necessary to delete the user database vendor instance and start that import again. If the schema repository failed import, in addition to removing the vendor database, the dbset name must be dropped: 
  # installutil dropdbset CQMS.clan-name.site-name
Back to the INDEX.
Change the properties of a replica.
As of CQ2001A, the only property of a replica that can be altered is the hostname on which it lives. The hostname where a replica lives is designated during replica creation. Thereafter, the replica is referred to by its dbset name: CQMS.clan-name.site-name. If necessary, a CQ database can moved to a different machine. If done, you will need to alert the other sites. To avoid lost packets, run "mu syncreplica" immediately after this command. That is, don't wait until the next scheduled automatic sync. 
  # mu chreplica -clan clan -site site -family family -user admin -password password -host new-hostname replica-name
Back to the INDEX.
Remove a replica.
The mastership of all objects mastered by the replica to be removed need to be turned over to another site before the replica can be removed. In the following steps, SiteA is where the replica is to be removed and SiteB is the new master of that replica's objects.

1) At SiteA, ensure all users have completed all work.
2) At SiteA, relenquish mastership of everything to SiteB. Or at SiteB, TAKE mastership of everything from SiteA (only if SiteA no longer exists). Run this command for all user databases and the MASTR database. If forcefully taking mastership, skip to step (5). 
At SiteA:
  # mu chmaster -family family -user admin -password password SiteB-replica -all
Or, at SiteB:
  # mu chmaster -family family -user admin -password password SiteB-replica -all -force SiteA-replica
3) At SiteA, sync the replica with SiteB. 
  # mu syncreplica -export -family family -user admin -password password -fship SiteB-replica
4) At SiteB, ensure the change in mastership has been successfully integrated. 
  # mu syncreplica -import -family family -user admin -password password -receive
Test the new masterhsip.
5) At SiteB, remove the logical name of the replica. 
  # mu rmreplica -family family -user admin -password password SiteA-replica
6) At SiteB, tell the remaining replicas that SiteA is now out of the loop. If SiteB is the last replica in the family, after step (5), there is no need to complete this step. You can sync with a space-separated list of all other replicas in the family all at once. 
  # mu syncreplica -export -family family -user admin -password password -fship other-replicas
7) At SiteA, remove the physical databases from the system.

Back to the INDEX.
Synchronize replicas.
From time to time, it may be necessary to manually sync replicas inbetween the times in which they would have been synced automatically. 
At the exporting site:
  # mu syncreplica -export -family family -user admin -password password -fship remote-replica
At the importing site:
  # mu lspacket -short
  # mu syncreplica -import -family family -user admin -password password packet-pathname
Other options:
-max.size   max-packet-size On export, limit packet sizes to max-packet-size(k, m, or g)
-max.size   max-packet-size   -limit   num-packets Same as above, except limit the total number of packets to num-packets.
-scl.ass   storage-class Export or import, specify the storage class. Available only for -ship and -fship.
-pex.pire   date-time On Export, designate how long store-and-forward should retry sending. Default is 14 days. Never expire is 0. See the syncreplica man page for date-time syntax.
-not.ify   email-address The deliver-failure message is sent to email-address.
-sh.ip On export, let store-and-forward create the packet in the outgoing bay but not deliver it.
-fsh.ip On export, let store-and-forward create the packet and send it immediately.
-tape   raw-tape-device On export, create the packet on the raw-tape-device.
-out   packet-file Designate your own packet-file for ftp or other transmission later.
Alternatively, at the import site, you can simply use the -receive option in lieu of the full-packet-pathname to process all packets destined for local replicas. If you encounter "packet was not applied", see recoverpacket or chepoch.

Back to the INDEX.
Describe a replica.
Describes mastership properties of a replica. There is no -long, -short, or -fmt option like the CC command. 
  # mu describe -family family -user admin -password password { -all | -local | replica-name }
Back to the INDEX.
Restore a replica from backup.
Run this command immediately following the retrieval of a replica from backup. This will instruct the other replicas in the family to transmit changes that you are missing. However, this assumes that the last synchronization to the remote site was more recent than the backup. 
  # mu restorereplica -family family -user admin -password password
The restorereplica command will look for update packets from all members of the family. If you only want to use a subset of the family members (perhaps the last update was only sent to one site), use the -replace option. Unlike other mu operations, that option requires the remote site names as it arguments, not the remote replica names.
The restorereplica command will lock the database until the -completed operation. 
  # mu restorereplica -family family -user admin -password password -completed
Back to the INDEX.
List replicas.
This command lists information about all replicas known to the current replica. You must be a CQ superuser to run this command. The command can be abbreviated "lsrep". 
  # mu lsreplica -family family -user admin -password password replica
If -clan nor -site are specified, the current clan and site are assumed. The schema repository's family name is MASTR.
Options:
-cl.an   clan-name Name of the replica's clan.
-site   site-name Name of the replica's site.
-l.ong List creation information and synchronization server.
-s.hort List the replica names only.
-fmt   format-string See the lsrpelica man page for formatting options.
-working.master Lists the working schema repository for the clan you specified.
-sib.lings For a user database, lists the family members of the current replica, but does not list the current replica itself. Same for schema repository.
-infa.mily   family Lists replicas of the specified replica when -family is MASTR.
Back to the INDEX.
Determine ID block size and threshold.
Updated: 09/28/11
Version: 7.1.1
The following commands can be used to get the id-block size, threshold and hostname. 
	multiutil lsreplica -clan TRAINING -site s1 -u admin -p "" -fam SAMPL –long
- or -
	multiutil idblockinfo -clan XYZ_CLAN -site XYZ_HQ -fam XYZ_DB -u admin "" -ser req
Back to the INDEX.
Programmatically determine the local replica's name.
Updated: 05/15/12
Version: 7.0.1.8 
	$replica_name = $session->GetLocalReplica->GetDisplayName;
Back to the INDEX.

ejostrander@cox.net
Return to the home page .

This page last modified: 05/04/2016