#license files /etc/vx/elm /etc/vx/licenses/db /etc/vx/licenses/lic /etc/vx/licenses/reg /nbucache*.dat /nbucache*.lrg /nbucache.txt /usr/lib/elm /usr/lib/locale /usr/openv/netbackup/bin/admincmd/bpminlicense /usr/openv/var /usr/openv/var/license.txt /usr/share/lib/zoneinfo /var/adm/VRTSshrd/VRTSlic/db /var/adm/VRTSshrd/VRTSlic/lic /var/adm/VRTSshrd/VRTSlic/reg ================================================================================ http://esoftware.veritas.com/veritassearch/login.jsp - products User: amercustomer Password: 1Media4U http://esoftware.veritas.com/CD/MTV/Q16200F.vcs.4.1.00.0.sol.tar.gz http://seer.entsupport.symantec.com/docs #docs http://ftp.support.veritas.com/pub/support/products #product patches and documents http://ftp.support.veritas.com/pub/products/vmtools #products for Windows VxExplorer.zip ================================================================================ #Run the gather script /opt/VRTSspt/VRTSexplorer/VRTSexplorer #Run the runbook /opt/VRTSarb/makerb =========================================================================== #get the grab/support script #Unix ftp ftp.veritas.com ftp bin cd /pub/support IBM AIX: VRTSspt-aix.tar.Z HP-UX: VRTSspt-hpux.tar.Z Redhat or IBM Linux: VRTSspt-linux.tar.Z Sun Solaris: VRTSspt-sol.tar.Z cd /opt; zcat VRTSspt*tar.Z | tar xvf - more /opt/VRTSspt*/README.VRTSspt #Run, then push the output for the case to Veritas #assume you used defaults /opt/VRTSspt/VRTSexplorer/VRTSexplorer =========================================================================== #Windows http://ftp.support.veritas.com/pub/products/vmtools VxExplorer.zip output to: c:/SERVICE-NUMBER/HOSTNAME/MMDDYYYY__HH-MM-SS/CAB/ =========================================================================== #push the file #Unix cd /tmp #assuming you took defaults for output ftp ftp.veritas.com # trouble tickets, point patches #or windows ftp ftp.support.veritas.com or (same site, diff name) ftp.entsupport.symantec.com User: iosupport Pass: iS*pp8rT1 (eff 7/1/????, works on 8/6/2008) bin hash cd pub/support/incoming put explorer-$CASE_NUMBER.tar.gz =========================================================================== #notes for windows Please generate and send the output of the VxExplorer utility by following the below steps: 1. Download VxExplorer from ftp://ftp.veritas.com/pub/support/vmtools/ (See Note 1). 2. Extract the file onto a server 3. Execute VxExplorer.exe. The VxExplorer utility will open a window containing several fields that can be modified. 4. Add the support case ID into the Incident/Titan/Ref # field. The case ID should have been provided by a support representative (The Name and Comments fields are optional). 5. Unless otherwise instructed, accept all of the default options and click OK. VxExplorer will begin gathering logs from the system (See Note 2). Once VxExplorer has completed, a *.CAB file will be created using the name of the Support case ID. By default, the file is placed directly under the C: drive in a folder named after the support case ID (For example, if VxExplorer is run on a server called "Server," and the case number is "111222333", the resulting file and path will be C:\111222333\SERVER\09052007__10-50-23\Cab\111222333_SERVER_BIGFiles.CAB). To post the file to the incoming FTP site, use the following link: ftp://ftp.entsupport.symantec.com/pub/support/incoming/ User Name: iosupport PW: iS*pp8rT1 (eff.: 7/1) This is a "blind" FTP site. An "access denied" error will appear when opening the window. However, the option to paste a file should still be available (directories cannot be posted, however!) (See Note 3). *** Note1: It is critical that the latest version of VxExplorer is used, unless otherwise instructed. Outdated versions of VxExplorer will not gather all of the logs properly. In the event that File Transfer Protocol (FTP) access is not available, contact technical Support to request that the file be sent through email. *** Note2: In many cases, VxExplorer may appear to "hang" and may be listed in the Windows Task Manager as "Not Responding." This happens because VxExplorer executes at a low priority to avoid impacting the available resources of the server. In some cases, VxExplorer may take 30-45 minutes to complete. If VxExplorer hangs for an extremely long time (say, over an hour), it may be because it cannot gain access to one or more logs. Try rerunning VxExplorer with some of the options unchecked (in particular, the "OS Information," "Application Log" and the "System Log"). *** Note3: Please send an email to technical support with the names of any files that are uploaded to the FTP site, as well as the time and date of the error. Otherwise, technical support will not be aware that they are available. =========================================================================== #NBU ftp://ftp.support.veritas.com/pub/support/outgoing/nbsupport.200.windows.tar.gz ftp://ftp.support.veritas.com/pub/support/outgoing/nbsupport.200.solaris.tar.z ftp://ftp.support.veritas.com/pub/support/outgoing/nbsupport.200.hp.tar.z ================================================================================ man vxintro > 4.0 cd /opt/VRTS/install; ./installvm -configure vxsvc (start daemon) vea (run gui - vmsa replacement) www.sun.com/software/download/index.html devfsadm -sv (shows what devfsadm would do if it needs to do anything) cd /var/vx/isis tail vxisis.log tail command.log (/var/adm/vx/veacmdlog) Revert a VXVM disk back to AIX LVM control: chpv -C powerpath6 #add/list licenses >= 3.5 vxlicinst -k vxlicrep vxlictest #translates a key or product and lists what vxlicrep would vea (run gui - vmsa replacement vxva replacement) vxsvc (start daemon) # to deactivate OLD keys, go to /etc/vx/license and remove old keys # then vxlicinst -k NEWKEY < 3.5 vxlicense -c vxlicense -p vmsa Windows c:/Program Files/Common Files/VERITAS Shared/vrtslic/lic ------------------------------------------------------------------------------- Troubleshoot /var/adm/messages disabled path 32/0x160 belonging to dmpnode 239/0x20 0x20 == 32 decimal ls -l /dev/vcs/[r]dmp brw------ 1 root other 239, 32 xxxxxxxx c1t15d0s0 ------------------------------------------------------------------------------- #remove ALL disk info, if you want to totally redo VMs info umount all stuff deport all stuff vxdisk rm ALL_DISKS rm /dev/vx/*dmp/* rm /dev/vx/*dsk/* rm /etc/vx/*.info vxconfigd -k ------------------------------------------------------------------------------- #get HPUX to see new disks ioscan -fnC disk insf -C disk vxdctl enable ------------------------------------------------------------------------------- #dirty volume/plex vxvol -o bg -f start ------------------------------------------------------------------------------- #make a simple rootdg w/o encapsulating the root disks - <=3.5 This procedure was given by veritas and is tested. This Procedure can be run in multiuser mode and doesn't require any reboots and doesn't encapsulate the rootdisks. This procedure eases recovery in case of rootdisk failures and makes veritas upgrades a lot easier than with encapsulated rootdisk. root=c0t0d0s6 rootmir=c1t0d0s6 #this is required only for vxinstall rm /etc/vx/reconfig.d/state.d/install-db vxdctl init #if error about config daemon not being accessible, then vxconfigd vxdctl init vxdg init rootdg #disregard errors about "not currently in configuration" vxdctl add disk $root type=simple vxdctl add disk $rootmir type=simple vxdisk -f init $root type=simple vxdisk -f init $rootmir type=simple vxdg adddisk root=$root vxdg adddisk rootmir=$rootmir vxdctl enable ------------------------------------------------------------------------------- #eeprom /etc/vx/bin/vxeeprom #mirror the rootvol vxassist -g rootdg mirror rootvol rootmirror vxassist -g rootdg mirror swapvol rootmirror #recover diskgoups volumes (start them up) vxrecover -sb -g diskgroup #Remove a plex and remake it vxplex dis spool-02 vxedit -r rm spool-02 vxmake sd disk02-01 disk02,0,23023573 vxmake plex spool-02 sd=disk02-01 vxplex att spool spool-02 #create a VXFS filesystem with mount point vxdiskadm (add disks to diskgroup) vxassist -g coins_dg make coins-u-vol 250g layout=stripe,nomirror,nolog stw idth=128k usetype=fsgen nstripe=5 coins_dg01 coins_dg02 coins_dg03 coins_dg04 c oins_dg05 coins_dg06 coins_dg07 coins_dg08 coins_dg09 coins_dg10 /usr/lib/fs/vxfs/mkfs -F vxfs /dev/vx/rdsk/coins_dg/coins-u-vol #/usr/lib/fs/vxfs/mkfs -F vxfs /dev/vx/rdsk/coins_dg/coins-u-vol 52428800 #create a UFS filesystem with mount point vxprint -g bighaus_1 -ht | more mkfs -m /dev/vx/rdsk/bighaus_1/export_home /usr/lib/fs/ufs/mkfs -F ufs -M /export/home /dev/vx/rdsk/bighaus_1/export_home 129769472 #list free space - shows physical disk that is not used vxdg -g DISKGROUP free #show all disks and what group they belong to /usr/sbin/vxdisk -o alldgs list # change between enclosure base to OS native and back (on the fly) vxddladm set namingscheme=ebn #frame/enclosure EMCx|HDSx cbn vxddladm set namingscheme=osn #native OS name - ctd|hdisk|powerpath # cleanup disk udid_mismatch vxdisk updateudid c2t66d0 c2t67d0 vxdctl enable vxdisk -e list vxdisk scandisks vxdg -C -o updateid import $DG Windows vxtool [disk, kernel, verify, getioparams] -------------------------------------------------------------------------------- #fix a messed up install or system #Remove files involved with installation that were created when you loaded VxVM but are no longer needed using the following command: rm -rf /etc/vx/reconfig.d/state.d/install-db #Start some VERITAS Volume Manager I/O daemons using the following command: vxiod set 10 #Start the VERITAS Volume Manager configuration daemon, vxconfigd, in disabled mode using the following command: vxconfigd -m disable #Initialize the vxconfigd daemon using the following command: vxdctl init #Initialize the DMP subsystem using the following command: vxdctl initdmp #Enable vxconfigd using the following command: vxdctl enable #Restart vxconfigd vxconfigd -k -------------------------------------------------------------------------------- #list free space within a diskgroup vxassist -g DISKGROUP maxsize #how much you can grow a volume vxassist -g rootdg maxgrow mydata Volume mydata can be extended by 4096 to: 20840656 (10176Mb+208 sectors) #add 3g to volume and to FS (growby/growto) /etc/vx/bin/vxresize -g diskgroup -F fstype VOL +3g /etc/vx/bin/vxresize -g diskgroup -F vxfs mydata +5553600 OR Volume & Filesystem Growth Procudure: 1) You can use vxassist to estimate the max size of a given volume based on the disks you wish to add: vxassist -g rootdg maxgrow vol01 disk01 disk02 disk03 2) Next, actually grow the volume (NOT THE FS) via the command (assuming maxgrow outputed 10639360 as the maxsize): vxassist -g rootdg growto vol01 10639360 disk01 disk02 disk03 3) Now VxVM grinds away, monitor with vxtask. 4) Now Grow the Filesystem, for UFS use: #for VXFS use: /usr/lib/fs/vxfs/fsadm -b 10639360 -r /dev/vx/rdsk/rootdg/vol01 /mnt #for UFS use: /usr/lib/fs/ufs/mkfs -F ufs -M /export /dev/vx/rdsk/rootdg/vol01 10639360 Grow only the volume #now growby that size vxassist -g rootdg growby mydata 20840656 #now growto that size vxassist -g rootdg growto mydata 20840656 #add 3g to volume vxassist -g diskgroup resize VOL 3g # change volume to oracle:dba vxinfo -U gen -g diskgroup volume vxprint -l VOLUME vxedit -g DISKGROUP -v set user=oracle group=dba mode=0600 VOLUME # delete a volume and all associated plexes, and subdisks # first remove all reference in the diskgroup # then remove the disk from the disk group (if you desire - this is # not always what you need/want to do) vxedit -rf -g diskgroup rm volumename vxdg -g diskgroup rmdisk emc29 # delete a diskgroup and all associated volumes, plexes, and subdisks vxedit -rf diskgroup # convert ufs to vxfs /opt/VRTSvxfs/sbin/vxfsconvert Install vxfs package, run vxfsconvert (give it the proper arguments - see the man on it: man -M /opt/VRTS/man vxfsconvert) and it will make ufs into vxfs. Then enter the needed cron entry (see the solaris install guide) so it is defragged routinely. # convert a RAID layout vxassist convert VOLUME layout=stripe-mirror #fix a license issue in single user # vxvol -g rootdg startall vxvm:vxvol: ERROR: changing plex var-01: License has expired or is not available for operation vxvm:vxvol: ERROR: changing plex swapvol-01: License has expired or is not available for operation # vxdctl license init # vxvol -g rootdg startall #list VRTS versions pkginfo -l `pkginfo | grep VRTS | awk '{print $2}'` #list metadata about a disk /etc/vx/diag.d/vxdmpinq -d /dev/rdsk/c4t0d201s2 /usr/lib/vxvm/diag.d/vxprivutil scan $DEVICE /usr/lib/vxvm/diag.d/vxasldebug #=============================================================================== #Put vxconfigd in debug vxconfigd -k -x log -x 9 -x timestamp -x \ logfile=/var/tmp/vxconfigd.debug.log > /dev/null 2>&1 #Run the cable pull test. #Copy /var/tmp/syslog.log.cablepull. #Disable vxconfigd debug log: vxconfigd -k #Run the vxexplorer. /opt/VRTSspt/VRTSexplorer/explorer #=============================================================================== #run debug #ii.) Enable debug #To enable detail logging on the agent: haconf -makerw hatype -modify DiskGroup LogDbg -add 1 2 3 4 5 6 7 8 9 10 DBG_AGDEBUG DBG_AGINFO DBG_AGTRACE haconf -dump -makero #iii.) Try to online the service group through VCS and wait for the DiskGroup resource to fail. #iv. ) Disable debug #To remove detail logging on the agent: haconf -makerw hatype -modify DiskGroup LogDbg -delete -keys #[ this clears all the debug tags that are currently enabled ] haconf -dump -makero #iv.) Send us engine_A.log and DiskGroup_A.log #=============================================================================== #deport/import disk group vxdg deport dgroup vxdg import dgroup vxdg -C import dgroup vxdg -C -f import dgroup vxdiskadm vxdg -h mynode -n mynode-new deport mynode (renames mynode to mynode-new) vxdg -C import mynode-new (re-enables mynode-new, also had crow in one disk dgid...) vxdg -h mynode -n mynode deport mynode-new (renames mynode-new to mynode) vxdg -C import mynode (re-enables mynode) #rename a volume, plex, subdisk vxedit -g diskgroup oldname newname vxedit -g ids1 -v rename db13 d13 vxedit -g ids1 -p rename p1 p1new vxedit -g ids1 -s rename sd11 sd11new #volume needs to be resync'd (may need to start vol) vxvol start vol vxvol -g group resync vol #change from stale to active #vxmend -g rootdg fix clean var-01 #vxmend -g rootdg fix stale var-01 vxmend -g rootdg fix active var-01 vxplex att var var-01 or vxvol resync var #Fix a failing disk that is not really bad vxedit -g group set failing=false VM_DISK #clean from tutil0 from ATT1 vxmend -g rootdg clear all mydata #files to watch /etc/path_to_inst /etc/name_to_major /etc/name_to_sysnum #ssaadm display c4 ^controller number SPARCstorage Array 100 Configuration (ssaadm version: 1.15 96/03/17) Controller path:/devices/io-unit@f,e2200000/sbi@0,0/SUNW,soc@2,0/SUNW,pln@a0000000,78c2d3:ctlr DEVICE STATUS TRAY 1 TRAY 2 TRAY 3 slot 1 Drive: 0,0 (FW) Drive: 2,0 (FW) Drive: 4,0 (FW) 2 Drive: 0,1 (FW) Drive: 2,1 (FW) Drive: 4,1 (FW) 3 Drive: 0,2 (FW) Drive: 2,2 (FW) Drive: 4,2 (FW) 4 Drive: 0,3 (FW) Drive: 2,3 (FW) Drive: 4,3 (FW) 5 Drive: 0,4 (FW) Drive: 2,4 (FW) Drive: 4,4 (FW) 6 Drive: 1,0 (FW) Drive: 3,0 (FW) Drive: 5,0 (FW) 7 Drive: 1,1 (FW) Drive: 3,1 (FW) Drive: 5,1 (FW) 8 Drive: 1,2 (FW) Drive: 3,2 (FW) Drive: 5,2 (FW) 9 Drive: 1,3 (FW) Drive: 3,3 (FW) Drive: 5,3 (FW) 10 Drive: 1,4 (FW) Drive: 3,4 (FW) Drive: 5,4 (FW) CONTROLLER STATUS Vendor: SUN Product ID: SSA100 Product Rev: 1.0 Firmware Rev: 3.12 Serial Num: 00000078C2D3 Accumulate Performance Statistics: Enabled In /var/adm/messages: Apr 17 11:19:56 tiger0 unix: WARNING: /io-unit@f,e3200000/sbi@0,0/SUNW,soc@2,0/S UNW,pln@a0000000,777d82/ssd@4,0 (ssd352): ^^^^ world wide number ^,^ target,unit Apr 17 11:19:56 tiger0 unix: disk not responding to selection vxtask list vxtask monitor task_nu #monitor tasks #save config file for DR rebuild vxdisk list vxdisk -e list vxdg list vxprint -g DISKGROUP -Qqvpshm vxprint -g DISKGROUP -Qqvpshmr (3.X versions) #mirror rootdisk vxmirror -V -g rootdg rootdisk rootmirror #get rootdisk config for DR #pre 3.x /usr/sbin/vxprint -ht -e '("rootdisk" in v_plex.pl_sd.sd_disk)||("rootdisk" in pl_sd.sd_disk)||("rootdisk" in sd_disk) #3.x: added 'r' /usr/sbin/vxprint -hrt -e '("rootdisk" in v_plex.pl_sd.sd_disk)||("rootdisk" in pl_sd.sd_disk)||("rootdisk" in sd_disk) The "database file not found" message refers to /var/vxvm/tempdb/rootdg. Any number of things could cause this file to be missing or unreadable. If the /var/vxvm/tempdb directory exists, you should be able to boot into single user mode, and run a "vxconfigd -kdx cleartempdir" to rebuild the temp database. ================================================================================ How to Determine Whether Node is Master or Slave for CFS On one node, determine whether it is the master or the slave node by using the command: vxdctl -c mode In the output, look for: cluster active - MASTER or: cluster active - SLAVE ================================================================================ Deporting and Importing Shared Disk Groups Shared disk groups used in a VERITAS Storage Foundation for Oracle RAC conguration are congured for Autoimport at the time of CVM startup. If the user manually deports the shared disk group on the CVM master, the disk group is deported on all nodes. To reimport the disk group, the user must import the disk group as shared from the CVM master. To deport a shared disk group, on the CVM master use the command: vxdg deport shared_disk_group To import a shared disk group, on the CVM master use the command: vxdg -s import shared_disk_group To import a disk group as stand-alone, deport it from the CVM master and, then, on any node use the command: vxdg -C import shared_disk_group To reimport a disk group as shared, deport from the stand-alone node and then, on the CVM master node, use the command: vxdg -C -s import shared_disk_group =============================================================================== Disable a DMP feature called Idle LUN probing - do this on NodeA and NodeB. vxdmpadm settune dmp_probe_idle_lun=off Check by vxdmpadm gettune all =============================================================================== All PREP Utility functionality is launched from a single url (https://sfprep.symantec.com). From there, the customer can decide to produce two different types of output. A report that provides specific details about what is missing from a server prior to performing an install or upgrade. It is based on collected information from the users environment (using the on-server collection tool) and automatically compares it to the latest HCL. 2. A Checklist that provides general information about what is needed for an install or upgrade. It is based on a set of responses to 6 on-line questions. The output consists of a checklist of corresponding HCL parameters. Both the Report and the Checklist contain clickable links to specific OS patches, Symantec and third party documentation, Symantec technical tips, and Support related Late Breaking News. http://sfprep.symantec.com/main.php on windows: use the "Configuration Checker" in the install dir. =============================================================================== http://seer.entsupport.symantec.com/docs/269233.htm Exact Error Message VxVM vxdg ERROR V-5-1-10127 associating disk-media with : Serial Split Brain detected. Run vxsplitlines Details: Background: The Serial Split Brain condition arises because VERITAS Volume Manager (tm) increments the serial ID in the disk media record of each imported disk in all the disk group configurations on those disks. A new serial (SSB) ID has been included as part of the new disk group version=110 in Volume Manager 4 to assist with recovery of the disk group from this condition. The value that is stored in the configuration database represents the serial ID that the disk group expects a disk to have. The serial ID that is stored in a disk's private region is considered to be its actual value. If some disks went missing from the disk group (due to physical disconnection or power failure) and those disks were imported by another host, the serial IDs for the disks in their copies of the configuration database, and also in each disk's private region, are updated separately on that host. When the disks are subsequently reimported into the original shared disk group, the actual serial IDs on the disks do not agree with the expected values from the configuration copies on other disks in the disk group. The disk group cannot be reimported because the databases do not agree on the actual and expected serial IDs. You must choose which configuration database to use. This is a true serial split brain condition, which Volume Manager cannot correct automatically. In this case, the disk group import fails, and the vxdg utility outputs error messages similar to the following before exiting: VxVM vxconfigd NOTICE V-5-0-33 Split Brain. da id is 0.1, while dm id is 0.0 for DM VxVM vxdg ERROR V-5-1-587 Disk group : import failed: Serial Split Brain detected. Run vxsplitlines The import does not succeed even if you specify the -f flag to vxdg. Although it is usually possible to resolve this conflict by choosing the version of the configuration database with the highest valued configuration ID (shown as config_tid in the output from the vxprivutil dumpconfig ), this may not be the correct thing to do in all circumstances. To resolve conflicting configuration information, you must decide which disk contains the correct version of the disk group configuration database. To assist you in doing this, you can run the vxsplitlines command to show the actual serial ID on each disk in the disk group and the serial ID that was expected from the configuration database. For each disk, the command also shows the vxdg command that you must run to select the configuration database copy on that disk as being the definitive copy to use for importing the disk group. The following example shows the result of JBOD losing access to one of the four disks in the disk group: # vxdisk -o alldgs list DEVICE TYPE DISK GROUP STATUS c2t1d0s2 auto:cdsdisk - (dgD280silo1) online c2t2d0s2 auto:cdsdisk d2 dgD280silo1 online c2t3d0s2 auto:cdsdisk d3 dgD280silo1 online c2t9d0s2 auto:cdsdisk d4 dgD280silo1 online - - d1 dgD280silo1 failed was:c2t1d0s2 # vxreattach -c c2t1d0s2 dgD280silo1 d1 # vxreattach -br c2t1d0s2 VxVM vxdg ERROR V-5-1-10127 associating disk-media d1 with c2t1d0s2: Serial Split Brain detected. Run vxsplitlines # /usr/lib/vxvm/bin/vxsplitlines -g dgD280silo1 VxVM vxsplitlines NOTICE V-5-2-2708 There are 1 pools. The Following are the disks in each pool. Each disk in the same pool has config copies that are similar. VxVM vxsplitlines INFO V-5-2-2707 Pool 0. c2t1d0s2 d1 To see the configuration copy from this disk issue /etc/vx/diag.d/vxprivutil dumpconfig /dev/vx/dmp/c2t1d0s2 To import the diskgroup with config copy from this disk use the following command; /usr/sbin/vxdg -o selectcp=1092974296.21.gopal import dgD280silo1 The following are the disks whose serial split brain (SSB) IDs don't match in this configuration copy: d2 At this stage, you need to gain confidence prior to running the recommended command by generating the following outputs : In this example, the disk group split so that one disk (d1) appears to be on one side of the split. You can specify the -c option to vxsplitlines to print detailed information about each of the disk IDs from the configuration copy on a disk specified by its disk access name: # vxsplitlines -g dgD280silo1 -c c2t3d0s2 VxVM vxsplitlines INFO V-5-2-2701 DANAME(DMNAME) || Actual SSB || Expected SSB VxVM vxsplitlines INFO V-5-2-2700 c2t1d0s2( d1 ) || 0.0 || 0.0 ssb ids match VxVM vxsplitlines INFO V-5-2-2700 c2t2d0s2( d2 ) || 0.1 || 0.0 ssb ids don't match VxVM vxsplitlines INFO V-5-2-2700 c2t3d0s2( d3 ) || 0.1 || 0.0 ssb ids don't match VxVM vxsplitlines INFO V-5-2-2700 c2t9d0s2( d4 ) || 0.1 || 0.0 ssb ids don't match VxVM vxsplitlines INFO V-5-2-2706 This output can be verified by using vxdisk list on each disk. A summary is shown below: # vxdisk list c2t1d0s2 # vxdisk list c2t3d0s2 Device: c2t1d0s2 Device: c2t3d0s2 disk: name= id=1092974296.21.gopal disk: name=d3 id=1092974311.23.gopal group: name=dgD280silo1 id=1095738111.20.gopal group: name=dgD280silo1 id=1095738111.20.gopal ssb: actual_seqno=0.0 ssb: actual_seqno=0.1 # vxdisk list c2t2d0s2 # vxdisk list c2t9d0s2 Device: c2t2d0s2 Device: c2t9d0s2 disk: name=d2 id=1092974302.22.gopal disk: name=d4 id=1092974318.24.gopal group: name=dgD280silo1 id=1095738111.20.gopal group: name=dgD280silo1 id=1095738111.20.gopal ssb: actual_seqno=0.1 ssb: actual_seqno=0.1 Note that though some disks SSB IDs might match that does not necessarily mean that those disks' config copies have all the changes. From some other configuration copies, those disks' SSB IDs might not match. To see the configuration from this disk, run /etc/vx/diag.d/vxprivutil dumpconfig /dev/rdsk/c2t3d0s2 > dumpconfig_c2t3d0s2 If the other disks in the disk group were not imported on another host, Volume Manager resolves the conflicting values of the serial IDs by using the version of the configuration database from the disk with the greatest value for the updated ID (shown as update_tid in the output from /etc/vx/diag.d/vxprivutil dumpconfig /dev/rdsk/). In this example , looking through the dumpconfig, there are the following update_tid and ssbid values: dumpconfig c2t3d0s2 dumpconfig c2t9d0s2 config:tid=0.1058 Config:tid=0.1059 dm d1 dm d1 update_tid=0.1038 Update_tid=0.1059 ssbid=0.0 ssbid=0.0 dm d2 dm d2 update_tid=0.1038 Update_tid=0.1038 ssbid=0.0 ssbid=0.0 dm d3 dm d3 update_tid=0.1053 Update_tid=0.1053 ssbid=0.0 ssbid=0.0 dm d4 dm d4 update_tid=0.1053 Update_tid=0.1059 ssbid=0.0 ssbid=0.1 Using the output from the dumpconfig for each disk determines which config output to use by running the command: # cat dumpconfig_c2t3d0s2 | vxprint -D - -ht Before deciding on which option to use for import, ensure the disk group is currently in a valid deport state: # vxdisk -o alldgs list DEVICE TYPE DISK GROUP STATUS c2t1d0s2 auto:cdsdisk - (dgD280silo1) online c2t2d0s2 auto:cdsdisk - (dgD280silo1) online c2t3d0s2 auto:cdsdisk - (dgD280silo1) online c2t9d0s2 auto:cdsdisk - (dgD280silo1) online At this stage, your knowledge of how the serial split brain condition came about may be a little clearer and you should have chosen a configuration from one disk to be used to import the disk group. In this example, the following command imports the disk group using the configuration copy from d2: # /usr/sbin/vxdg -o selectcp=1092974302.22.gopal import dgD280silo1 Once the disk group has been imported, Volume Manager resets the serial IDs to 0 for the imported disks. The actual and expected serial IDs for any disks in the disk group that are not imported at this time remain unchanged. # vxprint -htg dgD280silo1 dg dgD280silo1 default default 26000 1095738111.20.gopal dm d1 c2t1d0s2 auto 2048 35838448 - dm d2 c2t2d0s2 auto 2048 35838448 - dm d3 c2t3d0s2 auto 2048 35838448 - dm d4 c2t9d0s2 auto 2048 35838448 - v SNAP-vol_db2silo1.1 - DISABLED ACTIVE 1024000 SELECT - fsgen pl SNAP-vol_db2silo1.1-01 SNAP-vol_db2silo1.1 DISABLED ACTIVE 1024000 STRIPE 2/1024 RW sd d3-01 SNAP-vol_db2silo1.1-01 d3 0 512000 0/0 c2t3d0 ENA sd d4-01 SNAP-vol_db2silo1.1-01 d4 0 512000 1/0 c2t9d0 ENA dc SNAP-vol_db2silo1.1_dco SNAP-vol_db2silo1.1 SNAP-vol_db2silo1.1_dcl v SNAP-vol_db2silo1.1_dcl - DISABLED ACTIVE 544 SELECT - gen pl SNAP-vol_db2silo1.1_dcl-01 SNAP-vol_db2silo1.1_dcl DISABLED ACTIVE 544 CONCAT - RW sd d3-02 SNAP-vol_db2silo1.1_dcl-01 d3 512000 544 0 c2t3d0 ENA v orgvol - DISABLED ACTIVE 1024000 SELECT - fsgen pl orgvol-01 orgvol DISABLED ACTIVE 1024000 STRIPE 2/128 RW sd d1-01 orgvol-01 d1 0 512000 0/0 c2t1d0 ENA sd d2-01 orgvol-01 d2 0 512000 1/0 c2t2d0 ENA # vxrecover -g dgD280silo1 -sb # mount -F vxfs /dev/vx/dsk/dgD280silo1/orgvol /orgvol UX:vxfs mount: ERROR: V-3-21268: /dev/vx/dsk/dgD280silo1/orgvol is corrupted. needs checking # fsck -F vxfs /dev/vx/rdsk/dgD280silo1/orgvol log replay in progress replay complete - marking super-block as CLEAN # mount -F vxfs /dev/vx/dsk/dgD280silo1/orgvol /orgvol # df /orgvol /orgvol (/dev/vx/dsk/dgD280silo1/orgvol): 1019102 blocks 127386 files # vxdisk -o alldgs list DEVICE TYPE DISK GROUP STATUS c2t1d0s2 auto:cdsdisk d1 dgD280silo1 online c2t2d0s2 auto:cdsdisk d2 dgD280silo1 online c2t3d0s2 auto:cdsdisk d3 dgD280silo1 online c2t9d0s2 auto:cdsdisk d4 dgD280silo1 online # vxprint -htg dgD280silo1 dg dgD280silo1 default default 26000 1095738111.20.gopal dm d1 c2t1d0s2 auto 2048 35838448 - dm d2 c2t2d0s2 auto 2048 35838448 - dm d3 c2t3d0s2 auto 2048 35838448 - dm d4 c2t9d0s2 auto 2048 35838448 - v SNAP-vol_db2silo1.1 - ENABLED ACTIVE 1024000 SELECT SNAP-vol_db2silo1.1-01 fsgen pl SNAP-vol_db2silo1.1-01 SNAP-vol_db2silo1.1 ENABLED ACTIVE 1024000 STRIPE 2/1024 RW sd d3-01 SNAP-vol_db2silo1.1-01 d3 0 512000 0/0 c2t3d0 ENA sd d4-01 SNAP-vol_db2silo1.1-01 d4 0 512000 1/0 c2t9d0 ENA dc SNAP-vol_db2silo1.1_dco SNAP-vol_db2silo1.1 SNAP-vol_db2silo1.1_dcl v SNAP-vol_db2silo1.1_dcl - ENABLED ACTIVE 544 SELECT - gen pl SNAP-vol_db2silo1.1_dcl-01 SNAP-vol_db2silo1.1_dcl ENABLED ACTIVE 544 CONCAT - RW sd d3-02 SNAP-vol_db2silo1.1_dcl-01 d3 512000 544 0 c2t3d0 ENA v orgvol - ENABLED ACTIVE 1024000 SELECT orgvol-01 fsgen pl orgvol-01 orgvol ENABLED ACTIVE 1024000 STRIPE 2/128 RW sd d1-01 orgvol-01 d1 0 512000 0/0 c2t1d0 ENA sd d2-01 orgvol-01 d2 0 512000 1/0 c2t2d0 ENA =============================================================================== backup Veritas VM info -what disks have config copy on them? vxdisk list $DG | egrep "config disk.*clean online" -config backups #DG backups /etc/vx/cbr/bk/$DG.$DGID.$HOSTNAME$DG/$DGID.$HOSTNAME.dginfo.$NUM #disk backups /etc/vx/cbr/bk/$DG.$DGID.$HOSTNAME$DG/$DG.$DGID.$HOSTNAME.diskinfo.$NUM #binary config backup /etc/vx/cbr/bk/$DG.$DGID.$HOSTNAME$DG/$DG.$DGID.$HOSTNAME.binconfig.$NUM #vxprint -m save /etc/vx/cbr/bk/$DG.$DGID.$HOSTNAME$DG/$DG.$DGID.$HOSTNAME.cfgrec.$NUM -make a backup /usr/lib/vxvm/bin/vxconfigbackup -l /etc/dgbackups -restore a backup /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups $DG -commit the restore /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups -c $DG #ON WINDOWS VxCBR -a -pc:/ backup #makes a dir "cbr" under c:/ VxCBR -a -pc:/temp backup #makes a backup under c:/temp -get emailed on errors either alias the root user or vi /etc/init.d/S95vxvm-recover add: "vxrelocd root storageproblems &" ^^^^^^^^^^^^^^^ =============================================================================== Diskgroup splitbrain with GCO - figure out by running: vxsplitlines -g $DG =============================================================================== =============================================================================== =============================================================================== ===============================================================================