In 11g R2 RAC, Voting disk is stored on an ASM diskgroup. . But CSSD needs the voting files before ASM is online. At its startup, CSSD scans the device headers of all devices specified in the gpnp profile xml file, tag “DiscoveryString” which contains the same value as specified in asm_diskstring parameter for ASM instance.
If it can find at least more than half the number of total no. of voting files, the party takes place, otherwise, CSSD will cycle with appropriate error messages in $GRID_HOME/log/hostname/log/cssd/ocssd.log for each loop.
To verify it, I modified ASM_DISKSTRING to such a value that the ASM disk containing voting disk was not included and tried to restart crs on a node.. Looking up the CSSD logfile, I saw that the CSSD had trouble identifying its voting files.
Now, I had to change back the ASM disk_string parameter without having ASM running, and with no CSSD available, which is necessary to start it? How do we tell the CSSD, that’s running fairly in advance of ASM, to scan the right devices? I used gpnptool to edit the gpnp profile and restored the discovery string to the appropriate value. Now I could get crs running on my machine.
Overview:
– Move voting disk to a new diskgroup VOTE.
– Set ASM_DISKTRING to all the disks which are members of all the other diskgroups (DATA/FRA) using SQL
— stop and restart crs on the node –
– check that HAS services have started but rest of the services are not up
–Check the ocssd.log — Scans all the disks which are part of the discovery string but does not find voting disk
– Edit the gpnp profile to modify discovery string for asm
– Try to restart crs
– check that all the daemons have started and cluster servicves are up
Implementation:
– cretae a diskgroup with external redundancy to be used to store VD
– Move voting disk to the diskgroup vote – fails because
— diskgroup not mounted on all the nodes – mount it
— diskgroup compatibility < 11.2.0.0 – Modify it
[root@host01 ~]# crsctl replace votedisk +vote
Failed to create voting files on disk group vote.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.
– Mount vote diskgroup on all the nodes
— Modify diskgroup compatibility to 11.2.0.0
— Move voting disk to the diskgroup vote – succeeds
[root@host01 ~]# crsctl replace votedisk +vote
Successful addition of voting disk 443f1c60e16f4fa5bfbfeaae6b2f919d.
Successful deletion of voting disk 3d9da0d16baa4f10bf4e4b9b4aa688c6.
Successful deletion of voting disk 369621097c034f6dbf29a8cc97dc4bbc.
Successful deletion of voting disk 407b5d6588ea4f6fbf1503f7f6cc2951.
Successfully replaced voting disk group with +vote.
CRS-4266: Voting file(s) successfully replaced
– check that voting disk has been moved to vote diskgroup
[root@host01 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
– —– —————– ——— ———
1. ONLINE 443f1c60e16f4fa5bfbfeaae6b2f919d (ORCL:ASMDISK014) [VOTE]
Located 1 voting disk(s).
– Find out the disks in the other diskgroups – DATA and FRA
[grid@host01 ~]$ asmcmd lsdsk -G data
Path
ORCL:ASMDISK01
ORCL:ASMDISK010
ORCL:ASMDISK011
ORCL:ASMDISK012
[grid@host01 ~]$ asmcmd lsdsk -G fra
Path
ORCL:ASMDISK02
ORCL:ASMDISK03
– Set ASM_DISKTRING to all the disks which are members of DATA/FRA diskgroups
SQL> sho parameter asm_diskstring
NAME TYPE VALUE
———————————— ———– ————
asm_diskstring string
SQL> alter system set asm_diskstring='ORCL:ASMDISK01', 'ORCL:ASMDISK010','ORCL:ASMDISK011','ORCL:ASMDISK012', 'ORCL:ASMDISK02','ORCL:ASMDISK03';
– check that discovery string in gpnp profile points to the disks as specified
[root@host01 ~]# cd /u01/app/11.2.0/grid/gpnp/host01/profiles/peer/ vi profile.xml
DiscoveryString=”ORCL:ASMDISK01,ORCL:ASMDISK010,ORCL:ASMDISK011,ORCL:ASMDISK012,ORCL:ASMDISK02,ORCL:ASMDISK03“
— stop and restart crs on the node –
]
[root@host01 ~]# crsctl stop crs [root@host01 ~]# crsctl start crs
– check that has has started but rest of the services are not up
[root@host01 ~]# crsctl check has
CRS-4638: Oracle High Availability Services is online
[root@host01 ~]# crsctl check crs
]CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[root@host01 ~]# crsctl check css
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
[root@host01 ~]# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
–Check the ocssd.log
– Scans all the disks which are part of the discovery string but does not find voting disk
[root@host01 ~]# vi $ORACLE_HOME/log/host01/cssd/ocssd.log
2013-01-21 12:55:14.759: [ CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK01:0:0
2013-01-21 12:55:14.759: [ SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d0410 for disk :ORCL:ASMDISK01:
2013-01-21 12:55:14.759: [ CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK010:0:0
2013-01-21 12:55:14.759: [ SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d0c10 for disk :ORCL:ASMDISK010:
2013-01-21 12:55:14.759: [ CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK011:0:0
2013-01-21 12:55:14.759: [ SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d1508 for disk :ORCL:ASMDISK011:
2013-01-21 12:55:14.759: [ CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK012:0:0
2013-01-21 12:55:14.759: [ SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d1e00 for disk :ORCL:ASMDISK012:
2013-01-21 12:55:14.760: [ CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK02:0:0
2013-01-21 12:55:14.760: [ SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d26f8 for disk :ORCL:ASMDISK02:
2013-01-21 12:55:14.760: [ CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK03:0:0
@
2013-01-21 12:55:14.760: [ CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK03:0:0
2013-01-21 12:55:14.760: [ SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d2ff0 for disk :ORCL:ASMDISK03:
2013-01-21 12:55:14.760: [ CSSD][2985733008]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting
.
.
.
2013-01-21 12:55:14.760: [ CSSD][2985733008]clssnmvDiskVerify: Successful discovery of 0 disks
2013-01-21 12:55:14.760: [ CSSD][2985733008]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2013-01-21 12:55:14.760: [ CSSD][2985733008]clssnmvFindInitialConfigs: No voting files found
– Copy the existing gpnp profile to profile.bak and edit the copy to modify discovery string for asm
[root@host01 ~]# cd /u01/app/11.2.0/grid/gpnp/host01/profiles/peer/ [root@host01 peer]# cp profile.xml profile.bak
– remove the oracle signature from the file –
[root@host01 peer]# gpnptool unsign -p=profile.bak
Warning: some command line parameters were defaulted. Resulting command line:
/u01/app/11.2.0/grid/bin/gpnptool.bin unsign -p=profile.bak -o-
– change the DiscoveryString itself –
[root@host01 peer]# gpnptool edit -asm_dis='ORCL:*' -p=profile.bak -o=profile.bak -ovr
Resulting profile written to “profile.bak”.
Success.
– sign the profile xml file with the wallet (Notice: the path is only the directory to the wallet, NOT the wallet file itself) –
[root@host01 peer]# gpnptool sign -p=profile.bak -w=file:/u01/app/11.2.0/grid/gpnp/host01/wallets/peer/ -o=profile.new
Resulting profile written to “profile.new”.
Success.
– move the original profile.xml out of the way
[root@host01 peer]# mv profile.new profile.xml
– check that discovery string has been modified
[root@host01 peer]# vi profile.xml
DiscoveryString=”ORCL:*”
– Try to restart crs
[root@host01 peer]# ps -ef |grep d.bin
kill all the processes
[root@host01 peer]# crsctl start crs
– check that all the daemons have started and cluster servicves are up
[root@host01 peer]# ps -ef |grep d.bin [root@host01 peer]#crsctl stat res -t
Conclusion:
To access voting disk, gpnp profile is read to find the location of voting disk : found as asm
The asm disks as per the discovery string in gpnp profile are scanned to search for voting disk.
– Note : GPNP profile on other nodes also contains erroneous discovery string.
Hence copy the profile from the current node to the other nodes also
References:
————————————————————————————————–
Related links:
——————
~
Hi Anju,
I am facing a problem with my gpnptool utility.
I am able to fir commands such as gpnptool get or gpnptool find.
However I get below error while I fire gpnptool getpval.
[root@host01 ~]# /scratch/12.1.0/grid/bin/gpnptool getpval
Error: Can’t open profile ‘profile.xml’ for read: file not found
Please suggest.
Regards
Krunal
Hi Krunal
Pls issue command from folder/gpnp//profiles/peer/.
Regards
Anju Garg
Hi Anju,
If for some reason, asm_diskstring is not set in gpnp,without that will the CSSD start.? Because i have seen in some examples, the string is just a pair of quotes
Hi Karthik,
If asm_diskstring is not set i.e. it equals the null string, ASM discovery finds all disks in an operating system-specific location to which the Automatic Storage Management instance has read/write access.
Pls refer to : https://docs.oracle.com/cd/B19306_01/server.102/b14237/initparams011.htm#REFRN10248
Hope it answers your question.
Your comments and suggestions are always welcome.
Regards
Anju
Hi Anju,
Thank you very much for the explanation. I have one more question on this. I understand the mechanism to find voting disk is similar to find the disk containing spfile. Say there are 100 asm disks, css may probably stop searching for spfile if it finds a matching header in say 20th disk(remaining disks will not be scanned).But if it is voting disk, then css has to scan all the 100 asm disks because no.of voting disks may be more than one right? In that case cluster startup time increases (not desirable) – is that correct?
Hi KArthik
You are probably correct. I have not tried this scenario.
Regards
Anju Garg