Beginning with the version 11g Release 2, the ASM spfile is stored automatically in the first disk group created during Grid Infrastructure installation.
Since voting disk/OCR are stored on ASM, ASM needs to be started on the node. To startup ASM, its SPfile is needed. But SPFILE is again located on ASM diskgroup only. How does clusterware resolve this issue?
- When a node of an Oracle Clusterware cluster restarts, OHASD is started by platform-specific means. OHASD accesses OLR (Oracle Local Registry) stored on the local file system to get the data needed to complete OHASD initialization
- OHASD brings up GPNPD and CSSD. CSSD accesses the GPNP Profile stored on the local file system which contains the following vital bootstrap data;
a. ASM_DISKSTRING parameter (if specified) to locate the disks on which ASM disks are configured.
b. ASM SPFILE location : Name of the diskgroup containing ASM spfile
a. ASM_DISKSTRING parameter (if specified) to locate the disks on which ASM disks are configured.
b. ASM SPFILE location : Name of the diskgroup containing ASM spfile
c. Location of Voting Files : ASM
– CSSD scans the headers of all ASM disks ( as indicated in ASM_DISKSTRING in GPnP profile) to identify the disk containing the voting file. Using the pointers in ASM disk headers, the Voting Files locations on ASM Disks are accessed by CSSD and CSSD is able to complete initialization and start or join an existing cluster.
– To read the ASM spfile during the ASM instance startup, it is not necessary to open the disk group. All information necessary to access the data is stored in the device’s header. OHASD reads the header of the ASM disk containing ASM SPfile (as read from GPnP profile) and using the pointers in disk header, contents of ASM spfile are read. Thereafter, ASM instance is started.
– With an ASM instance operating and its Diskgroups mounted, access to Clusterware’s OCR is available to CRSD.
– OHASD starts CRSD with access to the OCR in an ASM Diskgroup.
– Clusterware completes initialization and brings up other services under its control.
Demonstration :
In my environment, the ASM disk group DATA created with EXTERNAL redundancy is used exclusively for ASM spfile, voting and OCR files:
- Let us read gpnp profile to find out the location of ASM SPfile
[grid@host01 peer]$ cd /u01/app/11.2.0/grid/gpnp/host01/profiles/peer
gpnptool getpval -asm_spf
Warning: some command line parameters were defaulted. Resulting command line:
/u01/app/11.2.0/grid/bin/gpnptool.bin getpval -asm_spf -p=profile.xml -o-
+DATA/cluster01/asmparameterfile/registry.253.793721441
– Let us find out the disks in DATA diskgroup
[grid@host01 peer]$ asmcmd lsdsk -G DATA
Path
ORCL:ASMDISK01
ORCL:ASMDISK010
ORCL:ASMDISK02
ORCL:ASMDISK03
ORCL:ASMDISK04
ORCL:ASMDISK09
– Let us find out which ASM disk maps to which partition
Note down major/minor device numbers of the disks in DATA diskgroup
[root@host01 ~]# ls -lr /dev/oracleasm/disks/*
brw-rw—- 1 grid asmadmin 8, 26 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK09
brw-rw—- 1 grid asmadmin 8, 25 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK08
brw-rw—- 1 grid asmadmin 8, 24 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK07
brw-rw—- 1 grid asmadmin 8, 23 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK06
brw-rw—- 1 grid asmadmin 8, 22 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK05
brw-rw—- 1 grid asmadmin 8, 21 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK04
brw-rw—- 1 grid asmadmin 8, 19 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK03
brw-rw—- 1 grid asmadmin 8, 18 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK02
brw-rw—- 1 grid asmadmin 8, 31 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK014
brw-rw—- 1 grid asmadmin 8, 30 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK013
brw-rw—- 1 grid asmadmin 8, 29 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK012
brw-rw—- 1 grid asmadmin 8, 28 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK011
brw-rw—- 1 grid asmadmin 8, 27 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK010
brw-rw—- 1 grid asmadmin 8, 17 Nov 8 09:35 /dev/oracleasm/disks/ASMDISK01
- Let us find out the major/minor device numbers of various disk partitions
[root@host01 ~]# ls -lr /dev/sdb*
brw-r—– 1 root disk 8, 25 Nov 8 09:35 /dev/sdb9
brw-r—– 1 root disk 8, 24 Nov 8 09:35 /dev/sdb8
brw-r—– 1 root disk 8, 23 Nov 8 09:35 /dev/sdb7
brw-r—– 1 root disk 8, 22 Nov 8 09:35 /dev/sdb6
brw-r—– 1 root disk 8, 21 Nov 8 09:35 /dev/sdb5
brw-r—– 1 root disk 8, 20 Nov 8 09:35 /dev/sdb4
brw-r—– 1 root disk 8, 19 Nov 8 09:35 /dev/sdb3
brw-r—– 1 root disk 8, 18 Nov 8 09:35 /dev/sdb2
brw-r—– 1 root disk 8, 31 Nov 8 09:35 /dev/sdb15
brw-r—– 1 root disk 8, 30 Nov 8 09:35 /dev/sdb14
brw-r—– 1 root disk 8, 29 Nov 8 09:35 /dev/sdb13
brw-r—– 1 root disk 8, 28 Nov 8 09:35 /dev/sdb12
brw-r—– 1 root disk 8, 27 Nov 8 09:35 /dev/sdb11
brw-r—– 1 root disk 8, 26 Nov 8 09:35 /dev/sdb10
brw-r—– 1 root disk 8, 17 Nov 8 09:35 /dev/sdb1
brw-r—– 1 root disk 8, 16 Nov 8 09:35 /dev/sdb
– Now we can find out the partitions mapping to various ASM disks by matching their
major/minor device numbers
ASMDISK01 8,17 /dev/sdb1
ASMDISK02 8,18 /dev/sdb2
ASMDISK03 8,19 /dev/sdb3
ASMDISK04 8,21 /dev/sdb5
ASMDISK09 8,26 /dev/sdb10
ASMDISK10 8,27 /dev/sdb11
– Let’s scan the headers of those devices:
[root@host01 ~]# kfed read /dev/sdb1 | grep -E ‘spf|ausize’
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.spfile: 0 ; 0x0f4: 0x00000000
kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000
[root@host01 ~]# kfed read /dev/sdb2 | grep -E ‘spf|ausize’
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.spfile: 0 ; 0x0f4: 0x00000000
kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000
[root@host01 ~]# kfed read /dev/sdb3 | grep -E ‘spf|ausize’
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.spfile: 16 ; 0x0f4: 0x00000010
kfdhdb.spfflg: 1 ; 0x0f8: 0x00000001
[root@host01 ~]# kfed read /dev/sdb5 | grep -E ‘spf|ausize’
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.spfile: 0 ; 0x0f4: 0x00000000
kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000
[root@host01 ~]# kfed read /dev/sdb10 | grep -E ‘spf|ausize’
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.spfile: 0 ; 0x0f4: 0x00000000
kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000
[root@host01 ~]# kfed read /dev/sdb11 | grep -E ‘spf|ausize’
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.spfile: 0 ; 0x0f4: 0x00000000
kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000
In the output above, we see that
the device /dev/sdb3 contains a copy of the ASM spfile (spfflg=1).
The ASM spfile location starts at the disk offset of 16 (spfile=16)
Considering the allocation unit size (kfdhdb.ausize = 1M), let’s dump the ASM spfile from the device:
[root@host01 ~]# dd if=/dev/sdb3 of=spfileASM_Copy2.ora skip=16 bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.170611 seconds, 6.1 MB/s
[root@host01 ~]# strings spfileASM_Copy2.ora
+ASM1.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value
+ASM2.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value
+ASM3.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value
+ASM3.asm_diskgroups=’FRA’#Manual Mount
+ASM2.asm_diskgroups=’FRA’#Manual Mount
+ASM1.asm_diskgroups=’FRA’#Manual Mount
*.asm_power_limit=1
*.diagnostic_dest=’/u01/app/grid’
*.instance_type=’asm’
*.large_pool_size=12M
*.remote_login_passwordfile=’EXCLUSIVE’
a:O/
The same technique is used to access the Clusterware voting files which are also stored in an ASM disk group. In this case, Clusterware does not need a running ASM instance to access the cluster voting files:
Let’s check the location of voting disk :
[grid@host01 peer]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
– —– —————– ——— ———
1. ONLINE 243ec3b2a3cf4fbbbfed6f20a1ef4319 (ORCL:ASMDISK01) [DATA]
Located 1 voting disk(s).
– Since above query shows that voting disk is stored on ASMDISK01 which maps to /dev/sdb1,
we will scan the header of /dev/sdb1
[root@host01 ~]# kfed read /dev/sdb1 | grep vf
kfdhdb.vfstart: 96 ; 0x0ec: 0x00000060
kfdhdb.vfend: 128 ; 0x0f0: 0x00000080
Here we can see that voting disk resides on /dev/sdb1 .
Once the voting disk is accessible and ASM is started using the SPfile read above, rest of the resources on the node can be started after reading the Oracle Local Registry (OLR) on the node.
References:
http://aychin.wordpress.com/2011/01/22/oracle-11gr2-asm-spfile-eng/
http://asmsupportguy.blogspot.in/2014/03/asm-spfile-in-disk-group.html
http://community.oraclepressbooks.com/downloads/tcoug-top-10-asm-questions.pdf
Related links:
11g R2 RAC: GPNP Profile Demystified
11g R2 RAC: Let’s Edit GPNP Profile
11g R2 RAC: Let’s Lose GPNP Profile
11g R2 RAC : Voting Disk Demystified
——————
This is a great level of explaination for clusterware startup and helped me to clear my doubts…..Thanks very much.
Regards..
Manish Mittal
Thanks Manish!
I keep updating my blog regularly. U can subscribe my blog by email to get notified whenever I post an article.
I get the foll.:
$ gpnptool getpval -asm_spf
Error: Can’t open profile ‘profile.xml’ for read: file not found
$
Can you pls help me understand as to what could be the reason? The above is when entire RAC stack is up on the node. Believe the above should return correct value even when RAC is down on the node – isnt it?
Thanks.
Hi Pcora
You should be in the folder where profile.xml file is located
i.e. GRID_HOME/gpnp//profiles/peer
when u issue above command.
Try and let me know if it works.
Regards
Moreover you should be logged in as root user.
thanks Anju. It worked as oracle user as well root user.
cheers
Sorry for pinging again and again
root@host01 ~]# kfed read /dev/sdb3 | grep -E ‘spf|ausize’
kfdhdb.ausize: 1048576 ; 0x0bc: 0×00100000
kfdhdb.spfile: 16 ; 0x0f4: 0×00000010
kfdhdb.spfflg: 1 ; 0x0f8: 0×00000001
In the output above, we see that
the device /dev/sdb1 contains a copy of the ASM spfile (spfflg=1).
The ASM spfile location starts at the disk offset of 16 (spfile=16)
IT SHOULD BE:-
the device /dev/sdb3 contains a copy of the ASM spfile (spfflg=1).
superbbbbbb…..i google for this and i can’t find this much crystal clear explanation.thank u so much……..
Many thanks. Simply Superb
Thanx Ravi for your time and feedback.
regards
Anju
its a lovely explanation
Hi,
Since oracle has Provisioned GPNP in 11GR2, we are able to find the spfile with the help of profile.xml & kfed,,
With no GPNP component how was this piece of finding SPFILE before ASM instance is OPEN is handled in Oracle 10G?
Regards,
Viswa.
Hi Viswa,
Prior to 11gR2, ASM spfile could not be stored in ASM diskgroup. It had to be stored in a cluster file system.
Regards
Anju
Great stuff!!! clearly explained!! tx and keep posting good stuff, it is helping many DBA to clear doubts!!!
Superb.. Great explanation.. Clearly understandable.. Thanks you so much.
Your blog is amazing. so is your way of explanation. Thanks for sharing your knowledge.
Thanks Venkat for your time and feedback.
Your comments and suggestions are always welcome.
regards
Anju Garg