11g R2 RAC : REAPIR ORACLE CLUSTR REGISTRY(OCR)

In this post, I will demonstrate how we can use ocrconfig -repair command to repair ocr configuration on a node which was not up when the configuration was modified on the other nodes.

Current scenario:
3 node cluster
Nodes: host01, host02, host03
Nodes host02 and host03 are up
Node host01 is down
OCR is stored on ASM diskgroup DATA

Overview:
- Store OCR on additionally on FRA diskgroup
- This information is modified in /etc/oracle/ocr.loc on nodes host02 and host03 which are up
- This information is not modified in /etc/oracle/ocr.loc on node host01 which is down.
– Startup Node host01
– Clusterware does not come up on host01
– Check alert log and crsd log on host01
– Repair OCR configuration on host01 so that /etc/oracle/ocr.loc on host01 gets updated
– STart clusterware on host01 – succeeds

Implementation:
 
- Store OCR on additionally on FRA diskgroup
[root@host02 ~]# ocrconfig -add +FRA
- Check that new OCR location is added  in /etc/oracle/ocr.loc on nodes host02 and host03 which are up
[root@host02 ~]# cat /etc/oracle/ocr.loc
#Device/file  getting replaced by device +FRA
ocrconfig_loc=+DATA
ocrmirrorconfig_loc=+FRA
[root@host03 ~]# cat /etc/oracle/ocr.loc
#Device/file  getting replaced by device +FRA
ocrconfig_loc=+DATA
ocrmirrorconfig_loc=+FRA

- Check that new OCR location is not added  in /etc/oracle/ocr.loc on node host01 which was down
[root@host01 ~]# cat /etc/oracle/ocr.loc
ocrconfig_loc=+DATA
local_only=FALSE
– Bring up host01 . 
– Check that clusterware has not come up there
[root@host01 host01]# crsctl stat res -t
- Check the alert log of host01
[root@host01 host01]# tailf /u01/app/11.2.0/grid/log/host01/alerthost01.log
[ohasd(4914)]CRS-2765:Resource ‘ora.crsd’ has failed on server ‘host01‘.
2013-01-18 23:35:01.950

- Check the crsd  log of host01 – Indicates that local and master information of OCR configuration does not match

 [root@host01 crsd]# vi /u01/app/11.2.0/grid/log/host01/crsd/crsd.log
[  OCRMAS][2876611472]th_calc_av:5′: Rturn persisted AV [186646784] [11.2.0.1.0]
2013-01-18 23:35:13.931: [  OCRSRV][2876611472]th_not_master_change: Master change callback not registered
2013-01-18 23:35:13.931: [  OCRMAS][2876611472]th_master:91: Comparing device hash ids between local and master failed
2013-01-18 23:35:13.931: [  OCRMAS][2876611472]th_master:91 Local dev (1862408427, 1028247821, 0, 0, 0)
2013-01-18 23:35:13.931: [  OCRMAS][2876611472]th_master:91 Master dev (1862408427, 1897369836, 0, 0, 0)
2013-01-18 23:35:13.931: [  OCRMAS][2876611472]th_master:9: Shutdown CacheLocal. my hash ids don’t match

– Repair OCR configuration on host01 
[root@host01 crsd]# ocrconfig -repair -add +FRA
- Check that new OCR location is added  in /etc/oracle/ocr.loc on node host01 
[root@host01 crsd]# cat /etc/oracle/ocr.loc
#Device/file  getting replaced by device +FRA
ocrconfig_loc=+DATA
ocrmirrorconfig_loc=+FRA

– Shutdown and restart cluster on host01
[root@host01 crsd]# crsctl stop crs -f
[root@host01 crsd]# crsctl start crs
[root@host01 crsd]# crsctl start cluster
– Check that crsd is started on host03
[root@host01 host01]# tailf /u01/app/11.2.0/grid/log/host01/ alerthost01.log
[crsd(7297)]CRS-1012:The OCR service started on node host01.
2013-01-18 23:46:07.609
[crsd(7297)]CRS-1201:CRSD started on node host01.
[root@host02 ~]# crsctl stat res -t
——————————————————————————————-
Related links:

                                          ——————-

4 thoughts on “11g R2 RAC : REAPIR ORACLE CLUSTR REGISTRY(OCR)

  1. Hi,

    Just wondering if ocrconfig in 11g has possibility to automatically “synchronize” local OCR.
    For example we could have situation:

    Scenario:

    Nodes: host01, host02, host03
    Nodes host02 and host03 are up
    Node host01 is down
    OCR is stored on ASM diskgroup DATA

    When host01 was down, OCR has been mirrored to OCRDG and FRADG disk groups in ASM using 2 separated commands:

    ocrconfig -add +FRADG
    ocrconfig -add +OCRDG

    So when host01 went up we have to do ???

    ocrconfig -repair -add +FRADG
    ocrconfig -repair -add +OCRDG

    Regards,
    Tom

    1. Hi,

      Your question is not clear. Obviously we have to issue following commands on host01 to repair OCR configuration
      ocrconfig -repair -add +FRADG
      ocrconfig -repair -add +OCRDG

      What do you mean by “automatically “synchronize” local OCR”?

      Regards
      Anju

Your comments and suggestions are welcome!