11G R2 RAC

                                                               —————

9 thoughts on “11G R2 RAC

  1. Hi mam,

    I got error while i restart the cluster in RAC.

    1. [root@host01 ~]# crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online

    2. [root@host01 ~]# crsctl stat res -t
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4000: Command Status failed, or completed with errors.

    3. ASMCMD> ls -l
    State Type Rebal Name
    MOUNTED NORMAL N DATA/
    MOUNTED NORMAL N FRA/
    MOUNTED NORMAL N MY_NEW_DISK/

    pls help me i googled it but not got any appropriate answers…………..

    1. Hi Bharat,

      You can

      – have a look at the alert log
      – crsctl start cluster
      – Restart crs (crsctl stop/start crs )
      – Reboot node

      Pls try above and let me know.

      Regards
      Anju Garg

      1. Hi mam,

        I did above all steps but still not resolved, i,m sending u all grid logs,

        pls check it and do the needfull.

        *********************************************************

        1.alert.host.log

        [ctssd(5545)]CRS-2406:The Cluster Time Synchronization Service timed out on host host01. Details in /u01/app/11.2.0/grid/log/host01/ctssd/octssd.log.
        2014-05-25 23:07:31.354
        [cssd(5361)]CRS-1612:Network communication with node host02 (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.490 seconds
        2014-05-25 23:07:35.362
        [ctssd(5545)]CRS-2406:The Cluster Time Synchronization Service timed out on host host01. Details in /u01/app/11.2.0/grid/log/host01/ctssd/octssd.log.
        2014-05-25 23:07:37.787
        [cssd(5361)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:ASMDISK01 will be considered not functional in 12800 milliseconds
        2014-05-25 23:07:37.883
        [cssd(5361)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:ASMDISK02 will be considered not functional in 12800 milliseconds
        2014-05-25 23:07:37.883
        [cssd(5361)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:ASMDISK03 will be considered not functional in 12800 milliseconds
        2014-05-25 23:07:38.786
        [cssd(5361)]CRS-1611:Network communication with node host02 (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.480 seconds
        2014-05-25 23:07:43.365
        [ctssd(5545)]CRS-2406:The Cluster Time Synchronization Service timed out on host host01. Details in /u01/app/11.2.0/grid/log/host01/ctssd/octssd.log.
        2014-05-25 23:07:44.292
        [cssd(5361)]CRS-1610:Network communication with node host02 (2) missing for 90% of timeout interval. Removal of this node from cluster in 2.470 seconds
        2014-05-25 23:07:45.411
        [cssd(5361)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:ASMDISK01 will be considered not functional in 5770 milliseconds
        2014-05-25 23:07:45.411
        [cssd(5361)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:ASMDISK02 will be considered not functional in 5770 milliseconds
        2014-05-25 23:07:45.411
        [cssd(5361)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:ASMDISK03 will be considered not functional in 5770 milliseconds
        2014-05-25 23:07:46.891
        [cssd(5361)]CRS-1632:Node host02 is being removed from the cluster in cluster incarnation 288448387
        2014-05-25 23:07:47.836
        [cssd(5361)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01 .
        2014-05-25 23:07:48.778
        [ctssd(5545)]CRS-2407:The new Cluster Time Synchronization Service reference node is host host01.
        2014-05-26 21:16:10.035
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.036
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.036
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.036
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.037
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.037
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.037
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.037
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:10.036
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:11.351
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:11.353
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:11.354
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:16.971
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:16.972
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:16.972
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:21.155
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:22.205
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:22.205
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:25.354
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:26.363
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:26.363
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:30.392
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:31.399
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:31.399
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:34.423
        [cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:35.470
        [cssd(5361)]CRS-1605:CSSD voting file is online: ORCL:ASMDISK03; details in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:35.470
        [cssd(5361)]CRS-1605:CSSD voting file is online: ORCL:ASMDISK02; details in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:16:38.490
        [cssd(5361)]CRS-1605:CSSD voting file is online: ORCL:ASMDISK01; details in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
        2014-05-26 21:19:37.934
        [cssd(5361)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01 host02 .
        [root@host01 host01]#

        2. crsd

        p/11.2.0/grid host host01 bin crs to /u01/app/11.2.0/grid/crs/init/
        2014-05-19 23:17:00.995: [ clsdmt][3045804944]Writing PID [13477] to the file [/u01/app/11.2.0/grid/crs/init/host01.pid]
        2014-05-19 23:17:01.833: [ default][3046098640] CRS Daemon Starting
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: AGENT 1
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: AGFW 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CLSFRAME 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CLSVER 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CLUCLS 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: COMMCRS 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: COMMNS 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSAPP 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSCCL 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSCEVT 0
        2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSCOMM 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSD 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSEVT 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSMAIN 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSOCR 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSPE 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSPLACE 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSRES 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSRPT 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSRTI 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSSE 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSSEC 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSTIMER 0
        2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSUI 0
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: CSSCLNT 0
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRAPI 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRASM 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRCAC 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRCLI 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRMAS 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRMSG 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCROSD 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRRAW 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRSRV 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRUTL 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: SuiteTes 1
        2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: UiServer 0
        2014-05-19 23:17:01.837: [ CRSMAIN][3046098640] Checking the OCR device
        2014-05-19 23:17:01.838: [ CRSMAIN][3046098640] Connecting to the CSS Daemon
        2014-05-19 23:17:01.849: [ CSSCLNT][3045804944]clssnsquerymode: not connected to CSSD
        2014-05-19 23:17:01.863: [ CRSMAIN][3046098640] Initializing OCR
        2014-05-19 23:17:01.866: [ OCRAPI][3046098640]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
        2014-05-19 23:17:01.866: [ OCRAPI][3046098640]Check namebufs
        2014-05-19 23:17:01.866: [ OCRAPI][3046098640]Finished checking namebufs
        2014-05-19 23:17:01.867: [ GIPC][3046098640] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3229], original from [clsss.c : 5011]
        2014-05-19 23:17:01.869: [ GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
        2014-05-19 23:17:01.869: [ GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:534] ‘/u01/app/11.2.0/grid’ in effect as GPnP home base.
        2014-05-19 23:17:01.876: [ GIPC][3046098640] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680], original from [clsss.c : 5011]
        2014-05-19 23:17:01.878: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3866] Init gpnp local security key providers (2) fatal if both fail
        2014-05-19 23:17:01.878: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3869] Init gpnp local security key proveders 1 of 2: file wallet (LSKP-FSW)
        2014-05-19 23:17:01.879: [ GPnP][3046098640]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/app/11.2.0/grid/gpnp/host01/wallets/peer/

        2014-05-19 23:17:01.879: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3891] Init gpnp local security key provider 1 of 2: file wallet (LSKP-FSW) OK
        2014-05-19 23:17:01.879: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3897] Init gpnp local security key proveders 2 of 2: OLR wallet (LSKP-CLSW-OLR)
        [ CLWAL][3046098640]clsw_Initialize: OLR initlevel [30000]
        2014-05-19 23:17:01.886: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3919] Init gpnp local security key provider 2 of 2: OLR wallet (LSKP-CLSW-OLR) OK
        2014-05-19 23:17:01.886: [ GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1950]
        2014-05-19 23:17:01.905: [ GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1950]
        2014-05-19 23:17:01.916: [ GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=13477, tl=3, f=0
        2014-05-19 23:17:01.938: [ CLSINET][3046098640] no network information found, grv 5
        2014-05-19 23:17:01.938: [ OCRAPI][3046098640]clsu_get_private_ip_addresses: clsinet_GetNetData failed (). Return [7]
        2014-05-19 23:17:01.938: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
        2014-05-19 23:17:01.940: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
        [ OCRAPI][3046098640]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
        2014-05-19 23:17:01.942: [ OCRAPI][3046098640]a_init:13!: Clusterware init unsuccessful : [44]
        2014-05-19 23:17:01.942: [ CRSOCR][3046098640] OCR context init failure. Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]
        2014-05-19 23:17:01.942: [ CRSD][3046098640][PANIC] CRSD exiting: Could not init OCR, code: 44
        2014-05-19 23:17:01.942: [ CRSD][3046098640] Done.

        [ clsdmt][3046591376]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=host01DBG_CRSD))
        2014-05-19 23:17:03.187: [ clsdmt][3046591376]PID for the Process [13489], connkey 1
        2014-05-19 23:17:03.187: [ clsdmt][3046591376]Creating PID [13489] file for home /u01/app/11.2.0/grid host host01 bin crs to /u01/app/11.2.0/grid/crs/init/
        2014-05-19 23:17:03.187: [ clsdmt][3046591376]Writing PID [13489] to the file [/u01/app/11.2.0/grid/crs/init/host01.pid]
        2014-05-19 23:17:03.942: [ default][3046885072] CRS Daemon Starting
        2014-05-19 23:17:03.943: [ default][3046885072] ENV Logging level for Module: AGENT 1
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: AGFW 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CLSFRAME 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CLSVER 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CLUCLS 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: COMMCRS 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: COMMNS 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSAPP 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSCCL 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSCEVT 0
        2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSCOMM 0
        2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSD 0
        2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSEVT 0
        2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSMAIN 0
        2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSOCR 0
        2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSPE 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSPLACE 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSRES 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSRPT 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSRTI 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSSE 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSSEC 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSTIMER 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSUI 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CSSCLNT 0
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRAPI 1
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRASM 1
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRCAC 1
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRCLI 1
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRMAS 1
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRMSG 1
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCROSD 1
        2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRRAW 1
        2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: OCRSRV 1
        2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: OCRUTL 1
        2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: SuiteTes 1
        2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: UiServer 0
        2014-05-19 23:17:03.946: [ CRSMAIN][3046885072] Checking the OCR device
        2014-05-19 23:17:03.947: [ CRSMAIN][3046885072] Connecting to the CSS Daemon
        2014-05-19 23:17:03.962: [ CRSMAIN][3046885072] Initializing OCR
        2014-05-19 23:17:03.965: [ OCRAPI][3046885072]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
        2014-05-19 23:17:03.965: [ OCRAPI][3046885072]Check namebufs
        2014-05-19 23:17:03.965: [ OCRAPI][3046885072]Finished checking namebufs
        2014-05-19 23:17:03.966: [ GIPC][3046885072] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3229], original from [clsss.c : 5011]
        2014-05-19 23:17:03.968: [ GPnP][3046885072]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
        2014-05-19 23:17:03.968: [ GPnP][3046885072]clsgpnp_Init: [at clsgpnp0.c:534] ‘/u01/app/11.2.0/grid’ in effect as GPnP home base.
        2014-05-19 23:17:03.975: [ GIPC][3046885072] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680], original from [clsss.c : 5011]
        2014-05-19 23:17:03.977: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3866] Init gpnp local security key providers (2) fatal if both fail
        2014-05-19 23:17:03.977: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3869] Init gpnp local security key proveders 1 of 2: file wallet (LSKP-FSW)
        2014-05-19 23:17:03.978: [ GPnP][3046885072]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/app/11.2.0/grid/gpnp/host01/wallets/peer/

        2014-05-19 23:17:03.978: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3891] Init gpnp local security key provider 1 of 2: file wallet (LSKP-FSW) OK
        2014-05-19 23:17:03.978: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3897] Init gpnp local security key proveders 2 of 2: OLR wallet (LSKP-CLSW-OLR)
        [ CLWAL][3046885072]clsw_Initialize: OLR initlevel [30000]
        2014-05-19 23:17:03.986: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3919] Init gpnp local security key provider 2 of 2: OLR wallet (LSKP-CLSW-OLR) OK
        2014-05-19 23:17:03.986: [ GPnP][3046885072]clsgpnp_getCK: [at clsgpnp0.c:1950]
        2014-05-19 23:17:04.009: [ GPnP][3046885072]clsgpnp_getCK: [at clsgpnp0.c:1950]
        2014-05-19 23:17:04.022: [ GPnP][3046885072]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=13489, tl=3, f=0
        2014-05-19 23:17:04.044: [ CLSINET][3046885072] no network information found, grv 5
        2014-05-19 23:17:04.044: [ OCRAPI][3046885072]clsu_get_private_ip_addresses: clsinet_GetNetData failed (). Return [7]
        2014-05-19 23:17:04.044: [GIPCXCPT][3046885072] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
        2014-05-19 23:17:04.045: [ CRSMAIN][3046591376] Policy Engine is not initialized yet!
        2014-05-19 23:17:04.050: [GIPCXCPT][3046885072] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
        [ OCRAPI][3046885072]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
        2014-05-19 23:17:04.054: [ OCRAPI][3046885072]a_init:13!: Clusterware init unsuccessful : [44]
        2014-05-19 23:17:04.054: [ CRSOCR][3046885072] OCR context init failure. Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]
        2014-05-19 23:17:04.054: [ CRSD][3046885072][PANIC] CRSD exiting: Could not init OCR, code: 44
        2014-05-19 23:17:04.054: [ CRSD][3046885072] Done.

        [root@host01 crsd]#

        3. cssd

        2, birth 5
        2014-05-26 22:04:01.316: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:01.316: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:03.390: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:03.390: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:04.176: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:04.176: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:05.468: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:05.468: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:07.537: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:07.537: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:09.181: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:09.181: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:09.627: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:09.627: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:11.697: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:11.697: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:13.783: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:13.783: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:14.189: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:14.189: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:15.851: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:15.851: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:17.948: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:17.948: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:19.192: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:19.192: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:20.018: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:20.018: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:22.098: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:22.098: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:24.179: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:24.179: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:24.215: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:24.215: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:26.254: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:26.254: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:28.306: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:28.306: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:29.222: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:29.222: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:30.399: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:30.399: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:32.466: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:32.466: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:34.223: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:34.223: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:34.529: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:34.529: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:36.599: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:36.599: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:38.689: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:38.689: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:39.231: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:39.231: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:40.736: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:40.736: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:42.814: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:42.814: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:43.230: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:43.230: [ CSSD][2831141776]clssnmSendingThread: sent 4 status msgs to all nodes
        2014-05-26 22:04:44.913: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:44.913: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:46.998: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:46.999: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:48.230: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:48.230: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:49.099: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:49.099: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:51.182: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:51.182: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:53.238: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:53.238: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:53.256: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:53.256: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:55.301: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:55.301: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:57.384: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:57.384: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:04:58.257: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:04:58.257: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:04:59.474: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:04:59.474: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:01.561: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:01.561: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:03.259: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:05:03.259: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:05:03.607: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:03.607: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:05.688: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:05.688: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:07.767: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:07.767: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:08.262: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:05:08.262: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:05:09.832: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:09.832: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:11.901: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:11.901: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:13.274: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:05:13.274: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:05:13.998: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:13.998: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:16.054: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:16.054: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:18.110: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:18.110: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:18.299: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:05:18.299: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:05:20.201: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:20.201: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:22.296: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:22.296: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:23.309: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:05:23.309: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:05:24.339: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:24.339: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:26.409: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:26.409: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:28.319: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:05:28.319: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:05:28.472: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:28.472: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:30.544: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:30.544: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:32.604: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:32.604: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        2014-05-26 22:05:33.324: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
        2014-05-26 22:05:33.324: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
        2014-05-26 22:05:34.714: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
        2014-05-26 22:05:34.714: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
        [root@host01 cssd]#

        4. ctssd

        2014-05-26 22:06:18.770: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:19.773: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:19.773: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:19.773: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:20.775: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:20.775: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:20.775: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:21.778: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:21.778: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:21.778: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:22.781: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:22.781: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:22.781: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:23.783: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:23.783: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:23.784: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:24.786: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:24.786: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:24.786: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:25.788: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:25.788: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:25.788: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:26.790: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:26.790: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:26.790: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:27.793: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:27.793: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:27.793: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:28.795: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:28.795: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:28.795: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:29.798: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:29.798: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:29.798: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:30.800: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:30.800: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:30.800: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:31.802: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:31.802: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:31.802: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:32.805: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:32.805: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:32.805: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:33.807: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:33.807: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:33.807: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:34.810: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:34.810: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:34.810: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:35.813: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:35.813: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:35.813: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:36.814: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:36.814: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:36.815: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:37.817: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:37.817: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:37.817: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:38.819: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:38.819: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:38.819: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:39.822: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:39.822: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:39.822: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:40.473: [ CTSS][2976906128]ctss_checkcb: clsdm requested check alive. Returns [6e]
        2014-05-26 22:06:40.824: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:40.824: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:40.825: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:41.827: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:41.827: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:41.827: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:42.830: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:42.830: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:42.830: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:43.834: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:43.834: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:43.834: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:44.835: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:44.835: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:44.836: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:45.838: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:45.838: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:45.838: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:46.841: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:46.841: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:46.842: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:47.845: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:47.845: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:47.845: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:48.727: [ CTSS][3009448848]sclsctss_gvss1: NTP default config file found
        2014-05-26 22:06:48.727: [ CTSS][3009448848]sclsctss_gvss8: Return [0] and NTP status [2].
        2014-05-26 22:06:48.727: [ CTSS][3009448848]ctss_check_vendor_sw: Vendor time sync software is detected. status [2].
        2014-05-26 22:06:48.847: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:48.847: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:48.848: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:49.850: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:49.850: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:49.851: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:50.854: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:50.854: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:50.854: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:51.857: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:51.857: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:51.857: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:52.860: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:52.860: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:52.860: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:53.862: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:53.862: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:53.862: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:54.866: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:54.866: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:54.866: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:55.870: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:55.870: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:55.871: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:56.874: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:56.874: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:56.874: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:57.876: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:57.876: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:57.877: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:58.878: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:58.878: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:58.879: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:06:59.881: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:06:59.881: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:06:59.881: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:07:00.883: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:07:00.884: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:07:00.884: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        2014-05-26 22:07:01.886: [ CTSS][3019938704]ctsscomm_prh: Handler called
        [ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
        2014-05-26 22:07:01.886: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
        2014-05-26 22:07:01.886: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
        [root@host01 ctssd]#

        5. diskmon

        2014-05-04 18:17:15.357: [ DISKMON][6255:3031575440] dskm_rac_ini13: calling clssgsqgrp
        2014-05-04 18:17:15.358: [ DISKMON][6255:3031575440] dskm_rac_ini17: calling clssgspubdata
        2014-05-04 18:17:15.402: [ DISKMON][6255:3031575440] dskm_nfy_kgzf1: notified thread kgzf enabled
        2014-05-04 18:17:15.402: [ DISKMON][6255:3031575440] dskm_rac_ini30: calling clsslsmutexlock(MASTER#DISKMON#GROUP#MX)
        2014-05-04 18:17:15.403: [ CSSCLNT]clsslsNativeLock: CSS daemon has not entered ACTIVE state.Retrying.

        2014-05-04 18:17:15.404: [ DISKMON][6255:3000105872] dskm_proc_kgzf_handshake5: got the post from the hb thread
        2014-05-04 18:17:15.404: [ DISKMON][6255:3000105872] dskm_proc_kgzf_handshake9: done, kgzf enabled
        2014-05-04 18:17:15.404: [ DISKMON][6255:3000105872] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
        2014-05-04 18:17:16.418: [ DISKMON][6255:3031575440] dskm_rac_ini32: calling clssgsreg
        2014-05-04 18:17:16.460: [ DISKMON][6255:3031575440] dskm_rac_ini34: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 1 (GSDGRPSZ 512)
        2014-05-04 18:17:16.460: [ DISKMON][6255:3031575440] dskm_rac_ini34: following is the list of the nodes in the cluster whose master diskmon has registered in the group:
        2014-05-04 18:17:16.461: [ DISKMON][6255:3031575440] node 2 – host02 (hostname host02)
        2014-05-04 18:17:16.461: [ DISKMON][6255:3031575440] dskm_rac_ini63: calling clsslsunlock (proc retcode 0)
        [ DISKMON][5519]

        I/O Fencing and SKGXP HA monitoring daemon — Version 1.0.0.0
        Process 5519 started on 2014-05-07 at 22:17:38.682

        2014-05-07 22:17:38.730: [ DISKMON][5519] dskm main: starting up
        2014-05-07 22:17:38.732: [ DISKMON][5519:3030997904] dskm_rac_thrd_main: running
        2014-05-07 22:17:38.732: [ DISKMON][5519:3046409920] dskm_rac_thrd_creat2: got the post from the css event handling thread
        2014-05-07 22:17:38.733: [ DISKMON][5519:3030997904] dskm_clss_ini1: calling clssscbinit
        2014-05-07 22:17:38.733: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:38.734: [ DISKMON][5519:3020508048] dskm_oss_thrd_main: running
        2014-05-07 22:17:38.734: [ DISKMON][5519:3046409920] dskm_oss_thrd_creat2: got the post from the oss check status thread
        2014-05-07 22:17:38.734: [ DISKMON][5519:3046409920] dskm main: startup complete
        2014-05-07 22:17:38.734: [ DISKMON][5519:3046409920] listening on -> /var/tmp/.oracle/master_diskmon
        2014-05-07 22:17:38.794: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:38.794: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:40.018: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:40.019: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:40.019: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:40.522: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:40.523: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:40.530: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:41.033: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:41.037: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:41.037: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:41.559: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:41.561: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:41.561: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:42.526: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:42.527: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:42.527: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:43.901: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:43.902: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:43.902: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:45.216: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:45.217: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-07 22:17:45.217: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-07 22:17:45.307: [ DISKMON][5519:3046409920] dskm_slave_thrd_creat: thread created
        2014-05-07 22:17:45.338: [ DISKMON][5519:3010018192] dskm_slave_thrd_main1: slave 0 running
        2014-05-07 22:17:45.338: [ DISKMON][5519:3010018192] dskm_proc_identify8: client orarootagent/5285, diskmon kgzm 2.0, slave 0, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
        2014-05-07 22:17:45.338: [ DISKMON][5519:3010018192] dskm_send_version1:
        2014-05-07 22:17:45.342: [ DISKMON][5519:3010018192] dskm_send_version4: done
        2014-05-07 22:17:45.342: [ DISKMON][5519:3010018192] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
        2014-05-07 22:17:45.719: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
        2014-05-07 22:17:45.745: [ DISKMON][5519:3030997904] dskm_clss_ini5: successful clsssinit(), clssvers 2.1
        2014-05-07 22:17:45.745: [ DISKMON][5519:3030997904] dskm_clss_ini6: calling clssnsqlnum
        2014-05-07 22:18:22.126: [ DISKMON][5519:3030997904] dskm_clss_ini8: calling clsssattrib
        2014-05-07 22:18:22.126: [ DISKMON][5519:3030997904] dskm_clss_ini11: calling clssnsqname
        2014-05-07 22:18:22.127: [ DISKMON][5519:3030997904] dskm_clss_ini13: calling clsssattrib
        2014-05-07 22:18:22.127: [ DISKMON][5519:3030997904] dskm_clss_ini15: calling clssgsregnodegrp
        2014-05-07 22:18:22.129: [ DISKMON][5519:3030997904] dskm_clss_ini18: calling clssgsgrpstat
        2014-05-07 22:18:22.129: [ DISKMON][5519:3030997904] dskm_clss_ini21: calling clssgsgrpstat
        2014-05-07 22:18:38.875: [ DISKMON][5519:3046409920] dskm_slave_thrd_creat: thread created
        2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_slave_thrd_main1: slave 1 running
        2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_proc_identify8: client kgzf/5540, diskmon kgzm 2.0, slave 1, reid cid=189a56762cba4f48bf3555f4f97d97fc,icin=288448381,nmn=1,lnid=288448381,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
        2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_send_version1:
        2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_send_version4: done
        2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
        2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_proc_kgzf_handshake3: client kgzf/5540, kgzf version 2.0, slave 1
        2014-05-07 22:18:39.190: [ DISKMON][5519:3030997904] dskm_clss_ini20: node host01 (1) registered in cluster
        2014-05-07 22:18:39.190: [ DISKMON][5519:3030997904] dskm_clss_ini23: successful dskm_clss_ini
        2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_check_cell_config: CELL storage configuration file /etc/oracle/cell/network-config/cellinit.ora not found
        2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini3: calling clsssattrib
        2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini7: calling clsssattrib
        2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini9: calling clsssattrib
        2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini13: calling clssgsqgrp
        2014-05-07 22:18:39.192: [ DISKMON][5519:3030997904] dskm_nfy_kgzf1: notified thread kgzf enabled
        2014-05-07 22:18:39.192: [ DISKMON][5519:3030997904] dskm_rac_ini30: calling clsslsmutexlock(MASTER#DISKMON#GROUP#MX)
        2014-05-07 22:18:39.193: [ CSSCLNT]clsslsNativeLock: CSS daemon has not entered ACTIVE state.Retrying.

        2014-05-07 22:18:39.194: [ DISKMON][5519:2999528336] dskm_proc_kgzf_handshake5: got the post from the hb thread
        2014-05-07 22:18:39.194: [ DISKMON][5519:2999528336] dskm_proc_kgzf_handshake9: done, kgzf enabled
        2014-05-07 22:18:39.194: [ DISKMON][5519:2999528336] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
        2014-05-07 22:18:40.209: [ DISKMON][5519:3030997904] dskm_rac_ini32: calling clssgsreg
        2014-05-07 22:18:40.212: [ DISKMON][5519:3030997904] dskm_rac_ini34: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 1 (GSDGRPSZ 512)
        2014-05-07 22:18:40.212: [ DISKMON][5519:3030997904] dskm_rac_ini63: calling clsslsunlock (proc retcode 0)
        2014-05-18 21:06:34.341: [ DISKMON][5519:2999528336] dskm_proc_fence_entity13: fenced reid cid=189a56762cba4f48bf3555f4f97d97fc,icin=288448381,nmn=2,lnid=288448381,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100 (size 93, msgno 1, event 2)
        2014-05-18 21:06:34.341: [ DISKMON][5519:2999528336] dskm_send_entity_fenced1:
        2014-05-18 21:06:34.342: [ DISKMON][5519:2999528336] dskm_send_entity_fenced4: done
        2014-05-18 21:06:34.342: [ DISKMON][5519:2999528336] dskm_process_msg7: processed msg 1 type KGZM_FENCE_ENTITY (0x0007), retcode 0
        [ DISKMON][5366]

        I/O Fencing and SKGXP HA monitoring daemon — Version 1.0.0.0
        Process 5366 started on 2014-05-18 at 22:17:55.799

        2014-05-18 22:17:55.823: [ DISKMON][5366] dskm main: starting up
        2014-05-18 22:17:55.825: [ DISKMON][5366:3031305104] dskm_rac_thrd_main: running
        2014-05-18 22:17:55.831: [ DISKMON][5366:3046717120] dskm_rac_thrd_creat2: got the post from the css event handling thread
        2014-05-18 22:17:55.831: [ DISKMON][5366:3031305104] dskm_clss_ini1: calling clssscbinit
        2014-05-18 22:17:55.832: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
        2014-05-18 22:17:55.833: [ DISKMON][5366:3020815248] dskm_oss_thrd_main: running
        2014-05-18 22:17:55.833: [ DISKMON][5366:3046717120] dskm_oss_thrd_creat2: got the post from the oss check status thread
        2014-05-18 22:17:55.833: [ DISKMON][5366:3046717120] dskm main: startup complete
        2014-05-18 22:17:55.833: [ DISKMON][5366:3046717120] listening on -> /var/tmp/.oracle/master_diskmon
        2014-05-18 22:17:56.070: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-18 22:17:56.070: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-18 22:17:56.572: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
        2014-05-18 22:17:56.577: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-18 22:17:56.577: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-18 22:17:57.080: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
        2014-05-18 22:17:57.081: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-18 22:17:57.081: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-18 22:17:57.582: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
        2014-05-18 22:17:57.584: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-18 22:17:57.584: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-18 22:17:58.127: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
        2014-05-18 22:17:58.128: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-18 22:17:58.128: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-18 22:17:58.845: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
        2014-05-18 22:17:58.846: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
        2014-05-18 22:17:58.846: [ CSSCLNT]clsssInitNative: connect failed, rc 29
        2014-05-18 22:17:59.348: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
        2014-05-18 22:17:59.730: [ DISKMON][5366:3031305104] dskm_clss_ini5: successful clsssinit(), clssvers 2.1
        2014-05-18 22:17:59.730: [ DISKMON][5366:3031305104] dskm_clss_ini6: calling clssnsqlnum
        2014-05-18 22:18:01.687: [ DISKMON][5366:3046717120] dskm_slave_thrd_creat: thread created
        2014-05-18 22:18:01.687: [ DISKMON][5366:3010325392] dskm_slave_thrd_main1: slave 0 running
        2014-05-18 22:18:01.687: [ DISKMON][5366:3010325392] dskm_proc_identify8: client orarootagent/5264, diskmon kgzm 2.0, slave 0, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
        2014-05-18 22:18:01.687: [ DISKMON][5366:3010325392] dskm_send_version1:
        2014-05-18 22:18:01.691: [ DISKMON][5366:3010325392] dskm_send_version4: done
        2014-05-18 22:18:01.691: [ DISKMON][5366:3010325392] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
        2014-05-18 22:18:22.996: [ DISKMON][5366:3031305104] dskm_clss_ini8: calling clsssattrib
        2014-05-18 22:18:22.996: [ DISKMON][5366:3031305104] dskm_clss_ini11: calling clssnsqname
        2014-05-18 22:18:22.998: [ DISKMON][5366:3031305104] dskm_clss_ini13: calling clsssattrib
        2014-05-18 22:18:22.998: [ DISKMON][5366:3031305104] dskm_clss_ini15: calling clssgsregnodegrp
        2014-05-18 22:18:23.002: [ DISKMON][5

  2. Hi Anju,
    Thanks a million…for this wonderful blog. I am really able to utilize your blog for my oracle practice.
    Thanks,
    manju

Leave a Reply to bharat Cancel reply