INS-08109 unexpected error occured while validating inputs at state ‘InstallOptions’
Instance Recovery In RAC
Instance Recovery In RAC
PRVF-5636 : The DNS response time for an unreachable node exceeded 15000 ms
Recover Voting Disk – Scenario-I
Recover Voting Disk – Scenario-II
Recover Voting Disk – Scenario-I
Recover Voting Disk – Scenario-II
VIP Failover Demonstration
Workload Management in 11g R2 RAC : Failover
Workload Management in 11g R2 RAC : Load Balancing
Workload Management in 11g R2 RAC : Failover
Workload Management in 11g R2 RAC : Load Balancing
—————
wonderful JOB for RAC lovers
Hi mam,
I got error while i restart the cluster in RAC.
1. [root@host01 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
2. [root@host01 ~]# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
3. ASMCMD> ls -l
State Type Rebal Name
MOUNTED NORMAL N DATA/
MOUNTED NORMAL N FRA/
MOUNTED NORMAL N MY_NEW_DISK/
pls help me i googled it but not got any appropriate answers…………..
Hi Bharat,
You can
– have a look at the alert log
– crsctl start cluster
– Restart crs (crsctl stop/start crs )
– Reboot node
Pls try above and let me know.
Regards
Anju Garg
Hi mam,
I did above all steps but still not resolved, i,m sending u all grid logs,
pls check it and do the needfull.
*********************************************************
1.alert.host.log
[ctssd(5545)]CRS-2406:The Cluster Time Synchronization Service timed out on host host01. Details in /u01/app/11.2.0/grid/log/host01/ctssd/octssd.log.
2014-05-25 23:07:31.354
[cssd(5361)]CRS-1612:Network communication with node host02 (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.490 seconds
2014-05-25 23:07:35.362
[ctssd(5545)]CRS-2406:The Cluster Time Synchronization Service timed out on host host01. Details in /u01/app/11.2.0/grid/log/host01/ctssd/octssd.log.
2014-05-25 23:07:37.787
[cssd(5361)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:ASMDISK01 will be considered not functional in 12800 milliseconds
2014-05-25 23:07:37.883
[cssd(5361)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:ASMDISK02 will be considered not functional in 12800 milliseconds
2014-05-25 23:07:37.883
[cssd(5361)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:ASMDISK03 will be considered not functional in 12800 milliseconds
2014-05-25 23:07:38.786
[cssd(5361)]CRS-1611:Network communication with node host02 (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.480 seconds
2014-05-25 23:07:43.365
[ctssd(5545)]CRS-2406:The Cluster Time Synchronization Service timed out on host host01. Details in /u01/app/11.2.0/grid/log/host01/ctssd/octssd.log.
2014-05-25 23:07:44.292
[cssd(5361)]CRS-1610:Network communication with node host02 (2) missing for 90% of timeout interval. Removal of this node from cluster in 2.470 seconds
2014-05-25 23:07:45.411
[cssd(5361)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:ASMDISK01 will be considered not functional in 5770 milliseconds
2014-05-25 23:07:45.411
[cssd(5361)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:ASMDISK02 will be considered not functional in 5770 milliseconds
2014-05-25 23:07:45.411
[cssd(5361)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:ASMDISK03 will be considered not functional in 5770 milliseconds
2014-05-25 23:07:46.891
[cssd(5361)]CRS-1632:Node host02 is being removed from the cluster in cluster incarnation 288448387
2014-05-25 23:07:47.836
[cssd(5361)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01 .
2014-05-25 23:07:48.778
[ctssd(5545)]CRS-2407:The new Cluster Time Synchronization Service reference node is host host01.
2014-05-26 21:16:10.035
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.036
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.036
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.036
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.037
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.037
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.037
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.037
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:10.036
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:11.351
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:11.353
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:11.354
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:16.971
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:16.972
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:16.972
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:21.155
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:22.205
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:22.205
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:25.354
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:26.363
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:26.363
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:30.392
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:31.399
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK03; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:31.399
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK02; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:34.423
[cssd(5361)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:35.470
[cssd(5361)]CRS-1605:CSSD voting file is online: ORCL:ASMDISK03; details in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:35.470
[cssd(5361)]CRS-1605:CSSD voting file is online: ORCL:ASMDISK02; details in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:16:38.490
[cssd(5361)]CRS-1605:CSSD voting file is online: ORCL:ASMDISK01; details in /u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2014-05-26 21:19:37.934
[cssd(5361)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01 host02 .
[root@host01 host01]#
2. crsd
p/11.2.0/grid host host01 bin crs to /u01/app/11.2.0/grid/crs/init/
2014-05-19 23:17:00.995: [ clsdmt][3045804944]Writing PID [13477] to the file [/u01/app/11.2.0/grid/crs/init/host01.pid]
2014-05-19 23:17:01.833: [ default][3046098640] CRS Daemon Starting
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: AGENT 1
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: AGFW 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CLSFRAME 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CLSVER 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CLUCLS 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: COMMCRS 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: COMMNS 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSAPP 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSCCL 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSCEVT 0
2014-05-19 23:17:01.835: [ default][3046098640] ENV Logging level for Module: CRSCOMM 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSD 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSEVT 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSMAIN 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSOCR 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSPE 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSPLACE 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSRES 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSRPT 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSRTI 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSSE 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSSEC 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSTIMER 0
2014-05-19 23:17:01.836: [ CRSD][3046098640] ENV Logging level for Module: CRSUI 0
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: CSSCLNT 0
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRAPI 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRASM 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRCAC 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRCLI 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRMAS 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRMSG 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCROSD 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRRAW 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRSRV 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: OCRUTL 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: SuiteTes 1
2014-05-19 23:17:01.837: [ CRSD][3046098640] ENV Logging level for Module: UiServer 0
2014-05-19 23:17:01.837: [ CRSMAIN][3046098640] Checking the OCR device
2014-05-19 23:17:01.838: [ CRSMAIN][3046098640] Connecting to the CSS Daemon
2014-05-19 23:17:01.849: [ CSSCLNT][3045804944]clssnsquerymode: not connected to CSSD
2014-05-19 23:17:01.863: [ CRSMAIN][3046098640] Initializing OCR
2014-05-19 23:17:01.866: [ OCRAPI][3046098640]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2014-05-19 23:17:01.866: [ OCRAPI][3046098640]Check namebufs
2014-05-19 23:17:01.866: [ OCRAPI][3046098640]Finished checking namebufs
2014-05-19 23:17:01.867: [ GIPC][3046098640] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3229], original from [clsss.c : 5011]
2014-05-19 23:17:01.869: [ GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2014-05-19 23:17:01.869: [ GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:534] ‘/u01/app/11.2.0/grid’ in effect as GPnP home base.
2014-05-19 23:17:01.876: [ GIPC][3046098640] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680], original from [clsss.c : 5011]
2014-05-19 23:17:01.878: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3866] Init gpnp local security key providers (2) fatal if both fail
2014-05-19 23:17:01.878: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3869] Init gpnp local security key proveders 1 of 2: file wallet (LSKP-FSW)
2014-05-19 23:17:01.879: [ GPnP][3046098640]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/app/11.2.0/grid/gpnp/host01/wallets/peer/
2014-05-19 23:17:01.879: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3891] Init gpnp local security key provider 1 of 2: file wallet (LSKP-FSW) OK
2014-05-19 23:17:01.879: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3897] Init gpnp local security key proveders 2 of 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][3046098640]clsw_Initialize: OLR initlevel [30000]
2014-05-19 23:17:01.886: [ GPnP][3046098640]clsgpnp_InitCKProviders: [at clsgpnp0.c:3919] Init gpnp local security key provider 2 of 2: OLR wallet (LSKP-CLSW-OLR) OK
2014-05-19 23:17:01.886: [ GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1950]
2014-05-19 23:17:01.905: [ GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1950]
2014-05-19 23:17:01.916: [ GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=13477, tl=3, f=0
2014-05-19 23:17:01.938: [ CLSINET][3046098640] no network information found, grv 5
2014-05-19 23:17:01.938: [ OCRAPI][3046098640]clsu_get_private_ip_addresses: clsinet_GetNetData failed (). Return [7]
2014-05-19 23:17:01.938: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
2014-05-19 23:17:01.940: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
[ OCRAPI][3046098640]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2014-05-19 23:17:01.942: [ OCRAPI][3046098640]a_init:13!: Clusterware init unsuccessful : [44]
2014-05-19 23:17:01.942: [ CRSOCR][3046098640] OCR context init failure. Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]
2014-05-19 23:17:01.942: [ CRSD][3046098640][PANIC] CRSD exiting: Could not init OCR, code: 44
2014-05-19 23:17:01.942: [ CRSD][3046098640] Done.
[ clsdmt][3046591376]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=host01DBG_CRSD))
2014-05-19 23:17:03.187: [ clsdmt][3046591376]PID for the Process [13489], connkey 1
2014-05-19 23:17:03.187: [ clsdmt][3046591376]Creating PID [13489] file for home /u01/app/11.2.0/grid host host01 bin crs to /u01/app/11.2.0/grid/crs/init/
2014-05-19 23:17:03.187: [ clsdmt][3046591376]Writing PID [13489] to the file [/u01/app/11.2.0/grid/crs/init/host01.pid]
2014-05-19 23:17:03.942: [ default][3046885072] CRS Daemon Starting
2014-05-19 23:17:03.943: [ default][3046885072] ENV Logging level for Module: AGENT 1
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: AGFW 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CLSFRAME 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CLSVER 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CLUCLS 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: COMMCRS 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: COMMNS 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSAPP 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSCCL 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSCEVT 0
2014-05-19 23:17:03.944: [ default][3046885072] ENV Logging level for Module: CRSCOMM 0
2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSD 0
2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSEVT 0
2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSMAIN 0
2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSOCR 0
2014-05-19 23:17:03.944: [ CRSD][3046885072] ENV Logging level for Module: CRSPE 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSPLACE 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSRES 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSRPT 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSRTI 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSSE 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSSEC 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSTIMER 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CRSUI 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: CSSCLNT 0
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRAPI 1
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRASM 1
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRCAC 1
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRCLI 1
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRMAS 1
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRMSG 1
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCROSD 1
2014-05-19 23:17:03.945: [ CRSD][3046885072] ENV Logging level for Module: OCRRAW 1
2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: OCRSRV 1
2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: OCRUTL 1
2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: SuiteTes 1
2014-05-19 23:17:03.946: [ CRSD][3046885072] ENV Logging level for Module: UiServer 0
2014-05-19 23:17:03.946: [ CRSMAIN][3046885072] Checking the OCR device
2014-05-19 23:17:03.947: [ CRSMAIN][3046885072] Connecting to the CSS Daemon
2014-05-19 23:17:03.962: [ CRSMAIN][3046885072] Initializing OCR
2014-05-19 23:17:03.965: [ OCRAPI][3046885072]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2014-05-19 23:17:03.965: [ OCRAPI][3046885072]Check namebufs
2014-05-19 23:17:03.965: [ OCRAPI][3046885072]Finished checking namebufs
2014-05-19 23:17:03.966: [ GIPC][3046885072] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3229], original from [clsss.c : 5011]
2014-05-19 23:17:03.968: [ GPnP][3046885072]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2014-05-19 23:17:03.968: [ GPnP][3046885072]clsgpnp_Init: [at clsgpnp0.c:534] ‘/u01/app/11.2.0/grid’ in effect as GPnP home base.
2014-05-19 23:17:03.975: [ GIPC][3046885072] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680], original from [clsss.c : 5011]
2014-05-19 23:17:03.977: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3866] Init gpnp local security key providers (2) fatal if both fail
2014-05-19 23:17:03.977: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3869] Init gpnp local security key proveders 1 of 2: file wallet (LSKP-FSW)
2014-05-19 23:17:03.978: [ GPnP][3046885072]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/app/11.2.0/grid/gpnp/host01/wallets/peer/
2014-05-19 23:17:03.978: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3891] Init gpnp local security key provider 1 of 2: file wallet (LSKP-FSW) OK
2014-05-19 23:17:03.978: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3897] Init gpnp local security key proveders 2 of 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][3046885072]clsw_Initialize: OLR initlevel [30000]
2014-05-19 23:17:03.986: [ GPnP][3046885072]clsgpnp_InitCKProviders: [at clsgpnp0.c:3919] Init gpnp local security key provider 2 of 2: OLR wallet (LSKP-CLSW-OLR) OK
2014-05-19 23:17:03.986: [ GPnP][3046885072]clsgpnp_getCK: [at clsgpnp0.c:1950]
2014-05-19 23:17:04.009: [ GPnP][3046885072]clsgpnp_getCK: [at clsgpnp0.c:1950]
2014-05-19 23:17:04.022: [ GPnP][3046885072]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=13489, tl=3, f=0
2014-05-19 23:17:04.044: [ CLSINET][3046885072] no network information found, grv 5
2014-05-19 23:17:04.044: [ OCRAPI][3046885072]clsu_get_private_ip_addresses: clsinet_GetNetData failed (). Return [7]
2014-05-19 23:17:04.044: [GIPCXCPT][3046885072] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
2014-05-19 23:17:04.045: [ CRSMAIN][3046591376] Policy Engine is not initialized yet!
2014-05-19 23:17:04.050: [GIPCXCPT][3046885072] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
[ OCRAPI][3046885072]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2014-05-19 23:17:04.054: [ OCRAPI][3046885072]a_init:13!: Clusterware init unsuccessful : [44]
2014-05-19 23:17:04.054: [ CRSOCR][3046885072] OCR context init failure. Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]
2014-05-19 23:17:04.054: [ CRSD][3046885072][PANIC] CRSD exiting: Could not init OCR, code: 44
2014-05-19 23:17:04.054: [ CRSD][3046885072] Done.
[root@host01 crsd]#
3. cssd
2, birth 5
2014-05-26 22:04:01.316: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:01.316: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:03.390: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:03.390: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:04.176: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:04.176: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:05.468: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:05.468: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:07.537: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:07.537: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:09.181: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:09.181: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:09.627: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:09.627: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:11.697: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:11.697: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:13.783: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:13.783: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:14.189: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:14.189: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:15.851: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:15.851: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:17.948: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:17.948: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:19.192: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:19.192: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:20.018: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:20.018: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:22.098: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:22.098: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:24.179: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:24.179: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:24.215: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:24.215: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:26.254: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:26.254: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:28.306: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:28.306: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:29.222: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:29.222: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:30.399: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:30.399: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:32.466: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:32.466: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:34.223: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:34.223: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:34.529: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:34.529: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:36.599: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:36.599: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:38.689: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:38.689: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:39.231: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:39.231: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:40.736: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:40.736: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:42.814: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:42.814: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:43.230: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:43.230: [ CSSD][2831141776]clssnmSendingThread: sent 4 status msgs to all nodes
2014-05-26 22:04:44.913: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:44.913: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:46.998: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:46.999: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:48.230: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:48.230: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:49.099: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:49.099: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:51.182: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:51.182: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:53.238: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:53.238: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:53.256: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:53.256: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:55.301: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:55.301: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:57.384: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:57.384: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:04:58.257: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:04:58.257: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:04:59.474: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:04:59.474: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:01.561: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:01.561: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:03.259: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:05:03.259: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:05:03.607: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:03.607: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:05.688: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:05.688: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:07.767: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:07.767: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:08.262: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:05:08.262: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:05:09.832: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:09.832: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:11.901: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:11.901: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:13.274: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:05:13.274: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:05:13.998: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:13.998: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:16.054: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:16.054: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:18.110: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:18.110: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:18.299: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:05:18.299: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:05:20.201: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:20.201: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:22.296: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:22.296: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:23.309: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:05:23.309: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:05:24.339: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:24.339: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:26.409: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:26.409: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:28.319: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:05:28.319: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:05:28.472: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:28.472: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:30.544: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:30.544: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:32.604: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:32.604: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
2014-05-26 22:05:33.324: [ CSSD][2831141776]clssnmSendingThread: sending status msg to all nodes
2014-05-26 22:05:33.324: [ CSSD][2831141776]clssnmSendingThread: sent 5 status msgs to all nodes
2014-05-26 22:05:34.714: [ CSSD][2852121488]clssgmTagize: version(1), type(13), tagizer(0x80cf3ac)
2014-05-26 22:05:34.714: [ CSSD][2852121488]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 5
[root@host01 cssd]#
4. ctssd
2014-05-26 22:06:18.770: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:19.773: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:19.773: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:19.773: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:20.775: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:20.775: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:20.775: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:21.778: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:21.778: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:21.778: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:22.781: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:22.781: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:22.781: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:23.783: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:23.783: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:23.784: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:24.786: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:24.786: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:24.786: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:25.788: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:25.788: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:25.788: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:26.790: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:26.790: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:26.790: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:27.793: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:27.793: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:27.793: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:28.795: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:28.795: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:28.795: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:29.798: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:29.798: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:29.798: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:30.800: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:30.800: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:30.800: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:31.802: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:31.802: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:31.802: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:32.805: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:32.805: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:32.805: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:33.807: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:33.807: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:33.807: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:34.810: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:34.810: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:34.810: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:35.813: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:35.813: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:35.813: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:36.814: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:36.814: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:36.815: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:37.817: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:37.817: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:37.817: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:38.819: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:38.819: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:38.819: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:39.822: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:39.822: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:39.822: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:40.473: [ CTSS][2976906128]ctss_checkcb: clsdm requested check alive. Returns [6e]
2014-05-26 22:06:40.824: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:40.824: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:40.825: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:41.827: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:41.827: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:41.827: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:42.830: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:42.830: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:42.830: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:43.834: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:43.834: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:43.834: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:44.835: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:44.835: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:44.836: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:45.838: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:45.838: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:45.838: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:46.841: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:46.841: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:46.842: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:47.845: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:47.845: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:47.845: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:48.727: [ CTSS][3009448848]sclsctss_gvss1: NTP default config file found
2014-05-26 22:06:48.727: [ CTSS][3009448848]sclsctss_gvss8: Return [0] and NTP status [2].
2014-05-26 22:06:48.727: [ CTSS][3009448848]ctss_check_vendor_sw: Vendor time sync software is detected. status [2].
2014-05-26 22:06:48.847: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:48.847: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:48.848: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:49.850: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:49.850: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:49.851: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:50.854: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:50.854: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:50.854: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:51.857: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:51.857: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:51.857: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:52.860: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:52.860: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:52.860: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:53.862: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:53.862: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:53.862: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:54.866: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:54.866: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:54.866: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:55.870: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:55.870: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:55.871: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:56.874: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:56.874: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:56.874: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:57.876: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:57.876: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:57.877: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:58.878: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:58.878: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:58.879: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:06:59.881: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:06:59.881: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:06:59.881: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:07:00.883: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:07:00.884: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:07:00.884: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
2014-05-26 22:07:01.886: [ CTSS][3019938704]ctsscomm_prh: Handler called
[ CTSS][3019938704]ctss_process_request_handler: Master: Received sync message event
2014-05-26 22:07:01.886: [ CTSS][3019938704]ctsscomm_pi: Received sync msg
2014-05-26 22:07:01.886: [ CTSS][3019938704]ctsscomm_pi: Received from slave ( mode [0x66] nodenum [2] hostname [host02] )
[root@host01 ctssd]#
5. diskmon
2014-05-04 18:17:15.357: [ DISKMON][6255:3031575440] dskm_rac_ini13: calling clssgsqgrp
2014-05-04 18:17:15.358: [ DISKMON][6255:3031575440] dskm_rac_ini17: calling clssgspubdata
2014-05-04 18:17:15.402: [ DISKMON][6255:3031575440] dskm_nfy_kgzf1: notified thread kgzf enabled
2014-05-04 18:17:15.402: [ DISKMON][6255:3031575440] dskm_rac_ini30: calling clsslsmutexlock(MASTER#DISKMON#GROUP#MX)
2014-05-04 18:17:15.403: [ CSSCLNT]clsslsNativeLock: CSS daemon has not entered ACTIVE state.Retrying.
2014-05-04 18:17:15.404: [ DISKMON][6255:3000105872] dskm_proc_kgzf_handshake5: got the post from the hb thread
2014-05-04 18:17:15.404: [ DISKMON][6255:3000105872] dskm_proc_kgzf_handshake9: done, kgzf enabled
2014-05-04 18:17:15.404: [ DISKMON][6255:3000105872] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
2014-05-04 18:17:16.418: [ DISKMON][6255:3031575440] dskm_rac_ini32: calling clssgsreg
2014-05-04 18:17:16.460: [ DISKMON][6255:3031575440] dskm_rac_ini34: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 1 (GSDGRPSZ 512)
2014-05-04 18:17:16.460: [ DISKMON][6255:3031575440] dskm_rac_ini34: following is the list of the nodes in the cluster whose master diskmon has registered in the group:
2014-05-04 18:17:16.461: [ DISKMON][6255:3031575440] node 2 – host02 (hostname host02)
2014-05-04 18:17:16.461: [ DISKMON][6255:3031575440] dskm_rac_ini63: calling clsslsunlock (proc retcode 0)
[ DISKMON][5519]
I/O Fencing and SKGXP HA monitoring daemon — Version 1.0.0.0
Process 5519 started on 2014-05-07 at 22:17:38.682
2014-05-07 22:17:38.730: [ DISKMON][5519] dskm main: starting up
2014-05-07 22:17:38.732: [ DISKMON][5519:3030997904] dskm_rac_thrd_main: running
2014-05-07 22:17:38.732: [ DISKMON][5519:3046409920] dskm_rac_thrd_creat2: got the post from the css event handling thread
2014-05-07 22:17:38.733: [ DISKMON][5519:3030997904] dskm_clss_ini1: calling clssscbinit
2014-05-07 22:17:38.733: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:38.734: [ DISKMON][5519:3020508048] dskm_oss_thrd_main: running
2014-05-07 22:17:38.734: [ DISKMON][5519:3046409920] dskm_oss_thrd_creat2: got the post from the oss check status thread
2014-05-07 22:17:38.734: [ DISKMON][5519:3046409920] dskm main: startup complete
2014-05-07 22:17:38.734: [ DISKMON][5519:3046409920] listening on -> /var/tmp/.oracle/master_diskmon
2014-05-07 22:17:38.794: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:38.794: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:40.018: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:40.019: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:40.019: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:40.522: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:40.523: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:40.530: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:41.033: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:41.037: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:41.037: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:41.559: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:41.561: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:41.561: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:42.526: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:42.527: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:42.527: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:43.901: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:43.902: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:43.902: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:45.216: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:45.217: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-07 22:17:45.217: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-07 22:17:45.307: [ DISKMON][5519:3046409920] dskm_slave_thrd_creat: thread created
2014-05-07 22:17:45.338: [ DISKMON][5519:3010018192] dskm_slave_thrd_main1: slave 0 running
2014-05-07 22:17:45.338: [ DISKMON][5519:3010018192] dskm_proc_identify8: client orarootagent/5285, diskmon kgzm 2.0, slave 0, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
2014-05-07 22:17:45.338: [ DISKMON][5519:3010018192] dskm_send_version1:
2014-05-07 22:17:45.342: [ DISKMON][5519:3010018192] dskm_send_version4: done
2014-05-07 22:17:45.342: [ DISKMON][5519:3010018192] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
2014-05-07 22:17:45.719: [ DISKMON][5519:3030997904] dskm_clss_ini2: calling clsssinit
2014-05-07 22:17:45.745: [ DISKMON][5519:3030997904] dskm_clss_ini5: successful clsssinit(), clssvers 2.1
2014-05-07 22:17:45.745: [ DISKMON][5519:3030997904] dskm_clss_ini6: calling clssnsqlnum
2014-05-07 22:18:22.126: [ DISKMON][5519:3030997904] dskm_clss_ini8: calling clsssattrib
2014-05-07 22:18:22.126: [ DISKMON][5519:3030997904] dskm_clss_ini11: calling clssnsqname
2014-05-07 22:18:22.127: [ DISKMON][5519:3030997904] dskm_clss_ini13: calling clsssattrib
2014-05-07 22:18:22.127: [ DISKMON][5519:3030997904] dskm_clss_ini15: calling clssgsregnodegrp
2014-05-07 22:18:22.129: [ DISKMON][5519:3030997904] dskm_clss_ini18: calling clssgsgrpstat
2014-05-07 22:18:22.129: [ DISKMON][5519:3030997904] dskm_clss_ini21: calling clssgsgrpstat
2014-05-07 22:18:38.875: [ DISKMON][5519:3046409920] dskm_slave_thrd_creat: thread created
2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_slave_thrd_main1: slave 1 running
2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_proc_identify8: client kgzf/5540, diskmon kgzm 2.0, slave 1, reid cid=189a56762cba4f48bf3555f4f97d97fc,icin=288448381,nmn=1,lnid=288448381,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_send_version1:
2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_send_version4: done
2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
2014-05-07 22:18:38.878: [ DISKMON][5519:2999528336] dskm_proc_kgzf_handshake3: client kgzf/5540, kgzf version 2.0, slave 1
2014-05-07 22:18:39.190: [ DISKMON][5519:3030997904] dskm_clss_ini20: node host01 (1) registered in cluster
2014-05-07 22:18:39.190: [ DISKMON][5519:3030997904] dskm_clss_ini23: successful dskm_clss_ini
2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_check_cell_config: CELL storage configuration file /etc/oracle/cell/network-config/cellinit.ora not found
2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini3: calling clsssattrib
2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini7: calling clsssattrib
2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini9: calling clsssattrib
2014-05-07 22:18:39.191: [ DISKMON][5519:3030997904] dskm_rac_ini13: calling clssgsqgrp
2014-05-07 22:18:39.192: [ DISKMON][5519:3030997904] dskm_nfy_kgzf1: notified thread kgzf enabled
2014-05-07 22:18:39.192: [ DISKMON][5519:3030997904] dskm_rac_ini30: calling clsslsmutexlock(MASTER#DISKMON#GROUP#MX)
2014-05-07 22:18:39.193: [ CSSCLNT]clsslsNativeLock: CSS daemon has not entered ACTIVE state.Retrying.
2014-05-07 22:18:39.194: [ DISKMON][5519:2999528336] dskm_proc_kgzf_handshake5: got the post from the hb thread
2014-05-07 22:18:39.194: [ DISKMON][5519:2999528336] dskm_proc_kgzf_handshake9: done, kgzf enabled
2014-05-07 22:18:39.194: [ DISKMON][5519:2999528336] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
2014-05-07 22:18:40.209: [ DISKMON][5519:3030997904] dskm_rac_ini32: calling clssgsreg
2014-05-07 22:18:40.212: [ DISKMON][5519:3030997904] dskm_rac_ini34: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 1 (GSDGRPSZ 512)
2014-05-07 22:18:40.212: [ DISKMON][5519:3030997904] dskm_rac_ini63: calling clsslsunlock (proc retcode 0)
2014-05-18 21:06:34.341: [ DISKMON][5519:2999528336] dskm_proc_fence_entity13: fenced reid cid=189a56762cba4f48bf3555f4f97d97fc,icin=288448381,nmn=2,lnid=288448381,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100 (size 93, msgno 1, event 2)
2014-05-18 21:06:34.341: [ DISKMON][5519:2999528336] dskm_send_entity_fenced1:
2014-05-18 21:06:34.342: [ DISKMON][5519:2999528336] dskm_send_entity_fenced4: done
2014-05-18 21:06:34.342: [ DISKMON][5519:2999528336] dskm_process_msg7: processed msg 1 type KGZM_FENCE_ENTITY (0x0007), retcode 0
[ DISKMON][5366]
I/O Fencing and SKGXP HA monitoring daemon — Version 1.0.0.0
Process 5366 started on 2014-05-18 at 22:17:55.799
2014-05-18 22:17:55.823: [ DISKMON][5366] dskm main: starting up
2014-05-18 22:17:55.825: [ DISKMON][5366:3031305104] dskm_rac_thrd_main: running
2014-05-18 22:17:55.831: [ DISKMON][5366:3046717120] dskm_rac_thrd_creat2: got the post from the css event handling thread
2014-05-18 22:17:55.831: [ DISKMON][5366:3031305104] dskm_clss_ini1: calling clssscbinit
2014-05-18 22:17:55.832: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
2014-05-18 22:17:55.833: [ DISKMON][5366:3020815248] dskm_oss_thrd_main: running
2014-05-18 22:17:55.833: [ DISKMON][5366:3046717120] dskm_oss_thrd_creat2: got the post from the oss check status thread
2014-05-18 22:17:55.833: [ DISKMON][5366:3046717120] dskm main: startup complete
2014-05-18 22:17:55.833: [ DISKMON][5366:3046717120] listening on -> /var/tmp/.oracle/master_diskmon
2014-05-18 22:17:56.070: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-18 22:17:56.070: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-18 22:17:56.572: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
2014-05-18 22:17:56.577: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-18 22:17:56.577: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-18 22:17:57.080: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
2014-05-18 22:17:57.081: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-18 22:17:57.081: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-18 22:17:57.582: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
2014-05-18 22:17:57.584: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-18 22:17:57.584: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-18 22:17:58.127: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
2014-05-18 22:17:58.128: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-18 22:17:58.128: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-18 22:17:58.845: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
2014-05-18 22:17:58.846: [ CSSCLNT]clssscConnect: gipc request failed with 29 (0x16)
2014-05-18 22:17:58.846: [ CSSCLNT]clsssInitNative: connect failed, rc 29
2014-05-18 22:17:59.348: [ DISKMON][5366:3031305104] dskm_clss_ini2: calling clsssinit
2014-05-18 22:17:59.730: [ DISKMON][5366:3031305104] dskm_clss_ini5: successful clsssinit(), clssvers 2.1
2014-05-18 22:17:59.730: [ DISKMON][5366:3031305104] dskm_clss_ini6: calling clssnsqlnum
2014-05-18 22:18:01.687: [ DISKMON][5366:3046717120] dskm_slave_thrd_creat: thread created
2014-05-18 22:18:01.687: [ DISKMON][5366:3010325392] dskm_slave_thrd_main1: slave 0 running
2014-05-18 22:18:01.687: [ DISKMON][5366:3010325392] dskm_proc_identify8: client orarootagent/5264, diskmon kgzm 2.0, slave 0, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
2014-05-18 22:18:01.687: [ DISKMON][5366:3010325392] dskm_send_version1:
2014-05-18 22:18:01.691: [ DISKMON][5366:3010325392] dskm_send_version4: done
2014-05-18 22:18:01.691: [ DISKMON][5366:3010325392] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
2014-05-18 22:18:22.996: [ DISKMON][5366:3031305104] dskm_clss_ini8: calling clsssattrib
2014-05-18 22:18:22.996: [ DISKMON][5366:3031305104] dskm_clss_ini11: calling clssnsqname
2014-05-18 22:18:22.998: [ DISKMON][5366:3031305104] dskm_clss_ini13: calling clsssattrib
2014-05-18 22:18:22.998: [ DISKMON][5366:3031305104] dskm_clss_ini15: calling clssgsregnodegrp
2014-05-18 22:18:23.002: [ DISKMON][5
Hi Bharat,
Did you make any changes to the network configuration?
Anju Garg
yes,
i enter SCAN IP in DNS servers…………
Please check all your DNS entries carefully (including the ones for private interconnect)
Regards
Anju Garg
Hi Anju,
Thanks a million…for this wonderful blog. I am really able to utilize your blog for my oracle practice.
Thanks,
manju
Thanks Manju for your time and feedback!
Your comments and suggestions are always welcome.
Regards
Anju Garg