Category Archives: 11g R2 RAC

ORA-39070: Unable to open the log file

I received this error message when I was trying to perform a data pump export of SH schema in parallel in a RAC database. I proceeded as follows:

Current scenario:
Name of the cluster: cluster01
Number of nodes : 3 (host01, host02, host03)
RAC Database version: 11.2.0.3
Name of RAC database : orcl
Number of instances : 3

  • Created a directory object  pointing to shared storage which is accessible by all the three instances of the database
SQL>drop directory dp_shared_dir;
SQL>create directory DP_SHARED_DIR as '+DATA/orcl/';
SQL>grant read, write on directory dp_shared_dir to public;
  • Issued the command to export SH schema in parallel across all active Oracle RAC instances with parallelism = 6 which resulted in error ORA-39070
[oracle@host01 root]$ expdp system/oracle@orcl schemas=sh directory=dp_shared_dir parallel=6 cluster=y dumpfile='expsh%U.dmp' reuse_dumpfiles=y

Export: Release 11.2.0.3.0 - Production on Tue Dec 8 14:45:39 2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
 With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
 Data Mining and Real Application Testing options
 ORA-39002: invalid operation
 ORA-39070: Unable to open the log file.
 ORA-29283: invalid file operation
 ORA-06512: at "SYS.UTL_FILE", line 536
 ORA-29283: invalid file operation

Cause:
The error message indicates that Log file cannot be opened. Since directory parameter points to a shared location on an ASM disk group and log file is not supported on it, I received the above error.

Solution:
I modified my command and explicitly specified log file to be created on  local file system pointed to by the directory object DATA_PUMP_DIR. Subsequently, export was performed successfully.

[oracle@host01 root]$ expdp system/oracle@orcl schemas=sh directory=dp_shared_dir parallel=6 cluster=y logfile=data_pump_dir:expsh.log dumpfile='expsh%U.dmp' reuse_dumpfiles=y

Export: Release 11.2.0.3.0 - Production on Tue Dec 8 15:14:11 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
Starting "SYSTEM"."SYS_EXPORT_SCHEMA_10": system/********@orcl schemas=sh directory=dp_shared_dir parallel=6 cluster=y logfile=data_pump_dir:expsh.log dumpfile=expsh%U.dmp reuse_dumpfiles=y
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 273.8 MB
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
.....
.....
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type SCHEMA_EXPORT/TABLE/INDEX/DOMAIN_INDEX/INDEX
Processing object type SCHEMA_EXPORT/MATERIALIZED_VIEW
Processing object type SCHEMA_EXPORT/DIMENSION
Master table "SYSTEM"."SYS_EXPORT_SCHEMA_10" successfully loaded/unloaded
******************************************************************************
Dump file set for SYSTEM.SYS_EXPORT_SCHEMA_10 is:
+DATA/orcl/expsh01.dmp
+DATA/orcl/expsh02.dmp
+DATA/orcl/expsh03.dmp
+DATA/orcl/expsh04.dmp
+DATA/orcl/expsh05.dmp
+DATA/orcl/expsh06.dmp
Job "SYSTEM"."SYS_EXPORT_SCHEMA_10" successfully completed at 15:20:49

I hope it helps!!!

—————————————————————————————————————-

Related links:  

SERVICE CREATED USING EM DOES NOT UPDATE DATA DICTIONARY OR OCR

Recently, I discovered a bug in EM. If we create a database service using EM, it updates TAF settings (Failover type, failover method, retries and delay) only in tnsnames.ora on all the nodes . Data dictionary and OCR are not updated.

Current scenario:

Name of RAC database : orcl
No. of Nodes  : 2
Names of nodes : host01, host02

— Create database service orcl_em using EM --

Name of service orcl_em

Attributes :

Start service after creation : Yes
Update local naming parameter (tnsnames.ora) file  : Yes
preferred instance : orcl1
Available instance : orcl2
TAF policy         : Basic

– After the service has been created, check that service is running on preferred instance orcl1

[oracle@host01 ~]$ srvctl status service -s orcl_em -d orcl

Service orcl_em is running on instance(s) orcl1

– check that the tnsnames.ora entry contains TAF settings as specified :

orcl_em =
(DESCRIPTION =
(ADDRESS =
(PROTOCOL = TCP)
(HOST = cluster01-scan.cluster01.example.com)
(PORT = 1521))
(LOAD_BALANCE = YES)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl_em)
  (FAILOVER_MODE =
         (TYPE = SELECT)
         (METHOD = BASIC)
         (RETRIES = 180)
         (DELAY = 5))))

– check that data dictionary view is not aware of TAF settings

SQL> col name for a15
col failover_type for a15
col failover_method for a15

select name, FAILOVER_METHOD, FAILOVER_TYPE, FAILOVER_RETRIES, FAILOVER_DELAY
from dba_services
where upper(name) = 'ORCL_EM';

NAME            FAILOVER_TYPE   FAILOVER_METHOD FAILOVER_RETRIES FAILOVER_DELAY
--------------- --------------- --------------- ---------------- --------------
orcl_em

– check that the resource profile as stored in OCR also does not contain any information about TAF settings

[oracle@host01 admin]$ srvctl config service -s orcl_em -d orcl
Service name: orcl_em
Service is enabled
Server pool: orcl_orcl_em
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: BASIC
Edition:
Preferred instances: orcl1
Available instances: orcl2

It can be clearly seen that TAF attributes are updated only in tnsnames.ora . They have not been updated in data dictionary or OCR.

– Now let us create and start another database service orcl_manual using srvctl

[oracle@host01 admin]$ srvctl add service -s orcl_manual -r orcl1 -a orcl2 -m BASIC -e SELECT -w 100 -z 5

[oracle@host01 admin]$ srvctl start service -s orcl_manual -d orcl

– Check that TAF attributes have been updated in OCR

[oracle@host01 admin]$ srvctl config service -s orcl_manual -d orclService name: orcl_manual
Service is enabled
Server pool: orcl_orcl_manual
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: SELECT
Failover method: BASIC
TAF failover retries: 5
TAF failover delay: 100
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: orcl1
Available instances: orcl2

– Check that TAF settings have been updated in data dictionary too

SQL> select name, failover_type, failover_method, failover_retries, failover_delay
from dba_services
where upper(name) = 'ORCL_MANUAL';

NAME            FAILOVER_TYPE   FAILOVER_METHOD FAILOVER_RETRIES FAILOVER_DELAY
--------------- --------------- --------------- ---------------- --------------
orcl_manual     SELECT          BASIC                          5            100

Hence, database services for RAC database should always be created using srvctl and tnsnames.ora entry should be  added manually.

——————————————————————————————

Related Links:

Home

11g R2 RAC Index

11g R2 RAC :Services
Tracing Services in RAC

 

 

 

 

Job Failover In RAC

In RAC, if a node crashes, a jobs running on that node can automatically failover to another node provided the job has been mapped to a service by means of job class.

Let’s demonstrate this concept.

Current scenario:

Name of the cluster : cluster01.example.com
No. of nodes : 2 (host01, host02)
Name of the database : orcl

– Create  demo table hr.test 

 SQL> create table hr.test(dt date);

– Create a  database service job_serv with

. preferred instance : orcl1
. available instance : orcl2

[oracle@host01 ~]$ srvctl add service -s job_srv -d orcl -a orcl2 -r orcl1
[oracle@host01 ~]$ srvctl status service -d orcl
Service job_serv is running on instance(s) orcl1
Service job_srv is not running.

preferred node : host01, available node host02

– Create tnsnames.ora entry for service job_serv

job_srv = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = cluster01-scan.cluster01.example.com)(PORT = 1521))(LOAD_BALANCE = YES)(CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = job_srv))

–  Start the service and check that service is running on host01 (preferred instance)

[oracle@host01 ~]$ srvctl start service -d orcl
CRS-5702: Resource 'ora.orcl.job_serv.svc' is already running on 'host01'

[oracle@host01 ~]$ srvctl status service -d orcl
Service job_srv is running on instance(s) orcl1

– Create job class JC1 which maps to the service job_serv

SQL>BEGIN
sys.dbms_scheduler.create_job_class(
logging_level => DBMS_SCHEDULER.LOGGING_RUNS,
service => 'job_srv',
job_class_name => '"JC1"');
END;
/

– Create a job JOB1 belonging to job class JC1 which inserts a row in table hr.test every 3  seconds

SQL>BEGIN
sys.dbms_scheduler.create_job(
job_name => '"SYS"."JOB1"',
job_type => 'PLSQL_BLOCK',
job_action => 'begin
insert into hr.test values (sysdate);
end;',
repeat_interval => 'FREQ=SECONDLY;INTERVAL=3',
start_date => systimestamp at time zone 'Asia/Calcutta',
job_class => '"JC1"',
auto_drop => FALSE,
enabled => TRUE);
END;
/

– Check that  job is running i.e. rows are getting inserted  in table hr.test 

SQL> select to_char(dt, 'dd-mon-yyyy hh24:mi:ss') from hr.test;

TO_CHAR(DT,'DD-MON-YYYYHH24:M
-----------------------------
11-jan-2014 04:12:57
11-jan-2014 04:12:47
11-jan-2014 04:12:49

...

11-jan-2014 04:13:27
11-jan-2014 04:13:30
11-jan-2014 04:13:33

– Check that job is running  the on same node as the  service job_serv i.e. host01 

SQL> col job_name for a7
 col status for a15 
select JOB_NAME, status , INSTANCE_ID from ALL_SCHEDULER_JOB_RUN_DETAILS where job_name = 'JOB1';
 JOB_NAM STATUS          INSTANCE_ID 
------- --------------- ----------- 
JOB1    SUCCEEDED                 1 
JOB1    SUCCEEDED                 1 
JOB1    SUCCEEDED                 1
..

– To simulate the crash of host01, kill the database instance on host01

[root@host01 ~]# ps -ef |grep pmon_
grid      5493     1  0 03:43 ?        00:00:00 asm_pmon_+ASM1
oracle    6755     1  0 03:47 ?        00:00:00 ora_pmon_orcl1
root     25051  6336  0 04:17 pts/1    00:00:00 grep pmon_
[root@host01 ~]# kill -9 6755

– Check that job has failed over to host02

[oracle@host01 ~]$ srvctl status service -d orcl
Service job_srv is not running.
[oracle@host01 ~]$ srvctl status service -d orcl
Service job_srv is running on instance(s) orcl2

select JOB_NAME, status , INSTANCE_ID from ALL_SCHEDULER_JOB_RUN_DETAILS where job_name = 'JOB1' ;

JOB_NAM STATUS          INSTANCE_ID
------- --------------- -----------
JOB1    SUCCEEDED                 1
JOB1    SUCCEEDED                 1
JOB1    SUCCEEDED                 1
...

JOB1    SUCCEEDED                 2
JOB1    SUCCEEDED                 2
JOB1    SUCCEEDED                 2

– Check that the job is still running i.e. records are still getting inserted in the table hr.test

SQL> select to_char(dt, 'dd-mon-yyyy hh24:mi:ss') from hr.test;

TO_CHAR(DT,'DD-MON-YYYYHH24:M
-----------------------------
11-jan-2014 04:12:57
11-jan-2014 04:12:47
11-jan-2014 04:12:49

...

11-jan-2014 04:15:27
11-jan-2014 04:15:30
11-jan-2014 04:15:33
I hope this post was useful.
Your comments and sugegstions are always welcome.
————————————————————————————————————————
Related links:

Home
11G R2 RAC Index

11g R2 RAC : Services

 

—————————-

VIP FAILOVER DEMONSTRATION

Recently, a student of mine wanted me to demonstrate failover of VIP.

Here is the demonstration:

Current scenario:

2 node 11.2.0.1 RAC cluster with SCAN and GNS

nodes : host01, host02

Overview:

– check that VIP’s of both the nodes are running on respective nodes
— Stop clusterware on host01
— check that VIP of host01  has failed over to host02
— Trace route of request to connect  to VIP of host01 and verify that connection is not made as listener on host02 accepts requests    to its own VIP only
— Trace route of request to connect  to VIP of host02 and verify  that connection is made  as listener on host02 accepts requests    to host02’s VIP
— start clusterware on host01
— check that host01 vip is running on host01 again
— Trace route of request to connect  to VIP of host01 and verify that connection is made  as listener on host01 accepts requests   to its own  VIP

Implementation :

- check that VIP’s of both the nodes are running on respective nodes

[root@host02 ~]# srvctl status nodeapps -n host01

VIP 192.9.201.247 is enabled
VIP 192.9.201.247 is running on node: host01
Network is enabled
Network is running on node: host01

[root@host02 ~]# srvctl status nodeapps -n host02

VIP 192.9.201.235 is enabled
VIP 192.9.201.235 is running on node: host02
Network is enabled
Network is running on node: host02

- Stop clusterware on host01

[root@host01 ~]# crsctl stop crs

- check that VIP of host01 (192.9.201.247) has failed over to host02

[root@host02 ~]# srvctl status nodeapps -n host01

VIP 192.9.201.247 is enabled
VIP 192.9.201.247 is running on node: host02
Network is enabled
Network is not running on node: host01

– Trace route of request to connect  to VIP of host01 i.e. 192.9.201.147
– It can be seen there is no listener as listener on host02 accepts requests    to host02’s VIP (192.9.201.235) only

[root@host02 ~]# trcroute 192.9.201.247

Route of TrcRoute:
------------------

Node: Client            Time and address of entry into node:
-------------------------------------------------------------
08-JAN-2014 14:14:39 ADDRESS= PROTOCOL=TCP  HOST=192.9.201.247  PORT=1521

TNS-12224: TNS:no listener
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-03601: Failed in route information collection

– Trace route of request to connect  to VIP of host02 i.e. 192.9.201.235
– It can be seen that connection is made  as listener on host02 accepts requests    to host02’s VIP

[root@host02 ~]# trcroute 192.9.201.235

Route of TrcRoute:
------------------

Node: Client            Time and address of entry into node:
-------------------------------------------------------------
08-JAN-2014 14:14:51 ADDRESS= PROTOCOL=TCP  HOST=192.9.201.235  PORT=1521

Node: Server            Time and address of entry into node:
-------------------------------------------------------------
08-JAN-2014 14:14:52 ADDRESS= PROTOCOL=TCP  HOST=192.9.201.235  PORT=1521

– start clusterware on host01

[root@host01 ~]# crsctl start crs

– check that host01 vip (192.9.201.147) is running on host01 again

[root@host02 ~]# srvctl status nodeapps -n host01

VIP 192.9.201.247 is enabled
VIP 192.9.201.247 is running on node: host01
Network is enabled
Network is running on node: host01

– Trace route of request to connect  to VIP of host01 i.e. 192.9.201.247
– It can be seen that connection is made  as listener on host01 accepts requests   to its own  VIP

[root@host02 ~]# trcroute 192.9.201.247

Route of TrcRoute:
------------------

Node: Client            Time and address of entry into node:
-------------------------------------------------------------
08-JAN-2014 14:23:00 ADDRESS= PROTOCOL=TCP  HOST=192.9.201.247  PORT=1521

Node: Server            Time and address of entry into node:
-------------------------------------------------------------
08-JAN-2014 14:23:00 ADDRESS= PROTOCOL=TCP  HOST=192.9.201.247  PORT=1521

I hope this post was useful.

Your comments and suggestions are always welcome.
—————————————————————————————-

Related LInks:

Home

11g R2 RAC INDEX

Need For VIP In RAC

————————————————————————————–

 

11g R2 RAC: REBOOT-LESS FENCING WITH MISSING DISK HEARTBEAT

In my earlier post, I had discussed about reboot-less node fencing , a new feature introduced since 11.2.0.2. In this post, I will demonstrate reboot-less node fencing when disk heartbeat is lost.

– Check that clusterware version is 11.2.0.3

[root@host02 ~]# crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]

– check that both the nodes in the cluster are active

[root@host02 ~]# olsnodes -s
host01  Active
host02  Active

– Stop ISCSI service on node2

[root@host02 ~]# service iscsi stop
Logging out of session [sid: 1, target: iqn.2006-01.com.openfiler:tsn.e55ea88d0212, portal: 

192.9.201.182,3260]
Logout of [sid: 1, target: iqn.2006-01.com.openfiler:tsn.e55ea88d0212, portal: 192.9.201.182,3260]: 

successful
Stopping iSCSI daemon:

– Alert log of node2 –

– Note that instead of rebooting the node, CRSD resources are cleaned up

[cssd(2876)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK013; details at (:CSSNM00059:) 

...

[cssd(2876)]CRS-1606:The number of voting files available, 0, is less than the minimum number of 

voting files required, 1, resulting in CSSD termination to ensure data integrity; 

[cssd(2876)]CRS-1656:The CSS daemon is terminating due to a fatal error; 

[cssd(2876)]CRS-1652:Starting clean up of CRSD resources.
2013-10-09 11:04:30.795

...

[cssd(2876)]CRS-1654:Clean up of CRSD resources finished successfully.
2013-10-09 11:04:31.914

— Check that OHAS service is still up on host02

[root@host02 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

– Check that resources cssd , crsd and HAIP are down on host02[

[root@host02 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE                                                   
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE                                                   
ora.crf
      1        ONLINE  ONLINE       host02                                       
ora.crsd
      1        ONLINE  OFFLINE                                                   
ora.cssd
      1        ONLINE  OFFLINE                               STARTING            
ora.cssdmonitor
      1        ONLINE  ONLINE       host02                                       
ora.ctssd
      1        ONLINE  OFFLINE                                                   
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.drivers.acfs
      1        ONLINE  ONLINE       host02                                       
ora.evmd
      1        ONLINE  OFFLINE                                                   
ora.gipcd
      1        ONLINE  ONLINE       host02                                       
ora.gpnpd
      1        ONLINE  ONLINE       host02                                       
ora.mdnsd
      1        ONLINE  ONLINE       host02

–Check that host02 is no longer a part of the cluster

[root@host01 cluster01]# olsnodes -s
host01  Active
host02  Inactive

– Restart ISCSI service on host02

[root@host02 ~]# service iscsi start
iscsid dead but pid file exists
Turning off network shutdown. 

Starting iSCSI daemon:                                     [  OK  ]
                                                           [  OK  ]
Setting up iSCSI targets: Logging in to [iface: default, target: iqn.2006-

01.com.openfiler:tsn.e55ea88d0212, portal: 192.9.201.182,3260]
Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.e55ea88d0212, portal: 

192.9.201.182,3260]: successful
                                                           [  OK  ]

- Alert log of host02

– Note that as soon as ISCSI service is started, CSSD service starts immediately and host02 joins the cluster

[cssd(5481)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details 

...

2013-10-09 11:10:43.897
[cssd(5481)]CRS-1707:Lease acquisition for node host02 number 2 completed

2013-10-09 11:10:47.629
[cssd(5481)]CRS-1605:CSSD voting file is online: ORCL:ASMDISK013; details in 

/u01/app/11.2.0/grid/log/host02/cssd/ocssd.log.

2013-10-09 11:10:54.652
[cssd(5481)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01 host02 .

– check that resources haip, cssd and crsd have started on host02

[root@host02 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       host02                   Started             
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       host02                                       
ora.crf
      1        ONLINE  ONLINE       host02                                       
ora.crsd
      1        ONLINE  ONLINE       host02                                       
ora.cssd
      1        ONLINE  ONLINE       host02                                       
ora.cssdmonitor
      1        ONLINE  ONLINE       host02                                       
ora.ctssd
      1        ONLINE  ONLINE       host02                   OBSERVER            
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.drivers.acfs
      1        ONLINE  ONLINE       host02                                       
ora.evmd
      1        ONLINE  ONLINE       host02                                       
ora.gipcd
      1        ONLINE  ONLINE       host02                                       
ora.gpnpd
      1        ONLINE  ONLINE       host02                                       
ora.mdnsd
      1        ONLINE  ONLINE       host02

– Check that host02 has joined the cluster

[root@host02 ~]# olsnodes -s
host01  Active
host02  Active

References:

http://ora-ssn.blogspot.in/2011/09/reboot-less-node-fencing-in-oracle.html
http://www.trivadis.com/uploads/tx_cabagdownloadarea/Trivadis_oracle_clusterware_node_fencing_v.pdf
http://www.vmcd.org/2012/03/11gr2-rac-rebootless-node-fencing/

———————————————————————————————

Related Links:

Home

11g R2 RAC Index

11g R2 RAC: Node Eviction Due To Missing Network Heartbeat
11g R2 RAC :Reboot-less Node Fencing
 11g R2 RAC: Reboot-less Fencing With Missing Network Heartbeat

 

————–

11g R2 RAC: REBOOT-LESS FENCING WITH MISSING NETWORK HEARTBEAT

In my earlier post, I had discussed about reboot-less node fencing , a new feature introduced since 11.2.0.2. In this post, I will demonstrate reboot-less node fencing when network heartbeat is lost.

– Check that clusterware version is 11.2.0.3

[root@host02 ~]# crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]

– check that both the nodes in the cluster are active

[root@host02 ~]# olsnodes -s
host01  Active
host02  Active

– Find out pvt interconnect

[root@host02 ~]# oifcfg getif
eth0  192.9.201.0  global  public
eth1  10.0.0.0  global  cluster_interconnect

– Stop pvt interconnect on node2

[root@host02 ~]# ifdown eth1

– Alert log of node2 –

– Note that instead of rebooting the node, CRSD resources are cleaned up

[cssd(802)]CRS-1612:Network communication with node host01 (1) missing for 50% of timeout interval.
..
Removal of this node from cluster in 6.640 seconds
2013-10-09 10:45:53.924
..
[cssd(802)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going  down to preserve cluster integrity;

[cssd(802)]CRS-1656:The CSS daemon is terminating due to a fatal error; ..
[cssd(802)]CRS-1652:Starting clean up of CRSD resources.
2013-10-09 10:46:01.918
...

[cssd(802)]CRS-1654:Clean up of CRSD resources finished successfully.
2013-10-09 10:46:03.794

— Check that OHAS service  is still up on host02

[root@host02 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

– Check that resources cssd , crsd and HAIP are down on host02[

[root@host02 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1        ONLINE  OFFLINE
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE              
ora.crf
1        ONLINE  ONLINE       host02
ora.crsd
      1        ONLINE  OFFLINE      
ora.cssd
      1        ONLINE  OFFLINE                               STARTING
ora.cssdmonitor
1        ONLINE  ONLINE       host02
ora.ctssd
1        ONLINE  OFFLINE
ora.diskmon
1        OFFLINE OFFLINE
ora.drivers.acfs
1        ONLINE  ONLINE       host02
ora.evmd
1        ONLINE  INTERMEDIATE host02
ora.gipcd
1        ONLINE  ONLINE       host02
ora.gpnpd
1        ONLINE  ONLINE       host02
ora.mdnsd
1        ONLINE  ONLINE       host02

– Restart private interconnect on host02

[root@host02 ~]# ifup eth1

- Alert log of host02

– Note that as soon as private network is started, CSSD, CRSD and EVMD services start immediately and host02 joins the cluster

[cssd(2876)]CRS-1713:CSSD daemon is started in clustered mode
2013-10-09 10:47:04.944
...
[cssd(2876)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01 host02 .
2013-10-09 10:55:22.403
..
[crsd(3973)]CRS-1012:The OCR service started on node host02.
2013-10-09 10:56:25.304
...
[evmd(2753)]CRS-1401:EVMD started on node host02.
2013-10-09 10:56:41.996
...
[crsd(3973)]CRS-1201:CRSD started on node host02.
2013-10-09 10:56:45.274

– check that resources haip, cssd and crsd have started on host02

[root@host02 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1        ONLINE  ONLINE       host02                   Started
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       host02  
ora.crf
1        ONLINE  ONLINE       host02
ora.crsd
      1        ONLINE  ONLINE       host02                                       
ora.cssd
      1        ONLINE  ONLINE       host02        
ora.cssdmonitor
1        ONLINE  ONLINE       host02
ora.ctssd
1        ONLINE  ONLINE       host02                   OBSERVER
ora.diskmon
1        OFFLINE OFFLINE
ora.drivers.acfs
1        ONLINE  ONLINE       host02
ora.evmd
1        ONLINE  ONLINE       host02
ora.gipcd
1        ONLINE  ONLINE       host02
ora.gpnpd
1        ONLINE  ONLINE       host02
ora.mdnsd
1        ONLINE  ONLINE       host02

– Check that host02 has joined the cluster

[root@host02 ~]# olsnodes -s
host01  Active
host02  Active

References:

http://ora-ssn.blogspot.in/2011/09/reboot-less-node-fencing-in-oracle.html
http://www.trivadis.com/uploads/tx_cabagdownloadarea/Trivadis_oracle_clusterware_node_fencing_v.pdf
http://www.vmcd.org/2012/03/11gr2-rac-rebootless-node-fencing/

———————————————————————————————

Related Links:

Home

11g R2 RAC Index

11g R2 RAC: Node Eviction Due To Missing Network Heartbeat 
11g R2 RAC :Reboot-less Node Fencing
11g R2 RAC :Reboot-less  Fencing With Missing Disk Heartbeat

 

————–

11g R2 RAC : REBOOT-LESS NODE FENCING

Prior to 11.2.0.2, during failures of certain Oracle RAC-required subcomponents (e.g. private interconnect, voting disk etc.) , Oracle Clusterware tried to prevent a split-brain with a fast reboot of the server(s) without waiting for ongoing I/O operations or synchronization of the file systems. As a result, non-cluster-aware applications would be forcibly shut down. Moreover, during reboots, resources need to re-mastered across the surviving nodes . In a big cluster with many numbers of nodes, this can be potentially a very expensive operation.

This mechanism has been changed in version 11.2.0.2 (first 11g Release 2 patch set).

After deciding which node to evict,

– the clusterware will attempt to clean up the failure within the cluster by killing only the offending process(es) on that node . Especially I/O generating processes are killed .

– If all oracle resources/processes can be stopped and all IO generating processes can be killed,

  • clusterware resources will stop on the node
  • Oracle High Availability Services Daemon will keep on trying to restart the  Cluster Ready Services (CRS) stack again.
  • Once the conditions to start  CRS stack are re-established, all relevant cluster resources on that node will automatically start.

– If, for some reason, not all resources can be stopped or IO generating processes cannot be stopped completely (hanging in kernel mode, I/O path, etc.) ,

  • Oracle Clusterware will still perform a reboot or use IPMI to forcibly evict the node from the cluster as earlier.

This behavior change is particularly useful for non-cluster aware applications as the data will be protected by shutting down the cluster only on the node without rebooting the node itself.

I will demonstrate this functionality in two scenarios :

Failure of network heartbeat
Failure of DISK heartbeat

References:

http://ora-ssn.blogspot.in/2011/09/reboot-less-node-fencing-in-oracle.html
http://www.trivadis.com/uploads/tx_cabagdownloadarea/Trivadis_oracle_clusterware_node_fencing_v.pdf
http://www.vmcd.org/2012/03/11gr2-rac-rebootless-node-fencing/
http://www.vitalsofttech.com/grid-11gr2-ipmi-based-failure-isolation/
http://www.orafaq.com/node/2504
http://www.unbreakablecloud.com/wordpress/2011/02/05/oracle-grid-infrastructure-reboot-less-node-fencing/

———————————————————————————————

Related Links:

Home

11g R2 RAC Index

11g R2 RAC: Node Eviction Due To Missing Network Heartbeat 
 11g R2 RAC: Reboot-less Fencing With Missing Network Heartbeat
11g R2 RAC :Reboot-less  Fencing With Missing Disk Heartbeat

————–