Recently, I had trouble starting ASM proxy instance on one of the nodes in my 2 node flex cluster having nodes host01 and host02. As a result I could not access the volume I created on an ASM diskgroup. This post explains how I resolved it.
While connected to host01, I created a volume VOL1 on DATA diskgroup with corresponding volume device /dev/asm/vol1-106 .
[grid@host01 root]$ asmcmd volcreate -G DATA -s 300m VOL1 [grid@host01 root]$ asmcmd volinfo -G DATA VOL1 Diskgroup Name: DATA Volume Name: VOL1 Volume Device: /dev/asm/vol1-106 State: ENABLED Size (MB): 320 Resize Unit (MB): 32 Redundancy: MIRROR Stripe Columns: 4 Stripe Width (K): 128 Usage: ACFS Mountpath:
I created ACFS file system on the newly created volume
[root@host01 ~]# mkfs -t acfs /dev/asm/vol1-106
I also created corresponding mount point /mnt/acfsmounts/acfs1 on both the nodes in the cluster.
root@host01 ~]# mkdir -p /mnt/acfsmounts/acfs1 root@host02 ~]# mkdir -p /mnt/acfsmounts/acfs1
When I tried to mount the volume device, I could mount the volume device on host01 but not on host02 .
[root@host01 ~]#mount -t acfs /dev/asm/vol1-106 /mnt/acfsmounts/acfs1
[root@host01 ~]# mount | grep vol1
/dev/asm/vol1-106 on /mnt/acfsmounts/acfs1 type acfs (rw)
[root@host02 ~]# mount -t acfs /dev/asm/vol1-106 /mnt/acfsmounts/acfs1
mount.acfs: CLSU-00100: Operating System function: open64 failed with error data: 2
mount.acfs: CLSU-00101: Operating System error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/vol1-106)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/vol1-106. Verify the volume exists.
The corresponding volume device was visible on host01 but not on host02
[root@host01 ~]# cd /dev/asm [root@host01 asm]# ls vol1-106 [root@host02 ~]# cd /dev/asm [root@host02 asm]# ls
Since ADVM / ACFS utilize an ASM Proxy instance in a flex cluster to access metadata from a local / remote ASM instance , I checked whether ASM Proxy instance was running on both the nodes and realized that whereas ASM Proxy instance was running on host01, it was not running on host02
[root@host01 ~]# ps -elf | grep pmon | grep APX
0 S grid 27782 1 0 78 0 – 350502 – 10:09 ? 00:00:00 apx_pmon_+APX1
[root@host02 asm]# ps -elf | grep pmon | grep APX
[root@host01 ~]# srvctl status asm -proxy
ADVM proxy is running on node host01
[root@host01 ~]# crsctl stat res ora.proxy_advm -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.proxy_advm
ONLINE ONLINE host01 STABLE
ONLINE OFFLINE host02 STABLE
I tried to start ASM proxy instance manually on host02
[grid@host02 ~]$ . oraenv ORACLE_SID = [grid] ? +APX2 The Oracle base has been set to /u01/app/grid [grid@host02 ~]$ sqlplus / as sysasm SQL*Plus: Release 12.1.0.1.0 Production on Sat May 2 10:31:45 2015 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORA-00099: warning: no parameter file specified for ASMPROXY instance ORA-00443: background process "VUBG" did not start SQL> ho oerr ORA 00443 00443, 00000, "background process \"%s\" did not start" // *Cause: The specified process did not start. // *Action: Ensure that the executable image is in the correct place with // the correct protections, and that there is enough memory.
I checked the memory allocated to VM for host02 – It was 1.5 GB as against 2.5 GB assigned to VM for host01. I increased the memory of host02 to 2.5 GB and ASM proxy instance started automatically.
[root@host01 ~]# crsctl stat res ora.proxy_advm -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.proxy_advm ONLINE ONLINE host01 STABLE ONLINE ONLINE host02 STABLE
Hope it helps!
References:
Oracle documentation
———————————————————————————————————
Related Links :
12c RAC: ORA-15477: cannot communicate with the volume driver