RAC 12.2, part 2

Well, today was an interesting day. My ASM, configured with so much effort, stopped working:

root@rac1 grid]# . ~grid/.bashrc
[root@rac1 grid]# crsctl start ohasd
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
[root@rac1 grid]# crsctl start cluster
CRS-2672: Attempting to start 'ora.crf' on 'rac1'
CRS-2672: Attempting to start 'ora.storage' on 'rac1'
CRS-2676: Start of 'ora.crf' on 'rac1' succeeded
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: OperatCRS-2674: Start of 'ora.storage' on 'rac1' failed
CRS-2679: Attempting to clean 'ora.storage' on 'rac1'
CRS-2681: Clean of 'ora.storage' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-4000: Command Start failed, or completed with errors.
[root@rac1 grid]#

After digging through the log files, I found the following message:

ORA-27302: failure occurred at: sskgpreset1
 2017-08-27 18:26:16.056 : OCRRAW:1181941824: -- trace dump on error exit --
 2017-08-27 18:26:16.056 : OCRRAW:1181941824: Error [kgfoAl06] in [kgfokge] at kgfo.c:3083
 2017-08-27 18:26:16.056 : OCRRAW:1181941824: ORA-27140: attach to post/wait facility failed
 ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
 ORA-27301: OS failure message: Operat
 2017-08-27 18:26:16.056 : OCRRAW:1181941824: Category: 7
 2017-08-27 18:26:16.056 : OCRRAW:1181941824: DepInfo: 27140
 2017-08-27 18:26:16.057 : OCRRAW:1181941824: -- trace dump end --
 2017-08-27 18:26:16.057 : OCRASM:1181941824: proprasmo: kgfoCheckMount returned [7]
 2017-08-27 18:26:16.057 : OCRASM:1181941824: proprasmo: The ASM instance is down

There is a problem with connecting to ASM instance, which is up and running:

[grid@rac1 ~]$ ORACLE_SID="+ASM1"
[grid@rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Sun Aug 27 18:29:48 2017

Copyright (c) 1982, 2016, Oracle. All rights reserved.




Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select name,state from v$asm_diskgroup;

NAME STATE
------------------------------ -----------
ACFS MOUNTED
MGMT MOUNTED
OCR MOUNTED

Now, I was at a loss as to why CRS could not connect to the ASM instance. The user “grid” has all necessary groups:

[grid@rac1 ~]$ id
uid=1000(grid) gid=1000(grid) groups=1000(grid),10(wheel),984(vboxsf),54321(oinstall),54331(asmadmin),54332(asmdba)

The content of the $GRID_HOME/rdbms/lib/config.c looks like this:

/*
 * Assembler will not parse a file past the .end directive
 */
#endif

#define SS_DBA_GRP "asmdba"
#define SS_OPER_GRP ""
#define SS_ASM_GRP "asmadmin"
#define SS_BKP_GRP "asmadmin"
#define SS_DGD_GRP "asmadmin"
#define SS_KMT_GRP "asmadmin"
#define SS_RAC_GRP "asmadmin"

User “grid” is the member of all the groups listed above. So, let’s see what group is the ASM instance running as?

[root@rac1 lib]# ps -G asmdba
 PID TTY TIME CMD
[root@rac1 lib]# ps -G asmadmin
 PID TTY TIME CMD
[root@rac1 lib]# ps -G grid
 PID TTY TIME CMD
 2401 ? 00:00:00 sshd
 2402 pts/0 00:00:00 bash
 2576 ? 00:00:04 oraagent.bin
 2595 ? 00:00:01 mdnsd.bin
 2597 ? 00:00:05 evmd.bin
 2646 ? 00:00:02 gpnpd.bin
 2682 ? 00:00:01 evmlogger.bin
 2695 ? 00:00:04 gipcd.bin
 2940 ? 00:00:06 ocssd.bin
 3121 ? 00:00:00 asm_pmon_+asm1
 3127 ? 00:00:00 asm_clmn_+asm1
 3130 ? 00:00:00 asm_psp0_+asm1
 3133 ? 00:00:00 asm_vktm_+asm1
 3137 ? 00:00:01 asm_gen0_+asm1
 3141 ? 00:00:00 asm_mman_+asm1
 3143 ? 00:00:00 asm_scmn_+asm1
 3147 ? 00:00:01 asm_diag_+asm1
 3149 ? 00:00:00 asm_ping_+asm1
 3151 ? 00:00:00 asm_pman_+asm1
 3153 ? 00:00:02 asm_dia0_+asm1
 3155 ? 00:00:01 asm_lmon_+asm1
 3157 ? 00:00:01 asm_lmd0_+asm1
 3159 ? 00:00:03 asm_lms0_+asm1
 3163 ? 00:00:01 asm_lmhb_+asm1
 3165 ? 00:00:00 asm_lck1_+asm1
 3167 ? 00:00:00 asm_dbw0_+asm1
 3169 ? 00:00:00 asm_lgwr_+asm1
 3171 ? 00:00:00 asm_ckpt_+asm1
 3173 ? 00:00:00 asm_smon_+asm1
 3175 ? 00:00:00 asm_lreg_+asm1
 3177 ? 00:00:00 asm_pxmn_+asm1
 3179 ? 00:00:00 asm_rbal_+asm1
 3181 ? 00:00:00 asm_gmon_+asm1
 3183 ? 00:00:00 asm_mmon_+asm1
 3185 ? 00:00:00 asm_mmnl_+asm1
 3187 ? 00:00:02 asm_imr0_+asm1
 3189 ? 00:00:00 asm_lck0_+asm1
 3195 ? 00:00:02 asm_gcr0_+asm1
 6972 ? 00:00:00 sshd
 6973 pts/1 00:00:00 bash
 9317 pts/1 00:00:00 adrci
 9907 ? 00:00:00 sshd
 9908 pts/2 00:00:00 bash
10610 pts/2 00:00:00 perl
10646 ? 00:00:00 oracle_10646_+a
[root@rac1 lib]#

So, the ASM instance was running as a group “grid”, but the SYSASM group was asmadmin. I did make a configuration mistake by leaving the default group “grid” as the primary  group for user “grid”, so the database was started as “grid,grid”.  The first possible solution was to change the primary group for the user “grid” and hope for the best. If that fails, I would have to re-install.  The trick that got everything back up was to allow group access to the dba group:

SQL> show parameter allow_group

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
allow_group_access_to_sga boolean TRUE
SQL>

After setting that in the SPFILE for a single ASM instance, all was well:

[grid@rac1 ~]$ srvctl status filesystem
ACFS file system /oradata is mounted on nodes rac1,rac2
[grid@rac1 ~]$

Installing Oracle RDBMS after that was a breeze. This parameter is new in Oracle 12.2 and is set to “FALSE” by default. It increases security, so much that the clusterware doesn’t start, which prevents the SCOTT tables from falling into the enemy hands. Of course,  I should have heeded the warning duly shown by cluvfy, but I was lazy. However, I learned a lot.

Advertisements

About mgogala

I am a consultant with an extensive Oracle DBA experience. I have worked on very large databases. I have worked with OPS and RAC since its inception.I am also a published book writer, having published two books about PHP. This blog is about the challenges and adventures in my professional life. Sorry, no family pictures here.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s