PRKP-1037 : Failed to create cluster Database
CRS-0210: Could not find resource
CRS ownership - CRS did not start after node reboot
clsssInitNative: connect failed, rc 9,Received status 3 from CSS
Cluster Split based on the service name
oui-10009
This error occured while adding a new node in the existing cluster. The following workaround is used as follows:
Node 1:
cd $ORACLE_HOME/oui/bin
./addNode.sh -silent "CLUSTER_NEW_NODES={new nodename}" -logLevel trace –debug
You may want to browse "Node add" to get more information.
Node add
PRKP-1037 : Failed to create cluster Database
Step 1:
[oracle@orcl-experts.info bin]$ ./crs_stat
NAME=ora.SAAP.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
Step 2 : crs_unregister ora.SAAP.db
Step 3: srvctl add database -d SAAP -o /data01/app/oracle/product/10.2.0/db/
[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP1 -n sabharwal11
[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP2 -n sabharwal12
[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP3 -n sabharwal13
[oracle@sabharwal11 bin]$ srvctl stop database -d SAAP
[oracle@sabharwal11 bin]$ srvctl start database -d SAAP
[oracle@sabharwal11 bin]$ ./crs_stat
NAME=ora.SAAP.SAAP1.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal11
NAME=ora.SAAP.SAAP2.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal12
NAME=ora.SAAP.SAAP3.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal13
[oracle@sabharwal11 bin]$ srvctl status database -d SAAP
Instance SAAP1 is running on node sabharwal11
Instance SAAP2 is running on node sabharwal12
Instance SAAP3 is running on node sabharwal13
CRS-0210: Could not find resource
[oracle@orcl-experts1.com admin]$ srvctl stop nodeapps -n orcl-experts1.com
CRS-0210: Could not find resource 'ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr'.
[oracle@orcl-experts1.com admin]$ srvctl start nodeapps -n orcl-experts1.com
CRS-0210: Could not find resource ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr.
[oracle@orcl-experts1.com bin]$ ./crs_register ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr
[oracle@orcl-experts1.com bin]$ srvctl start nodeapps -n orcl-experts1.com
[oracle@orcl-experts1.com bin]$ srvctl stop nodeapps -n orcl-experts1.com
[oracle@orcl-experts1.com bin]$ srvctl start nodeapps -n orcl-experts1.com
NO Errors
CRS ownership - CRS did not start after node reboot - Metalink 428195.1
I had to deal with a maintenance which required disabling the auto start of CRS during server reboot:
#/etc/init.d/initcrs disable
I shutdown the RAC 2 node cluster and verified that there is no oracle process running and hand over the servers to Sysadmin. After the hardware maintenance before the 2nd reboot the sysadmin asked me to re enable the CRS auto start. In both nodes I did:
#/etc/init.d/initcrs enable
Sysadmin rebooted both the nodes. After 10 minutes I checked the process details and found that CRS stack is not running and the instances are not up. We had a deadline of 1 hour to bring up all the instances. Under the short time and pressure I started digging out the issue of why the CRS stack was not up. I tried to bring it up manually but it did not work.
I found that after remounting the file systems the permission of /oracle01 mount point (voting file and OCR file location) is:
-rw-r--r-- 1 root root 10240000 Apr 3 21:41 voting.dbf
Sysadmin and I decided to make an immediate change:
chown -R root:install /oracle01
and
we rebooted the machine.
It worked and during the node reboots process all the CRS and Oracle processes came up.
The maintenance was based on the mounting the existing file system on the new LUN.
clsssInitNative: connect failed, rc 9,Received status 3 from CSS. Waiting for good status .., Failure at final check of Oracle CRS stack. 10
Issue: I had issue while bringing up CRS in the 2nd node.The root.sh executions failed with the following message:
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Failure at final check of Oracle CRS stack.
10
crsd.log
2007-10-26 17:33:37.510: [ CSSCLNT][3086960320]clsssInitNative: connect failed, rc 9
2007-10-26 17:33:37.511: [ CRSRTI][3086960320]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2007-10-26 17:33:38.545: [ COMMCRS][116607920]clsc_connect: (0x8605510) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_oracl-experts.info2_crs))
Investigation:
I looked into the files under /var/tmp/.oracle in node1 - Looks everything fine but in the 2nd node , the CRS process was not able to write
Solution:
After long research, I found out that there is a major issue with the /var/tmp
[root@oracl-experts.info2 app]# chmod 1777 /var/tmp
[root@oracl-experts.info2 app]# ls -ld /var/tmp
drwxrwxrwt 3 root root 4096 Oct 26 17:33 /var/tmp
total 8
drwxrwxrwt 3 root root 4096 Oct 26 17:33 ..
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 soracl-experts.info2DBG_CSSD
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sOracle_CSS_LclLstnr_crs_2
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sOCSSD_LL_oracl-experts.info2_crs
srwxrwxrwx 1 root root 0 Oct 26 17:52 sprocr_local_conn_0_PROC
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 soracl-experts.info2DBG_EVMD
srwxrwxrwx 1 root root 0 Oct 26 17:52 soracl-experts.info2DBG_CRSD
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sSYSTEM.evm.acceptor.auth
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sCoracl-experts.info2_crs_evm
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sAoracl-experts.info2_crs_evm
srwxrwxrwx 1 root root 0 Oct 26 17:52 sora_crsqs
srwxrwxrwx 1 root root 0 Oct 26 17:52 sCRSD_UI_SOCKET
drwxrwxrwt 2 root root 4096 Oct 26 17:52 .
[root@oracl-experts.info2 app]# ./root.sh
WARNING: directory '/data01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/data01/crs/oracle/product' is not owned by root
WARNING: directory '/data01/crs/oracle' is not owned by root
WARNING: directory '/data01/crs' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/data01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/data01/crs/oracle/product' is not owned by root
WARNING: directory '/data01/crs/oracle' is not owned by root
WARNING: directory '/data01/crs' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname oracl-experts.info1 for node 1.
assigning default hostname oracl-experts.info2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node
node 1: oracl-experts.info1 oracl-experts.info1-priv oracl-experts.info1
node 2: oracl-experts.info2 oracl-experts.info2-priv oracl-experts.info2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
oracl-experts.info1
oracl-experts.info2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "eth0" is not public. Public interfaces should be used t o configure virtual IPs.
Cluster Split based on the service name
Cluster split - 2 nodes for DSS and 1 Node for OLTP
The requirement is to allocate 1 node exclusively to OLTP enviroment out of 3 node cluster.
The logic is based on the setting up the service_name
SQL> show parameter service_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
service_names string orclexperts.info
SQL>
SQL> alter system set service_names='OLTP_experts' sid='ORCLEXPERTS3';
System altered.
SQL>
[oracle@s262381ch3el101 ~]$ lsnrctl services | grep -i orclexperts
Service "ORCLEXPERTS.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 2 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "ORCLEXPERTSXDB.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "ORCLEXPERTS_XPT.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 2 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "OLTP_ORCLEXPERTS.info" has 1 instance(s).
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
[oracle@experts1 admin]$ sqlplus system/xxxx@OLTP_ORCLEXPERTS
SQL*Plus: Release 10.2.0.2.0 - Production on Wed Feb 6 22:35:44 2008
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
SQL>
SQL> show parameter service_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
service_names string OLTP_ORCLEXPERTS, ORCLEXPERTS.info
TNS:
OLTP_ORCLEXPERTS =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = orclexperts3-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = OLTP_ORCLEXPERTS.info)
(INSTANCE_NAME = ORCLEXPERTS3)
)
)
All OLTP connection strings changed to use the above string value.
No comments:
Post a Comment