Checking logical block corruption

a) $ rman target / nocatalog

b) run {
allocate channel d1 type disk;
backup check logical validate database;
release channel d1;
}

c) select * from V$DATABASE_BLOCK_CORRUPTION ;

d)If V$DATABASE_BLOCK_CORRUPTION contains rows please run this procedure to find the objects that contains the corrupted blocks:

drop table corrupted_objects;
create table corrupted_objects as
select owner, segment_name, partition_name, segment_type, 1 file#, 1 block# from dba_extents where 1=2;
declare
cursor C1 is select file#, block# from v$database_block_corruption;
begin
for r1 in c1 loop
insert into corrupted_objects
select owner, segment_name, partition_name,segment_type, r1.file#, r1.block# from dba_extents where file_id=r1.file#
and r1.block# between block_id and block_id + blocks - 1;
if sql%notfound then /* segment header block might be corrupt causing dba_extents not returning rows. */
insert into corrupted_objects select owner, segment_name, partition_name, segment_type, r1.file#, r1.block#
from dba_segments where header_file=r1.file# and header_block = r1.block#;
end if;
end loop;
end;
/


e) Select all the objects populated in the table corrupted_objects:


SQL> select distinct owner, segment_name, partition_name, segment_type from corrupted_objects;

restarting resource to clear OFFLINE State

$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.EPPROD.db application ONLINE ONLINE jury
ora....p1.inst application ONLINE ONLINE jasmine
ora....p2.inst application ONLINE ONLINE jury
ora....NE.lsnr application ONLINE ONLINE jasmine
ora....ine.gsd application ONLINE ONLINE jasmine
ora....ine.ons application ONLINE ONLINE jasmine
ora....ine.vip application ONLINE ONLINE jasmine
ora....RY.lsnr application ONLINE ONLINE jury
ora.jury.gsd application ONLINE OFFLINE
ora.jury.ons application ONLINE OFFLINE
ora.jury.vip application ONLINE ONLINE jury

$ crs_stat
NAME=ora.EPPROD.db
TYPE=application
TARGET=ONLINE
STATE=ONLINE on jury
..
..
..

NAME=ora.jury.LISTENER_JURY.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on jury

NAME=ora.jury.gsd
TYPE=application
TARGET=ONLINE
STATE=OFFLINE

NAME=ora.jury.ons
TYPE=application
TARGET=ONLINE
STATE=OFFLINE

NAME=ora.jury.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE on jury

$ crs_stop ora.jury.gsd
Target set to OFFLINE for `ora.jury.gsd`
$ crs_start ora.jury.gsd
Attempting to start `ora.jury.gsd` on member `jury`
Start of `ora.jury.gsd` on member `jury` succeeded.
$ crs_stop ora.jury.ons
Target set to OFFLINE for `ora.jury.ons`
$ crs_start ora.jury.ons
Attempting to start `ora.jury.ons` on member `jury`
Start of `ora.jury.ons` on member `jury` succeeded.
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.EPPROD.db application ONLINE ONLINE jury
ora....p1.inst application ONLINE ONLINE jasmine
ora....p2.inst application ONLINE ONLINE jury
ora....NE.lsnr application ONLINE ONLINE jasmine
ora....ine.gsd application ONLINE ONLINE jasmine
ora....ine.ons application ONLINE ONLINE jasmine
ora....ine.vip application ONLINE ONLINE jasmine
ora....RY.lsnr application ONLINE ONLINE jury
ora.jury.gsd application ONLINE ONLINE jury
ora.jury.ons application ONLINE ONLINE jury
ora.jury.vip application ONLINE ONLINE jury
$

Oracle Performance Diagnostic Guide

Oracle Performance Diagnostic Guide (OPDG)

--> MetaLink Doc ID: 390374.1

Configure ssh for RAC using rsa/dsa

From each node say md004a,md004b, logged in as oracle:

1) Execute below on md004a,md004b
mkdir ~/.ssh
chmod 755 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa

2) On md004a
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub > ~/.ssh/authorized_keys
ssh oracle@md004b cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
ssh oracle@md004b cat ~/.ssh/id_dsa.pub > ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys

3) on md004b
ssh oracle@md004a cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
ssh oracle@md004a cat ~/.ssh/id_dsa.pub > ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys

4) on md004a and md004b
cp id_rsa.pub authorized_keys2

Common RAC errors

oui-10009
PRKP-1037 : Failed to create cluster Database
CRS-0210: Could not find resource
CRS ownership - CRS did not start after node reboot
clsssInitNative: connect failed, rc 9,Received status 3 from CSS
Cluster Split based on the service name



oui-10009





This error occured while adding a new node in the existing cluster. The following workaround is used as follows:

Node 1:

cd $ORACLE_HOME/oui/bin

./addNode.sh -silent "CLUSTER_NEW_NODES={new nodename}" -logLevel trace –debug

You may want to browse "Node add" to get more information.

Node add





PRKP-1037 : Failed to create cluster Database

Step 1:

[oracle@orcl-experts.info bin]$ ./crs_stat
NAME=ora.SAAP.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

Step 2 : crs_unregister ora.SAAP.db

Step 3: srvctl add database -d SAAP -o /data01/app/oracle/product/10.2.0/db/

[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP1 -n sabharwal11
[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP2 -n sabharwal12
[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP3 -n sabharwal13

[oracle@sabharwal11 bin]$ srvctl stop database -d SAAP
[oracle@sabharwal11 bin]$ srvctl start database -d SAAP
[oracle@sabharwal11 bin]$ ./crs_stat
NAME=ora.SAAP.SAAP1.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal11

NAME=ora.SAAP.SAAP2.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal12

NAME=ora.SAAP.SAAP3.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal13


[oracle@sabharwal11 bin]$ srvctl status database -d SAAP
Instance SAAP1 is running on node sabharwal11
Instance SAAP2 is running on node sabharwal12
Instance SAAP3 is running on node sabharwal13

CRS-0210: Could not find resource



[oracle@orcl-experts1.com admin]$ srvctl stop nodeapps -n orcl-experts1.com

CRS-0210: Could not find resource 'ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr'.

[oracle@orcl-experts1.com admin]$ srvctl start nodeapps -n orcl-experts1.com


CRS-0210: Could not find resource ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr.



[oracle@orcl-experts1.com bin]$ ./crs_register ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr




[oracle@orcl-experts1.com bin]$ srvctl start nodeapps -n orcl-experts1.com
[oracle@orcl-experts1.com bin]$ srvctl stop nodeapps -n orcl-experts1.com
[oracle@orcl-experts1.com bin]$ srvctl start nodeapps -n orcl-experts1.com

NO Errors



CRS ownership - CRS did not start after node reboot - Metalink 428195.1

I had to deal with a maintenance which required disabling the auto start of CRS during server reboot:

#/etc/init.d/initcrs disable

I shutdown the RAC 2 node cluster and verified that there is no oracle process running and hand over the servers to Sysadmin. After the hardware maintenance before the 2nd reboot the sysadmin asked me to re enable the CRS auto start. In both nodes I did:

#/etc/init.d/initcrs enable

Sysadmin rebooted both the nodes. After 10 minutes I checked the process details and found that CRS stack is not running and the instances are not up. We had a deadline of 1 hour to bring up all the instances. Under the short time and pressure I started digging out the issue of why the CRS stack was not up. I tried to bring it up manually but it did not work.

I found that after remounting the file systems the permission of /oracle01 mount point (voting file and OCR file location) is:

-rw-r--r-- 1 root root 10240000 Apr 3 21:41 voting.dbf

Sysadmin and I decided to make an immediate change:

chown -R root:install /oracle01

and

we rebooted the machine.

It worked and during the node reboots process all the CRS and Oracle processes came up.

The maintenance was based on the mounting the existing file system on the new LUN.

clsssInitNative: connect failed, rc 9,Received status 3 from CSS. Waiting for good status .., Failure at final check of Oracle CRS stack. 10


Issue: I had issue while bringing up CRS in the 2nd node.The root.sh executions failed with the following message:


Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.

Failure at final check of Oracle CRS stack.
10


crsd.log

2007-10-26 17:33:37.510: [ CSSCLNT][3086960320]clsssInitNative: connect failed, rc 9

2007-10-26 17:33:37.511: [ CRSRTI][3086960320]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..

2007-10-26 17:33:38.545: [ COMMCRS][116607920]clsc_connect: (0x8605510) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_oracl-experts.info2_crs))


Investigation:

I looked into the files under /var/tmp/.oracle in node1 - Looks everything fine but in the 2nd node , the CRS process was not able to write


Solution:


After long research, I found out that there is a major issue with the /var/tmp


[root@oracl-experts.info2 app]# chmod 1777 /var/tmp

[root@oracl-experts.info2 app]# ls -ld /var/tmp

drwxrwxrwt 3 root root 4096 Oct 26 17:33 /var/tmp




total 8

drwxrwxrwt 3 root root 4096 Oct 26 17:33 ..
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 soracl-experts.info2DBG_CSSD
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sOracle_CSS_LclLstnr_crs_2
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sOCSSD_LL_oracl-experts.info2_crs
srwxrwxrwx 1 root root 0 Oct 26 17:52 sprocr_local_conn_0_PROC
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 soracl-experts.info2DBG_EVMD
srwxrwxrwx 1 root root 0 Oct 26 17:52 soracl-experts.info2DBG_CRSD
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sSYSTEM.evm.acceptor.auth
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sCoracl-experts.info2_crs_evm
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sAoracl-experts.info2_crs_evm
srwxrwxrwx 1 root root 0 Oct 26 17:52 sora_crsqs
srwxrwxrwx 1 root root 0 Oct 26 17:52 sCRSD_UI_SOCKET
drwxrwxrwt 2 root root 4096 Oct 26 17:52 .


[root@oracl-experts.info2 app]# ./root.sh
WARNING: directory '/data01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/data01/crs/oracle/product' is not owned by root
WARNING: directory '/data01/crs/oracle' is not owned by root
WARNING: directory '/data01/crs' is not owned by root
Checking to see if Oracle CRS stack is already configured


Setting the permissions on OCR backup directory

Setting up NS directories

Oracle Cluster Registry configuration upgraded successfully

WARNING: directory '/data01/crs/oracle/product/10.2.0' is not owned by root

WARNING: directory '/data01/crs/oracle/product' is not owned by root

WARNING: directory '/data01/crs/oracle' is not owned by root

WARNING: directory '/data01/crs' is not owned by root

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

assigning default hostname oracl-experts.info1 for node 1.

assigning default hostname oracl-experts.info2 for node 2.

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node :

node 1: oracl-experts.info1 oracl-experts.info1-priv oracl-experts.info1

node 2: oracl-experts.info2 oracl-experts.info2-priv oracl-experts.info2

clscfg: Arguments check out successfully.


NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Oracle Cluster Registry for cluster has already been initialized

Startup will be queued to init within 90 seconds.

Adding daemons to inittab

Expecting the CRS daemons to be up within 600 seconds.

CSS is active on these nodes.

oracl-experts.info1

oracl-experts.info2

CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

The given interface(s), "eth0" is not public. Public interfaces should be used t o configure virtual IPs.

Cluster Split based on the service name


Cluster split - 2 nodes for DSS and 1 Node for OLTP

The requirement is to allocate 1 node exclusively to OLTP enviroment out of 3 node cluster.


The logic is based on the setting up the service_name


SQL> show parameter service_name

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
service_names string orclexperts.info
SQL>


SQL> alter system set service_names='OLTP_experts' sid='ORCLEXPERTS3';

System altered.

SQL>
[oracle@s262381ch3el101 ~]$ lsnrctl services | grep -i orclexperts
Service "ORCLEXPERTS.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 2 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "ORCLEXPERTSXDB.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "ORCLEXPERTS_XPT.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 2 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "OLTP_ORCLEXPERTS.info" has 1 instance(s).
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...


[oracle@experts1 admin]$ sqlplus system/xxxx@OLTP_ORCLEXPERTS

SQL*Plus: Release 10.2.0.2.0 - Production on Wed Feb 6 22:35:44 2008

Copyright (c) 1982, 2005, Oracle. All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL>


SQL> show parameter service_name

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
service_names string OLTP_ORCLEXPERTS, ORCLEXPERTS.info


TNS:
OLTP_ORCLEXPERTS =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = orclexperts3-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = OLTP_ORCLEXPERTS.info)
(INSTANCE_NAME = ORCLEXPERTS3)
)
)



All OLTP connection strings changed to use the above string value.

Effective Leadership Tips

1. Be a leader, not a victim: Do not blame others or look for scapegoats. Your ability to be a leader starts with mastering yourself.

2. Beware of your blind spots: The key to teamwork is valuing others’ perspectives, and encouraging them to fill in where you have gaps.

3. Take care of important relationships: Treat others as though they are the most important thing. A Chinese proverb says: If you want one year’s prosperity, grow grain. If you want ten years’ prosperity, grow trees. If you want 100 years’ prosperity, grow people.

4. Balance short- and long-term results: The ‘golden egg’ or the ‘golden goose;’ which do you protect? Both, but with balance.

5. Know and live your mission: Create a personal mission statement that is short, memorable, passionate, measurable and for which you become accountable.

6. Do what matters most – now: Do things that will make your personal life more meaningful, and do what it takes to move to a higher level of professional success.

7. Be a master communicator: Practice the 2:1 rule; listen twice as much as you speak. From listening comes wisdom.

8. Be a great team player: When you have disagreements or conflicts with others, try this approach: “you see it differently, help me understand.”

9. Commit one hour per day to personal renewal: Take time for spiritual, physical, and professional growth. A great leader takes care of him/her self.

10. Live with integrity: Live and lead in honesty and forthrightness; be true to your word.

5 Tips on Effective Leadership

Communicate clearly and routinely. Lay out your company goals and principles in a mission statement and keep sharing your vision with your employees.

Involve employees in setting objectives. Give them feedback on how they are progressing toward meeting those targets.

Give your people authority, then hold them accountable. But don’t go after them personally when things go wrong. Find out first if the process is at fault.

Be accountable yourself. Install an advisory board or executive team to help you make good strategic decisions and give you feedback on your own performance.

Be trustworthy and extend trust to your employees. That will help you earn their loyalty and strengthen your company