EMC Avamar RMAN backup fails with: RMAN-06403, ORA-01034 and ORA-27101

Hello folks,

I had a really interesting (to me) Avamar RMAN backup failure after a system was rebooted. The system had been backing up perfectly for months, and then all of the sudden:

Recovery Manager: Release 11.2.0.3.0 – Production on Wed Apr 24 13:59:45 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN> @@RMAN-blahblah-1543-cred1.rman
2> connect target *;
3> **end-of-file**
4> run {
5> configure controlfile autobackup on;
6> set controlfile autobackup format for device type sbt to ‘CONTROLFILE.blahblah.%F’;
7> allocate channel c0 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
8> send channel ‘c0’ ‘”–libport=32110″ “–cacheprefix=blahblah_c0” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log0” “–ctlcallport=32108″‘;
9> allocate channel c1 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
10> send channel ‘c1’ ‘”–libport=32110″ “–cacheprefix=blahblah_c1” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log1” “–ctlcallport=32108″‘;
11> backup database plus archivelog delete input;
12> }
13>
connected to target database (not started)
using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 04/24/2013 13:59:45
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Recovery Manager complete.

Sorry for the craptastic formatting there, gotta love Oracle’s gigantic errors 🙂

Being the type of guy who tries to figure things out myself, …I did the usual and checked support.emc.com, legacy Powerlink and of course Google. In my search, I came up with this EMC KB article: esg130176. It stepped me through making sure the instance was running and that the tnsnames.ora & /etc/oratab were correctly formatted.

All of this seemed to check out, and I was even able to perform a manual RMAN backup to local disk, which boggled me further.

I tried to get our DBA’s to help out, but got the usual “everything looks good on our side” answer.

I resigned myself to creating an SR with EMC. The gentleman who returned my call took a quick look and immediately knew what was wrong. He mentioned that when he originally discovered this issue for another customer, it had taken a while to figure out what was going on.

Somehow since the last restart, someone had managed to add a / to the end of oracle’s ORACLE_HOME environment variable:

$ whoami
oracle
$ env |grep ORACLE_HOME
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1/

Avamar apparently strips off that erroneous trailing / as it sets its own ORACLE_HOME when it starts the job and thus breaks sqlplus, rman… all of the those tools we so dearly need.

I want my DBA’s to fix that environment variable, but that would require restarting this production database. To work around this, create a file called /usr/local/avamar/var/avoracle.cmd and put this inside it. This will FORCE Avamar to tack on the /.

–orahome=/u01/app/oracle/product/11.2.0/dbhome_1/

Obviously you need to make that match YOUR environment!

Try the backup again, and with any luck you’ll be “fixed”! I suggested to the EMC support engineer that he should add that to the above mentioned KB article, and he said he would do that.

Till next time…