EMC Avamar Windows Server 2008 R2 VSS backup fails with: System Writer is not present

Welcome back everyone,

Today’s random backup failure is brought to you by the number: infinity. Well, thats how many .NET temp files I seemed to have on a server that refused to complete its VSS backup.

On this particular SharePoint 2010 machine, when a VSS backup would run it bombed with this error:

2013-07-11 18:04:50 avvss Info : VSS: Creating vss version 6.0 or greater object
2013-07-11 18:04:50 avvss Info : Gathering writer metadata…
2013-07-11 18:04:51 avvss Error : Can not continue disaster recovery backup because the System Writer is not present, exiting.
2013-07-11 18:04:51 avvss Info : Final summary generated subwork 0, cancelled/aborted 0, snapview 0, exitcode 536870919
2013-07-11 18:04:56 avvss Info : uvss returning with exitcode 536870919

I tried all of the usual VSS writer DLL re-register & permissions fix tricks I knew (which technically aren’t recommended on Server 2008 R2!), but alas nothing would bring the System Writer back. Becoming almost apathetic about the issue, I then bumbled onto this TechNet Social post.

It gets interesting about halfway down with a post from “Rosaceae” & “Microbolt”. I’ve quoted their discussion below, should that link ever die.

I checked out my .NET Framework temp directories, and there were about 100k files in there going back to 2009. I cleaned them out, restarted the Cryptographic Service and wouldn’t you know it, the VSS System Writer came back and my backup was successful!

By the way, the Cryptographic Service is probably about the most unintuitive service name that could relate to a VSS component Microsoft could think of.

I’m going to keep my eye on this and see if I end up needing to relocate my .NET Framework temp files like Microbolt did, but I’m guessing not. It looked as if some developer was trying out some new/bad code and caused it.

So thanks to both of those people, I would’ve been stumped without it.

Till next time…

Hi,
I’ve got this problem about a month ago. I refer to MSP.
The problem was caused due to stack full. When we list system writer using “vssadmin list writers”, it will go through all the system files. To do that, the OS use a search algorithm with a stack which has a size limitation of 1000. When the stack was full, it failed to continue listing files and log an event in the application event log.
In my case, the following folder contains too many subdirectory and caused the problem.
C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files\*
1. Open C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\CONFIG\Web.config
2. Add tempDirectory attribute to compilation tag. For example:
< compilation tempDirectory=”c:\ASPTEMP”>
And also grant the folder with the same privilege with as “C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files”.
3. Restart the IIS Service.
4. Backup and delete all files under “C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files”.
5. restart the Cryptographic Service.
6. Try “vssadmin list writers” again.
Hopes this brings idea for you to solve it.

Thanks Rosaceae!
After going on a wild goose chase setting permissions and nothing working I got looking around in the .Net Folders per your advice. It looks like in my case I had the same issue with you except in the Framework instead of Framework64 (as most of my web apps are running x86).
I’ll share what I did incase it helps anyone (Ignore that last two of each step if you don’t have .Net 4.0 Installed):
Created 4 Folders:
C:\Asp.net Temp Files\2.0.50727\x86
C:\Asp.net Temp Files\2.0.50727\x64
C:\Asp.net Temp Files\4.0.30319\x86
C:\Asp.net Temp Files\4.0.30319\x64
Set Permissions on the folder (This is how I set them, may be different on your server. Check existing “Temporary ASP.NET Files” directory for permissions on your server
icacls “c:\Asp.net Temp Files” /grant “BUILTIN\Administrators:(OI)(CI)(F)”
icacls “c:\Asp.net Temp Files” /grant “NT AUTHORITY\SYSTEM:(OI)(CI)(M,WDAC,DC)”
icacls “c:\Asp.net Temp Files” /grant “CREATOR OWNER:(OI)(CI)(IO)(F)”
icacls “c:\Asp.net Temp Files” /grant “BUILTIN\IIS_IUSRS:(OI)(CI)(M,WDAC,DC)”
icacls “c:\Asp.net Temp Files” /grant “BUILTIN\Users:(OI)(CI)(RX)”
icacls “c:\Asp.net Temp Files” /grant “NT SERVICE\TrustedInstaller:(CI)(F)”
Add tempDirectory attribute to compilation tag. This will keep you from having the problem again in the future. Add the following attribute to these files:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\CONFIG\Web.config

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\CONFIG\Web.config

C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\CONFIG\Web.config

C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\CONFIG\Web.config

Restart IIS so that it will use the new Temp Directory
iisreset
Deleted old Temp Files
rmdir /s /q “C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files\root”
rmdir /s /q “C:\Windows\Microsoft.Net\Framework\v2.0.50727\Temporary ASP.NET Files\root”
rmdir /s /q “C:\Windows\Microsoft.Net\Framework64\v4.0.30319\Temporary ASP.NET Files\root”
rmdir /s /q “C:\Windows\Microsoft.Net\Framework\v4.0.30319\Temporary ASP.NET Files\root”
Restart Cryptographic Service
net stop cryptsvc
net start cryptsvc
Now if all goes well you should be able to see the “System Writer” again!
vssadmin list writers

EMC Avamar RMAN backup fails with: RMAN-06403, ORA-01034 and ORA-27101

Hello folks,

I had a really interesting (to me) Avamar RMAN backup failure after a system was rebooted. The system had been backing up perfectly for months, and then all of the sudden:

Recovery Manager: Release 11.2.0.3.0 – Production on Wed Apr 24 13:59:45 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN> @@RMAN-blahblah-1543-cred1.rman
2> connect target *;
3> **end-of-file**
4> run {
5> configure controlfile autobackup on;
6> set controlfile autobackup format for device type sbt to ‘CONTROLFILE.blahblah.%F’;
7> allocate channel c0 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
8> send channel ‘c0’ ‘”–libport=32110″ “–cacheprefix=blahblah_c0” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log0” “–ctlcallport=32108″‘;
9> allocate channel c1 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
10> send channel ‘c1’ ‘”–libport=32110″ “–cacheprefix=blahblah_c1” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log1” “–ctlcallport=32108″‘;
11> backup database plus archivelog delete input;
12> }
13>
connected to target database (not started)
using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 04/24/2013 13:59:45
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Recovery Manager complete.

Sorry for the craptastic formatting there, gotta love Oracle’s gigantic errors 🙂

Being the type of guy who tries to figure things out myself, …I did the usual and checked support.emc.com, legacy Powerlink and of course Google. In my search, I came up with this EMC KB article: esg130176. It stepped me through making sure the instance was running and that the tnsnames.ora & /etc/oratab were correctly formatted.

All of this seemed to check out, and I was even able to perform a manual RMAN backup to local disk, which boggled me further.

I tried to get our DBA’s to help out, but got the usual “everything looks good on our side” answer.

I resigned myself to creating an SR with EMC. The gentleman who returned my call took a quick look and immediately knew what was wrong. He mentioned that when he originally discovered this issue for another customer, it had taken a while to figure out what was going on.

Somehow since the last restart, someone had managed to add a / to the end of oracle’s ORACLE_HOME environment variable:

$ whoami
oracle
$ env |grep ORACLE_HOME
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1/

Avamar apparently strips off that erroneous trailing / as it sets its own ORACLE_HOME when it starts the job and thus breaks sqlplus, rman… all of the those tools we so dearly need.

I want my DBA’s to fix that environment variable, but that would require restarting this production database. To work around this, create a file called /usr/local/avamar/var/avoracle.cmd and put this inside it. This will FORCE Avamar to tack on the /.

–orahome=/u01/app/oracle/product/11.2.0/dbhome_1/

Obviously you need to make that match YOUR environment!

Try the backup again, and with any luck you’ll be “fixed”! I suggested to the EMC support engineer that he should add that to the above mentioned KB article, and he said he would do that.

Till next time…