EMC Avamar Windows Server 2008 R2 VSS backup fails with: System Writer is not present

Welcome back everyone,

Today’s random backup failure is brought to you by the number: infinity. Well, thats how many .NET temp files I seemed to have on a server that refused to complete its VSS backup.

On this particular SharePoint 2010 machine, when a VSS backup would run it bombed with this error:

2013-07-11 18:04:50 avvss Info : VSS: Creating vss version 6.0 or greater object
2013-07-11 18:04:50 avvss Info : Gathering writer metadata…
2013-07-11 18:04:51 avvss Error : Can not continue disaster recovery backup because the System Writer is not present, exiting.
2013-07-11 18:04:51 avvss Info : Final summary generated subwork 0, cancelled/aborted 0, snapview 0, exitcode 536870919
2013-07-11 18:04:56 avvss Info : uvss returning with exitcode 536870919

I tried all of the usual VSS writer DLL re-register & permissions fix tricks I knew (which technically aren’t recommended on Server 2008 R2!), but alas nothing would bring the System Writer back. Becoming almost apathetic about the issue, I then bumbled onto this TechNet Social post.

It gets interesting about halfway down with a post from “Rosaceae” & “Microbolt”. I’ve quoted their discussion below, should that link ever die.

I checked out my .NET Framework temp directories, and there were about 100k files in there going back to 2009. I cleaned them out, restarted the Cryptographic Service and wouldn’t you know it, the VSS System Writer came back and my backup was successful!

By the way, the Cryptographic Service is probably about the most unintuitive service name that could relate to a VSS component Microsoft could think of.

I’m going to keep my eye on this and see if I end up needing to relocate my .NET Framework temp files like Microbolt did, but I’m guessing not. It looked as if some developer was trying out some new/bad code and caused it.

So thanks to both of those people, I would’ve been stumped without it.

Till next time…

Hi,
I’ve got this problem about a month ago. I refer to MSP.
The problem was caused due to stack full. When we list system writer using “vssadmin list writers”, it will go through all the system files. To do that, the OS use a search algorithm with a stack which has a size limitation of 1000. When the stack was full, it failed to continue listing files and log an event in the application event log.
In my case, the following folder contains too many subdirectory and caused the problem.
C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files\*
1. Open C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\CONFIG\Web.config
2. Add tempDirectory attribute to compilation tag. For example:
< compilation tempDirectory=”c:\ASPTEMP”>
And also grant the folder with the same privilege with as “C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files”.
3. Restart the IIS Service.
4. Backup and delete all files under “C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files”.
5. restart the Cryptographic Service.
6. Try “vssadmin list writers” again.
Hopes this brings idea for you to solve it.

Thanks Rosaceae!
After going on a wild goose chase setting permissions and nothing working I got looking around in the .Net Folders per your advice. It looks like in my case I had the same issue with you except in the Framework instead of Framework64 (as most of my web apps are running x86).
I’ll share what I did incase it helps anyone (Ignore that last two of each step if you don’t have .Net 4.0 Installed):
Created 4 Folders:
C:\Asp.net Temp Files\2.0.50727\x86
C:\Asp.net Temp Files\2.0.50727\x64
C:\Asp.net Temp Files\4.0.30319\x86
C:\Asp.net Temp Files\4.0.30319\x64
Set Permissions on the folder (This is how I set them, may be different on your server. Check existing “Temporary ASP.NET Files” directory for permissions on your server
icacls “c:\Asp.net Temp Files” /grant “BUILTIN\Administrators:(OI)(CI)(F)”
icacls “c:\Asp.net Temp Files” /grant “NT AUTHORITY\SYSTEM:(OI)(CI)(M,WDAC,DC)”
icacls “c:\Asp.net Temp Files” /grant “CREATOR OWNER:(OI)(CI)(IO)(F)”
icacls “c:\Asp.net Temp Files” /grant “BUILTIN\IIS_IUSRS:(OI)(CI)(M,WDAC,DC)”
icacls “c:\Asp.net Temp Files” /grant “BUILTIN\Users:(OI)(CI)(RX)”
icacls “c:\Asp.net Temp Files” /grant “NT SERVICE\TrustedInstaller:(CI)(F)”
Add tempDirectory attribute to compilation tag. This will keep you from having the problem again in the future. Add the following attribute to these files:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\CONFIG\Web.config

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\CONFIG\Web.config

C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\CONFIG\Web.config

C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\CONFIG\Web.config

Restart IIS so that it will use the new Temp Directory
iisreset
Deleted old Temp Files
rmdir /s /q “C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files\root”
rmdir /s /q “C:\Windows\Microsoft.Net\Framework\v2.0.50727\Temporary ASP.NET Files\root”
rmdir /s /q “C:\Windows\Microsoft.Net\Framework64\v4.0.30319\Temporary ASP.NET Files\root”
rmdir /s /q “C:\Windows\Microsoft.Net\Framework\v4.0.30319\Temporary ASP.NET Files\root”
Restart Cryptographic Service
net stop cryptsvc
net start cryptsvc
Now if all goes well you should be able to see the “System Writer” again!
vssadmin list writers