Solution – SQL Server Backup Failing with EXCEPTION_ACCESS_VIOLATION
Posted by blakhani on March 25, 2015
Recently someone posted on Facebook group about a problem. He informed that whenever the backups are taken in SQL Server 2014 instance, it’s failing with error as below.
Msg 3013, Level 16, State 1, Line 4
BACKUP DATABASE is terminating abnormally.
When we open ERRORLOG we saw below
***Stack Dump being sent to D:\MSSQL\LOG\SQLDump0089.txt
SqlDumpExceptionHandler: Process 1144 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process.
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 01/01/15 18:05:25 spid 1144
* Private server build.
*
*
* Exception Address = 00007FF85B652D45 Module(sqlmin+00000000009D2D45)
* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
* Access Violation occurred reading address 00007FFA0C208980
* Input Buffer 116 bytes –
* backup database test to disk=’c:\temp\test.bak’
As we can see that Access Violation is occurring while running backup database command. Whenever there is a dump generated, there is a MDMP file is also generated. Here is the stack when we analyze the dump. Yon can refer http://mssqlwiki.com/2012/10/16/sql-server-exception_access_violation-and-sql-server-assertion link to know more about identifying stack.
sqlmin!PerfmonManager::AddInstance+0x4e0
sqlmin!BackupPerfmonCounter::AddInstance+0x1f
sqlmin!BackupDevicePerfmonCounter::Init+0x13c
sqlmin!BackupFileDesc::InitPerfCounters+0xc <<Performance Counters Initialization.
sqlmin!BackupMediaIo::Initialize+0x29
sqlmin!BackupMedium::CreateDeviceObject+0x224
sqlmin!BackupMedium::Open+0x27
sqlmin!BackupStream::OpenForBackup+0x27
sqlmin!BackupStream::ThreadMainRoutine+0x160
sqlmin!BackupThread::ThreadBase+0x51
sqlmin!SubprocEntrypoint+0xa7f << This is the child thread of main backup thread.
sqldk!SOS_Task::Param::Execute+0x21e
sqldk!SOS_Scheduler::RunTask+0xa8
sqldk!SOS_Scheduler::ProcessTasks+0x279
sqldk!SchedulerManager::WorkerEntryPoint+0x24c
sqldk!SystemThread::RunWorker+0x8f
sqldk!SystemThreadDispatcher::ProcessWorker+0x3ab
sqldk!SchedulerManager::ThreadEntryPoint+0x226
kernel32!BaseThreadInitThunk+0xd
ntdll!RtlUserThreadStart+0x1d
Here is the version of SQL Server.
Microsoft SQL Server 2014 – 12.0.2000.8 (X64)
Feb 20 2014 20:04:26
Copyright (c) Microsoft Corporation
Enterprise Edition: Core-based Licensing (64-bit) on Windows NT 6.3 <X64> (Build 9600: ) (Hypervisor)
This is identified as an issue with SQL Server 2014 RTM version. Here is the KB article – http://support.microsoft.com/en-us/kb/2973444 (FIX: "SQL Server performance counters are disabled" when you move the SQL Server resource in SQL Server 2014)
Fix: Cumulative Update 2 for SQL Server 2014 or any later version of SQL Server 2014.
If you are running higher version and above fix then you need to find out why performance monitor is not showing SQL Server related counters. The way to verify it would be to use DMV
select * from sys.dm_os_performance_counters
You should see many counters specifically look for “Backup/Restore Throughput/sec”. In SQL 2014 this is a new performance counter added to get backup/restore speed and that missing counter (Backup/Restore Throughput/sec) which is causing backup to fail.
Help: SQL Server 
In Depth SQL said
Did we tried to rebuild the performance counters before applying this fix and observed if it’s fixed ?
http://support.microsoft.com/en-us/kb/300956
blakhani said
Yes. that was tried. Due to the issue which i mentioned, it was disappearing again and again.
floraligon said
I do consider all of the ideas you’ve presented in your post. They are very convincing and will definitely work. Still, the posts are very quick for newbies.
Could you please lengthen them a bit from subsequent time? Thanks for the post.
Albert said
We have exactly the same issue in test box…
0:254> kc 1000
Call Site
sqlmin!GetObjOffsets
sqlmin!PerfmonManager::AddInstance
sqlmin!BackupPerfmonCounter::AddInstance
sqlmin!BackupDevicePerfmonCounter::Init
sqlmin!BackupMedium::CreateDeviceObject
sqlmin!BackupMedium::Open
sqlmin!BackupStream::ThreadMainRoutine
sqlmin!BackupThread::ThreadBase
sqlmin!SubprocEntrypoint
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
*** WARNING: Unable to verify timestamp for kernel32.dll
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
0:254> lmvm sqlmin
start end module name
00007ffc`cdbd0000 00007ffc`d1a7c000 sqlmin (pdb symbols) c:\websymbols\sqlmin.pdb\31D5B80990D64E0FB25CB5327135BF492\sqlmin.pdb
Loaded symbol image file: sqlmin.dll
Mapped memory image file: C:\Program Files\Microsoft SQL Server\MSSQL12.INSTANCE1\MSSQL\Binn\sqlmin.dll
Image path: C:\Program Files\Microsoft SQL Server\MSSQL12.INSTANCE1\MSSQL\Binn\sqlmin.dll
Image name: sqlmin.dll
Timestamp: Thu Jan 29 19:02:46 2015 (54C9E926)
CheckSum: 03DB4FF7
ImageSize: 03EAC000
File version: 2014.120.2480.0
Product version: 12.0.2480.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 0.0 Unknown
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft SQL Server
InternalName: SQLMIN
OriginalFilename: SQLMIN.DLL
ProductVersion: 12.0.2480.0
FileVersion: 2014.0120.2480.00 ((SQL14_RTM_QFE-CU).150128-1755)
FileDescription: SQL Server Windows NT – 64 Bit
LegalCopyright: Microsoft Corp. All rights reserved.
LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
Comments: SQL
CU2 was installed in either Test or Prod server, I am confused only Test Server has this issue reported since a restart, you can see above version details.
I am also confused, if this is a bug related to backup/restore, then the server restart will involve recovery process, and all databases are online now, just cannot do backup/restore, unless the recovery is not monitored by performance counters…
blakhani said
Do you have any issue with performance counter? How many rows do you see when you query
Select * from sys.dm_os_performance_counters
Albert said
I followed the steps in this article to add the missing performance counter “Backup/Restore Throughput/sec” back http://blogs.technet.com/b/pfelatam/archive/2011/08/08/sql-performance-counters-are-missing.aspx
now I have 32780 counters in sys.dm_os_performance_counters table,
and for “Backup/Restore Throughput/sec”, each instance has 1 record now, but the mini dump still being generated continually in log folder…
Albert said
I have checked the mini dump again, even the performance counter is in place, it is still looking for the performance counter…
0:406> kc 1000
Call Site
sqlmin!GetObjOffsets
sqlmin!PerfmonManager::AddInstance
sqlmin!BackupPerfmonCounter::AddInstance
sqlmin!BackupDevicePerfmonCounter::Init
sqlmin!BackupMedium::CreateDeviceObject
sqlmin!BackupMedium::Open
sqlmin!BackupStream::OpenForBackup
sqlmin!BackupStream::ThreadMainRoutine
sqlmin!BackupThread::ThreadBase
sqlmin!SubprocEntrypoint
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
*** WARNING: Unable to verify timestamp for kernel32.dll
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Albert said
When yesterday, after adding the missing counter, I was able to create an empty database “test” and do backup, then I restart the sql server service, now today, I cannot create new database, backup or restore databases, when I create new database “test2”, I got below dump:
0:000> kc 1000
Call Site
sqlmin!PerfmonManager::AddInstance
sqlmin!DBTABLE::Startup
sqlmin!DBMgr::StartupDB
sqllang!CStmtCreateDB::CreateLocalDatabaseFragment
sqllang!DBDDLAgent::CreateDatabase
sqllang!CStmtCreateDB::XretExecute
sqllang!CMsqlExecContext::ExecuteStmts
sqllang!CMsqlExecContext::FExecute
sqllang!CSQLSource::Execute
sqllang!process_request
sqllang!process_commands
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
*** WARNING: Unable to verify timestamp for kernel32.dll
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
It is related to performance counter… I notice all performance counters for ‘Backup/Restore Throughput/sec’, each instance only has one cntr_value in table, so it only successfully record cntr value once and fails afterwards… so strange…
Albert said
ignore the last sentence in above reply…
Albert said
Hello All, guess what, I installed a new instance(INSTANCE2) in same box without any problem, then I install CU6, I found I can tick INSTANCE1 which is the old instance, even @@version says it is on the latest version, but it is not, after applying the CU6 on INSTANCE1, the backup/restore issue is gone.
The more I use SQL Server I more miss Oracle now.
Albert said
Hi Blakhani,
Guess what, it is fixed now, how? I installed a new instance in same box, then install CU6, I found I can tick the old instance, which means the old instance didn’t have CU6 properly but no error, no log indicates that, after CU6 is installed, the issue is gone now.
what a pity, this is not the first time we have the problem, in future we have to run update twice to make sure it is correct.
blakhani said
Last patch can always be reinstalled so tick box enabled is normal behavior.
Good to know the issue is resolved.
Sajan Bhaukajee said
I blog often and I genuinely appreciate your content. This article has really peaked my interest. I’m going to bookmark your site and keep checking for new details about once a week. I subscribed to your RSS feed too.