Help: SQL Server

Sharing my knowlege about SQL Server Troubleshooting Skills

Solution – SQL Server Backup Failing with EXCEPTION_ACCESS_VIOLATION

Posted by blakhani on March 25, 2015


Recently someone posted on Facebook group about a problem. He informed that whenever the backups are taken in SQL Server 2014 instance, it’s failing with error as below.

Msg 3013, Level 16, State 1, Line 4
BACKUP DATABASE is terminating abnormally.

When we open ERRORLOG we saw below

***Stack Dump being sent to D:\MSSQL\LOG\SQLDump0089.txt
SqlDumpExceptionHandler: Process 1144 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process.
* ******************************************************************************* 

* BEGIN STACK DUMP:
*   01/01/15 18:05:25 spid 1144
* Private server build.
*
*
*   Exception Address = 00007FF85B652D45 Module(sqlmin+00000000009D2D45)
*   Exception Code    = c0000005 EXCEPTION_ACCESS_VIOLATION
*   Access Violation occurred reading address 00007FFA0C208980
* Input Buffer 116 bytes –
*             backup database test to disk=’c:\temp\test.bak’

As we can see that Access Violation is occurring while running backup database command. Whenever there is a dump generated, there is a MDMP file is also generated. Here is the stack when we analyze the dump. Yon can refer http://mssqlwiki.com/2012/10/16/sql-server-exception_access_violation-and-sql-server-assertion link to know more about identifying stack.

sqlmin!PerfmonManager::AddInstance+0x4e0
sqlmin!BackupPerfmonCounter::AddInstance+0x1f
sqlmin!BackupDevicePerfmonCounter::Init+0x13c
sqlmin!BackupFileDesc::InitPerfCounters+0xc                          <<Performance Counters Initialization.
sqlmin!BackupMediaIo::Initialize+0x29
sqlmin!BackupMedium::CreateDeviceObject+0x224
sqlmin!BackupMedium::Open+0x27
sqlmin!BackupStream::OpenForBackup+0x27
sqlmin!BackupStream::ThreadMainRoutine+0x160
sqlmin!BackupThread::ThreadBase+0x51
sqlmin!SubprocEntrypoint+0xa7f                                       << This is the child thread of main backup thread.
sqldk!SOS_Task::Param::Execute+0x21e
sqldk!SOS_Scheduler::RunTask+0xa8
sqldk!SOS_Scheduler::ProcessTasks+0x279
sqldk!SchedulerManager::WorkerEntryPoint+0x24c
sqldk!SystemThread::RunWorker+0x8f
sqldk!SystemThreadDispatcher::ProcessWorker+0x3ab
sqldk!SchedulerManager::ThreadEntryPoint+0x226
kernel32!BaseThreadInitThunk+0xd
ntdll!RtlUserThreadStart+0x1d

Here is the version of SQL Server.

Microsoft SQL Server 2014 – 12.0.2000.8 (X64)
Feb 20 2014 20:04:26
Copyright (c) Microsoft Corporation
Enterprise Edition: Core-based Licensing (64-bit) on Windows NT 6.3 <X64> (Build 9600: ) (Hypervisor)

This is identified as an issue with SQL Server 2014 RTM version. Here is the KB article – http://support.microsoft.com/en-us/kb/2973444 (FIX: "SQL Server performance counters are disabled" when you move the SQL Server resource in SQL Server 2014)

Fix: Cumulative Update 2 for SQL Server 2014 or any later version of SQL Server 2014.

If you are running higher version and above fix then you need to find out why performance monitor is not showing SQL Server related counters. The way to verify it would be to use DMV

select * from sys.dm_os_performance_counters

 

You should see many counters specifically look for “Backup/Restore Throughput/sec”. In SQL 2014 this is a new performance counter added to get backup/restore speed and that missing counter (Backup/Restore Throughput/sec) which is causing backup to fail.

  • Cheers,
  • Balmukund Lakhani
  • Twitter @blakhani
  • Author: SQL Server 2012 AlwaysOnPaperback, Kindle
  • Advertisements

    13 Responses to “Solution – SQL Server Backup Failing with EXCEPTION_ACCESS_VIOLATION”

    1. Did we tried to rebuild the performance counters before applying this fix and observed if it’s fixed ?

      http://support.microsoft.com/en-us/kb/300956

    2. floraligon said

      I do consider all of the ideas you’ve presented in your post. They are very convincing and will definitely work. Still, the posts are very quick for newbies.
      Could you please lengthen them a bit from subsequent time? Thanks for the post.

    3. Albert said

      We have exactly the same issue in test box…

      0:254> kc 1000
      Call Site
      sqlmin!GetObjOffsets
      sqlmin!PerfmonManager::AddInstance
      sqlmin!BackupPerfmonCounter::AddInstance
      sqlmin!BackupDevicePerfmonCounter::Init
      sqlmin!BackupMedium::CreateDeviceObject
      sqlmin!BackupMedium::Open
      sqlmin!BackupStream::ThreadMainRoutine
      sqlmin!BackupThread::ThreadBase
      sqlmin!SubprocEntrypoint
      sqldk!SOS_Task::Param::Execute
      sqldk!SOS_Scheduler::RunTask
      sqldk!SOS_Scheduler::ProcessTasks
      sqldk!SchedulerManager::WorkerEntryPoint
      sqldk!SystemThread::RunWorker
      sqldk!SystemThreadDispatcher::ProcessWorker
      *** WARNING: Unable to verify timestamp for kernel32.dll
      sqldk!SchedulerManager::ThreadEntryPoint
      kernel32!BaseThreadInitThunk
      ntdll!RtlUserThreadStart
      0:254> lmvm sqlmin
      start end module name
      00007ffc`cdbd0000 00007ffc`d1a7c000 sqlmin (pdb symbols) c:\websymbols\sqlmin.pdb\31D5B80990D64E0FB25CB5327135BF492\sqlmin.pdb
      Loaded symbol image file: sqlmin.dll
      Mapped memory image file: C:\Program Files\Microsoft SQL Server\MSSQL12.INSTANCE1\MSSQL\Binn\sqlmin.dll
      Image path: C:\Program Files\Microsoft SQL Server\MSSQL12.INSTANCE1\MSSQL\Binn\sqlmin.dll
      Image name: sqlmin.dll
      Timestamp: Thu Jan 29 19:02:46 2015 (54C9E926)
      CheckSum: 03DB4FF7
      ImageSize: 03EAC000
      File version: 2014.120.2480.0
      Product version: 12.0.2480.0
      File flags: 0 (Mask 3F)
      File OS: 40004 NT Win32
      File type: 0.0 Unknown
      File date: 00000000.00000000
      Translations: 0409.04b0
      CompanyName: Microsoft Corporation
      ProductName: Microsoft SQL Server
      InternalName: SQLMIN
      OriginalFilename: SQLMIN.DLL
      ProductVersion: 12.0.2480.0
      FileVersion: 2014.0120.2480.00 ((SQL14_RTM_QFE-CU).150128-1755)
      FileDescription: SQL Server Windows NT – 64 Bit
      LegalCopyright: Microsoft Corp. All rights reserved.
      LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
      Comments: SQL

      CU2 was installed in either Test or Prod server, I am confused only Test Server has this issue reported since a restart, you can see above version details.

      I am also confused, if this is a bug related to backup/restore, then the server restart will involve recovery process, and all databases are online now, just cannot do backup/restore, unless the recovery is not monitored by performance counters…

      • blakhani said

        Do you have any issue with performance counter? How many rows do you see when you query
        Select * from sys.dm_os_performance_counters

        • Albert said

          I followed the steps in this article to add the missing performance counter “Backup/Restore Throughput/sec” back http://blogs.technet.com/b/pfelatam/archive/2011/08/08/sql-performance-counters-are-missing.aspx
          now I have 32780 counters in sys.dm_os_performance_counters table,
          and for “Backup/Restore Throughput/sec”, each instance has 1 record now, but the mini dump still being generated continually in log folder…

        • Albert said

          I have checked the mini dump again, even the performance counter is in place, it is still looking for the performance counter…
          0:406> kc 1000
          Call Site
          sqlmin!GetObjOffsets
          sqlmin!PerfmonManager::AddInstance
          sqlmin!BackupPerfmonCounter::AddInstance
          sqlmin!BackupDevicePerfmonCounter::Init
          sqlmin!BackupMedium::CreateDeviceObject
          sqlmin!BackupMedium::Open
          sqlmin!BackupStream::OpenForBackup
          sqlmin!BackupStream::ThreadMainRoutine
          sqlmin!BackupThread::ThreadBase
          sqlmin!SubprocEntrypoint
          sqldk!SOS_Task::Param::Execute
          sqldk!SOS_Scheduler::RunTask
          sqldk!SOS_Scheduler::ProcessTasks
          sqldk!SchedulerManager::WorkerEntryPoint
          sqldk!SystemThread::RunWorker
          sqldk!SystemThreadDispatcher::ProcessWorker
          *** WARNING: Unable to verify timestamp for kernel32.dll
          sqldk!SchedulerManager::ThreadEntryPoint
          kernel32!BaseThreadInitThunk
          ntdll!RtlUserThreadStart

        • Albert said

          When yesterday, after adding the missing counter, I was able to create an empty database “test” and do backup, then I restart the sql server service, now today, I cannot create new database, backup or restore databases, when I create new database “test2”, I got below dump:
          0:000> kc 1000
          Call Site
          sqlmin!PerfmonManager::AddInstance
          sqlmin!DBTABLE::Startup
          sqlmin!DBMgr::StartupDB
          sqllang!CStmtCreateDB::CreateLocalDatabaseFragment
          sqllang!DBDDLAgent::CreateDatabase
          sqllang!CStmtCreateDB::XretExecute
          sqllang!CMsqlExecContext::ExecuteStmts
          sqllang!CMsqlExecContext::FExecute
          sqllang!CSQLSource::Execute
          sqllang!process_request
          sqllang!process_commands
          sqldk!SOS_Task::Param::Execute
          sqldk!SOS_Scheduler::RunTask
          sqldk!SOS_Scheduler::ProcessTasks
          sqldk!SchedulerManager::WorkerEntryPoint
          sqldk!SystemThread::RunWorker
          sqldk!SystemThreadDispatcher::ProcessWorker
          *** WARNING: Unable to verify timestamp for kernel32.dll
          sqldk!SchedulerManager::ThreadEntryPoint
          kernel32!BaseThreadInitThunk
          ntdll!RtlUserThreadStart

          It is related to performance counter… I notice all performance counters for ‘Backup/Restore Throughput/sec’, each instance only has one cntr_value in table, so it only successfully record cntr value once and fails afterwards… so strange…

        • Albert said

          ignore the last sentence in above reply…

      • Albert said

        Hello All, guess what, I installed a new instance(INSTANCE2) in same box without any problem, then I install CU6, I found I can tick INSTANCE1 which is the old instance, even @@version says it is on the latest version, but it is not, after applying the CU6 on INSTANCE1, the backup/restore issue is gone.

        The more I use SQL Server I more miss Oracle now.

    4. Albert said

      Hi Blakhani,

      Guess what, it is fixed now, how? I installed a new instance in same box, then install CU6, I found I can tick the old instance, which means the old instance didn’t have CU6 properly but no error, no log indicates that, after CU6 is installed, the issue is gone now.

      what a pity, this is not the first time we have the problem, in future we have to run update twice to make sure it is correct.

      • blakhani said

        Last patch can always be reinstalled so tick box enabled is normal behavior.
        Good to know the issue is resolved.

    5. I blog often and I genuinely appreciate your content. This article has really peaked my interest. I’m going to bookmark your site and keep checking for new details about once a week. I subscribed to your RSS feed too.

    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out / Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out / Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out / Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out / Change )

    Connecting to %s

     
    %d bloggers like this: