Help: SQL Server

Sharing my knowlege about SQL Server Troubleshooting Skills

Archive for the ‘SSMS’ Category

SQL SERVER – SSMS Database Expand Hang – High waits on PREEMPTIVE_OS_LOOKUPACCOUNTSID

Posted by blakhani on April 7, 2015


Recently I have had a friend who reported below issues.

  1. When I expand database, it takes a lot of time.
  2. When I expand Jobs under SQL Server Agent node in SQL Server Management Studio, it freezes and finally it fails with error  
    “An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
    Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding”

The symptom on the server that we were working on was that from Management Studio, when trying to browse SQL agent job, the interface would hang. I have asked to capture Profiler trace to find out which query is taking time and what is the waits for those query which are stuck.

In profiler trace, and the query to capture currently running queries (refer this blog) I found that it was running sp_help_job stored procedure from MSDB database. This is the procedure to get high level details about all jobs in MSDB database using msdb.dbo.sysjobs_view. When we looked further, we found that SQL Server is running function dbo.SQLAGENT_SUSER_SNAME and got stuck at SELECT @ret = SUSER_SNAME(@user_sid) statement. The wait for the session is was PREEMPTIVE_OS_LOOKUPACCOUNTSID and wait time was increasing. This wait is related to the communication/validation from Active Directory.

When we debugged further, here is the chain of reaching to function.

sp_help_job  
             >> sp_get_composite_job_info 
                     >> Query having – owner = dbo.SQLAGENT_SUSER_SNAME(sjv.owner_sid) 
                                >>  SELECT @ret = SUSER_SNAME(@user_sid)

This is getting stuck at PREEMPTIVE_OS_LOOKUPACCOUNTSID. We was identified that the function is used to convert SIDs stored in SQL Server table to the name by making call to Active Directory. Now the challenge was to find why and also was to identify if its happening with particular logins or all login. The complexity here was that SQL Server stores SID in varbinary format not in the format which OS would understand.

Luckily, I have had a blog post having script to convert the varbinary to well known format. So I have used that to convert SIDs obtained from below query

select    owner_sid 
from    msdb.dbo.sysjobs_view
where    owner_sid <> 0x01

 

Once we have SID value in OS understandable format, I used PsGetSID tool from sysinternals to get Windows account name. While running that it was taking a long time and finally it failed with below error.

Error querying SID:

The trust relationship between the primary domain and the trusted domain failed

So, it was something to do with two domains trust which seems to be broken. I asked him to work with this Windows Domain Admin team and networking team to get the issue resolved.

Same issue might happen during database expand also as, database owner is a SID stored in sys.databases and that has to be converted to name.

Hope this helps.

  • Cheers,
  • Balmukund Lakhani
  • Twitter @blakhani
  • Advertisements

    Posted in SQL Server Management Studio, SSMS, Troubleshooting | Tagged: , | 4 Comments »

    SSMS – FIX – Error – The log file is not using Unicode format

    Posted by blakhani on December 11, 2014


    While playing with SQL Server Agent Log setting, I changed some setting and found that I was not able to read SQLAgent Logs any more. Here is the error which I was getting in SSMS.

    I followed my own blog and executed T-SQL to read the Agent log and got same error.

    Msg 22004, Level 16, State 1, Line 13
    The log file is not using Unicode format.

    I looked around into the properties which I changed and one of them was “Write OEM error log”. As per MSDN documentation, this option writes the error log file as a non-Unicode file. This reduces the amount of disk space consumed by the log file. However, messages that include Unicode data may be more difficult to read when this option is enabled.

    And that’s precisely the error message says – “it’s not in unicode”.

    There are two ways to fix the error.

    Method 1 (UI Based)

    • Connect to Object Explorer to SSMS > Expand “SQL Server Agent” >  Right Click on “Error Logs” and choose “Configure”
    • Over there uncheck the box “Write OEM error log”

    Method 2 (T-SQL Based)

    Execute below query

    USE [msdb]
    GO
    EXEC msdb.dbo.sp_set_sqlagent_properties @oem_errorlog=0
    GO
    
    
    

    Both of the methods have same effect. Once done, restart SQL Agent service so that now the file is written in unicode format. You should be able to view the log in the viewer now.

    Hope this helps.

  • Cheers,
  • Balmukund Lakhani
  • Twitter @blakhani
  • Author: SQL Server 2012 AlwaysOnPaperback, Kindle
  • Posted in SQL Server, SQL Server Agent, SQL Server Management Studio, SSMS | Tagged: , , , , | Leave a Comment »

    SSMS Internals : What is the meaning of Wait Category in Activity Monitor Under Resource Waits

    Posted by blakhani on November 27, 2014


    Recently I was working with a friend on performance issue and she asked an simple question.

    I use activity monitor tool which is provided with management studio. When I look at the Resource Waits group, there is a column called Wait Category. What does that show and how data is picked? Here is the screenshot from my machine.

    At almost same time, same question popped in Facebook group (SQLBangalore) as well. So I decided to dig into more details. Captured the profiler and found that Activity Monitor uses two DMVs to show the values in that section (Resource Waits).

    • sys.dm_os_wait_stats is used to get information for completed waits
    • sys.dm_os_waiting_tasks for threads that are in an in-progress wait.

    Here are the various waits we could potentially see.

    Wait Category Count(Type)
    Backup 7
    Buffer I/O 12
    Buffer Latch 6
    Compilation 3
    Full Text Search 3
    Idle 26
    Latch 9
    Lock 21
    Logging 5
    Memory 6
    Network I/O 6
    Other 120
    Parallelism 3
    SQLCLR 17
    Transaction 18
    User Waits 1

     

    How did I find this? By running profiler! As per profiler, the waits related information in Activity Monitor (AM) is populated by procedure #am_generate_waitstats. This is created when AM is launched. The temporary table which is populated in #am_wait_types which has mapping of “display” values and DMV values.

     

    category_name wait_type ignore
    Backup BACKUP 0
    Backup BACKUP_CLIENTLOCK 0
    Backup BACKUP_OPERATOR 0
    Backup BACKUPBUFFER 0
    Backup BACKUPIO 0
    Backup BACKUPTHREAD 0
    Backup DISKIO_SUSPEND 0
    Buffer I/O ASYNC_DISKPOOL_LOCK 0
    Buffer I/O ASYNC_IO_COMPLETION 0
    Buffer I/O FCB_REPLICA_READ 0
    Buffer I/O FCB_REPLICA_WRITE 0
    Buffer I/O IO_COMPLETION 0
    Buffer I/O PAGEIOLATCH_DT 0
    Buffer I/O PAGEIOLATCH_EX 0
    Buffer I/O PAGEIOLATCH_KP 0
    Buffer I/O PAGEIOLATCH_NL 0
    Buffer I/O PAGEIOLATCH_SH 0
    Buffer I/O PAGEIOLATCH_UP 0
    Buffer I/O REPLICA_WRITES 0
    Buffer Latch PAGELATCH_DT 0
    Buffer Latch PAGELATCH_EX 0
    Buffer Latch PAGELATCH_KP 0
    Buffer Latch PAGELATCH_NL 0
    Buffer Latch PAGELATCH_SH 0
    Buffer Latch PAGELATCH_UP 0
    Compilation RESOURCE_SEMAPHORE_MUTEX 0
    Compilation RESOURCE_SEMAPHORE_QUERY_COMPILE 0
    Compilation RESOURCE_SEMAPHORE_SMALL_QUERY 0
    Full Text Search SOAP_READ 0
    Full Text Search SOAP_WRITE 0
    Full Text Search MSSEARCH 0
    Idle ONDEMAND_TASK_QUEUE 1
    Idle REQUEST_FOR_DEADLOCK_SEARCH 1
    Idle RESOURCE_QUEUE 1
    Idle LOGMGR_QUEUE 1
    Idle KSOURCE_WAKEUP 1
    Idle LAZYWRITER_SLEEP 1
    Idle BROKER_RECEIVE_WAITFOR 1
    Idle CHECKPOINT_QUEUE 1
    Idle CHKPT 1
    Idle BROKER_EVENTHANDLER 1
    Idle BROKER_TRANSMITTER 1
    Idle SERVER_IDLE_CHECK 1
    Idle SNI_HTTP_ACCEPT 1
    Idle SLEEP_BPOOL_FLUSH 1
    Idle SLEEP_DBSTARTUP 1
    Idle SLEEP_DCOMSTARTUP 1
    Idle SLEEP_MSDBSTARTUP 1
    Idle SLEEP_SYSTEMTASK 1
    Idle SLEEP_TASK 1
    Idle SLEEP_TEMPDBSTARTUP 1
    Idle SQLTRACE_BUFFER_FLUSH 1
    Idle TRACEWRITE 1
    Idle WAITFOR_TASKSHUTDOWN 1
    Idle WAIT_FOR_RESULTS 1
    Idle XE_DISPATCHER_WAIT 1
    Idle XE_TIMER_EVENT 1
    Latch VIEW_DEFINITION_MUTEX 0
    Latch DEADLOCK_ENUM_MUTEX 0
    Latch LATCH_DT 0
    Latch LATCH_EX 0
    Latch LATCH_KP 0
    Latch LATCH_NL 0
    Latch LATCH_SH 0
    Latch LATCH_UP 0
    Latch INDEX_USAGE_STATS_MUTEX 0
    Lock LCK_M_BU 0
    Lock LCK_M_IS 0
    Lock LCK_M_IU 0
    Lock LCK_M_IX 0
    Lock LCK_M_RIn_NL 0
    Lock LCK_M_RIn_S 0
    Lock LCK_M_RIn_U 0
    Lock LCK_M_RIn_X 0
    Lock LCK_M_RS_S 0
    Lock LCK_M_RS_U 0
    Lock LCK_M_RX_S 0
    Lock LCK_M_RX_U 0
    Lock LCK_M_RX_X 0
    Lock LCK_M_S 0
    Lock LCK_M_SCH_M 0
    Lock LCK_M_SCH_S 0
    Lock LCK_M_SIU 0
    Lock LCK_M_SIX 0
    Lock LCK_M_U 0
    Lock LCK_M_UIX 0
    Lock LCK_M_X 0
    Logging LOGBUFFER 0
    Logging LOGMGR 0
    Logging LOGMGR_FLUSH 0
    Logging LOGMGR_RESERVE_APPEND 0
    Logging WRITELOG 0
    Memory UTIL_PAGE_ALLOC 0
    Memory SOS_RESERVEDMEMBLOCKLIST 0
    Memory SOS_VIRTUALMEMORY_LOW 0
    Memory LOWFAIL_MEMMGR_QUEUE 0
    Memory RESOURCE_SEMAPHORE 0
    Memory CMEMTHREAD 0
    Network I/O ASYNC_NETWORK_IO 0
    Network I/O DBMIRROR_SEND 0
    Network I/O NET_WAITFOR_PACKET 0
    Network I/O OLEDB 0
    Network I/O MSQL_DQ 0
    Network I/O DTC_STATE 0
    Other EXECUTION_PIPE_EVENT_INTERNAL 0
    Other FAILPOINT 0
    Other INTERNAL_TESTING 0
    Other IO_AUDIT_MUTEX 0
    Other KTM_ENLISTMENT 0
    Other KTM_RECOVERY_MANAGER 0
    Other KTM_RECOVERY_RESOLUTION 0
    Other MSQL_SYNC_PIPE 0
    Other MIRROR_SEND_MESSAGE 0
    Other MISCELLANEOUS 0
    Other MSQL_XP 0
    Other REQUEST_DISPENSER_PAUSE 0
    Other PARALLEL_BACKUP_QUEUE 0
    Other PRINT_ROLLBACK_PROGRESS 0
    Other QNMANAGER_ACQUIRE 0
    Other QPJOB_KILL 0
    Other QPJOB_WAITFOR_ABORT 0
    Other QRY_MEM_GRANT_INFO_MUTEX 0
    Other QUERY_ERRHDL_SERVICE_DONE 0
    Other QUERY_EXECUTION_INDEX_SORT_EVENT_OPEN 0
    Other QUERY_NOTIFICATION_MGR_MUTEX 0
    Other QUERY_NOTIFICATION_SUBSCRIPTION_MUTEX 0
    Other QUERY_NOTIFICATION_TABLE_MGR_MUTEX 0
    Other QUERY_NOTIFICATION_UNITTEST_MUTEX 0
    Other QUERY_OPTIMIZER_PRINT_MUTEX 0
    Other QUERY_REMOTE_BRICKS_DONE 0
    Other QUERY_TRACEOUT 0
    Other RECOVER_CHANGEDB 0
    Other REPL_CACHE_ACCESS 0
    Other REPL_SCHEMA_ACCESS 0
    Other DBMIRROR_WORKER_QUEUE 0
    Other DBMIRRORING_CMD 0
    Other DBTABLE 0
    Other DAC_INIT 0
    Other DBCC_COLUMN_TRANSLATION_CACHE 0
    Other DBMIRROR_DBM_EVENT 0
    Other DBMIRROR_DBM_MUTEX 0
    Other DBMIRROR_EVENTS_QUEUE 0
    Other DEADLOCK_TASK_SEARCH 0
    Other DEBUG 0
    Other DISABLE_VERSIONING 0
    Other DLL_LOADING_MUTEX 0
    Other DROPTEMP 0
    Other DUMP_LOG_COORDINATOR 0
    Other DUMP_LOG_COORDINATOR_QUEUE 0
    Other DUMPTRIGGER 0
    Other EC 0
    Other EE_PMOLOCK 0
    Other EE_SPECPROC_MAP_INIT 0
    Other ENABLE_VERSIONING 0
    Other ERROR_REPORTING_MANAGER 0
    Other FSAGENT 1
    Other FT_RESTART_CRAWL 0
    Other FT_RESUME_CRAWL 0
    Other FULLTEXT GATHERER 0
    Other GUARDIAN 0
    Other HTTP_ENDPOINT_COLLCREATE 0
    Other HTTP_ENUMERATION 0
    Other HTTP_START 0
    Other IMP_IMPORT_MUTEX 0
    Other IMPPROV_IOWAIT 0
    Other ABR 0
    Other BROKER_REGISTERALLENDPOINTS 0
    Other BROKER_SHUTDOWN 0
    Other BROKER_TASK_STOP 1
    Other BAD_PAGE_PROCESS 0
    Other BROKER_CONNECTION_RECEIVE_TASK 0
    Other BROKER_ENDPOINT_STATE_MUTEX 0
    Other CURSOR 0
    Other CURSOR_ASYNC 0
    Other BUILTIN_HASHKEY_MUTEX 0
    Other CHECK_PRINT_RECORD 0
    Other BROKER_INIT 0
    Other BROKER_MASTERSTART 0
    Other SOSHOST_EVENT 0
    Other SOSHOST_INTERNAL 0
    Other SOSHOST_MUTEX 0
    Other SOSHOST_RWLOCK 0
    Other SOSHOST_SEMAPHORE 0
    Other SOSHOST_SLEEP 0
    Other SOSHOST_TRACELOCK 0
    Other SOSHOST_WAITFORDONE 0
    Other SOS_STACKSTORE_INIT_MUTEX 0
    Other SOS_SYNC_TASK_ENQUEUE_EVENT 0
    Other SHUTDOWN 0
    Other SOS_CALLBACK_REMOVAL 0
    Other SOS_DISPATCHER_MUTEX 0
    Other SOS_LOCALALLOCATORLIST 0
    Other SOS_OBJECT_STORE_DESTROY_MUTEX 0
    Other SOS_PROCESS_AFFINITY_MUTEX 0
    Other SNI_CRITICAL_SECTION 0
    Other SNI_HTTP_WAITFOR_0_DISCON 0
    Other SNI_LISTENER_ACCESS 0
    Other SNI_TASK_COMPLETION 0
    Other SEC_DROP_TEMP_KEY 0
    Other SEQUENTIAL_GUID 0
    Other VIA_ACCEPT 0
    Other SQLSORT_NORMMUTEX 0
    Other SQLSORT_SORTMUTEX 0
    Other WAITSTAT_MUTEX 0
    Other WCC 0
    Other WORKTBL_DROP 0
    Other XE_BUFFERMGR_ALLPROCECESSED_EVENT 0
    Other XE_BUFFERMGR_FREEBUF_EVENT 0
    Other XE_DISPATCHER_JOIN 0
    Other SQLTRACE_LOCK 0
    Other SQLTRACE_SHUTDOWN 0
    Other SQLTRACE_WAIT_ENTRIES 0
    Other SRVPROC_SHUTDOWN 0
    Other TEMPOBJ 0
    Other THREADPOOL 1
    Other TIMEPRIV_TIMEPERIOD 0
    Other XE_TIMER_MUTEX 0
    Other XE_TIMER_TASK_DONE 0
    Other XE_MODULEMGR_SYNC 0
    Other XE_OLS_LOCK 0
    Other XE_SERVICES_MUTEX 0
    Other XE_SESSION_CREATE_SYNC 0
    Other XE_SESSION_SYNC 0
    Other XE_STM_CREATE 0
    Parallelism CXPACKET 1
    Parallelism EXCHANGE 1
    Parallelism EXECSYNC 1
    SQLCLR CLR_AUTO_EVENT 1
    SQLCLR CLR_CRST 0
    SQLCLR CLR_JOIN 0
    SQLCLR CLR_MANUAL_EVENT 1
    SQLCLR CLR_MEMORY_SPY 0
    SQLCLR CLR_MONITOR 0
    SQLCLR CLR_RWLOCK_READER 0
    SQLCLR CLR_RWLOCK_WRITER 0
    SQLCLR CLR_SEMAPHORE 0
    SQLCLR CLR_TASK_START 0
    SQLCLR CLRHOST_STATE_ACCESS 0
    SQLCLR ASSEMBLY_LOAD 0
    SQLCLR FS_GARBAGE_COLLECTOR_SHUTDOWN 0
    SQLCLR SQLCLR_APPDOMAIN 0
    SQLCLR SQLCLR_ASSEMBLY 0
    SQLCLR SQLCLR_DEADLOCK_DETECTION 0
    SQLCLR SQLCLR_QUANTUM_PUNISHMENT 0
    Transaction TRAN_MARKLATCH_DT 0
    Transaction TRAN_MARKLATCH_EX 0
    Transaction TRAN_MARKLATCH_KP 0
    Transaction TRAN_MARKLATCH_NL 0
    Transaction TRAN_MARKLATCH_SH 0
    Transaction TRAN_MARKLATCH_UP 0
    Transaction TRANSACTION_MUTEX 0
    Transaction XACT_OWN_TRANSACTION 0
    Transaction XACT_RECLAIM_SESSION 0
    Transaction XACTLOCKINFO 0
    Transaction XACTWORKSPACE_MUTEX 0
    Transaction DTC_TMDOWN_REQUEST 0
    Transaction DTC_WAITFOR_OUTCOME 0
    Transaction MSQL_XACT_MGR_MUTEX 0
    Transaction MSQL_XACT_MUTEX 0
    Transaction DTC 0
    Transaction DTC_ABORT_REQUEST 0
    Transaction DTC_RESOLVE 0
    User Waits WAITFOR 1

     

    Hope this helps in understanding Activity monitor wait category little better.

  • Cheers,
  • Balmukund Lakhani
  • Twitter @blakhani
  • Author: SQL Server 2012 AlwaysOnPaperback, Kindle
  • Posted in Internals, SQL Server Management Studio, SSMS | Tagged: , , , , , , , | 3 Comments »