Author: Cindy Gross | Befriending Dragons

  • SQL Server Performance Tools – Boise Code Camp Presentation

    Today I am presenting about SQL Server Performance Tools at the Boise Code Camp. You can download the slides and supporting files here on this blog (at the bottom it says Attachment(s): PerformanceTools.zip ). The basic agenda of items covered is:

     

    ¢  Methodology

    ¢  SQLDiag

    ¢  PSSDiag

    ¢  SQLNexus

    ¢  Profiler

    ¢  PerfMon

    ¢  References

    The perfstats script I discussed can be found at:
    And the perfstats analysis tools are at:

    PerformanceTools.zip

  • What do those “IO requests taking longer than 15 seconds” messages on my SQL box mean?

    You may be sometimes seeing stuck/stalled IO messages on one or more of your SQL Server boxes. This is something it is important to understand so I am providing some background information on it.

     

    Here is the message you may see in the SQL error log:

    SQL Server has encountered xxx occurrence(s) of IO requests taking longer than 15 seconds to complete on file [mdf_or_ldf_file_path_name] in database [dbname] (dbid). The OS file handle is 0x…. The offset of the latest long IO is: 0x….”.

     

    The message indicates that SQL Server has been waiting on at least one I/O for 15 seconds or longer. The exact number of times you have exceeded this time for the specified file since the last message is included in the message. The messages will not be written more than once every five minutes. Keep in mind that read IOs on an average system should take no more than 10-20ms and writes should take no more than 3-5ms (the exact acceptable values vary depending on your business needs and technical configuration). So anything measured in seconds indicates a serious performance problem. The problem is NOT within SQL Server, this message indicates SQL has sent off an IO request and has waited more than 15 seconds for a response. The problem is somewhere in the disk IO subsystem. For example, the disk IO subsystem may have more load than it is designed to handle, there is a “bad” hardware or firmware somewhere along the path, filter drivers such as anti-virus software are interfering, your file layout is not optimal, or some IO subsystem setting such as HBA queue depth is not set optimally.

     

    Though the root cause is IO, you can see other symptoms that are a side effect and may lead you down the wrong troubleshooting path. For example, if enough IO is backed up behind the stalled IO then you may see blocking in SQL Server (because locks that are usually taken for very short periods of time are now held for seconds), new connections may not be allowed, and the CPU usage can increase (because many threads are waiting), and a clustered SQL Server can fail over (because the IsAlive checks which are just SQL queries fail to complete like all the other queued queries). You may see other errors returned to the user or in the various logs, such as timeouts.

     

    There are two ways to approach this problem. You can either reduce the IO on the system (change indexes or queries or archive data for example) or you can make the underlying system able to handle the IO load (fix hardware/firmware problems, change configurations, add disks or controllers, change the file layout, etc.).

     

    Background:

    ·         897284  Diagnostics in SQL Server 2000 SP4 and in later versions help detect stalled and stuck I/O operations

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;897284

    ·         Detecting and Resolving Stalled and Stuck I/O Issues in SQL Server 2000 SP 4 http://msdn.microsoft.com/en-us/library/aa175396(SQL.80).aspx

     

    Troubleshooting:

    ·         Every Windows 2003 SP1 or SP2 system should have this storport fix: 941276  A Windows Server 2003-based computer stops responding when the system is under a heavy load and when the Storport driver is being used http://support.microsoft.com/default.aspx?scid=kb;EN-US;941276

    ·         Use PerfMon to look at the disk counters for sec/read, sec/write, bytes/sec, current disk queue length, reads/sec, writes/sec

    ·         Collect data from sys.dm_io_virtual_file_stats and sys.dm_io_pending_io_requests.

    ·         Ask your storage admins to monitor the entire IO subsystem from the Windows system all the way through to the underlying disks.

  • How to Rename SQL Server

    How to rename a SQL Server varies a bit depending on the SQL version, whether it is clustered or not, and whether you want to rename the server/virtual server part of the name (works except for SQL 2000 clusters) or the instance part of the name (requires a reinstall). Also, you do not want to try renaming a server involved in replication as it will break replication (you have to drop/recreate all replication after a rename), and there are extra steps if mirroring is involved (stop mirroring before the rename, change configuration after). Be very careful to include the keyword “local” in the sp_addserver part of the steps (applies only to stand alone systems) and check @@SERVERNAME afterwards to make sure you have completed the steps correctly.

     

    SQL 2008:

    ·         How to: Rename a SQL Server Failover Cluster Instance http://msdn.microsoft.com/en-us/library/ms178083.aspx

    ·         How to: Rename a Computer that Hosts a Stand-Alone Instance of SQL Server http://msdn.microsoft.com/en-us/library/ms143799.aspx

     

    SQL 2005:

    ·         How to: Rename a SQL Server 2005 Virtual Server http://msdn.microsoft.com/en-us/library/ms178083(SQL.90).aspx

    ·         How to: Rename a Computer that Hosts a Stand-Alone Instance of SQL Server 2005 http://msdn.microsoft.com/en-us/library/ms143799(SQL.90).aspx

     

    SQL 2000:

    ·         The SQL Server Network Name resource cannot be renamed http://support.microsoft.com/kb/307336

    ·         Renaming a Server http://msdn.microsoft.com/en-us/library/aa197071(SQL.80).aspx

     

    From the BOL topics we can see that you can NOT rename the instance part of the name in SQL Server 2000, 2005, or 2008:

     

    SQL 2005 cluster:

    “The name of the virtual server is always the same as the name of the SQL Network Name (the SQL Virtual Server Network Name). Although you can change the name of the virtual server, you cannot change the instance name. For example, you can change a virtual server named VS1instance1 to some other name, such as SQL35instance1, but the instance portion of the name, instance1, will remain unchanged.”

     

    SQL 2005 standalone:

    “These steps can be used only to rename the part of the instance name that corresponds to the computer name. For example, you can change a computer named MB1 that hosts an instance of SQL Server named Instance1 to another name, such as MB2. However, the instance portion of the name, Instance1, will remain unchanged. In this example, the \ComputerNameInstanceName would be changed from \MB1Instance1 to \MB2Instance1.”

     

    SQL 2008 cluster:

    “Although you can change the name of the virtual server, you cannot change the instance name. For example, you can change a virtual server named VS1instance1 to some other name, such as SQL35instance1, but the instance portion of the name, instance1, will remain unchanged.”

     

    SQL 2008 standalone:

    “The following steps cannot be used to rename an instance of SQL Server. They can be used only to rename the part of the instance name that corresponds to the computer name. For example, you can change a computer named MB1 that hosts an instance of SQL Server named Instance1 to another name, such as MB2. However, the instance part of the name, Instance1, will remain unchanged. In this example, the \ComputerNameInstanceName would be changed from \MB1Instance1 to \MB2Instance1.”

     

     

  • SQL DAC

    When you start SQL Server (2005+) it creates a separate “Dedicated Administrator Connection” or DAC using a special TCP port. One sysadmin at a time can connect with this DAC connection by specifying Admin:ServerNameInstance. From SQLCMD you can either prefix the server name with Admin: or you can use the /A switch. From SSMS you can use Admin: to make a “Query Editor” connection but you cannot use it in Object Explorer. DAC should only be used when other connection methods fail and you must collect information or think you might be able to kill some SPIDs to improve the situation. On a local connection you can always use DAC (as long as you are a sysadmin and no one else is using it) but for remote connections (and all connections to a cluster are considered remote) you must have enabled remote connections for DAC:

    EXEC sp_configure ‘remote admin connection’, 1

    RECONFIGURE

     

    If you want to see who if anyone is connected using DAC, try this query:

    SELECT dec.local_tcp_port AS DAC_Port, des.login_name AS LoginName, des.nt_domain AS NTDomain,

          des.nt_user_name AS NTUserName, dec.session_id AS SPID,

          dec.connect_time AS ConnectTime, dec.last_read AS LastRead, dec.last_write AS LastWrite,

          des.host_name AS HostName, dec.client_net_address AS ClientIP, des.program_name AS AppName,

          e.state AS EndpointState, e.is_admin_endpoint AS IsAdminEndpoint

          FROM sys.dm_exec_connections dec

          JOIN sys.endpoints e ON e.endpoint_id = dec.endpoint_id

          JOIN sys.dm_exec_sessions des ON des.session_id = dec.session_id

          WHERE e.name = ‘Dedicated Admin Connection’

     

    If you have any Express Editions, you have to use trace flag 7806 to enable DAC for Express.

     

    Using a Dedicated Administrator Connection

    http://msdn.microsoft.com/en-us/library/ms189595.aspx

     

    How to: Use the Dedicated Administrator Connection with SQL Server Management Studio

    http://msdn.microsoft.com/en-us/library/ms178068.aspx

  • Remote Desktop Options

    You probably use remote desktop to connect to your servers or other desktops. You may see Remote Desktop referred to as RDP, RDS, or MSTSC. One common concern is how to resize the remote desktop connection. You can use mstsc.exe to open your connection and pass /f for full screen. You can also pass in the specific pixel count with /w: and /h:. If you have multiple monitors on both the local and remote machines, you can use /span to be able to use both monitors. If you prefer the GUI, when you launch mstsc.exe, click on the options button. On the display tab you have a slider bar of less…more for size. Something else you may find useful is making your local desktop drives available on your RDP session. On the “local resources” tab click on “more” and then check “drives” in the white box. Back on the general tab you can choose to save the current connection settings and use them again.

     

    If you want to see what documented options are available from the command line run mstsc.exe /? and take a look at the popup window. Alternatively you can look at this site: http://technet.microsoft.com/en-us/library/cc753907(WS.10).aspx.

     

    Additional RDP/MSTSC resources:

    ·         Remote Desktop Services (Terminal Services) Team Blog http://blogs.msdn.com/rds/default.aspx

    ·         Remote Desktop scripting http://blogs.msdn.com/rds/archive/2009/11/04/announcing-the-launch-of-remote-desktop-services-script-center-to-ease-management.aspx,  http://technet.microsoft.com/en-us/scriptcenter/ee364707.aspx

    ·         MSTSC.EXE – no more /console switch in RDC 6.1 http://blogs.technet.com/askperf/archive/2008/01/04/mstsc-exe-no-more-console-switch-in-rdc-6-1.aspx

  • How to slipstream SP1 based CU5 into SQL Server 2008

    Starting with SQL Server 2008 SP1 you can “slipstream” SP1 and/or SP1 + SP1 based CUs to avoid a multi-step installation process.

     

    http://blogs.msdn.com/petersad/archive/2009/04/16/create-a-merged-slipstream-drop-containing-sql-server-2008-server-pack-1-and-a-cumulative-update-cu-based-on-server-pack-1.aspx

     

    To install a slipstreamed instance with SQL08+SP1+SP1-based-CU5 I followed these steps based on the above blog by Peter Saddow:

    1)      You will probably want to rename the directory (and therefore update the CUSource entry in the Defaultsetup.ini files) to indicate exactly which CU you have slipstreamed.

    2)      Instead of the blog reference in his step 1, you can optionally follow the more official KB 955392, choosing option/procedure 2 (Create a merged drop) to create the slipstream files of RTM+SP1.

    a.       To make sure I got the commands exactly right I cut/pasted them into a .cmd file and ran that.

    b.      Do NOT do step 7 “Start the Setup program” because you still need to add the CU files.

    3)      Download your CU of choice which is likely different than what is listed in the blog. For SP1 based CU5 you will download the first file (SQL_Server_2008_SP1_Cumlative_Update_5) from http://support.microsoft.com/kb/975977.

    4)      For CU5, run 399273_intl_x64_zip.exe, 399273_intl_ia64_zip.exe, and 399273_intl_ia64_zip.exe to extract the files SQLServer2008-KB975977-{architecture}.exe (which are the actual setup files for each architecture). You will need the password from the hotfix email you got from the 975977 link/email.

    5)      In the blog’s step 4 which tells you to extract files from the setup files, use your actual KB number. For SP1-based-CU5 the KB is 975977 so the commands will be SQLServer2008-KB975977-{architecture}.exe /x:c:SQLServer2008_FullSP1_CUCU

    6)      Run setup with the “run as admin” option on Windows 2008/Vista/Windows 7.

    7)      If you later move the setup files to a different location you will need to update the Defaultsetup.ini file in each of the architecture directories (x86, x64, ia64).

  • How to Find the Amount of Fragmentation on Your SQL Server Instance

    Need to find the amount of fragmentation in all your indexes in all databases for a given instance of SQL Server 2005 or later? You can use the undocumented/unsupported sp_MSforeachdb to loop through all the databases. For more information on fragmentation see my previous blog http://blogs.msdn.com/cindygross/archive/2009/11/20/sql-server-and-fragmentation.aspx.

     

    — Cindy Gross 2009

    — find fragmentation on all indexes in all databases on this instances

    — to find fragmentation on just one db comment out the exec master… line and the last quote

    — you must use the db_id() rather than NULL as the first parameter of the DMV or it will try to do a join across all dbs

    SELECT @@SERVERNAME, @@VERSION, GETDATE() as BatchStartTime

    exec master.sys.sp_MSforeachdb ‘ USE [?];

     

    DECLARE @starttime datetime, @endtime datetime

    SELECT @starttime = GETDATE()

    SELECT db_name() as CurrentDB, @starttime as DBStartTime

    SELECT db_name() as DatabaseName, OBJECT_NAME (a.object_id, DB_ID()) as ObjectName,

    a.index_id, b.name as IndexName,

    avg_fragmentation_in_percent, page_count, index_depth, index_type_desc, alloc_unit_type_desc

    — , record_count, avg_page_space_used_in_percent –(null in limited)

    FROM sys.dm_db_index_physical_stats (db_id(), NULL, NULL, NULL, NULL) AS a

    JOIN sys.indexes AS b

    ON a.object_id = b.object_id AND a.index_id = b.index_id

    –WHERE index_id > 0 — exclude heaps

    ORDER BY DatabaseName asc, avg_fragmentation_in_percent desc, ObjectName asc, IndexName asc

    SELECT @endtime = GETDATE()

    SELECT @starttime as StartTime, @endtime as EndTime, DATEDIFF(MINUTE,@starttime,@endtime) as TotalMinutes

    ‘

  • How and Why to Enable Instant File Initialization

    See my new blog post (written with Denzil Ribeiro) about “How and Why to Enable Instant File Initialization” on our PFE blog. Keep an eye on the PFE blog for more posts from my team in the near future.

  • Professional SQL Server 2008 Internals and Troubleshooting

    Our new book, Professional SQL Server 2008 Internals and Troubleshooting, will be shipping soon! Order now! 🙂 Christian Bolton, Justin Langford, Brent Ozar, and James Rowland-Jones have each written several chapters in this book. Steven Wort, Jonathan Kehayias and I each contributed a chapter as well. The 1st half of the book introduces you to how things work within SQL Server at a level that will make it easier to understand the rest of the book. The 2nd half of the book focuses on troubleshooting common SQL Server problems.

    Download the first chapter and find out more about the book here: http://sqlservertroubleshooting.com/.

    Chapters include:

    1. SQL Server Architecture
    2. Understanding Memory
    3. SQL Server Waits and Extended Events
    4. Working with Storage
    5. CPU and Query Processing
    6. Locking and Latches
    7. Knowing Tempdb
    8. Defining Your Approach to Troubleshooting
    9. Viewing Server Performance with PerfMon and the PAL Tool
    10. Tracing SQL Server with SQL Trace and Profiler
    11. Consolidating Data Collection with SQLDiag and the PerfStats Script
    12. Introducing RML Utilities for Stress Testing and Trace File Analysis
    13. Bringing It All Together with SQL Nexus
    14. Using Management Studio Reports and the Performance Dashboard
    15. Using SQL Server Management Data Warehouse
    16. Shortcuts to Efficient Data Collection and Quick Analysis

    http://rcm.amazon.com/e/cm?lt1=_blank&bc1=000000&IS2=1&bg1=FFFFFF&fc1=000000&lc1=319D3A&t=cinisthou-20&o=1&p=8&l=as1&m=amazon&f=ifr&asins=0470484284

  • SQL Server’s Default Trace

    Are you familiar with SQL Server’s default trace setting? It can be helpful with finding basic who/when type information on major events. For example, you may want to know who was creating and dropping databases on a given instance.

     

    SQL Server has a couple of options that might help you find out more about when/by who the database is being created and dropped. One is Policy Based Management but you would need to configure it ahead of time. Another option is to run a profiler trace that captures information such as CREATE, ALTER, DROP DATABASE. Some of the DMVs might have the execution information if you capture it fast enough after it happens. XEvents can be used in SQL 2008 to find all sorts of information. However, the one that might be most appropriate in this case is the Default Trace.

     

    1)      Make sure the default trace is enabled in your configuration options for this instance. If it is not enabled, you can enable it through the sp_configure settings.

    — Check to see if the default trace is enabled (0=off, 1=on)

    EXEC sp_configure ‘default trace enabled’

    GO

    — Enabled the default trace

    EXEC sp_configure ‘default trace enabled’, 1

    GO

    RECONFIGURE

    2)      The trace files will eventually overwrite themselves, so check for the output soon after the problem occurs (perhaps make periodic copies of the files). They will be under the log directory where SQL Server is installed. For example, for my SQL 2008 instance named WASH the output files are in C:Program FilesMicrosoft SQL ServerMSSQL10.WASHMSSQLLog. The files will be named log_xxx.trc and there will be up to 5 of them.  

    3)      Find the trace which covers the time period when the database was created or dropped. You can either open it in the Profiler GUI or you can use the query below to pull out the appropriate data. Look for the create and/or drop events and see who executed them from what workstation and at what time. Some applications will send their “application name” so you may be able to tell that as well.

     

    Key Points:

    ·         You cannot control what is captured by the default trace, how many files it captures before rolling over, or any other options. Your only option is to turn it on or off. If you want a similar trace that differs in any way you can create your own and configure it to start when SQL Server starts (or whatever time period is appropriate).

    ·         The trace file name/number will continue to increase until you delete the files.

    ·         The trace does NOT capture all events, it is very lightweight.

     

    References:

    ·         SQL Server 2008 Internals – Chapter 1 page 73

    ·         Searching for a Trace – Solving the mystery of SQL Server 2005’s default trace enabled option http://www.sqlmag.com/Articles/ArticleID/48939/pg/1/1.html

    ·         SQL Server Default Trace http://blogs.technet.com/beatrice/archive/2008/04/29/sql-server-default-trace.aspx

    ·         Default Trace in SQL Server 2005 http://blogs.technet.com/vipulshah/archive/2007/04/16/default-trace-in-sql-server-2005.aspx

    ·         Default Trace in SQL Server 2005 http://www.mssqltips.com/tip.asp?tip=1111

     

    Query:

    — Example of using the default trace to find out more about who/when/why a database is dropped or created

     

    — Get current file name for existing traces

    SELECT * FROM ::fn_trace_getinfo(0)

     

    — CHANGE THIS VALUE to the current file name

    DECLARE @Path nvarchar(2000)

    SELECT  @Path = ‘C:Program FilesMicrosoft SQL ServerMSSQL10.WASHMSSQLLoglog_120.trc’

     

    — Get information most relevant to CREATE/DROP database

    SELECT SPID, LoginName, NTUserName, NTDomainName, HostName, ApplicationName, StartTime, ServerName, DatabaseName

          ,CASE EventClass

                WHEN 46 THEN ‘CREATE’

                WHEN 47 THEN ‘DROP’

                ELSE ‘OTHER’

           END AS EventClass

          , CASE ObjectType

                WHEN 16964 THEN ‘DATABASE’

                ELSE ‘OTHER’

           END AS ObjectType

          –,*

    FROM fn_trace_gettable

    (@Path, default)

    WHERE ObjectType = 16964 /* Database */ AND EventSubClass = 1 /* Committed */

    ORDER BY StartTime

    GO

     

    /* BOL

     

    == Event Class

    46 Object:Created

     Indicates that an object has been created, such as for CREATE INDEX, CREATE TABLE, and CREATE DATABASE statements.

    47 Object:Deleted

     Indicates that an object has been deleted, such as in DROP INDEX and DROP TABLE statements.

     

     == Object Type

     16964 Database

     

    == EventSubClass

     int Type of event subclass.

    0=Begin

    1=Commit

    2=Rollback

     */