How to improve mysql performance using CACHING

A deep analisys of a performance bottleneck could be long and is based on different kind of data (OS metrics, IO performance tests, DB wait stats and so on…). Before starting with this (or simply after a new delivery) some simple checks could help you to correct rapidly some of the common (and bigger) bottlenecks you can find in a low-performance installation.
One of this is cache sizing.

A cache disabled or not correctly configured is a common reason of low-performance problems.
MySQL engine use a result-caching to improve query speed. A query result is cached in memory to avoid slow IO reading operations. Mysql use the query_cache_size parameter to size the engine cache. A result is saved in the cache only if it’s larger then the query_cache_min_res_unit parameter but smaller then the query_cache_limit.

A query result is kept in memory until:
– the result is valid (data are not modified)
– there are no more-used results using spaces in cache dedicated memory

A good sizing of cache instance parameters is based on instance activity. My suggestion are:
– Start with a simple configuration using system memory available (avoiding paging). note: by default mysql cache is DISABLED
– Start keeping query_cache_min_res_unit little.
– Check istance metrics after some days of normal database activity to understand how you can tune better cache parameters

This is an example of an installed configuration:

mysql> show variables like '%query_cache%';
| Variable_name                | Value    |
| have_query_cache             | YES      |
| query_cache_limit            | 2097152  |
| query_cache_min_res_unit     | 4096     |
| query_cache_size             | 33554432 |
| query_cache_type             | ON       |
5 rows in set (0.00 sec)

After some days of normal instance activity start watching some metrics to understand whats heppening.
Those values are self explained

mysql> show status like 'QCache%';
| Variable_name           | Value    |
| Qcache_free_blocks      | 957      |
| Qcache_free_memory      | 15523704 |
| Qcache_hits             | 1978478  |
| Qcache_inserts          | 132298   |
| Qcache_lowmem_prunes    | 6037     |
| Qcache_not_cached       | 5969     |
| Qcache_queries_in_cache | 3389     |
| Qcache_total_blocks     | 8396     |
8 rows in set (0.00 sec)

Talking about performances there are no “top configuration” usable. You have to work with users and system integrators to understand:
– Any application suggested cache configuration
– In low performance situations: what is the performance target ?
– Are there any test query usable to test configuration tuning?

For a deeper explanation of all mysql cache parameters start from here:


Anayze SQL default trace to investigate instance events

Quering default trace is the best way to investigate unusual or critical events heppened in SQL server and not logged in errorlog files.
It’s not difficult to find useful informations there but the trace is full of codes to translate to make it more readable and useful. This is my query, based on sys.fn_trace_gettable function and sys.trace_events system table.

DECLARE @TraceFileName NVARCHAR(512)  --Find the trace file name
SELECT @TraceFileName = path
  FROM sys.traces
 WHERE id = 1
 SELECT StartTime,  --then we can quering, translating the EventClass as EventClass , 
    SPID,           --this is the SPID of the session:   
    ObjectName      --can be used for following analysis 	
   FROM sys.fn_trace_gettable(@TraceFileName,default) TG
   left join
   sys.trace_events TE on TG.EventClass=TE.trace_event_id
   where is not null

If you are looking for the query that generated an event (log file grow, deleted object and so on…) you can look for it quering the inputbuffer for the SPID.
NOTE: the SPID can be re-used during the instance life and the inputbuffer is cleaned and re-used for following queries

dbcc inputbuffer([SPID])

The following is the full list of events available to filter in the previous query if you need to monitor a particular kind of event.
Not all are used in the default trace. If you need not traced event you have to create a custom trace.

select trace_event_id,name from sys.trace_events 

trace_event_id name
-------------- -------------------------------
10             RPC:Completed
11             RPC:Starting
12             SQL:BatchCompleted
13             SQL:BatchStarting
14             Audit Login
15             Audit Logout
16             Attention
17             ExistingConnection
18             Audit Server Starts And Stops
19             DTCTransaction
20             Audit Login Failed
21             EventLog
22             ErrorLog
23             Lock:Released
24             Lock:Acquired
25             Lock:Deadlock
26             Lock:Cancel
27             Lock:Timeout
28             Degree of Parallelism
33             Exception
34             SP:CacheMiss
35             SP:CacheInsert
36             SP:CacheRemove
37             SP:Recompile
38             SP:CacheHit
40             SQL:StmtStarting
41             SQL:StmtCompleted
42             SP:Starting
43             SP:Completed
44             SP:StmtStarting
45             SP:StmtCompleted
46             Object:Created
47             Object:Deleted
50             SQLTransaction
51             Scan:Started
52             Scan:Stopped
53             CursorOpen
54             TransactionLog
55             Hash Warning
58             Auto Stats
59             Lock:Deadlock Chain
60             Lock:Escalation
61             OLEDB Errors
67             Execution Warnings
68             Showplan Text (Unencoded)
69             Sort Warnings
70             CursorPrepare
71             Prepare SQL
72             Exec Prepared SQL
73             Unprepare SQL
74             CursorExecute
75             CursorRecompile
76             CursorImplicitConversion
77             CursorUnprepare
78             CursorClose
79             Missing Column Statistics
80             Missing Join Predicate
81             Server Memory Change
82             UserConfigurable:0
83             UserConfigurable:1
84             UserConfigurable:2
85             UserConfigurable:3
86             UserConfigurable:4
87             UserConfigurable:5
88             UserConfigurable:6
89             UserConfigurable:7
90             UserConfigurable:8
91             UserConfigurable:9
92             Data File Auto Grow
93             Log File Auto Grow
94             Data File Auto Shrink
95             Log File Auto Shrink
96             Showplan Text
97             Showplan All
98             Showplan Statistics Profile
100            RPC Output Parameter
102            Audit Database Scope GDR Event
103            Audit Schema Object GDR Event
104            Audit Addlogin Event
105            Audit Login GDR Event
106            Audit Login Change Property Event
107            Audit Login Change Password Event
108            Audit Add Login to Server Role Event
109            Audit Add DB User Event
110            Audit Add Member to DB Role Event
111            Audit Add Role Event
112            Audit App Role Change Password Event
113            Audit Statement Permission Event
114            Audit Schema Object Access Event
115            Audit Backup/Restore Event
116            Audit DBCC Event
117            Audit Change Audit Event
118            Audit Object Derived Permission Event
119            OLEDB Call Event
120            OLEDB QueryInterface Event
121            OLEDB DataRead Event
122            Showplan XML
123            SQL:FullTextQuery
124            Broker:Conversation
125            Deprecation Announcement
126            Deprecation Final Support
127            Exchange Spill Event
128            Audit Database Management Event
129            Audit Database Object Management Event
130            Audit Database Principal Management Event
131            Audit Schema Object Management Event
132            Audit Server Principal Impersonation Event
133            Audit Database Principal Impersonation Event
134            Audit Server Object Take Ownership Event
135            Audit Database Object Take Ownership Event
136            Broker:Conversation Group
137            Blocked process report
138            Broker:Connection
139            Broker:Forwarded Message Sent
140            Broker:Forwarded Message Dropped
141            Broker:Message Classify
142            Broker:Transmission
143            Broker:Queue Disabled
144            Broker:Mirrored Route State Changed
146            Showplan XML Statistics Profile
148            Deadlock graph
149            Broker:Remote Message Acknowledgement
150            Trace File Close
151            Database Mirroring Connection
152            Audit Change Database Owner
153            Audit Schema Object Take Ownership Event
154            Audit Database Mirroring Login
155            FT:Crawl Started
156            FT:Crawl Stopped
157            FT:Crawl Aborted
158            Audit Broker Conversation
159            Audit Broker Login
160            Broker:Message Undeliverable
161            Broker:Corrupted Message
162            User Error Message
163            Broker:Activation
164            Object:Altered
165            Performance statistics
166            SQL:StmtRecompile
167            Database Mirroring State Change
168            Showplan XML For Query Compile
169            Showplan All For Query Compile
170            Audit Server Scope GDR Event
171            Audit Server Object GDR Event
172            Audit Database Object GDR Event
173            Audit Server Operation Event
175            Audit Server Alter Trace Event
176            Audit Server Object Management Event
177            Audit Server Principal Management Event
178            Audit Database Operation Event
180            Audit Database Object Access Event
181            TM: Begin Tran starting
182            TM: Begin Tran completed
183            TM: Promote Tran starting
184            TM: Promote Tran completed
185            TM: Commit Tran starting
186            TM: Commit Tran completed
187            TM: Rollback Tran starting
188            TM: Rollback Tran completed
189            Lock:Timeout (timeout > 0)
190            Progress Report: Online Index Operation
191            TM: Save Tran starting
192            TM: Save Tran completed
193            Background Job Error
194            OLEDB Provider Information
195            Mount Tape
196            Assembly Load
198            XQuery Static Type
199            QN: Subscription
200            QN: Parameter table
201            QN: Template
202            QN: Dynamics
212            Bitmap Warning
213            Database Suspect Data Page
214            CPU threshold exceeded
215            PreConnect:Starting
216            PreConnect:Completed
217            Plan Guide Successful
218            Plan Guide Unsuccessful
235            Audit Fulltext

How to make your databases smaller and faster: find unused indexes

It’s a boring job but sometimes a good DBA has to do it.

Applications change and you have to understand what become unuseful in your databases: we are talking about unused indexes. In any SQL server database indexes take up a lot of space and have to be updated every time an application runs an update on an indexed table. You have to rebuild and reorganize them… and you have to backup them, every day.

SQL Server gives you a good instrument to understand which indexes are really used. SYS.DM_DB_INDEX_USAGE_STATS is a dynamic management view used by SQL engine to collect information about this.
The first time a new index is used SQL server adds a new line to this table with many different counters. Those counters are used to collect data about the index every time it’s used. On every instance restart SQL reset index counters.
Querying this view is easy to understad which indexes are really used:
– Indexes not listed here are never used
– Indexes with blank counters are never used since the last instance restart

Those are a queries ready to make this kind of analisys. Remembar that not every application use alway ALL their indexes. Some indexes are used only when specific application functions are turned on. For this reason talk ALWAYS with application support guys before dropping anything.

--Indexes Never Used

--Indexes never used since the last restart

FROM [sys].[indexes] i
JOIN [sys].[objects] o
LEFT JOIN [sys].[dm_db_index_usage_stats] u
AND i.[index_id] = u.[index_id]
AND u.[database_id] = @dbid 
WHERE o.[type] <> 'S'
AND i.[type_desc] <> 'HEAP'
AND u.[user_seeks] + u.[user_scans] + u.[user_lookups] = 0
ORDER BY i.indexname asc

Queries to see rapidly what your SQL Server is doing NOW

1) Blocked And Blocking queries.
If this query returns no rows you have no blocked queries in this moment. Run it more then once to see any few-seconds blocking queries. NOTE: This exclude ONLY problems with long-locking running queries. Cumulative short-term locking contentions need other kinds of debug (see point 2)

BlockedSPID=left(blocked.session_id,5) ,
FROM sys.dm_exec_requests blocked
JOIN sys.dm_exec_requests blocking
ON blocked.blocking_session_id = blocking.session_id
FROM sys.dm_exec_sql_text(blocked.sql_handle)
) blockedsql
FROM sys.dm_exec_sql_text(blocking.sql_handle)
) blockingsql

2) Time-Wait analysis
SQL Server collects informations about time wait events of your instance for every session. Every event (IO,CPU Processing,Locking and so on) is collected and showed in some dynamic management views from instance start/restart. To see what’s heppening now you can reset one af this views and collect for a short time windows events details for debug purpose. To understand the meaning of every SQL Wait Events see:
Following you can see a good wait analysis script to cross informations for a fast debug (source:

DBCC SQLPERF ('sys.dm_os_wait_stats', CLEAR); --reset DM view

FROM sys.dm_os_waiting_tasks [owt]
INNER JOIN sys.dm_exec_sessions [es] ON
	[owt].[session_id] = [es].[session_id]
INNER JOIN sys.dm_exec_requests [er] ON
	[es].[session_id] = [er].[session_id]
OUTER APPLY sys.dm_exec_sql_text ([er].[sql_handle]) [est]
OUTER APPLY sys.dm_exec_query_plan ([er].[plan_handle]) [eqp]
WHERE [es].[is_user_process] = 1
ORDER BY [owt].[session_id], [owt].[exec_context_id];

3) Open transactions with plan and sql texts
It’s really simple to see informations about current sessions using the old and trusty exec sp_who2 or the dynamic management view sys.dm_exec_requests
But if you need exactly what statements are running and wich plan are they using you need a more complicate query.
This is a good script from useful to see current transactions with detailed informations about every sessions running.

 SELECT s_tst.[session_id],
   s_es.[login_name] AS [Login Name],
   DB_NAME (s_tdt.database_id) AS [Database],
   s_tdt.[database_transaction_begin_time] AS [Begin Time],
   s_tdt.[database_transaction_log_record_count] AS [Log Records],
   s_tdt.[database_transaction_log_bytes_used] AS [Log Bytes],
   s_tdt.[database_transaction_log_bytes_reserved] AS [Log Rsvd],
   s_est. AS [Last T-SQL Text],
   s_eqp.[query_plan] AS [Last Plan]
FROM sys.dm_tran_database_transactions s_tdt
   JOIN sys.dm_tran_session_transactions s_tst
      ON s_tst.[transaction_id] = s_tdt.[transaction_id]
   JOIN sys.[dm_exec_sessions] s_es
      ON s_es.[session_id] = s_tst.[session_id]
   JOIN sys.dm_exec_connections s_ec
      ON s_ec.[session_id] = s_tst.[session_id]
   LEFT OUTER JOIN sys.dm_exec_requests s_er
      ON s_er.[session_id] = s_tst.[session_id]
   CROSS APPLY sys.dm_exec_sql_text (s_ec.[most_recent_sql_handle]) AS s_est
   OUTER APPLY sys.dm_exec_query_plan (s_er.[plan_handle]) AS s_eqp
ORDER BY [Begin Time] ASC;