VMSTAT is a tool on Unix type plaforms that provides an impressive amount of resource information, but is formatted in such a way that it can be difficult to read much less determine when the snapshot occurred if looking at historic snapshots. Here is an example

[root@new-host ~]# vmstat 2
procs ———–memory———- —swap– —–io—- –system– —–cpu——
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 1916352 39376 906944 0 0 186 39 1024 1139 1 2 93 4 0
1 0 0 1916352 39384 906968 0 0 0 76 1015 1203 0 2 95 4 0
0 0 0 1916260 39392 906976 0 0 0 58 1014 1118 0 2 98 0 0

Through some simple Unix scripting commands, you can get more user-friendly information from this tool.

The vmstat_tool.sh script below works for most Linux versions:

#!/bin/sh
#vmstat_tool.sh
#vmstat parser script. Makes the information easily readable.

#Ensure that the number of loops and delay parameters
#were passed to this script
if [ $# -ne 2 ]
then
echo “Usage: $0 ”
exit
fi

export LPCNT=$1
export DLAY=$2

c=0

while [ "$c" -lt $LPCNT ]
do
date
vmstat | tail -n 1 | awk ‘{printf(“%d processes are waiting\n%d processes are blocked\n%d virtual memory used, %d idle memory, %d buffer memory, %d cache memory\n%d memory swapped in from disk, %d memory swapped out to disk\n%d blocks per second written\n%d blocks per second read\n%d interrupts per second\n%d context switches per second\nCPU usage: %d%% (user, non-kernel), %d%% (system, kernel), %d%% idle, %d%% wait\n\n”,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)}’

sleep $DLAY
c=$((c+1))
done

Sample Output (Single Loop):

Sat Feb 18 11:15:48 PST 2012
1 processes are waiting
0 processes are blocked
0 virtual memory used, 1990308 idle memory, 32964 buffer memory, 843564 cache memory
0 memory swapped in from disk, 0 memory swapped out to disk
928 blocks per second written
75 blocks per second read
1062 interrupts per second
1128 context switches per second
CPU usage: 4% (user, non-kernel), 2% (system, kernel), 77% idle, 17% wait

As you can see. the output is much more readable than the regular vmstat output.

Should you need to do more than just a cursory examination of the VMSTAT output, you can use these scripts to generate an output file for examination with the following method:

nohup vmstat_tool.sh 288 300 > ./vmstat.log &

In the above example, you are using the native nohup command to run the vmstat_tool.sh script to loop every 5 minutes (300 seconds) for 24 hours (288 5-minute loops). The output will to the vmstat.log script in the local directory and will run in the background by using ‘&’ at the end of the command. Examining the output is a simple matter of using the native cat and grep commands in the following manner:

cat ./vmstat.log | grep “processes are waiting” | more

Sample Output:

0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked

You can now see the output of the first line of the VMSTAT tool scrolled one screen at a time. If you see a line that has some abnormal values, you can then use the vi utility to zero in on that particular value and get the timestamp from it.

info@enkitec.com

• CPU Wait (new metric in 11g Performance page)
• Latency issues on virtualized environments

So if you are attending the RMOUG training days, stop by at my session @ room 401 Thursday 9:45 am-10:45 am

There will also be a RAC Attack at the exhibition area so that must be fun

and don’t forget to follow RMOUG’s twitter @RMOUG_ORG for updates during the conference!

See yah!

Update: 

I had a great time at RMOUG, was able to meet old/new friends and had a good number of attendees. The organizers really did a good job  Here’s the updated PPT I used during the presentation.
That’s the most recent, the one on the RMOUG is not yet updated. For the scripts http://karlarao.wordpress.com/scripts-resources/ they are all at the “PerformanceAndTroubleshooting” folder

Thanks to all who attended my presentation, I know that’s a bit too much technical stuff for 1hour presentation squeezing all 101 slides and all that info. The best way is to go through my notes on the slides, check out all the links I pointed, and try to poke around on the scripts. And I assure you will learn tons of stuff & greatly appreciate the material  :)

SQLTXPLAIN:

I’ve been using SQLTXPLAIN for quite a while, and I can say I’m a really big fan. It’s a cool tool to use for systematic SQL Tuning so I got them all neatly documented here http://karlarao.tiddlyspot.com/#SQLTXPLAIN.

BTW, Carlos Sierra, the author of the tool will present at Hotsos 2012 http://www.hotsos.com/sym12/sym_speakers_sierra.html so I may drop by the conference and say hello

The wiki link will probably show you all the things you need to know about SQLTXPLAIN. My personal use case favorites would be the following:

• Mining 10053 output
  • On the execution plan view of the main HTML report whenever you hover on each step of the plan it will pop up an info about the Query Block (QB) details which you can mine on the raw 10053 text file that is also on the zip file. This is helpful when troubleshooting the specific part of the SQL where it has gone wrong.
• Instant SQL Monitor output
  • The HTML and text versions of SQL Monitor is readily available for the particular SQL_ID you are troubleshooting
• Compare good/bad plan of SQLs
  • This is pretty handy because the output report of the SQLT-compare highlights the difference in red text. So instead of chasing some delta numbers, you can just go straight to the red text!
• Fix bad stats
  • Let’s say you have a prod and dev that have completely different run times. And you found out that the problem is the stats, you can do a SQLT on the good plan and transfer the stats to the other environment. You will be using the sqltimp.sql for this
• Transfer stored outline/profiles
  • This makes use of exp/imp to transfer and outline + DBMS_SQLTUNE for the SQL Profile
  • Manual SQL Profile is readily available as sqlt_<sqlt_id>_sqlprof.sql or can be created using sqlt/utl/coe_xfr_sql_profile.sql
• Reproducing an execution plan from one system to another and Creating test cases
  • I usually pull SQLT from the server I’m troubleshooting and load it on my laptop. And from there I can have the same optimizer environment then I can do anything I want with it. (well you can also do this from prod then load the SQLT to dev/QA environment)
  • I mainly use the SQLT-tc (test case builder) and that is totally different from the TCB (test case builder) which uses the DBMS_SQLDIAG.. wait, am I confusing you with the TCB terms?
    • Let me give you a little background

Each SQLTXPLAIN run (START sqltxtract.sql SQL_ID) packages everything in one zip file

[oracle@karl sqlt_s54491-good]$ du -sm sqlt_s54491.zip
 47 sqlt_s54491.zip

and inside that zip file includes the test case zip file that has a *_tc.zip extension (sqlt_s54491_tc.zip)
Note that this is different from the *_tcb.zip extension (sqlt_s54491_tcb.zip) which is using the DBMS_SQLDIAG TCB (test case builder) package

[oracle@karl sqlt_s54491-good]$ unzip -l sqlt_s54491.zip
 Archive: sqlt_s54491.zip
 Length Date Time Name
 --------- ---------- ----- ----
 104557695 11-27-2011 02:36 sqlt_s54491_10053_explain.trc
 2587 11-27-2011 09:18 sqlt_s54491_driver.zip
 660035 11-27-2011 02:35 sqlt_s54491_lite.html
 28711 11-27-2011 09:18 sqlt_s54491_log.zip
 34589692 11-27-2011 02:35 sqlt_s54491_main.html
 316568 11-27-2011 09:18 sqlt_s54491_opatch.zip
 40487 11-27-2011 02:35 sqlt_s54491_p73080644_sqlprof.sql
 21526 11-27-2011 02:35 sqlt_s54491_readme.html
 461504 11-27-2011 02:35 sqlt_s54491_sql_detail_active.html
 385239 11-27-2011 02:35 sqlt_s54491_sql_monitor_active.html
 424680 11-27-2011 02:35 sqlt_s54491_sql_monitor.html
 57213 11-27-2011 02:35 sqlt_s54491_sql_monitor.txt
 333117 11-27-2011 02:35 sqlt_s54491_sta_report_mem.txt
 1488 11-27-2011 02:35 sqlt_s54491_sta_script_mem.sql
 616556 11-27-2011 02:36 sqlt_s54491_tcb.zip         <-- ''DBMS_SQLDIAG TCB'' (test case builder) package zip file
 8659 11-27-2011 09:18 sqlt_s54491_tc_script.sql
 7793 11-27-2011 09:18 sqlt_s54491_tc_sql.sql
 4254816 11-27-2011 09:18 sqlt_s54491_tc.zip         <-- ''SQLT-tc test case'' zip file
 33308174 11-27-2011 09:18 sqlt_s54491_trc.zip
 --------- -------
 180076540 19 files

• And from the wiki, you will see the related wiki topics that starts with “testcase -”. Actually, you can generate test cases using the following methods:
  • by Manually reproducing the SQL
  • by using SQLT-tc 
  • by using the DBMS_SQLDIAG TCB (test case builder) package
  • by using the DBMS_SQLDIAG TCB from ADRCI

• And last but not the least, the most awesome part is 1) using the generated SQLT zip file, 2) building the testcase using SQLT-tc, and 3) then generating a DB Optimizer VST on the SQL_ID!
  • Right after the DB Optimizer http://dboptimizer.com/db-optimizer was released last 2010, and right after I got the tool for $99! http://www.freelists.org/post/oracle-l/Special-Offer-for-readers-of-OracleL I was chatting with Kyle Hailey about the possibilities of making a VST out of SQLTXPLAIN.. and all along I was right

DB Optimizer:

Below you will see that the highlighted table name starts with the schema TC<sqlt_id>, that means it was generated from SQLT-tc and I was also able to reproduce underlying objects of the SQL_ID and the optimizer environment on my laptop.

The cool thing about DB Optimizer is that it automates a lot of tedious tasks for you. It takes away the pain of manual diagramming just like what I’m doing here http://goo.gl/VZFB that turns out just look like this http://goo.gl/ttrh

I’m a very visual guy, and I can have a better understanding of the problem and solve things quickly (and creatively) by making use of visualization.

The highlighted table on the diagram below shows a missing join condition. This version of the SQL takes FOREVER!… here with DB Optimizer, right after it prints the VST diagram and correlating it with the “merge join cartesian” on the SQLT.. it’s pretty easy to identify the missing join (see highlighted SQL on the left) which is the root cause of the performance problem.

and a quick modification on the SQL made it run for just 10 seconds! that’s the power of SQLTXPLAIN and DB Optimizer VST 

You may also want to check out this ultimate SQL tune off by Kyle Hailey and Jonathan Lewis to know more about DB Optimizer http://dboptimizer.com/2010/06/14/jonathan-lewis-kyle-hailey-webinar-replay/

And my example usage of the tool on a high load SAP OLTP environment – DB Optimizer example – 3mins to 10secs

That’s it for now, hope I’ve shared you some good stuff

February 16th, in Denver at RMOUG Training Days 2012 - Topic: DIY Exadata
February 23rd, in Redwood Shores at NoCOUG Winter Conference - Topic: DIY Exadata
March 7th, in Dallas at Hotsos Symposium 2012 - Topic: Exadata Optimization – Case Studies

Hope to see you at one of these events.