XCTEND lines are reported as “COMMIT/ROLLBACK (synthetic)”. Using Goolge and Metalink I can’t find any other resources describing “COMMIT/ROLLBACK (synthetic)”. This term seems not be widely used, although Hotsos uses the same term. Could you please elaborate what exactly that is and why it possibly happens?

To understand what “synthetic” means, let’s have a look to two small trace files.The first one is generated by tracing the execution of the following SQL statements in SQL*Plus:

UPDATE scott.emp SET sal = sal*1.15;
COMMIT;

The relevant part of the trace file is the following. Notice that:

• There are two cursors: the first one is the update, the second one is the commit.
• In the second one, because it is a commit, between the PARSE and the EXEC lines there is a XCTEND line. Note that the database engine emmits a XCTEND line for every commit or rollback. To differentiate the two, the attribute “rlbk” is used: 0=commit, 1=rollback.
• The “log file sync” wait is associated to the second cursor, the commit.

=====================
PARSING IN CURSOR #3 len=35 dep=0 uid=84 oct=6 lid=84 tim=1250674381587415 hv=950048100 ad='38e58340' sqlid='crk1wdnwa15b4'
UPDATE scott.emp SET sal = sal*1.15
END OF STMT
PARSE #3:c=0,e=338,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1494045816,tim=1250674381587404
EXEC #3:c=2999,e=2831,p=0,cr=7,cu=4,mis=0,r=14,dep=0,og=1,plh=1494045816,tim=1250674381590917
STAT #3 id=1 cnt=0 pid=0 pos=1 obj=0 op='UPDATE  EMP (cr=7 pr=0 pw=0 time=0 us)'
STAT #3 id=2 cnt=14 pid=1 pos=1 obj=73268 op='TABLE ACCESS FULL EMP (cr=7 pr=0 pw=0 time=14 us cost=3 size=70 card=14)'
WAIT #3: nam='SQL*Net message to client' ela= 11 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1250674381592362
WAIT #3: nam='SQL*Net message from client' ela= 1812 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1250674381594288
CLOSE #3:c=0,e=28,dep=0,type=0,tim=1250674381594476
=====================
PARSING IN CURSOR #2 len=6 dep=0 uid=84 oct=44 lid=84 tim=1250674381594857 hv=255718823 ad='0' sqlid='8ggw94h7mvxd7'
COMMIT
END OF STMT
PARSE #2:c=0,e=146,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=0,tim=1250674381594814
XCTEND rlbk=0, rd_only=0, tim=1250674381595557
EXEC #2:c=1000,e=740,p=0,cr=0,cu=1,mis=0,r=0,dep=0,og=0,plh=0,tim=1250674381596086
WAIT #2: nam='log file sync' ela= 1974 buffer#=1980 p2=0 p3=0 obj#=-1 tim=1250674381598390
WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1250674381598664
WAIT #2: nam='SQL*Net message from client' ela= 2770 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1250674381601434
CLOSE #2:c=0,e=17,dep=0,type=3,tim=1250674381601595
=====================

The second trace file is generated by tracing the execution of the same operations from a Java program. The following is the method that contains the update and the commit:

private static void test(Connection connection) throws Exception
{
  Statement statement = connection.createStatement();
  statement.execute("UPDATE scott.emp SET sal = sal * 1.15");
  statement.close();
  connection.commit();
}

The relevant part of the trace file is the following. Notice that:

• There is only one cursor: the update. No cursor related to the commit is available.
• Just after the CLOSE line there is a XCTEND line with the attribute “rlbk” set to 0. Obviously a commit was performed.
• The “log file sync” wait is associated to cursor number 0. Note that the database engine associate to cursor number 0 all lines that cannot be associated to other cursors. In this case a cursor with the commit statement is missing, hence it is not possible to associate the wait to it.

=====================
PARSING IN CURSOR #2 len=37 dep=0 uid=84 oct=6 lid=84 tim=1250673660406187 hv=517367075 ad='38d919e4' sqlid='cc8438wgdct93'
UPDATE scott.emp SET sal = sal * 1.15
END OF STMT
PARSE #2:c=39994,e=40693,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=1494045816,tim=1250673660406167
EXEC #2:c=17997,e=18116,p=0,cr=7,cu=4,mis=0,r=14,dep=0,og=1,plh=1494045816,tim=1250673660427450
STAT #2 id=1 cnt=0 pid=0 pos=1 obj=0 op='UPDATE  EMP (cr=7 pr=0 pw=0 time=0 us)'
STAT #2 id=2 cnt=14 pid=1 pos=1 obj=73268 op='TABLE ACCESS FULL EMP (cr=7 pr=0 pw=0 time=5 us cost=3 size=70 card=14)'
WAIT #2: nam='SQL*Net message to client' ela= 5 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1250673660432401
WAIT #2: nam='SQL*Net message from client' ela= 12925 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1250673660446291
CLOSE #2:c=0,e=45,dep=0,type=0,tim=1250673660447335
XCTEND rlbk=0, rd_only=0, tim=1250673660449381
WAIT #0: nam='log file sync' ela= 1325 buffer#=1217 p2=0 p3=0 obj#=-1 tim=1250673660454674
WAIT #0: nam='SQL*Net message to client' ela= 4 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1250673660454885
=====================

The interesting thing to note about the second case is that a commit was performed without executing a COMMIT statement. This is possible because at the OCI level a commit can be performed by calling the function OCITransCommit. In other words, without having to execute a statement.

Now, back to the question. When TVD$XTAT processes a trace file like the second one, it automatically generates a cursor related to the commit operation. The text of the cursor will be “COMMIT (synthetic)”. So, the term “synthetic” is only added to point out that it is a generated cursor. In addition, TVD$XTAT also associate the waits associated to the cursor 0 to the generated cursor. This is very important because in some situations, for example when commits are executed very often or when long rollbacks are executed, the time needed for the commit/rollback is not negligible. As a result, if they were not accounted, the unaccounted-for time could be relevant.

BTW, it is not a coincidence that the Method R Profiler (a.k.a. Hotsos Profiler) and TVD$XTAT uses the same term. I fact, while comparing the two profilers, I noticed that in Method R Profiler the generated statements were called “synthetic”. I found the idea good and since I was looking for a method to mark such statements as well, I borrowed their term.

The detailed change log since Beta 8 is the following:

• Added formatting for bind variable values of type DATE
• Added support for several execution plans for a single cursor
• Added number of executions and hash value to execution plans
• Added detection of incomplete execution plans
• Added support for RPC bind variables
• Added command-line option to control logging level
• Added warning for 11.1.0.7 trace files (because of bug# 7522002 timing information might be wrong)
• Improved data type detection to distinguish VARCHAR2 from NVARCHAR2 and CHAR from NCHAR
• Improved handling of incorrectly formatted input lines
• Changed logging formatter (time is displayed with the following pattern HH:mm:ss)
• Reduced memory utilization for the processing of large trace files
• Fix to prevent poor performance for the processing of large trace files
• Fix to replace special characters not supported by XML (the unicode character FFFD is used istead of the special ones)
• Fix in template to correctly handle space character in SQL text and bind variable values
• Fix to ignore timestamp lines not generated by SQL trace

As always, your feedback is welcome!

[13/02/2009 10:06:38:079] (EMERGENCY) ajp12: can not connect to host 127.0.0.1:3501
[13/02/2009 10:06:38:181] (EMERGENCY) ajp12: connection fail
[13/02/2009 10:06:38:181] (ERROR) an error returned handling request via protocol "ajpv12"

Mhmm… a listener should be available on port 3501. But, no such listener was available on my system (note that port 3500 is used for the HTTP listener):

oracle@helicon:/u00/app/oracle/product/ad4j/ [rdbms11107] netstat -l --numeric-ports | grep 350[01]
tcp        0      0 *:3500                      *:*                         LISTEN

A quick search in Metalink revealed that at least another person has hit the same issue few days ago (see bug# 8235076). Since OSS is still working on it, I’ll wait to see what the findings are. In the mean time, I was able to successfully install it on my Windows laptop.

Anyway, if somebody of you managed to successfully install AD4J on Linux, please, let me know!

ADDENDA (February 16th, 2009): Because of the comments of Charles and Michael I spent a bit more time looking at the problem. In fact, the first time I stopped immediately after seeing the bug in Metalink… I was lazy . Hey, I try to optimize my worktime as well. Anyway, both suggested to manually start jserv. When I tried to do so, I received an error (at last). Based on it the problem was self explanatory! The java environment was causing the problem… In fact, with the default installation of CentOS 4.4 only the package java-1.4.2-gcj-compat was installed. After downloading and installing the most recent version of HotSpot (build 1.6.0_12-b04) the problem was solved.

• I prepared the test environment by executing the following SQL statements:

ALTER SYSTEM FLUSH SHARED_POOL;

CREATE TABLE t
AS
SELECT rownum AS id, rpad('*',100,'*') AS pad
FROM dual
CONNECT BY level <= 1000;

ALTER TABLE t ADD CONSTRAINT t_pk PRIMARY KEY (id);

EXECUTE dbms_stats.gather_table_stats(ownname=>user,tabname=>'t')

VARIABLE id NUMBER

• I enabled SQL trace:

EXECUTE dbms_monitor.session_trace_enable(waits=>false,binds=>false)

• I execute three time the same query that, by the way, takes advantage of extended cursor sharing (a.k.a. adaptive cursor sharing and intelligent cursor sharing):

EXECUTE :id := 10;

SELECT count(pad) FROM t WHERE id < :id;

EXECUTE :id := 990;

SELECT count(pad) FROM t WHERE id < :id;

SELECT count(pad) FROM t WHERE id < :id;

• I disabled SQL trace and displayed the name of the trace file:

EXECUTE dbms_monitor.session_trace_disable

SELECT value
FROM v$diag_info
WHERE name = 'Default Trace File';

Now, let's check the content of the trace file...

• For the first query, the following information is provided. Notice that in the line PARSE the value of "plh" is "0"; in the line EXEC the value of "plh" is "4270555908"; in the lines PARSE and EXEC the value of "mis" is "1" (the cursor was hard parsed); according to the execution plan (the line beginning with STAT; notice that I manually removed the runtime statistics and query optimizer estimations to keep the output more readable) the table was read through an index range scan.

PARSING IN CURSOR #2 len=39 dep=0 uid=83 oct=3 lid=83 tim=1231427339018492 hv=1107655156 ad='2e70b9cc' sqlid='asth1mx10aygn'
SELECT count(pad) FROM t WHERE id < :id
END OF STMT
PARSE #2:c=0,e=0,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=1231427339018492
EXEC #2:c=2000,e=0,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=4270555908,tim=1231427339018492
FETCH #2:c=0,e=0,p=0,cr=3,cu=0,mis=0,r=1,dep=0,og=1,plh=4270555908,tim=1231427339018492
STAT #2 id=1 cnt=1 pid=0 pos=1 obj=0 op='SORT AGGREGATE'
STAT #2 id=2 cnt=9 pid=1 pos=1 obj=72373 op='TABLE ACCESS BY INDEX ROWID T'
STAT #2 id=3 cnt=9 pid=2 pos=1 obj=72374 op='INDEX RANGE SCAN T_PK'
FETCH #2:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=4270555908,tim=1231427339018492
CLOSE #2:c=0,e=0,dep=0,type=0,tim=1231427339018492

• For the second query, the following information is provided. Notice that in the lines PARSE and EXEC the value of "plh" is "4270555908" (the same as the previous query) and the value of "mis" is "0" (the cursor was not hard parsed), i.e. a cursor already present in the library cache was reused.

PARSING IN CURSOR #4 len=39 dep=0 uid=83 oct=3 lid=83 tim=1231427339018492 hv=1107655156 ad='2e70b9cc' sqlid='asth1mx10aygn'
SELECT count(pad) FROM t WHERE id < :id
END OF STMT
PARSE #4:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4270555908,tim=1231427339018492
EXEC #4:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4270555908,tim=1231427339018492
FETCH #4:c=0,e=0,p=0,cr=19,cu=0,mis=0,r=1,dep=0,og=1,plh=4270555908,tim=1231427339018492
STAT #4 id=1 cnt=1 pid=0 pos=1 obj=0 op='SORT AGGREGATE'
STAT #4 id=2 cnt=989 pid=1 pos=1 obj=72373 op='TABLE ACCESS BY INDEX ROWID T'
STAT #4 id=3 cnt=989 pid=2 pos=1 obj=72374 op='INDEX RANGE SCAN T_PK'
FETCH #4:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=4270555908,tim=1231427339018492
CLOSE #4:c=0,e=0,dep=0,type=0,tim=1231427339018492

• For the third query, the one taking advantage of extended cursor sharing, the following information is provided. Notice that in the line PARSE the value of "plh" is "4270555908" (the same as the previous query) and the value of "mis" is "0" (cursor was not hard parsed); in the line EXEC the value of "plh" is "2966233522" and the value of "mis" is "1" (cursor was hard parsed); according to the execution plan the table was read through a full table scan, i.e. not the same execution plan as the previous queries.

PARSING IN CURSOR #13 len=39 dep=0 uid=83 oct=3 lid=83 tim=1231427339018492 hv=1107655156 ad='2e70b9cc' sqlid='asth1mx10aygn'
SELECT count(pad) FROM t WHERE id < :id
END OF STMT
PARSE #13:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4270555908,tim=1231427339018492
EXEC #13:c=0,e=0,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=2966233522,tim=1231427339018492
FETCH #13:c=0,e=0,p=0,cr=19,cu=0,mis=0,r=1,dep=0,og=1,plh=2966233522,tim=1231427339018492
STAT #13 id=1 cnt=1 pid=0 pos=1 obj=0 op='SORT AGGREGATE'
STAT #13 id=2 cnt=989 pid=1 pos=1 obj=72373 op='TABLE ACCESS FULL T'
FETCH #13:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=2966233522,tim=1231427339018492
CLOSE #13:c=0,e=0,dep=0,type=0,tim=1231427339018492

In summary, according to this test:

• The value of "plh" in the line EXEC provides the hash value of the execution plan used to execute a SQL statement.
• The value of "plh" in the line PARSE cannot be trusted.

• System agent gather performance figures about the operating system.
• Web server agents gather performance figures about a web server.
• Application server agents gather detailed performance figures about an application server.

Since the agents are specific to each component, Quest Software clearly states which components are officially supported. For example, the current release (4.4) supports the following application servers: BEA WebLogic Server, IBM WebSphere Application Server, Oracle Application Server, JBoss Application Server and Apache Tomcat Server. Even if those are among the most commonly used, other application servers exist… For this reason PerformaSure also supports a generic agent called X-Agent. Although the X-Agent does not provide as much information as the other agents do, it might be very useful in some situations. The problem is that the documentation does not provide instructions on how to configure the it. Instead, it provides the following information:

“The X-Agent needs to be configured by Quest Consulting. For more information about customizing the X-Agent and modifying the startup sequence of the application server or standalone Java application that you want to instrument, contact your Quest sales representative.”

Ups! Not very useful…

To show you how such a configuration is performed, let me describe what I did few days ago on my Windows XP laptop to configure the X-Agent for Sun Java System Application Server 9.1 Update 2.

• First of all, I opened a command prompt and I defined two environment variables pointing to the installation directories of Java and PerformaSure

c:tmp>set JAVA_HOME=C:Javajdk1.5.0
c:tmp>set PERFORMASURE_HOME=C:JavaquestPerformaSure4.4.0

• Then, in the same command prompt, I called the script preinstrumentor.cmd to create a JAR file that will be used during the startup of the application server

c:tmp>call "%PERFORMASURE_HOME%scriptspreinstrumentor.cmd" "%JAVA_HOME%"
2008-12-26 23:09:00.703 INFO  Loading Agent startup configuration from C:Java
  questPerformaSure4.4.0configagent.config
2008-12-26 23:09:02.031 INFO  PerformaSure 4.4 (Build PAS_440-20061024-1025) on
  Windows XP x86 5.1 using Sun Microsystems Inc. Java HotSpot(TM) Client VM
  version 1.5.0_10-b03 (extra info: mixed mode, sharing); Agent Type:
  Preinstrumentor
2008-12-26 23:09:02.062 INFO  Configured to connect to Nexus at localhost:41705
2008-12-26 23:09:02.187 INFO  Connected to Nexus at localhost:41705
2008-12-26 23:09:03.203 INFO  Agent started
2008-12-26 23:09:03.203 INFO  Creating bootstrap classes in C:JavaquestPerfo
  rmaSure4.4.0scripts..bootstrapC--Java-jdk-1.5.0.jar
2008-12-26 23:09:05.796 INFO  Preinstrumentation complete
2008-12-26 23:09:05.812 INFO  Preinstrumentor Agent shutdown completed

• The last operation I performed through the command prompt was the “generation” of two Java options (-Xbootclasspath and -javaagent) to be used in the configuration of the application server (it is not necessary to do it in this way, but with that method it’s possible to avoid annoying typos…)

C:Tmp>call "%PERFORMASURE_HOME%scriptsmakebootstrappath.cmd" "%JAVA_HOME%"
C:Tmp>echo -Xbootclasspath/p:"%PS_BOOTSTRAP_PATH%"
-Xbootclasspath/p:"C:JavaquestPerformaSure4.4.0bootstrapC--Java-jdk-1.5.0.jar"
C:Tmp>echo -javaagent:"%PERFORMASURE_HOME%libperformasure-agent.jar"
-javaagent:"C:JavaquestPerformaSure4.4.0libperformasure-agent.jar"

• To finish the configuration I added, through the administration console of the application server, the following JVM options (click here to see an image of the relevant part of the console)

-Dperformasure.agent.appserverinfo=kalgan:Generic:SunAS9
-Dperformasure.debug=0
-Xbootclasspath/p:"C:JavaquestPerformaSure4.4.0bootstrapC--Java-jdk-1.5.0.jar"
-javaagent:"C:JavaquestPerformaSure4.4.0libperformasure-agent.jar"

Notice that the value of the variable performasure.agent.appserverinfo has the following format: <server>:Generic:<version>.

After these operations I restarted the application server and, as a result, I was able to use PerformaSure to profile the applications running into the JVM of the application server.

The change log since Beta 7 is the following:

• Improved generation of synthetic cursors (particularly for RPC and XCTEND)
• Added support for 11.1.0.7 formatting of XCTEND
• Added check for questionable values for cpu and elapsed time
• Extended the list of recognized data types
• Fix to avoid Java 1.6.0 bug 6506304
• Fix to correctly handle attribution of bind variables
• Fix to correctly handle bind variables containing invalid numbers
• Fix to correctly handle bind variables containing the value NULL
• Fix to avoid parse errors for bind variables containing kkscoacd lines
• Fix to avoid warnings about freed cursors
• Fix to avoid warnings for lines beginning with “*** TRACE CONTINUED FROM FILE”

Please, do not hesitate to contact me if you have any kind of problem while using it.

Why Is TKPROF Not Enough?

In late 1999, I had my first encounter with extended SQL trace, through MetaLink note Interpreting Raw SQL_TRACE and DBMS_SUPPORT.START_TRACE output (39817.1). From the beginning, it was clear that the information it provided was essential for understanding what an application is doing when it is connected to an Oracle database engine. At the same time, I was very disappointed that no tool was available for analyzing extended SQL trace files for the purpose of leveraging their content. I should note that TKPROF at that time did not provide information about wait events. After spending too much time manually extracting information from the raw trace files, I decided to write my own analysis tool: TVD$XTAT.

Currently, TKPROF provides information about wait events, but it still has three major problems that are addressed in TVD$XTAT:

• As soon as the argument sort is specified, the relationship between SQL statements is lost.
• Data is provided only in aggregated form. Consequently, useful information is lost.
• No information about bind variables is provided.

TVD$XTAT is freeware. You can download it (presently, version 4.0 beta 7) from this page. TOP fully describes how to use it to identify performance problems. The installation is documented in the README file.

It goes without saying that all feedbacks about TVD$XTAT are highly welcome.