--- /dev/null
+
+====================================================================
+
+11 May 2009
+
+Protocols 1 through 3 supported Memcheck only. Protocol 4 provides
+XML output for Memcheck, Helgrind and Ptrcheck. Technically there are
+three variants of Protocol 4, one for each tool, since they produce
+different errors. The three variants differ only in the definition of
+the ERROR nonterminal and are otherwise identical.
+
+NOTE that Protocol 4 (for the current svn trunk, which will eventually
+become 3.5.x) is still under development. The text herein should not
+be regarded as the final definition.
+
+
+Identification of Protocols
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In Protocols 1 through 3, a <protocolversion>INT<protocolversion>
+close to the start of the stream makes it possible for parsers to
+ascertain the version, so they can tell whether or not they can handle
+it. The presence of support for multiple tools brings a complication,
+though: it is not enough merely to state the protocol version -- the
+tool name must also be stated. Hence in Protocol 4, the
+<protocolversion>INT<protocolversion> is followed immediately by
+<protocoltool>TEXT</protocoltool>, to identify the tool.
+
+This duplicates the tool name present later in the preamble, but it
+was felt important to place the tool name right at the front along
+with the protocol number, for easy determination of parseability.
+
+
+How this specification is structured
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The TOPLEVEL nonterminal specifies top level XML output structure. It
+is common to all error producing tools.
+
+TOPLEVEL references TOOLSPECIFICs for each tool, and these are defined
+differently for each tool. Each TOOLSPECIFIC is an error, which is
+tool-specific. For Helgrind, a TOOLSPECIFIC may also contain a
+so-called thread-announcement record (described below).
+
+Overall there is a very high degree of format commonality between the
+three tools. Once a GUI is able to display the output correctly for
+one tool, it should be easy to extend it for the other two.
+
+
+Protocol 4 changes for Memcheck
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Protocol 4 for Memcheck is similar to Protocol 3, but has a number
+of changes to make it fit in the common framework:
+
+- the SUPPCOUNTS nonterminal now appears after the "Zero or more
+ ERRORs" block, and not before it.
+
+- the abovementioned "Zero or more ERRORs" block now becomes
+ "Zero or more of (either ERROR or ERRORCOUNTS)".
+
+- ERRORs for Memcheck may contain a SUPPRESSION field, which gives
+ the corresponding suppression for it.
+
+- ERRORs for Memcheck now use the XWHAT and XAUXWHAT nonterminals, as
+ well as WHAT and XWHAT.
+
+- The ad-hoc blocks <leakedbytes> and <leakedblocks> used by Memcheck
+ have been moved inside the XWHAT for the relevant error kinds. This
+ facilitates a common definition of ERROR across all three tools.
+
+The first two changes are required in order to correct a longstanding
+design flaw in the way Memcheck interacts with Valgrind's error
+management mechanism. See bug #186790
+(https://bugs.kde.org/show_bug.cgi?id=186790). The third change was
+requested in #191189 (https://bugs.kde.org/show_bug.cgi?id=191189).
+
+For GUI authors upgrading from Protocol 3 or earlier, the most
+significant new concept to grasp is the relationship between WHAT and
+XWHAT, and between AUXWHAT and XAUXWHAT.
+
+The definition of Protocol 4 now follows. It is structured similarly
+to that of the previous protocols, except that there is a separate
+definition of a nonterminal called TOOLSPECIFIC for each of Memcheck,
+Helgrind and Ptrcheck. The XWHAT and XAUXWHAT nonterminals also have
+tool-specific components. Apart from that, the structure is common
+to all supported tools.
+
+
+====================================================================
+
+TOPLEVEL
+--------
+
+The first line output is always this:
+
+ <?xml version="1.0"?>
+
+All remaining output is contained within the tag-pair
+<valgrindoutput>.
+
+Inside that, the first entity is an indication of the protocol
+version. This is provided so that existing parsers can identify XML
+created by future versions of Valgrind merely by observing that the
+protocol version is one they don't understand. Hence TOPLEVEL is:
+
+ <?xml version="1.0"?>
+ <valgrindoutput>
+ <protocolversion>INT<protocolversion>
+ <protocoltool>TEXT</protocoltool>
+ PROTOCOL
+ </valgrindoutput>
+
+Valgrind versions 3.0.0 and 3.0.1 emit protocol version 1. Versions
+3.1.X and 3.2.X [and 3.3.X ??] emit protocol version 2. 3.4.X emits
+protocol version 3. 3.5.X emits version 4.
+
+The TEXT in <protocoltool> is either "memcheck", "helgrind" or
+"exp-ptrcheck" and determines the allowed format of the ERROR
+nonterminal. Note that <protocoltool> is only present when the
+protocol version is 4 or above.
+
+
+PROTOCOL for version 4
+----------------------
+
+This is the main top-level construction. Roughly speaking, it
+contains a preamble, a program-started marker, the errors from the run
+of the program, a program-ended marker, and any further errors
+resulting from post-run analysis (eg, memory leak detection). Hence
+the following in sequence:
+
+* Various preamble lines which give version info for the various
+ components. The text in them can be anything; it is not intended
+ for interpretation by the GUI:
+
+ <preamble>
+ <line>Misc version/copyright text</line> (zero or more of)
+ </preamble>
+
+* The PID of this process and of its parent:
+
+ <pid>INT</pid>
+ <ppid>INT</ppid>
+
+* The name of the tool being used:
+
+ <tool>TEXT</tool>
+
+ This can be anything, and it doesn't have to match the
+ <protocoltool> entry, although that might be wise.
+
+* Zero or more bindings of environment variable names to actual
+ values. These describe precisely the instantiations of %q format
+ specifiers used in the --xml-file= argument for the run, if any.
+ There is one <logfilequalifier> entry for each %q expanded:
+
+ <logfilequalifier> <var>VAR</var> <value>$VAR</value>
+ </logfilequalifier>
+
+* OPTIONALLY, if --xml-user-comment=STRING was given:
+
+ <usercomment>STRING</usercomment>
+
+ STRING is not escaped in any way, so that it itself may be a piece
+ of XML with arbitrary tags etc.
+
+* The program and args: first those pertaining to Valgrind itself, and
+ then those pertaining to the program to be run under Valgrind (the
+ client):
+
+ <args>
+ <vargv>
+ <exe>TEXT</exe>
+ <arg>TEXT</arg> (zero or more of)
+ </vargv>
+ <argv>
+ <exe>TEXT</exe>
+ <arg>TEXT</arg> (zero or more of)
+ </argv>
+ </args>
+
+* The following, indicating that the program has now started:
+
+ <status> <state>RUNNING</state>
+ <time>human-readable-time-string</time>
+ </status>
+
+ The format of this string is not defined, but it is expected to be
+ human-understandable. In current Valgrind versions it is the
+ elapsed wallclock time since process start.
+
+* Zero or more of (either ERRORCOUNTS or TOOLSPECIFIC).
+
+* The following, indicating that the program has now finished, and
+ that the any final wrapup (eg, for Memcheck, leak checking) is happening.
+
+ <status> <state>FINISHED</state>
+ <time>human-readable-time-string</time>
+ </status>
+
+* Zero or more of (either ERRORCOUNTS or TOOLSPECIFIC). In Memcheck's
+ case these will be complaints from the leak checker. For Ptrcheck
+ and Helgrind we don't expect any output here (but the spec does not
+ guarantee that either).
+
+* SUPPCOUNTS, indicating how many times each suppression was used.
+
+
+That's it. The tool-specific definitions for TOOLSPECIFIC are below;
+however let's first continue with some smaller nonterminals used in
+the construction of errors for all the tool types.
+
+
+====================================================================
+
+Nonterminals used in construction of ERRORs
+-------------------------------------------
+
+STACK
+-----
+STACK indicates locations in the program being debugged. A STACK
+is one or more FRAMEs. The first is the innermost frame, the
+next its caller, etc.
+
+ <stack>
+ one or more FRAME
+ </stack>
+
+
+FRAME
+-----
+FRAME records a single program location:
+
+ <frame>
+ <ip>HEX64</ip>
+ optionally <obj>TEXT</obj>
+ optionally <fn>TEXT</fn>
+ optionally <dir>TEXT</dir>
+ optionally <file>TEXT</file>
+ optionally <line>INT</line>
+ </frame>
+
+Only the <ip> field is guaranteed to be present. It indicates a
+code ("instruction pointer") address.
+
+The optional fields, if present, appear in the order stated:
+
+* obj: gives the name of the ELF object containing the code address
+
+* fn: gives the name of the function containing the code address
+
+* dir: gives the source directory associated with the name specified
+ by <file>. Note the current implementation often does not
+ put anything useful in this field.
+
+* file: gives the name of the source file containing the code address
+
+* line: gives the line number in the source file
+
+
+ERRORCOUNTS
+-----------
+This specifies, for each error that has been so far presented,
+the number of occurrences of that error.
+
+ <errorcounts>
+ zero or more of
+ <pair> <count>INT</count> <unique>HEX64</unique> </pair>
+ </errorcounts>
+
+Each <pair> gives the current error count <count> for the error with
+unique tag </unique>. The counts do not have to give a count for each
+error so far presented - partial information is allowable.
+
+As at Valgrind rev 3793, error counts are only emitted at program
+termination. However, it is perfectly acceptable to periodically emit
+error counts as the program is running. Doing so would facilitate a
+GUI to dynamically update its error-count display as the program runs.
+
+
+SUPPCOUNTS
+----------
+A SUPPCOUNTS block appears exactly once, after the program terminates.
+It specifies the number of times each error-suppression was used.
+Suppressions not mentioned were used zero times.
+
+ <suppcounts>
+ zero or more of
+ <pair> <count>INT</count> <name>TEXT</name> </pair>
+ </suppcounts>
+
+The <name> is as specified in the suppression name fields in .supp
+files.
+
+
+SUPPRESSION
+-----------
+These are optionally emitted as part of ERRORs, and specify the
+suppression that would be needed to suppress the containing error.
+For convenience, the suppression is presented twice, once in
+a structured nicely wrapped up in tags, and once as raw text
+suitable for direct copying and pasting into a suppressions file.
+
+ <suppression>
+ <sname>TEXT</sname> name of the suppression
+ <skind>TEXT</skind> kind, eg "Memcheck:Param"
+ <skaux>TEXT</skaux> (optional) aux kind, eg "write(buf)"
+ SFRAME (one or more) frames
+ <rawtext> CDATAS </rawtext>
+ </suppression>
+
+where CDATAS is a sequence of one or more <![CDATA[ .. ]]> blocks
+holding the raw text. Unfortunately, CDATA provides no way to escape
+the ending marker "]]>", which means that if the raw data contains
+such a sequence, it has to be split between two CDATA blocks, one
+ending with data "]]" and the other beginning with data "<". This is
+why the spec calls for one or more CDATA blocks rather than exactly
+one.
+
+Note that, so far, we cannot envisage a circumstance in which a
+generated suppression would contain the string "]]>", since neither
+"]" nor ">" appear to turn up in mangled symbol names. Hence it is
+not envisaged that there will ever be more than one CDATA block, and
+indeed the implementation as of Valgrind 3.5.0 will only ever generate
+one block (it ignores any possible escaping problems). Nevertheless
+the specification allows multiple blocks, as a matter of safety.
+
+
+SFRAME
+------
+Either
+
+ <sframe> <obj>TEXT</obj> </sframe>
+
+eg denoting "obj:/usr/X11R6/lib*/libX11.so.6.2", or
+
+ <sframe> <fun>TEXT</fun> </sframe>
+
+eg denoting "fun:*libc_write"
+
+
+WHAT and XWHAT
+--------------
+
+WHAT supplies a single line of text, which is a human-understandable,
+primary description of an error.
+
+XWHAT is an extended version of WHAT. It also contains a piece of
+text intended for human reading, but in addition may contain arbitrary
+other tagged data. This extra data is tool-specific. One of its
+purposes is to supply GUIs with links to other data in the sequence of
+TOOLSPECIFICs, that are associated with the error. Another purpose is
+wrap certain quantities (numbers, file names, etc) embedded in the
+message, so that the GUIs can get hold of them without having to parse
+the text itself.
+
+For example, we could get:
+
+ <what>Possible data race on address 0x12345678</what>
+
+or alternatively
+
+ <xwhat>
+ <text>Possible data race by thread #17 on address 0x12345678</text>
+ <threadid>17</threadid>
+ </xwhat>
+
+And presumably the <threadid>17</threadid> refers to some previously
+emitted entity in the stream of TOOLSPECIFICs for this tool.
+
+In an XWHAT, the <text> tag-pair is mandatory. GUIs which don't want
+to handle the extra fields can just ignore them and display the text
+part. In this way they have the option to present at least something
+useful to the user even in the case where the extra fields can't be
+handled, for whatever reason.
+
+A corollary of this is that the degenerate extended case
+
+ <xwhat> <text>T</text> </xwhat>
+
+is exactly equivalent to
+
+ <what>T</what>
+
+
+AUXWHAT and XAUXWHAT
+--------------------
+
+AUXWHAT is exactly like WHAT: a single line of text. It provides
+additional, secondary description of an error, that should be shown to
+the user.
+
+XAUXWHAT relates to AUXWHAT in the same way XWHAT relates to WHAT: it
+wraps up extra tagged info along with the line of text that would be
+in the AUXWHAT.
+
+
+====================================================================
+
+ERROR definition -- common structure
+------------------------------------
+
+ERROR defines an error, and is the most complex nonterminal. For all
+of the tools, the structure is common, and always conforms to the
+following:
+
+ <error>
+ <unique>HEX64</unique>
+ <tid>INT</tid>
+ <kind>KIND</kind>
+
+ (either WHAT or XWHAT)
+ optionally: (either WHAT or XWHAT)
+
+ STACK
+
+ zero or more: (either AUXWHAT or XAUXWHAT or STACK)
+
+ optionally: SUPPRESSION
+ </error>
+
+
+* Each error contains a unique, arbitrary 64-bit hex number. This is
+ used to refer to the error in ERRORCOUNTS nonterminals (see above).
+
+* The <tid> tag indicates the Valgrind thread number. This value
+ is arbitrary but may be used to determine which threads produced
+ which errors (at least, the first instance of each error).
+
+* The <kind> tag specifies one of a small number of fixed error types,
+ so that GUIs may roughly categorise errors by type if they want.
+ The tags themselves are tool-specific and are defined further
+ below, for each tool.
+
+* The "(either WHAT or XWHAT)" gives a primary description of the
+ error. WHAT and XWHAT are defined earlier in this file. Any XWHATs
+ appearing here may contain tool-specific subcomponents.
+
+* Optionally, a second line of primary description may be present.
+
+* A STACK gives the primary source location for the error.
+
+* There then follow zero or more of "(either AUXWHAT or XAUXWHAT or
+ STACK)". These give further (auxiliary) information about the
+ error, possibly including stack traces. They should be shown to the
+ user in the order they appear. AUXWHAT and XAUXWHAT are defined
+ earlier in this file. Any XAUXWHATs appearing here may contain
+ tool-specific subcomponents.
+
+* Optionally, as the last field, a SUPPRESSION may be provided. This
+ contains a suppression that would hide the error.
+
+
+====================================================================
+
+TOOLSPECIFIC definition for Memcheck
+------------------------------------
+
+For Memcheck, a TOOLSPECIFIC is simply an ERROR:
+
+TOOLSPECIFIC = ERROR
+
+
+ERROR details for Memcheck
+--------------------------
+
+XWHATs (for definition, see above) may contain the following extra
+components (along with the mandatory <text>...</text> component):
+
+* <leakedbytes>INT</leakedbytes>
+
+* <leakedblocks>INT</leakedblocks>
+
+These fields are used in errors that have a <kind> tag specifying a
+KIND of the form "Leak_*", to indicate the number of leaked bytes and
+blocks.
+
+
+XAUXWHATs (for definition, see above) may contain the following extra
+components (along with the mandatory <text>...</text> component):
+
+* <file>TEXT</file>, as defined in FRAME
+
+* <line>INT</line>, as defined in FRAME
+
+* <dir>TEXT</dir>, as defined in FRAME
+
+
+KIND for Memcheck
+-----------------
+
+This is a small enumeration indicating roughly the nature of an error.
+The possible values are:
+
+ InvalidFree
+
+ free/delete/delete[] on an invalid pointer
+
+ MismatchedFree
+
+ free/delete/delete[] does not match allocation function
+ (eg doing new[] then free on the result)
+
+ InvalidRead
+
+ read of an invalid address
+
+ InvalidWrite
+
+ write of an invalid address
+
+ InvalidJump
+
+ jump to an invalid address
+
+ Overlap
+
+ args overlap other otherwise bogus in eg memcpy
+
+ InvalidMemPool
+
+ invalid mem pool specified in client request
+
+ UninitCondition
+
+ conditional jump/move depends on undefined value
+
+ UninitValue
+
+ other use of undefined value (primarily memory addresses)
+
+ SyscallParam
+
+ system call params are undefined or point to
+ undefined/unaddressible memory
+
+ ClientCheck
+
+ "error" resulting from a client check request
+
+ Leak_DefinitelyLost
+
+ memory leak; the referenced blocks are definitely lost
+
+ Leak_IndirectlyLost
+
+ memory leak; the referenced blocks are lost because all pointers
+ to them are also in leaked blocks
+
+ Leak_PossiblyLost
+
+ memory leak; only interior pointers to referenced blocks were
+ found
+
+ Leak_StillReachable
+
+ memory leak; pointers to un-freed blocks are still available
+
+
+====================================================================
+
+TOOLSPECIFIC definition for Ptrcheck
+------------------------------------
+
+For Ptrcheck, a TOOLSPECIFIC is simply an ERROR:
+
+TOOLSPECIFIC = ERROR
+
+
+ERROR details for Ptrcheck
+--------------------------
+
+Ptrcheck does not produce any XWHAT records, despite the fact that
+"ERROR definition -- common structure" says that tools may do so.
+
+
+XAUXWHATs (for definition, see above) may contain the following extra
+components (along with the mandatory <text>...</text> component):
+
+* <file>TEXT</file>, as defined in FRAME
+
+* <line>INT</line>, as defined in FRAME
+
+* <dir>TEXT</dir>, as defined in FRAME
+
+
+KIND for Ptrcheck
+-----------------
+This is a small enumeration indicating roughly the nature of an error.
+The possible values are:
+
+ SorG
+
+ Stack or global array inconsistency (roughly speaking, an
+ overrun of a stack or global array). The <auxwhat> blocks give
+ further details.
+
+ Heap
+
+ Usage of a pointer derived from a heap block, to access
+ outside that heap block
+
+ Arith
+
+ Doing arithmetic on pointers in a way that cannot possibly
+ result in another valid pointer. Eg, adding two pointer values.
+
+ SysParam
+
+ Special case of "Heap", in which the invalidly-addressed memory
+ is presented as an argument to a system call which reads or
+ writes memory.
+
+
+====================================================================
+
+TOOLSPECIFIC definition for Helgrind
+-------------------------------------
+
+For Helgrind, a TOOLSPECIFIC may be one of two things:
+
+TOOLSPECIFIC = either ERROR or ANNOUNCETHREAD
+
+
+ANNOUNCETHREAD
+--------------
+
+The definition is
+
+ <announcethread>
+ <hthreadid>INT</hthreadid>
+ STACK
+ </announcethread>
+
+This states the creation point of a thread, and gives it a unique
+"hthreadid", which may be referred to in subsequent ERRORs. Note that
+
+1. The appearance of ANNOUNCETHREAD does not mean that the thread was
+ actually created at that point relative to any preceding or
+ following ERRORs in the output stream -- in general the thread will
+ have been created arbitrarily earlier. Helgrind only "announces" a
+ thread when it needs to refer to it for the first time, in a
+ subsequent ERROR.
+
+2. The "hthreadid" is a number which uniquely identifies the thread
+ for the run - no other thread will have the same hthreadid. The
+ hthreadid is a Helgrind-specific piece of information and is
+ unrelated to the <tid> fields in the common part of an ERROR.
+ Be careful not to confuse the two.
+
+
+ERROR details for Helgrind
+--------------------------
+
+XWHATs (for definition, see above) may contain the following extra
+components (along with the mandatory <text>...</text> component):
+
+* <hthreadid>INT</hthreadid> fields. These refer to ANNOUNCETHREADs
+ appearing previously in the scheme, and state the creation points of
+ the thread(s) concerned in the ERROR. Hence it should be possible
+ for GUIs to show users stacks of the creation points of all threads
+ involved in each ERROR.
+
+
+XAUXWHATs (for definition, see above) may contain the following extra
+components (along with the mandatory <text>...</text> component):
+
+* <hthreadid>INT</hthreadid>, same meaning as when referred to in
+ XWHAT
+
+* <file>TEXT</file>, as defined in FRAME
+
+* <line>INT</line>, as defined in FRAME
+
+* <dir>TEXT</dir>, as defined in FRAME
+
+
+KIND for Helgrind
+-----------------
+This is a small enumeration indicating roughly the nature of an error.
+The possible values are:
+
+ Race
+
+ Data race. Helgrind will try to show the stacks for both
+ conflicting accesses if it can; it will always show the stack
+ for at least one of them.
+
+ UnlockUnlocked
+
+ Unlocking a not-locked lock
+
+ UnlockForeign
+
+ Unlocking a lock held by some other thread
+
+ UnlockBogus
+
+ Unlocking an address which is not known to be a lock
+
+ PthAPIerror
+
+ One of the POSIX pthread_ functions that are intercepted
+ by Helgrind, failed with an error code. Usually indicates
+ something bad happening.
+
+ LockOrder
+
+ An inconsistency in the acquisition order of locks was observed;
+ dangerous, as it can potentially lead to deadlocks
+
+ Misc
+
+ One of various miscellaneous noteworthy conditions was observed
+ (eg, thread exited whilst holding locks, "impossible" behaviour
+ from the underlying threading library, etc)
projects / perl / modules / Test-Valgrind.git / commitdiff