XML ZONE TIP: Command-line XML Processing
Working with XML Documents from the Shell
David Mertz, Ph.D.
Line commander, Gnosis Software, Inc.
April, 2003
Most of the time, processing XML documents utilizes heavy-duty
APIs and custom applications. However, the tradition of using
small tools with I/O piped between them is a very fine one on
Unix-like platforms. XML need not be entirely left out of
quick-and-dirty processing with one-liners that is especially
useful during development and debugging cycles.
INTRODUCTION
------------------------------------------------------------------------
As much as I hate to say it, XML tools simply have not
reached the level of convenience of the text utilities
available at a Unix-like command-line. For line-oriented,
whitespace- or comma-delimited text files, it is quite amazing
what you can accomplish with clever combinations of 'sed',
'grep', 'xargs', 'wc', 'cut', pipes, and short shell scripts.
In my opinion, it is not that XML is inherently resistent to
the modular treatment flat text files enjoy. We just need to
learn from experience the best ways to componentize XML tools.
For example, in writing this tip, I had a few realistic sample
tasks in mind; but what I found was that even those tools that
have command-line facilities have not yet learned to "play
nice" with each other. Working with multiple tools is not
intractable, it just requires a little bit of wrapping.
A fact to note is that quite a few people have written
versions--in various programming languages--of similar simple
tools. Each version behaves a bit differently, but they tend
to accomplish the same overall task. For this tip, I look at
the tools 'xml_indexer', 'xmlcat', and 'xpath'--the first two
come from my Gnosis Utilities, the last is a Perl module
written by Matt Sergeant (get it from CPAN).
FINDING WORDS IN XML PROSE
------------------------------------------------------------------------
I have written previously about 'xml_indexer', which creates an
index of the words within XML documents by their XPath. For
example, you can index then search an XML document with:
#--------- Indexing and searching on XPaths --------------#
% xml_indexer chap.xml
% indexer events were
/Users/dqm/chap.xml::/chapter/sect1[2]/sect2[1]/para[1]
/Users/dqm/chap.xml::/chapter/sect1[2]/sect2[4]/para[3]
1 files matched wordlist: ['events', 'mostly']
Processed in 0.062 seconds (SlicedZPickleIndexer)
These commands display the elements within the XML document
'chap.xml' that contain the words 'events' and 'were' (not
necessarily in order or proximity). If other XML documents were
added to the index, matching occurrences in them would appear
also. New searches are almost instantaneous, even if multiple
documents are indexed, by the way.
While it tells you a little bit to know that words occur at
particular XPaths within particular documents, the point of a
search is usually to see (or further process) the actual content
matches. For that, you need to employ a command-line 'xpath'
tool; I have installed Perl's XML:XPath, whose behavior I like.
You -can- cut-and-paste discovered XPaths into the tool 'xpath'.
E.g.:
#------------ Manually looking at an XPath ---------------#
% xpath chap.xml '/chapter/sect1[2]/sect2[4]/para[3]'
Found 1 nodes:
-- NODE --
It is not particularly remarkable that...
...
This points to a nice modularity in the tools. Moreover, if the
XPath passed to 'xpath' had wildcards in it, it might have
matched more than just the one node. Unfortunately, the output of
'indexer' does not have quite the right form to pipe to 'xpath',
to automate looking at the nodes with matched words: 'indexer'
separates the filename from the XPath with "::", and 'xpath' only
looks at one XPath at a time. We can do better.
A FIRST LITTLE SHELL SCRIPT
------------------------------------------------------------------------
There might be a way to manage the above "impedence mismatch"
using clever combinations of 'xargs', 'apply', pipes, and the
like. But I found it easier to write a short (and reusable)
shell script:
#------------------ find-xml-elements --------------------#
#!/bin/sh
for hit in `indexer $@ 2> /dev/null`
do
echo $hit | sed 's/::/ /' > loc.tmp
cat loc.tmp | xargs xpath 2> /dev/null
echo
done
rm loc.tmp
As with other well-designed command-line tools, 'indexer' and
'xpath' send informational messages to STDERR, the actual results
to STDOUT. For my script, I am not interested in the STDERR
messages. Now I can find all the nodes in which a list of words
occur as easily as:
#------------ Searching XML elements for words -----------#
% find-xml-elements events were
Lest we forget some events in a recent decade...
...
Salem and by HUAC.
It is not particularly remarkable that...
...
being uncovered.
So far, so good. What our search outputs is a series of XML
snippets, where each top-level element contains the searched
words. However, the result is generally not quite a
well-formed XML document, since it is multiply-rooted.
COMPARING XML DOCUMENT AND EXTRACTING TEXT
------------------------------------------------------------------------
One difficulty in analyzing XML data is that XML documents can
contain variations in formatting that are irrelevant to their
semantic content. Some whitespace is "ignorable", the order of
attributes is discarded during parsing, empty elements may be
either self-closed or have an end-tag, and entities can be
encoded in a few different ways. In truth, even much whitespace
that is non-ignorable from a parser's perspective is nonetheless
insignificant from an application point-of-view; "pretty"
newlines and indenting are useful for people, and many
applications (optionally) perform such stylistic formatting.
There are a rather large number of tools that have been written
to compare XML documents in a semantically useful way. Most of
them have chosen the obvious name 'xmldiff', or something close
to it (use Google to find versions for various programming
languages). Underlying such a comparison of XML documents is a
-canonicalization- of the layout of each document. Once
inflexible algorithmic decisions have been made about the exact
rendering of an XML document, semantically similar documents are
easier to compare with generic tools like 'diff'.
I use a Python script I wrote called 'xmlcat'. The tool is not
complicated--it acts much like the standard 'cat' utility, but
canonicalizes XML documents along the way. In a chance to use my
favorite word, I can note that the operation of 'xmlcat' is
idempotent. The reason I like 'xmlcat' over similar tools like
'xmlpp' (see Resources) is that it adds an option inspired by the
web browser 'lynx'. If you pass the '--dump' argument to
'xmlcat', it outputs only the textual content of an XML document,
eliminating the tags (using vertical whitespace is a moderately
pretty way). For data-oriented XML, this capability is of
little use, but for marked-up prose, it is handy.
A SECOND SHELL SCRIPT FOR VIEWING TEXT
------------------------------------------------------------------------
If you search XML documents of prose for content words, most
likely you are interested in the content more than you are the
markup. Filtering with 'xmlcat --dump' is exactly the trick to
remove unwanted XML tags. However, directly piping the output
of 'find-xml-elements' to 'xmlcat' is not quite right, since
the output of 'find-xml-elements' is not quite an entire
well-formed XML documents (it is fragments, as noted). A short
shell script solves the problem:
#-------------------- find-xml-text ----------------------#
#!/bin/sh
for hit in `indexer $@ 2> /dev/null`
do
echo $hit | sed 's/::/ /' > loc.tmp
cat loc.tmp | xargs xpath 2> /dev/null | xmlcat --dump
echo
done
rm loc.tmp
The output from 'find-xml-text' plays nice with standard text
utilites. For example, I would like to display all the
paragraphs that contain some search terms, but remove any left
indent from their lines and limit line-length:
#--------- Searching XML element text for words ----------#
% find-xml-text events were | sed 's/^ *//' | fmt -w 70
Lest we forget some events in a recent decade...
...
...those in Salem and by HUAC.
It is not particularly remarkable...
...
...being uncovered.
RESOURCES
------------------------------------------------------------------------
Kip Hampton wrote a worthwhile article last year looking at
Perl tools for command-line XML processing:
http://www.xml.com/lpt/a/2002/04/17/perl-xml.html
The Perl tools 'xmldiff' (compare XML documents) and 'xmlpp' (XML
pretty printer) can be found at:
http://software.decisionsoft.com/tools.html
Gnosis Utilies includes several of the utilities discussed in
this article, download it from:
http://gnosis.cx/download/Gnosis_Utils-current.tar.gz
_XML Matters #10_ discusses full text indexing of XML documents
by XPath:
http://www-106.ibm.com/developerworks/xml/library/x-matters10.html
ABOUT THE AUTHOR
------------------------------------------------------------------------
{Picture of Author: http://gnosis.cx/cgi-bin/img_dqm.cgi} David
Mertz uses a wholly unstructured brain to write about structured
document formats. David may be reached at mertz@gnosis.cx; his
life pored over at http://gnosis.cx/publish/. And buy his book:
http://gnosis.cx/TPiP/.