Lyrics:

I first met XPath in 2007, but we didn't become friends until just recently. For the most part I had avoided it; when forced to use it, I made do with trial and error. XPath just didn't really make sense to me.

But then I came across a peculiar parsing problemtoo complex for CSS selectors, too simple to warrant hand-rolled codeand decided to give XPath another shot. I discovered, much to my surprise and glee, that it does make sense, and once it makes sense, it's actually quite useful.

This is my story.The problemSay you're working on a website full of song lyrics, and in order to maintain a consistent reading experience, you want to capitalize the first word of every line. If the lyrics are stored in plain text this is pretty straightforward:lyrics.gsub!(/^./) { |character| character.upcase }But it gets more interesting if the lyrics are stored as an HTML fragment. A DOM doesn't have any built-in concept of 'lines.' You can't just break it up with a simple regular expression.

So the first thing we'll need to do is define, for ourselves, what 'the beginning of a line' means in a DOM. Here's a simple version:The first text node inside a

tagThe first text node following a tagSo, in the simplest case:

This is the beginning of a line.This is too.

This is the beginning of a line. This is not.

tags, text nodes, or is it a free-for-all? This is specified by the node test, which is the only required part of the location step. In our above example, node() is the most permissive node test: it selects everything. text() would only select text nodes; element() would only select elements; and explicitly specified node names like p and em above, of course, would only select elements with those names.Are there additional filters I want to add?Maybe I only want to select the first child of every node in the current context, or I only want to select tags that have an href attribute. For this sort of assertion, we can use predicates, which filter the matched nodes based on additional tree traversals. So I can filter the matched nodes based on a property of those nodes' children, parents, or siblings.

Our example doesn't have any predicates, so let's add one to only match tags that have an href attribute:/descendant-or-self::node()/child::p/child::em/child::aAs you can see, the predicate looks just like a location step in brackets, although the 'node test' portion of a predicate has much more functionality than the node test of a true location step.A better way to think about XPathSo instead of thinking of XPath as beefed-up CSS selectors, a more helpful analogy is to chained jQuery traversals. For instance, we could approximate our expression above in jQuery like so:$(document).find('*').
children('p').
children('em').
children('a').filter(')
In our analogy, the jQuery traversal method we use is the equivalent of the axis: .children() corresponds to the XPath child axis, and .find() to the descendant axis.
The selector we pass to the jQuery methods is transparently equivalent to the XPath node test, although jQuery doesn't allow us to select text nodes at all.

The jQuery .filter() method stands in for the XPath predicate, although in this case again jQuery is much less powerful than XPath.

Note that the .children('em') call above will match all children of all the matched

tags; this is exactly how XPath axes work as well.OK, back to the beginning-of-line thingNow that we've got a solid grasp of how XPath actually works, let's put it to work to solve the beginning-of-line problem. To start, we'll simplify the problem even further and just look for the first text node in each paragraph:/descendant-or-self::node()/child::p/child::text()Translated into English, we're saying:Find all the nodes in the documentFind all the children of those nodes that are paragraphsFind all the children of those paragraphs that are text nodesAnd filter those text nodes down to only the first matchNote that the position() function is evaluated within each matching node from the context: so the filter is saying 'first text child of each paragraph,' not 'first text child of a paragraph in the entire document.'

This is a good start, but it's not going to match text nodes that are deeply nested inside paragraphs. To do that, let's change our child axis to descendant:/descendant-or-self::node()/child::p/descendant::text()Now we're really cooking! The beauty of putting a position filter on a descendant axis is that it gives us exactly what we want, namely 'the first text node descendant of each paragraph.'

Now we just need to account for line breaks. To start, let's just build a separate expression that matches the first text node following a line break. I'll divide this one into multiple lines (perfectly legal in XPath) since it's a bit long:/descendant-or-self::node()/
child::br/
following-sibling::node()/
descendant-or-self::text()Again, breaking it down:Find all the nodesFind children of those nodes that are br elementsFind all the siblings that follow the br elementFilter those down to only the immediate next sibling of each brFind all the text nodes that are either the first sibling we already matched,
or one of its descendantsFilter those text nodes down to only the first one we matchedVoila! Now we've got a way to select both newlines inside p tags and newlines after br tags. Time to combine this all into a single expression:(/descendant-or-self::node()/child::p|
/descendant-or-self::node()/child::br/following-sibling::node())/
descendant-or-self::text()To put the finishing touches on it, let's tighten up the expression a little bit with some shorthand, and collapse a couple of the steps:(//p|//br/following-sibling::node())/
descendant-or-self::text()That's a pretty concise expression of a fairly sophisticated concept. And if we wanted to handle more ways of generating 'lines' on the screen, we could simply add more element names to the list of possible matches.So what does all this get us?Hopefully at this point XPath is making some sense, but why go through all that when we had a perfectly good, easily understandable Ruby implementation?

For the most part, Ruby is best used for high-level code like business logic, integrating various application components, and writing rich domain models. The best Ruby code is about intent, not implementation. That's why it's so unsatisfying to use Ruby to do something as low-level and application-agnostic as traversing a DOM tree looking for nodes with a certain property.

One good reason to use XPath might be speed: Nokogiri's XPath traversal is implemented by libxml, and native code is fast! However, for this particular task, XPath is actually considerably slower than the pure-Ruby implementation. Interestingly, that's not true if you take out the part and only look for text at the beginning of paragraphs. My guess is that the following-sibling axis is the culprit, since it has to select all the following siblings of the br tags, and then filter them down to only the first sibling.

So XPath may be slower or may be faster depending on exactly what you're doing, but its only hard to read if youre new to it. It's a purpose-built tool, allowing you to traverse the DOM in any way you'd like with expressions that are compact and idiomatic.