The Complete Guide to Parsing XML with LXML in Python

XML is ubiquitous these days. Chances are, you‘ll eventually need to parse, extract data from, or modify XML documents in your Python projects. But working with XML-based data can be tricky and complex.

In this comprehensive 4500+ word guide, you‘ll learn how to use one of the most powerful Python libraries for XML processing – LXML.

Whether you are working with configurations, documents, web APIs, databases or any other source of XML data, being able to slice and dice XML efficiently can save you tons of time and effort.

Let‘s get started!

XML – The Data Powerhouse

Before we dive into Python tools, let‘s do a quick primer on XML itself.

XML stands for Extensible Markup Language. It provides a standard, platform-independent way of annotating, structuring and organizing data.

Some key facts about XML:

• Created in 1996, now widely adopted
• Human and machine readable
• Used for documents, configs, data storage and much more
• Powers many standard protocols and formats (SOAP, SVG, RSS etc)
• Popular in Java, .NET and other ecosystems

Like HTML, XML uses elements and attributes:

<book category="tech">
  <title>Python 101</title>
  <author>John Doe</author>  
</book>

But XML allows you to define your own element names and structure.

Here are some common uses of XML:

• Configuration files – App configs, building automation, device settings etc.
• Office documents – DocX, spreadsheet ML, Adobe InDesign etc.
• Publishing – eBooks, scholarly articles, magazines, journals
• Web services – SOAP, WSDL for APIs, feeds like RSS/Atom
• Databases – Exporting and storing relational data as XML
• Graphics – SVG vector images, X3D 3D graphics etc.

XML provides a vendor-neutral way of representing rich, hierarchical data which can be easily parsed and understood by different systems.

The Problem of XML Processing

But while XML solves many problems, it brings its own challenges!

• Verbose and cumbersome – XML files are often bloated and bulky with all the tag markup.
• Hard to query – Navigating complex nested structures is tricky.
• Namespace collisions – Identical element names from different vocabularies.
• No native validation – Unlike JSON Schema, XML itself doesn‘t validate.
• XPointer and XPath – Query expressions are powerful but confusing.

This is where XML parsing libraries come in. They abstract away the complexity and allow you to focus on the data itself.

In Python, we have several choices including the builtin xml.etree, third-party LXML and BeautifulSoup.

Let‘s look at why LXML rises above the rest for performance and functionality.

Introducing LXML – A Powerful Ally

LXML is a Pythonic binding for the C libraries libxml2 and libxslt. It provides:

• Blistering parsing speed – Even faster than native C or C++ XML tools.
• Lightning fast XPath queries – Supports complex element extraction.
• Robust – Handles malformed markup without crashing.
• Feature rich – Plug into a vast XML ecosystem.
• Pythonic – Conforms to Python idioms wherever possible.

Some key benefits of LXML:

• Uses ElementTree API – Easy to migrate from standard library.
• Transparent XPath support – Filter elements by complex criteria.
• Powerful objectify API – Access elements as object attributes.
• Sandboxed XSLT – Safely transform XML using XSL stylesheets.
• Pythonic element trees – No DOM! Access your XML intuitively.
• Full programmatic construction of XML docs.
• Incremental event-based document building.
• Support for XML namespaces via prefixes.
• Integrates with other Python XML tools like Requests.

Let‘s contrast LXML with a few alternatives:

Library	Pros	Cons
xml.etree (builtin)	Ships with Python, simple API	Slow, limited functionality
BeautifulSoup	HTML parsing, fuzzy matching	Not designed for XML
xmltodict	XML to Python dicts	Loss of structure, limited XML support

So while the other libraries have some use cases, for industrial strength XML wrangling in Python, LXML is hard to beat.

According to benchmarks, LXML can parse XML 2-10x faster than standard ElementTree! The C acceleration makes a huge difference.

Note that LXML requires libxml2 and libxslt as dependencies. Make sure you have these installed first.

Let‘s dive in and see how to put LXML through its paces!

Installing LXML

The best way to install LXML is via pip:

pip install lxml

This will fetch the latest stable version from PyPI and install it in your environment.

For some additional speed, consider compiling libxml2 and libxslt from source with optimizations enabled. But the PyPI LXML works fine for most applications.

After installation, import the etree module:

from lxml import etree

This provides access to all functionality including XML, XSLT, DOM and XMLSchema classes.

Let‘s start parsing!

Loading and Parsing XML with LXML

XML can be loaded from different sources:

• Local files
• Remote websites
• Database blobs
• Strings in memory
• Standard input streams

LXML provides flexibility to load XML documents from all these sources.

The result is an XML ElementTree object which represents the parsed document as nested Python objects.

Let‘s look at a couple common ways to parse XML into an ElementTree:

From Local File

Pass a file path to parse():

doc = etree.parse(‘data.xml‘) # Parse file

We can also specify the parser for more control:

parser = etree.XMLParser(remove_blank_text=True) # No whitespace text nodes
doc = etree.parse(‘data.xml‘, parser) 

The parser allows you to:

• Strip comments, blank text, PI nodes
• Validate against DTD
• Resolve entities
• Recover from errors
• Collect diagnostics

So make sure to configure an XMLParser for your specific use case.

From Remote Server

To fetch and parse XML from a URL:

import requests

resp = requests.get(‘https://www.example.com/data.xml‘)  
doc = etree.XML(resp.content)

We use the handy requests module to download the content first.

This can be combined with a custom XMLParser as well.

From String

If you already have the XML content in a string, use:

xml_string = ‘<data>...</data>‘
doc = etree.XML(xml_string)

So LXML makes it easy to parse XML from diverse sources into a common ElementTree object.

Troubleshooting Load Errors

When loading real-world XML, you may encounter:

• IOError – File not found, permissions issue etc. Add error handling.
• SyntaxError – Invalid XML syntax. Enable recover/resolve in parser.
• Encoding errors – Declare encoding in parser if needed.

So be ready to catch exceptions and handle problems with bad XML!

With our document loaded, let‘s start querying it.

XPath Queries with LXML

XPath is a language for addressing and filtering nodes in an XML document.

LXML has built-in support for XPath expressions when working with ElementTree objects.

For example, consider this sample XML:

<bookstore>

  <book category="cooking">
    <title lang="en">Everyday Italian</title>  
    <author>Giada De Laurentiis</author>
    <year>2005</year>
    <price>30.00</price>
  </book>

  <book category="children">
    <title lang="en">Harry Potter</title>
    <author>J K. Rowling</author>
    <year>2005</year> 
    <price>29.99</price> 
  </book>

</bookstore>

Let‘s grab all the book titles:

titles = doc.xpath(‘//book/title/text()‘) 

The xpath() method returns a list of matching elements.

Some key parts of our XPath query:

• //book – Selects all <book> nodes recursively
• /title – Selects the <title> child element
• /text() – Returns the text inside

This gives us the readability of XPath with the power of LXML!

Other examples:

# All authors
authors = doc.xpath(‘//author/text()‘) 

# Books after 2000
recent = doc.xpath(‘//book[year>2000]‘) 

# All prices 
prices = doc.xpath(‘//price/text()‘)

We can also match attributes:

english_books = doc.xpath(‘//book/title[@lang="en"]‘)

LXML has support for the full XPath 1.0 standard including operators, functions and more.

Some common uses:

• Filtering lists of elements
• Extracting text and data
• Testing node characteristics

Keep your XPath expressions short and simple. And watch out for injection attacks when using user-supplied input in XPaths!

Namespaces in XPath Queries

Namespaces allow mixing of vocabularies in XML.

We can map a namespace prefix to a URI when querying:

ns = {‘x‘: ‘http://books.com‘}

x.xpath(‘//x:book‘, namespaces=ns)

LXML will handle matching our custom x prefixed elements.

Namespaces are very common in real-world XML, so this is important functionality that LXML provides out of the box.

Now let‘s look at how to process our queried XML elements in Python.

Extracting and Processing XML Data

Once we have selected elements using XPath, we can easily extract data or manipulate the XML further in Python.

Looping Over Elements

A common pattern is looping through elements:

for book in books:
  print(book.xpath(‘./title/text()‘)[0]) 

for author in authors:
  print(author.text)

This allows us to iterate over elemento trees and work with their child nodes.

Extracting Text and Attributes

Use the .text attribute to get element inner text:

title = title_element.text

And .attrib for attributes:

lang = title_element.attrib[‘lang‘] 

We can then process the text in any way needed:

for title in titles:
  print(title.upper()) # Uppercase title

Modifying the XML Tree

Elements support helper methods to modify the XML:

book.remove(book.find(‘year‘)) # Delete 

title.set(‘lang‘, ‘fr‘) # Set attribute

new_node = etree.Element(‘rating‘) 
new_node.text = ‘5‘
book.append(new_node) # Insert new element

So you can programmatically create, update or delete nodes.

Serializing Back to XML

Once done, serialize the ElementTree back into XML:

xml_string = etree.tostring(doc)
print(xml_string.decode(‘utf-8‘)) 

doc.write(‘output.xml‘) # Write to file

You can also pretty print with indentation:

print(etree.tostring(doc, pretty_print=True).decode(‘utf-8‘))

This allows piping the processed XML to any destination.

So LXML gives the flexibility to not just read XML documents, but also modify, enrich and write them back out. You get a full workflow for XML transformations.

Validating XML Against Schemas

Unlike JSON, plain XML does not validate structure and datatypes by default.

We need schemas like DTD and XMLSchema to define validation rules.

LXML allows validating documents against schemas:

xmlschema = etree.XMLSchema(file=‘schema.xsd‘)

valid = xmlschema.validate(doc)  

if not valid:
  print(xmlschema.error_log) # Prints errors

This applies constraints during parsing for invalid documents.

You can also validate after parsing:

is_valid = xmlschema.assertValid(doc)

So LXML provides ways to go beyond basic well-formed XML and enforce validation rules.

When to Use LXML vs xml.etree

The Python standard library comes with xml.etree.ElementTree for XML parsing.

LXML has much more functionality, but xml.etree may suffice for simple cases like:

• Prototyping and learning XML
• Parsing very small files
• Read-only XML processing
• Cost of dependencies unacceptable

Some signs it‘s time to upgrade to LXML:

• XML files > 100 KB
• Making frequent XPath queries
• Need for speed and performance
• Modifying and regenerating XML a lot
• Dealing with malformed documents
• Namespace heavy documents

As a rule of thumb, any non-trivial XML wrangling merits LXML. The C-based implementation pays for itself very quickly in terms of developer time saved!

LXML Gotchas and Tips

Here are some common pitfalls to watch out for:

• Don‘t manipulate the same tree from multiple threads concurrently. Use a lock.
• Remember that XPath indexing starts from 1, not 0!
• Disable entity expansion if not needed to prevent billion laugh attacks.
• Catch ParserError and XSLTApplyError explicitly.
• When replacing elements, reuse the old nodes if possible.
• If validation drags, try disabling DTD loading.
• Avoid XPath injection attacks when using dynamic queries.

If performance drags, look at compiling libxml2 and libxslt from source with optimizations enabled.

For more handy tips, check the LXML gotchas page.

XML Parsing With LXML – In Summary

We‘ve covered a lot of ground here! Let‘s recap the key takeaways:

• LXML provides supercharged XML parsing functionality by building on mature C libraries.
• It parses XML from files, urls, strings into ElementTree objects.
• Use XPath queries to extract and filter elements flexibly.
• Extract text, loop over sub-elements and process XML programmatically.
• Modify, add and delete nodes to transform documents.
• Output the modified XML by serializing back to string or files.
• Validate against schemas to enforce structure.

LXML combines speed, power and simplicity when working with real-world XML. While it has a learning curve, it pays for itself in increased productivity and less debugging time.

For production scenarios, accept no substitutes!

I hope you‘ve found this guide useful. Please leave a comment if you have any other LXML tips or feedback to share. Happy XML hacking!