Skip to content

Latest commit

 

History

History
125 lines (96 loc) · 4.01 KB

README.md

File metadata and controls

125 lines (96 loc) · 4.01 KB
Raw

Skyscraper - HTML scraping with XPath

Dependency Status License MIT Crates.io doc.rs

Rust library to scrape HTML documents with XPath expressions.

This library is major-version 0 because there are still todo! calls for many xpath features. If you encounter one that you feel should be prioritized, open an issue on GitHub.

See the Supported XPath Features section for details.

HTML Parsing

Skyscraper has its own HTML parser implementation. The parser outputs a tree structure that can be traversed manually with parent/child relationships.

Example: Simple HTML Parsing

use skyscraper::html::{self, parse::ParseError};
let html_text = r##"
<html>
    <body>
        <div>Hello world</div>
    </body>
</html>"##;
 
let document = html::parse(html_text)?;

Example: Traversing Parent/Child Relationships

// Parse the HTML text into a document
let text = r#"<parent><child/><child/></parent>"#;
let document = html::parse(text)?;
 
// Get the children of the root node
let parent_node: DocumentNode = document.root_node;
let children: Vec<DocumentNode> = parent_node.children(&document).collect();
assert_eq!(2, children.len());
 
// Get the parent of both child nodes
let parent_of_child0: DocumentNode = children[0].parent(&document).expect("parent of child 0 missing");
let parent_of_child1: DocumentNode = children[1].parent(&document).expect("parent of child 1 missing");
 
assert_eq!(parent_node, parent_of_child0);
assert_eq!(parent_node, parent_of_child1);

XPath Expressions

Skyscraper is capable of parsing XPath strings and applying them to HTML documents.

Below is a basic xpath example. Please see the docs for more examples.

use skyscraper::html;
use skyscraper::xpath::{self, XpathItemTree, grammar::{XpathItemTreeNodeData, data_model::{Node, XpathItem}}};
use std::error::Error;

fn main() -> Result<(), Box<dyn Error>> {
    let html_text = r##"
    <html>
        <body>
            <div>Hello world</div>
        </body>
    </html>"##;

    let document = html::parse(html_text)?;
    let xpath_item_tree = XpathItemTree::from(&document);
    let xpath = xpath::parse("//div")?;
   
    let item_set = xpath.apply(&xpath_item_tree)?;
   
    assert_eq!(item_set.len(), 1);
   
    let mut items = item_set.into_iter();
   
    let item = items
        .next()
        .unwrap();

    let element = item
        .as_node()?
        .as_tree_node()?
        .data
        .as_element_node()?;

    assert_eq!(element.name, "div");
    Ok(())
}

Supported XPath Features

Below is a non-exhaustive list of all the features that are currently supported.

  1. Basic xpath steps: /html/body/div, //div/table//span
  2. Attribute selection: //div/@class
  3. Text selection: //div/text()
  4. Wildcard node selection: //body/*
  5. Predicates:
    1. Attributes: //div[@class='hi']
    2. Indexing: //div[1]
  6. Functions:
    1. fn:root()
    2. contains(haystack, needle)
  7. Forward axes:
    1. Child: child::*
    2. Descendant: descendant::*
    3. Attribute: attribute::*
    4. DescendentOrSelf: descendant-or-self::*
    5. (more coming soon)
  8. Reverse axes:
    1. Parent: parent::*
    2. (more coming soon)
  9. Treat expressions: /html treat as node()

This should cover most XPath use-cases. If your use case requires an unimplemented feature, please open an issue on GitHub.