Skip to content

trafilatura

2,988 4 63 Apache-2.0
1.9.0 (2 May 2024) Jul 17 2019 431.5 thousand (month)

Trafilatura is a Python package and command-line tool designed to gather text on the Web. It includes discovery, extraction and text processing components. Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats.

Going from raw HTML to essential parts can alleviate many problems related to text quality, first by avoiding the noise caused by recurring elements (headers, footers, links/blogroll etc.) and second by including information such as author and date in order to make sense of the data. The extractor tries to strike a balance between limiting noise (precision) and including all valid parts (recall). It also has to be robust and reasonably fast, it runs in production on millions of documents.

This tool can be useful for quantitative research in corpus linguistics, natural language processing, computational social science and beyond: it is relevant to anyone interested in data science, information extraction, text mining, and scraping-intensive use cases like search engine optimization, business analytics or information security.

Example Use


# it can be used to clean HTML files
from trafilatura import clean_html

html = '<html><head><title>My Title</title></head><body><p>This is some <b>bold</b> text.</p></body></html>'
cleaned_html = clean_html(html)
print(cleaned_html)

# can strip away tags:
clean_html(html, tags_to_remove=["title"])
# or attributes
clean_html(html, attributes_to_remove=["title"])

Alternatives / Similar


1,595 2024.2.26 (2 months ago) Dec 14 2008 compare
13,679 0.2.8 (5 years ago) Dec 28 2012 compare
129,031 2021.12.17 (2 years ago) Feb 22 2012 compare
3,438 0.11.0 (1 year, 7 months ago) Oct 20 2013 compare
2,579 0.8.1 (3 years ago) Jun 30 2011 compare
828 0.16.0 (10 months ago) Oct 27 2015 compare
49,385 0.4.1700 (4 days ago) Sep 01 2012 compare
10,575 1.1.9 (5 years ago) Aug 24 2018 compare
182 2.0.7 (1 year, 6 months ago) Dec 11 2020 compare

Other Languages

2,482 v1.3.0 (2 months ago) Apr 20 2016 compare
2,065 v4.4.10 (5 months ago) Oct 26 2013 compare
330 2.0.39 (11 days ago) Sep 11 2013 compare
767 3.5.4 (3 years ago) Mar 02 2013 compare
Was this page helpful?