news-please - an integrated web crawler and information extractor for news that just works

Process Common Crawl data with Python and Spark

A very simple news crawler with a funny name

A python utility for downloading Common Crawl data

Price Crawler - Tracking Price Inflation

Statistics of Common Crawl monthly archives mined from URL index files

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.

[码云](https://gitee.com/generals-space/site-mirror-py) 通用爬虫, 仿站工具, 整站下载

Simple multi threaded tool to extract domain related data from commoncrawl.org

A News Article Collection Library

Python tools to retrieve text from CommonCrawl WARC files based on cdx index.

super-Django-CC is a simle web interface for commoncrawl.org

Lightweight Python utility for retrieving individual pages from the Common Crawl archives.

Crawls the web to generate a huge dataset for training

Collected data about from three sources, one opinion-based social media in twitter, research data in New York Times, and the third is the common crawl data for the same topic or key phrase, and from similar time periods. Processed the three data sets collected individually using classical big data methods like Map Reduce in Google Dataproc Clust…

👻 GhostPath — A powerful modular reconnaissance toolkit built for hackers, OSINT professionals & bug bounty hunters — passive + active recon in a sleek CLI shell. Discover subdomains, probe paths, mine archives and hunt certificates — all from one interactive terminal interface.

A python module to download pages from commoncrawl