Elegant Scraper and Crawler Framework for Golang

A Chrome DevTools Protocol driver for web automation and scraping.

Declarative web scraping

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Extract structured data from web sites. Web sites scraping.

Open source SEO audit tool.

Second-order subdomain takeover scanner

🕸️ Crawl in the web network

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Download a large list of files concurrently

Go process used to crawl websites

Scraply a simple dom scraper to fetch information from any html based website

A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.

Fast, highly configurable, cloud native dark web crawler.

🌱 goClone - clone websites in seconds

A Content Discovery and Development Platform. Empowering Cybersecurity, AI, Marketing, and Finance professionals and researchers to discover, analyze, and interact with the web in all its dimensions.

Supacrawler's ultralight engine for scraping and crawling the web. Written in go for maximum performance and concurrency.

Raven is a powerful and customizable web crawler written in Go.