Web crawler and scraper for Rust
-
Updated
Dec 6, 2025 - Rust
#
Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
Here are 101 public repositories matching this topic...
An ergonomic Python HTTP Client with TLS fingerprint
python tls rust http crawler scraper https websocket http2 http-client fingerprint websocket-client web-scraping akamai ja3 tls-client tls-fingerprint web-scraping-python ja4 akamai-fingerprint
-
Updated
Dec 4, 2025 - Rust
An ergonomic Rust HTTP Client with TLS fingerprint
tls rust http client crawler scraper https websocket http2 http-client fingerprint web-scraper websocket-client akamai ja3 tls-fingerprint ja4
-
Updated
Dec 5, 2025 - Rust
A rock-solid cryptocurrency crawler library.
-
Updated
Mar 19, 2023 - Rust
🕷️ The pipeline for the OSCAR corpus
-
Updated
Nov 9, 2025 - Rust
Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.
-
Updated
Aug 5, 2025 - Rust
✴️ An experimental graph database
-
Updated
Jul 20, 2023 - Rust
Spider ported to Python
-
Updated
Jan 28, 2025 - Rust
A command line tool based on the crypto-crawler library.
-
Updated
Mar 19, 2023 - Rust
🎓a Better BlackBoard for PKUers. 北京大学教学网命令行工具（🖥️Win/🐧Linux/🍏Mac）, 支持查看/提交作业、下载课程回放.
-
Updated
Sep 11, 2025 - Rust
Spider ported to Node.js
-
Updated
Jan 28, 2025 - Rust
CLI to download all images/webms in a 4chan thread
-
Updated
Aug 22, 2022 - Rust
⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations
rust crawler scanner bruteforce rust-lang subdomain brute-force pentesting pentest rust-crate pentest-tool subdomain-scanner subdomain-enumeration subdomain-bruteforcing zonetransfer pentesting-tool subdomain-finder searchengines
-
Updated
Nov 27, 2025 - Rust
Rust Web Crawler saving pages on Redis
-
Updated
Jun 13, 2021 - Rust
Crawling and scraping the Web for fun and profit
-
Updated
Apr 13, 2023 - Rust
Python, Javascript, and Rust libraries for the Spider Cloud API.
crawler scraper ai spider html-to-markdown web-scraping ai-agents supabase ai-scraping llm-webcrawler
-
Updated
Nov 22, 2025 - Rust
A smart web crawler built in Rust that uses Claude AI to select the most relevant URLs from website sitemaps based on crawling objectives.
-
Updated
Jul 9, 2025 - Rust
Download Webtoon
crawler scraper manga hentai manga-downloader manga-scraper hentai-downloader manhwa-scraper manhua-scraper doujin-downloader hentai-scraper tachiyomi-alternative
-
Updated
Nov 7, 2025 - Rust
decentralized web search engine, web archive and feed reader
-
Updated
Jul 21, 2025 - Rust
- Followers
- 531 followers
- Website
- github.com/topics/crawler
- Wikipedia
- Wikipedia