MetaHarvester

MetaHarvester is a BitTorrent metadata crawler toolkit.

This repository contains two parts:

• dht package (module root): DHT crawler/server implementation.
  • BEP-5
  • BEP-9
  • BEP-10
• bep3 package: decoupled BEP-3 implementation.
  • metainfo parsing
  • tracker announce client
  • peer wire protocol codec/session
  • piece manager and file storage mapping

See doc/BEP3_DESIGN.md for architecture details.

Requirements

• Go 1.25+

Installation

go get metaharvester@latest

Quick Start (DHT Spider)

package main

import (
    "fmt"

    "metaharvester"
)

func main() {
    downloader := dht.NewWire(65536, 1024, 256)

    go func() {
        for resp := range downloader.Response() {
            fmt.Println(resp.InfoHash, len(resp.MetadataInfo))
        }
    }()
    go downloader.Run()

    config := dht.NewCrawlConfig()
    config.OnAnnouncePeer = func(infoHash, ip string, port int) {
        downloader.Request([]byte(infoHash), ip, port)
    }

    d := dht.New(config)
    d.Run()
}

More examples are in sample/.

Metadata crawler demo (config-driven, metadata only):

go run ./sample/magnetcrawler -config sample/magnetcrawler/config.example.json

Quick Start (BEP-3)

Parse .torrent metadata:

go run ./sample/bep3 <path-to-file.torrent>

Core code is under bep3/.

Notes

• Default crawl config may use around 300MB RAM. Tune MaxNodes and BlackListMaxSize for your machine.
• NAT traversal is still not implemented.

TODO

• NAT Traversal.
• Optimization.

License

MIT, see LICENSE.