From db9480cd57bd658a800b83f69e615dc9967ae764 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Sun, 15 Jan 2023 13:16:36 -0500 Subject: [PATCH 01/18] init disc. and route section --- content/concepts/discovery/_index.md | 5 +++++ content/concepts/discovery/kaddht.md | 7 +++++++ content/concepts/discovery/mdns.md | 7 +++++++ content/concepts/discovery/overview.md | 2 +- content/concepts/discovery/pubsub.md | 7 +++++++ content/concepts/discovery/rendezvous.md | 7 +++++++ 6 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 content/concepts/discovery/_index.md create mode 100644 content/concepts/discovery/kaddht.md create mode 100644 content/concepts/discovery/mdns.md create mode 100644 content/concepts/discovery/pubsub.md create mode 100644 content/concepts/discovery/rendezvous.md diff --git a/content/concepts/discovery/_index.md b/content/concepts/discovery/_index.md new file mode 100644 index 00000000..33ea2c43 --- /dev/null +++ b/content/concepts/discovery/_index.md @@ -0,0 +1,5 @@ +--- +title : "Peer Discovery and Routing" +description: "Peer discovery and routing protocols are used to discover and announce services to other peers." +weight: 7 +--- diff --git a/content/concepts/discovery/kaddht.md b/content/concepts/discovery/kaddht.md new file mode 100644 index 00000000..732de995 --- /dev/null +++ b/content/concepts/discovery/kaddht.md @@ -0,0 +1,7 @@ +--- +title: "Kad-DHT" +description: "." +weight: 226 +--- + +Coming soon! diff --git a/content/concepts/discovery/mdns.md b/content/concepts/discovery/mdns.md new file mode 100644 index 00000000..5643dad8 --- /dev/null +++ b/content/concepts/discovery/mdns.md @@ -0,0 +1,7 @@ +--- +title: "mDNS" +description: "mDNS uses a multicast system of DNS records over a local network to enable peer discovery." +weight: 224 +--- + +Coming soon! diff --git a/content/concepts/discovery/overview.md b/content/concepts/discovery/overview.md index 703c4aa2..55beaa8b 100644 --- a/content/concepts/discovery/overview.md +++ b/content/concepts/discovery/overview.md @@ -1,5 +1,5 @@ --- -title: "What is Peer Discovery" +title: "What is Discovery & Routing" description: "Peer discovery protocols are used to discover and announce services to other peers." weight: 221 --- diff --git a/content/concepts/discovery/pubsub.md b/content/concepts/discovery/pubsub.md new file mode 100644 index 00000000..37d74d84 --- /dev/null +++ b/content/concepts/discovery/pubsub.md @@ -0,0 +1,7 @@ +--- +title: "PubSub" +description: "." +weight: 225 +--- + +Coming soon! diff --git a/content/concepts/discovery/rendezvous.md b/content/concepts/discovery/rendezvous.md new file mode 100644 index 00000000..fe3ca878 --- /dev/null +++ b/content/concepts/discovery/rendezvous.md @@ -0,0 +1,7 @@ +--- +title : "Rendezvous" +description: "The rendezvous protocol can facilitate the routing and discovery of nodes in a peer-to-peer network using a common location." +weight: 223 +--- + +Coming soon! From 9ca1b62c10fdd57f62e4d3fae7bc3af230cc5c03 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Sun, 15 Jan 2023 13:20:47 -0500 Subject: [PATCH 02/18] transform discovery overview to include route --- content/concepts/discovery/overview.md | 52 ++++++++++++++++++-------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/content/concepts/discovery/overview.md b/content/concepts/discovery/overview.md index 55beaa8b..8cc67596 100644 --- a/content/concepts/discovery/overview.md +++ b/content/concepts/discovery/overview.md @@ -6,25 +6,45 @@ weight: 221 ## Overview -Distributed networks require a way to discover peers. Peer discovery in distributed networks -is the process of using a discovery protocol to discover and announce services to other -available peers. Peer discovery requires each node in a network to be able to discover other -nodes without the need for a central server. +Peer discovery and routing are two essential aspects of P2P networking. In a distributed network, +each node must be able to discover and communicate with other nodes without the need for a central +server. -## Peer discovery in libp2p +### Peer Discovery -Peers can discover other peers in libp2p using a variety of mechanisms, such as -exchanging peer addresses over the network, querying a directory service, or using a -[DHT (distributed hash table)](/concepts/fundamentals/dht) to store and retrieve information about -available peers. The process of peer discovery varies on the composition of protocols being -used to connect peers. +Peer discovery is the process of finding and announcing services to other available peers in a +distributed network. It allows a node to discover other nodes in the network and learn about their +services. Peer discovery can be done using various protocols, such as broadcasting a +message to all peers in the network or using a bootstrap node to provide a list of known peers. -Generally, peer discovery is a function in respect to a topic that returns a list of -peers, usually their multiaddr and port information. +### Peer Routing + +Peer routing, on the other hand, refers to finding a specific peer's location +in the network. This is typically done by maintaining a routing table or a similar data structure that +keeps track of the network topology. Algorithms such as kbucket and "routing by gossip" can be used to find +the closest peers to a given peer ID. + +A peer may use a routing algorithm to find the location of a specific peer and then use that +information to discover new peers in the vicinity. Additionally, a peer may parallel use both peer routing +and peer discovery mechanisms to find new peers and route data to them. In practice, the distinction between +peer routing and peer discovery is not always clear-cut, and it's worth noting that in a real-world implementation, +discovery and routing usually happen concurrently. + +## Discovery and Routing in libp2p + +libp2p is a peer-to-peer networking stack that provides a set of modules for different network-level +functionality, including peer discovery and routing. Peers in a libp2p network can discover other +peers using various mechanisms, such as exchanging peer addresses over the network, querying a directory +service, or using a Distributed Hash Table (DHT) to store and retrieve information about available peers. The methods for peer discovery in libp2p are, but are not limited to: -- [rendezvous](rendezvous); -- [mDNS](mdns); -- [publish/subscribe](/concepts/pubsub/overview.md) -- and using a [DHT](/concepts/introduction/protocols/dht.md). +- [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer addresses in a secure and + private manner +- [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to discover other + peers on the local network. +- [Publish/Subscribe](pubsub.md): a protocol that allows peers to subscribe to specific topics and receive + updates from other peers that are publishing information on those topics. +- [DHT](kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns each piece of + content a unique identifier and stores the content on the peer whose identifier is closest to the + content's identifier. From f78aaa539bb352c8a461ba0b6fbab63ff0cee7ac Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Sun, 15 Jan 2023 13:27:51 -0500 Subject: [PATCH 03/18] rename dir --- content/concepts/{discovery => discovery-routing}/_index.md | 0 content/concepts/{discovery => discovery-routing}/kaddht.md | 0 content/concepts/{discovery => discovery-routing}/mdns.md | 0 content/concepts/{discovery => discovery-routing}/overview.md | 0 content/concepts/{discovery => discovery-routing}/pubsub.md | 0 content/concepts/{discovery => discovery-routing}/rendezvous.md | 0 6 files changed, 0 insertions(+), 0 deletions(-) rename content/concepts/{discovery => discovery-routing}/_index.md (100%) rename content/concepts/{discovery => discovery-routing}/kaddht.md (100%) rename content/concepts/{discovery => discovery-routing}/mdns.md (100%) rename content/concepts/{discovery => discovery-routing}/overview.md (100%) rename content/concepts/{discovery => discovery-routing}/pubsub.md (100%) rename content/concepts/{discovery => discovery-routing}/rendezvous.md (100%) diff --git a/content/concepts/discovery/_index.md b/content/concepts/discovery-routing/_index.md similarity index 100% rename from content/concepts/discovery/_index.md rename to content/concepts/discovery-routing/_index.md diff --git a/content/concepts/discovery/kaddht.md b/content/concepts/discovery-routing/kaddht.md similarity index 100% rename from content/concepts/discovery/kaddht.md rename to content/concepts/discovery-routing/kaddht.md diff --git a/content/concepts/discovery/mdns.md b/content/concepts/discovery-routing/mdns.md similarity index 100% rename from content/concepts/discovery/mdns.md rename to content/concepts/discovery-routing/mdns.md diff --git a/content/concepts/discovery/overview.md b/content/concepts/discovery-routing/overview.md similarity index 100% rename from content/concepts/discovery/overview.md rename to content/concepts/discovery-routing/overview.md diff --git a/content/concepts/discovery/pubsub.md b/content/concepts/discovery-routing/pubsub.md similarity index 100% rename from content/concepts/discovery/pubsub.md rename to content/concepts/discovery-routing/pubsub.md diff --git a/content/concepts/discovery/rendezvous.md b/content/concepts/discovery-routing/rendezvous.md similarity index 100% rename from content/concepts/discovery/rendezvous.md rename to content/concepts/discovery-routing/rendezvous.md From 8da59b068fafd92338664edff715b2e01e9959ed Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Tue, 17 Jan 2023 05:32:07 -0500 Subject: [PATCH 04/18] Update overview.md --- .../concepts/discovery-routing/overview.md | 68 ++++++++++--------- 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index 8cc67596..22b2237d 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -1,50 +1,54 @@ --- title: "What is Discovery & Routing" -description: "Peer discovery protocols are used to discover and announce services to other peers." +description: "Peer discovery and routing protocols are used to discover and announce services to other peers and find a peer's location, respectively." weight: 221 --- ## Overview -Peer discovery and routing are two essential aspects of P2P networking. In a distributed network, -each node must be able to discover and communicate with other nodes without the need for a central -server. +Peer discovery and routing are two essential aspects of P2P networking. In a P2P network, +each node must be able to discover and communicate with other nodes without the need for +a central server. ### Peer Discovery -Peer discovery is the process of finding and announcing services to other available peers in a -distributed network. It allows a node to discover other nodes in the network and learn about their -services. Peer discovery can be done using various protocols, such as broadcasting a -message to all peers in the network or using a bootstrap node to provide a list of known peers. +Peer discovery is the process of finding and announcing services to other available +peers in a P2P network. It allows a peer to discover other peers in the network and +learn about their services. Peer discovery can be done using various protocols, such +as broadcasting a message to all peers in the network or using a bootstrap node to +provide a list of known peers. ### Peer Routing Peer routing, on the other hand, refers to finding a specific peer's location -in the network. This is typically done by maintaining a routing table or a similar data structure that -keeps track of the network topology. Algorithms such as kbucket and "routing by gossip" can be used to find -the closest peers to a given peer ID. +in the network. This is typically done by maintaining a routing table or a similar +data structure that keeps track of the network topology. Algorithms such as kbucket +and "routing by gossip" can be used to find the closest peers to a given peer ID. -A peer may use a routing algorithm to find the location of a specific peer and then use that -information to discover new peers in the vicinity. Additionally, a peer may parallel use both peer routing -and peer discovery mechanisms to find new peers and route data to them. In practice, the distinction between -peer routing and peer discovery is not always clear-cut, and it's worth noting that in a real-world implementation, -discovery and routing usually happen concurrently. +A peer may use a routing algorithm to find the location of a specific peer and then +use that information to discover new peers in the vicinity. Additionally, a peer may +parallel use both peer routing and peer discovery mechanisms to find new peers and +route data to them. In practice, the distinction between peer routing and peer +discovery is not always clear-cut, and it's worth noting that in a real-world +implementation, discovery and routing usually happen concurrently. ## Discovery and Routing in libp2p -libp2p is a peer-to-peer networking stack that provides a set of modules for different network-level -functionality, including peer discovery and routing. Peers in a libp2p network can discover other -peers using various mechanisms, such as exchanging peer addresses over the network, querying a directory -service, or using a Distributed Hash Table (DHT) to store and retrieve information about available peers. - -The methods for peer discovery in libp2p are, but are not limited to: - -- [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer addresses in a secure and - private manner -- [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to discover other - peers on the local network. -- [Publish/Subscribe](pubsub.md): a protocol that allows peers to subscribe to specific topics and receive - updates from other peers that are publishing information on those topics. -- [DHT](kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns each piece of - content a unique identifier and stores the content on the peer whose identifier is closest to the - content's identifier. +libp2p provides a set of modules for different network-level functionality, +including peer discovery and routing. Peers in libp2p can discover other +peers using various mechanisms, such as exchanging peer addresses over the +network, querying a directory service, or using a Distributed Hash Table (DHT) +to store and retrieve information about available peers. + +These methods include, but are not limited to: + +- [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer addresses + in a secure and private manner. +- [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to + discover other peers on the local network. +- [Publish/Subscribe](pubsub.md): a protocol that allows peers to subscribe to specific + topics and receive updates from other peers that are publishing information on those + topics. +- [DHT](kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns + each piece of content a unique identifier and stores the content on the peer whose + identifier is closest to the content's identifier. From 374339d445b8d4d9f374f24c97420f5ad9d7f8c4 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 19 Jan 2023 17:25:48 -0500 Subject: [PATCH 05/18] Apply suggestions from code review Co-authored-by: Marco Munizaga --- content/concepts/discovery-routing/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index 22b2237d..1cc60787 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -27,7 +27,7 @@ and "routing by gossip" can be used to find the closest peers to a given peer ID A peer may use a routing algorithm to find the location of a specific peer and then use that information to discover new peers in the vicinity. Additionally, a peer may -parallel use both peer routing and peer discovery mechanisms to find new peers and +in parallel use both peer routing and peer discovery mechanisms to find new peers and route data to them. In practice, the distinction between peer routing and peer discovery is not always clear-cut, and it's worth noting that in a real-world implementation, discovery and routing usually happen concurrently. From f31b1923fe3d8945a48753ca7df292e89a44951d Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 19 Jan 2023 17:33:25 -0500 Subject: [PATCH 06/18] tweaks --- .../concepts/discovery-routing/overview.md | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index 1cc60787..876a37d1 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -10,15 +10,14 @@ Peer discovery and routing are two essential aspects of P2P networking. In a P2P each node must be able to discover and communicate with other nodes without the need for a central server. -### Peer Discovery +### Peer discovery Peer discovery is the process of finding and announcing services to other available -peers in a P2P network. It allows a peer to discover other peers in the network and -learn about their services. Peer discovery can be done using various protocols, such +peers in a P2P network. Peer discovery can be done using various protocols, such as broadcasting a message to all peers in the network or using a bootstrap node to provide a list of known peers. -### Peer Routing +### Peer routing Peer routing, on the other hand, refers to finding a specific peer's location in the network. This is typically done by maintaining a routing table or a similar @@ -27,22 +26,27 @@ and "routing by gossip" can be used to find the closest peers to a given peer ID A peer may use a routing algorithm to find the location of a specific peer and then use that information to discover new peers in the vicinity. Additionally, a peer may -in parallel use both peer routing and peer discovery mechanisms to find new peers and -route data to them. In practice, the distinction between peer routing and peer +use both peer routing and peer discovery mechanisms in parallel to find new peers and +route data to them. + +{{< alert icon="" context="note">}} +In practice, the distinction between peer routing and peer discovery is not always clear-cut, and it's worth noting that in a real-world implementation, discovery and routing usually happen concurrently. +{{< /alert >}} -## Discovery and Routing in libp2p +## Discovery and routing in libp2p libp2p provides a set of modules for different network-level functionality, including peer discovery and routing. Peers in libp2p can discover other -peers using various mechanisms, such as exchanging peer addresses over the -network, querying a directory service, or using a Distributed Hash Table (DHT) +peers using various mechanisms, such as exchanging peer +[multiaddresses](../introduction/core-abstractions/addressing.md) over the +network, querying a directory service, or using a distributed hash table (DHT) to store and retrieve information about available peers. These methods include, but are not limited to: -- [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer addresses +- [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer multiaddresses in a secure and private manner. - [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to discover other peers on the local network. From dca5ba67a21b2050b281d201f5a6d00861c4458a Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 2 Feb 2023 15:11:14 -0500 Subject: [PATCH 07/18] Apply suggestions from code review Co-authored-by: Prithvi Shahi <50885601+p-shahi@users.noreply.github.com> --- content/concepts/discovery-routing/overview.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index 876a37d1..e8601edf 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -50,9 +50,6 @@ These methods include, but are not limited to: in a secure and private manner. - [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to discover other peers on the local network. -- [Publish/Subscribe](pubsub.md): a protocol that allows peers to subscribe to specific - topics and receive updates from other peers that are publishing information on those - topics. - [DHT](kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns each piece of content a unique identifier and stores the content on the peer whose identifier is closest to the content's identifier. From 0c3e6b0e830dab2c311c9f355aea1027a18c715b Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 2 Feb 2023 15:18:20 -0500 Subject: [PATCH 08/18] rmv pubsub --- content/concepts/discovery-routing/pubsub.md | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 content/concepts/discovery-routing/pubsub.md diff --git a/content/concepts/discovery-routing/pubsub.md b/content/concepts/discovery-routing/pubsub.md deleted file mode 100644 index 37d74d84..00000000 --- a/content/concepts/discovery-routing/pubsub.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: "PubSub" -description: "." -weight: 225 ---- - -Coming soon! From ea65642ba3288eb640737096675265fd15608c16 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Mon, 13 Feb 2023 14:10:10 -0500 Subject: [PATCH 09/18] feat: rendezvous (#259) --- .../concepts/discovery-routing/rendezvous.md | 83 ++++++++++++++++++- 1 file changed, 81 insertions(+), 2 deletions(-) diff --git a/content/concepts/discovery-routing/rendezvous.md b/content/concepts/discovery-routing/rendezvous.md index fe3ca878..ed7a75df 100644 --- a/content/concepts/discovery-routing/rendezvous.md +++ b/content/concepts/discovery-routing/rendezvous.md @@ -1,7 +1,86 @@ --- title : "Rendezvous" -description: "The rendezvous protocol can facilitate the routing and discovery of nodes in a peer-to-peer network using a common location." +description: "The rendezvous protocol can facilitate the routing and discovery of nodes in a P2P network using a common location." weight: 223 --- -Coming soon! +## What is Rendezvous? + +A rendezvous protocol is a routing protocol that enables nodes and resources +in a P2P network to discover each other. Rendezvous is used +as a common location (point) to route between two routes. + +Rendezvous points are typically nodes that are well-connected and stable in +a network and can handle large amounts of traffic and data. They +serve as a hub for nodes to discover. + +
+ Rendezvous is not decentralized + + It is important to note that Rendezvous is not decentralized but rather + federated. While this has its use cases, it also introduces a single + point of failure into the network. This can be contrasted with fully decentralized + solutions like DHT and Gossipsub. DHT (Distributed Hash Table) and Gossipsub are + decentralized alternatives to Rendezvous. + + [DHT](kaddht.md) is a distributed network protocol used to store and + retrieve data in a P2P network efficiently. It is like a hash table mapping keys + to values, allowing for fast lookups and efficient data distribution across the network. + + [Gossipsub](pubsub.md), on the other hand, is a pub-sub (publish-subscribe) protocol + that is used to distribute messages and data across a network. It uses a gossip-based + mechanism to propagate messages throughout the network, allowing fast and efficient + distribution without relying on a central control point. + +
+ +## Rendezvous in libp2p + +{{< alert icon="💡" context="info" text="The current rendezvous implementation replaces the initial ws-star-rendezvous implementation with rendezvous daemons and a fleet of p2p-circuit relays." />}} + +The libp2p rendezvous protocol can be used for different use cases. E.g. it can +be used during bootstrap to discover circuit relays that provide connectivity +for browser nodes. Generally, a peer can use known rendezvous points to find +peers that provide network services. Rendezvous is also used throughout the +lifetime of an application for peer discovery by registering and polling +rendezvous points. In an application-specific setting, rendezvous points can be +used to progressively discover peers that can answer specific queries or host +shards of content. + +The libp2p rendezvous protocol allows peers to connect to a rendezvous point and +register their presence by sending a `REGISTER` message in one or more +namespaces. Any node implementing the rendezvous protocol can act as a +rendezvous point, and any peer can connect to a rendezvous point. However, only +peers initiating a registration can register themselves at a rendezvous point. + +By registering with a rendezvous point, peers allow for their discovery by other peers who +query the rendezvous point. The query may: + +- provide namespace(s), such as `test-app`; +- optionally provide a maximum number of peers to return; +- can include a cookie that is obtained from the response to a previous query, + thus the current query only contain registrations that weren't part of the + previous response. + > This simplifies discovery as it reduces the overhead of queried peers and allows for + > the pagination of query responses. + +There is a default peer registration lifetime of 2 hours. Peers can optionally specify the +lifetime using a TTL parameter in the `REGISTER` message, with an upper bound of 72 hours. + +The rendezvous protocol runs over libp2p streams using the protocol ID `/rendezvous/1.0.0`. + + + +### Rendezvous and publish-subscribe + +For effective discovery, rendezvous can be combined with [libp2p publish/subscribe](../messaging/pubsub/overview). +At a basic level, rendezvous can bootstrap pubsub by discovering peers subscribed to a topic. The rendezvous would +be responsible for publishing packets, subscribing, or unsubscribing from packet shapes. + +Pubsub can also be used as a mechanism for building rendezvous services, where a number +of rendezvous points can federate using pubsub for internal distribution while still +providing a simple interface to clients. + + + +{{< alert icon="💡" context="note" text="See the rendezvous technical specification for more details." />}} From 2b0f65bf2b41f3b55031864c21e6040a119aac5c Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Fri, 24 Feb 2023 19:43:51 -0500 Subject: [PATCH 10/18] feat: mdns (#260) --- content/concepts/discovery-routing/mDNS.md | 29 ++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 content/concepts/discovery-routing/mDNS.md diff --git a/content/concepts/discovery-routing/mDNS.md b/content/concepts/discovery-routing/mDNS.md new file mode 100644 index 00000000..3ae73796 --- /dev/null +++ b/content/concepts/discovery-routing/mDNS.md @@ -0,0 +1,29 @@ +--- +title: "mDNS" +description: "mDNS uses a multicast system of DNS records over a local network to enable peer discovery." +weight: 224 +--- + +## What is mDNS? + +mDNS, or multicast Domain Name System, is a way for nodes to use the multicast +system of DNS records [RFC 6762](https://www.rfc-editor.org/rfc/rfc6762) over a +local network to discover and communicate with nodes. Nodes broadcast topics +they're interested in. The discovery, however, is limited to the peers in the +local network. mDNS is commonly used on home networks to allow devices such as +computers, printers, and smart TVs to find each other and connect. It uses a +protocol called multicast to broadcast messages on the network, allowing devices +to discover each other and exchange information. + +## mDNS in libp2p + +In libp2p, mDNS is used for peer discovery, allowing peers to find and +communicate with each other on the same local network without any configuration. +The basic mDNS node discovery flow begins when a peer starts or detects a +network change and then sends a query message to all peers. The node adds the +response it receives from other peers into a local database. + +To learn more about +definitions, specific fields, and peer discovery, [visit the mDNS libp2p +specification](https://github.com/libp2p/specs/blob/master/discovery/mdns.md). + From 4755885d458316e47a4a753582a755a1c80bca87 Mon Sep 17 00:00:00 2001 From: Max Inden Date: Wed, 1 Mar 2023 19:16:12 +0100 Subject: [PATCH 11/18] Rephrase mDNS section --- content/concepts/discovery-routing/mDNS.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/content/concepts/discovery-routing/mDNS.md b/content/concepts/discovery-routing/mDNS.md index 3ae73796..f03ae0a6 100644 --- a/content/concepts/discovery-routing/mDNS.md +++ b/content/concepts/discovery-routing/mDNS.md @@ -6,14 +6,12 @@ weight: 224 ## What is mDNS? -mDNS, or multicast Domain Name System, is a way for nodes to use the multicast -system of DNS records [RFC 6762](https://www.rfc-editor.org/rfc/rfc6762) over a -local network to discover and communicate with nodes. Nodes broadcast topics -they're interested in. The discovery, however, is limited to the peers in the -local network. mDNS is commonly used on home networks to allow devices such as -computers, printers, and smart TVs to find each other and connect. It uses a -protocol called multicast to broadcast messages on the network, allowing devices -to discover each other and exchange information. +mDNS, or multicast Domain Name System, is a way for nodes to use IP multicast to +publish and receive DNS records [RFC +6762](https://www.rfc-editor.org/rfc/rfc6762) within a local network. Nodes +broadcast topics they're interested in. mDNS is commonly used on home networks +to allow devices such as computers, printers, and smart TVs to discover each +other and connect. ## mDNS in libp2p From faf6f5971c2d72e76c3e442076de433ba799fd24 Mon Sep 17 00:00:00 2001 From: Max Inden Date: Wed, 1 Mar 2023 19:19:31 +0100 Subject: [PATCH 12/18] Rephrase mdns in libp2p section --- content/concepts/discovery-routing/mDNS.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/content/concepts/discovery-routing/mDNS.md b/content/concepts/discovery-routing/mDNS.md index f03ae0a6..675f76b8 100644 --- a/content/concepts/discovery-routing/mDNS.md +++ b/content/concepts/discovery-routing/mDNS.md @@ -15,11 +15,10 @@ other and connect. ## mDNS in libp2p -In libp2p, mDNS is used for peer discovery, allowing peers to find and -communicate with each other on the same local network without any configuration. -The basic mDNS node discovery flow begins when a peer starts or detects a -network change and then sends a query message to all peers. The node adds the -response it receives from other peers into a local database. +In libp2p, mDNS is used for peer discovery, allowing peers to find each other on +the same local network without any configuration. In the basic mDNS node +discovery flow a node broadcasts a request which is consecutively replied to by +other nodes within the network with their multiaddresses. To learn more about definitions, specific fields, and peer discovery, [visit the mDNS libp2p From 9bfde993fb5e656bd9fd3d6478731260a176a6e0 Mon Sep 17 00:00:00 2001 From: Prithvi Shahi Date: Wed, 1 Mar 2023 14:45:57 -0800 Subject: [PATCH 13/18] add Kad DHT doc and remove duplicate mdns doc --- content/concepts/discovery-routing/kaddht.md | 73 +++++++++++++++++++- content/concepts/discovery-routing/mdns.md | 7 -- 2 files changed, 70 insertions(+), 10 deletions(-) delete mode 100644 content/concepts/discovery-routing/mdns.md diff --git a/content/concepts/discovery-routing/kaddht.md b/content/concepts/discovery-routing/kaddht.md index 732de995..e99b3923 100644 --- a/content/concepts/discovery-routing/kaddht.md +++ b/content/concepts/discovery-routing/kaddht.md @@ -1,7 +1,74 @@ --- -title: "Kad-DHT" -description: "." +title: "Kademlia DHT" +description: "The libp2p Kad-DHT subsystem is an implementation of the Kademlia +DHT, a distributed hash table." weight: 226 --- -Coming soon! +## Overview + +The Kademlia Distributed Hash Table (DHT), or Kad-DHT, is a distributed hash table +that is designed for P2P networks. + +Kad-DHT in libp2p is a subsystem based on the +[Kademlia whitepaper](https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf). + +Kad-DHT offers a way to find nodes and data on the network by using a +[routing table](https://en.wikipedia.org/wiki/Routing_table) that organizes peers based +on how similar their keys are. + +
+ A deeper look + + The routing table is organized based on a prefix length and a distance metric. + The prefix length helps to group similar keys, and the distance metric helps to + find the closest peers to a specific key in the routing table. The table maintains + a list of `k` closest peers for each possible prefix length between `0` and `L-1`, + where `L` is the length of the keyspace, determined by the length of the hash + function used. **Kad-DHT uses SHA-256**, with a keyspace of 256 bits, trying to maintain + `k` peers with a shared key prefix for every prefix length between `0` and `255` in + its routing table. + + The prefix length measures the proximity of two keys in the routing table and + divides the keyspace into smaller subspaces, called "buckets", each containing nodes + that share a common prefix of bits in their SHA-256 hash. The prefix length is the + number of bits that are the same in the two keys' SHA-256 hash. The more leading bits + that are the same, the longer the prefix length and the closer the proximity of the + two keys are considered to be. + + The distance metric is a way to calculate the distance between two keys by + taking the bitwise exclusive-or (XOR) of the SHA-256 hash of the two keys. The + result is a measure of the distance between the two keys, where a distance of + `0` means the keys are identical, and a distance of `1` means that only one + bit is different, meaning the two keys are close to each other (i.e. their + SHA-256 hashes are similar). + + This design allows for efficient and effective lookups in the routing table when + trying to find nodes or data that share similar prefixes. + +
+ +## Peer routing + +The Kad-DHT uses a process called "peer routing" to discover nodes in the +network. When looking for a peer, the local node contacts the `k` closest nodes to +the remote peer's ID asking them for closer nodes. The local node repeats the +process until it finds the peer or determines that it is not in the network. + +## Content provider routing + +Kad-DHT also includes a feature for content provider discovery, where nodes can +look up providers for a given key. The local node again contacts the `k` closest +nodes to the key asking them for either providers of the key and/or closer nodes +to the key. The local node repeats the process until it finds providers for the +key or determines that it is not in the network. + +## Bootstrap process + +To maintain a healthy routing table and discover new nodes, the Kad-DHT includes +a bootstrap process that runs periodically. The process starts by generating a random peer +ID and looking it up via the peer routing process. The node then adds the closest peers it +discovers to its routing table and repeats the process multiple times. This process also +includes looking up its own peer ID to improve awareness of nodes close to itself. + +{{< alert icon="💡" context="note" text="See the Kademlia DHT technical specification for more details." />}} diff --git a/content/concepts/discovery-routing/mdns.md b/content/concepts/discovery-routing/mdns.md deleted file mode 100644 index 5643dad8..00000000 --- a/content/concepts/discovery-routing/mdns.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: "mDNS" -description: "mDNS uses a multicast system of DNS records over a local network to enable peer discovery." -weight: 224 ---- - -Coming soon! From 9d150327285c4511edcdd77f6ac7aeed046d07e6 Mon Sep 17 00:00:00 2001 From: Prithvi Shahi Date: Wed, 1 Mar 2023 14:51:26 -0800 Subject: [PATCH 14/18] update mdns link in overview --- content/concepts/discovery-routing/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index e8601edf..93c11d9d 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -48,7 +48,7 @@ These methods include, but are not limited to: - [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer multiaddresses in a secure and private manner. -- [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to +- [mDNS](mDNS.md): a multicast Domain Name System (DNS) protocol that allows peers to discover other peers on the local network. - [DHT](kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns each piece of content a unique identifier and stores the content on the peer whose From a2f5e7fbaf9f6da26ccc4a68353e2d77e1a3972b Mon Sep 17 00:00:00 2001 From: Prithvi Shahi Date: Wed, 1 Mar 2023 14:56:11 -0800 Subject: [PATCH 15/18] update mdns link in overview --- content/concepts/discovery-routing/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index 93c11d9d..e8601edf 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -48,7 +48,7 @@ These methods include, but are not limited to: - [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer multiaddresses in a secure and private manner. -- [mDNS](mDNS.md): a multicast Domain Name System (DNS) protocol that allows peers to +- [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to discover other peers on the local network. - [DHT](kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns each piece of content a unique identifier and stores the content on the peer whose From 03c8a71cd1fe8f1d7646c930a2221dd1635fe75a Mon Sep 17 00:00:00 2001 From: Prithvi Shahi Date: Wed, 1 Mar 2023 15:00:05 -0800 Subject: [PATCH 16/18] links --- content/concepts/discovery-routing/overview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index e8601edf..114b3cf5 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -40,16 +40,16 @@ implementation, discovery and routing usually happen concurrently. libp2p provides a set of modules for different network-level functionality, including peer discovery and routing. Peers in libp2p can discover other peers using various mechanisms, such as exchanging peer -[multiaddresses](../introduction/core-abstractions/addressing.md) over the +[multiaddresses](../fundamentals/addressing.md) over the network, querying a directory service, or using a distributed hash table (DHT) to store and retrieve information about available peers. These methods include, but are not limited to: -- [Rendezvous](rendezvous.md): a protocol that allows peers to exchange peer multiaddresses +- [Rendezvous](./rendezvous.md): a protocol that allows peers to exchange peer multiaddresses in a secure and private manner. -- [mDNS](mdns.md): a multicast Domain Name System (DNS) protocol that allows peers to +- [mDNS](./mDNS.md): a multicast Domain Name System (DNS) protocol that allows peers to discover other peers on the local network. -- [DHT](kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns +- [DHT](./kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns each piece of content a unique identifier and stores the content on the peer whose identifier is closest to the content's identifier. From f42d98d2bb87e54c9ac89de836f4aba8d182f4c0 Mon Sep 17 00:00:00 2001 From: Prithvi Shahi Date: Wed, 1 Mar 2023 15:08:12 -0800 Subject: [PATCH 17/18] update --- content/concepts/discovery-routing/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index 114b3cf5..53937a22 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -21,9 +21,9 @@ provide a list of known peers. Peer routing, on the other hand, refers to finding a specific peer's location in the network. This is typically done by maintaining a routing table or a similar -data structure that keeps track of the network topology. Algorithms such as kbucket -and "routing by gossip" can be used to find the closest peers to a given peer ID. +data structure that keeps track of the network topology. +Different algorithms can be used to find the "closest" neighboring peers to a given peer ID. A peer may use a routing algorithm to find the location of a specific peer and then use that information to discover new peers in the vicinity. Additionally, a peer may use both peer routing and peer discovery mechanisms in parallel to find new peers and From 9e25484f4a2f6cb6512229d88ad232092d319883 Mon Sep 17 00:00:00 2001 From: Prithvi Shahi Date: Wed, 1 Mar 2023 15:17:35 -0800 Subject: [PATCH 18/18] fix links --- content/concepts/discovery-routing/overview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/concepts/discovery-routing/overview.md b/content/concepts/discovery-routing/overview.md index 53937a22..29bd0d88 100644 --- a/content/concepts/discovery-routing/overview.md +++ b/content/concepts/discovery-routing/overview.md @@ -40,16 +40,16 @@ implementation, discovery and routing usually happen concurrently. libp2p provides a set of modules for different network-level functionality, including peer discovery and routing. Peers in libp2p can discover other peers using various mechanisms, such as exchanging peer -[multiaddresses](../fundamentals/addressing.md) over the +[multiaddresses](./../../fundamentals/addressing) over the network, querying a directory service, or using a distributed hash table (DHT) to store and retrieve information about available peers. These methods include, but are not limited to: -- [Rendezvous](./rendezvous.md): a protocol that allows peers to exchange peer multiaddresses +- [Rendezvous](./../rendezvous): a protocol that allows peers to exchange peer multiaddresses in a secure and private manner. -- [mDNS](./mDNS.md): a multicast Domain Name System (DNS) protocol that allows peers to +- [mDNS](./../mdns): a multicast Domain Name System (DNS) protocol that allows peers to discover other peers on the local network. -- [DHT](./kaddht.md): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns +- [DHT](./../kaddht): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns each piece of content a unique identifier and stores the content on the peer whose identifier is closest to the content's identifier.