From 82c6d72e2f859e65a0aa4d1d9b596328629e73ff Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Mon, 18 Dec 2017 14:11:42 -0800 Subject: [PATCH 1/3] Document network diagnostic tool --- _data/toc.yaml | 2 + engine/userguide/networking/network-debug.md | 212 +++++++++++++++++++ 2 files changed, 214 insertions(+) create mode 100644 engine/userguide/networking/network-debug.md diff --git a/_data/toc.yaml b/_data/toc.yaml index 8d9dfb3533a1..c22ad4c8889e 100644 --- a/_data/toc.yaml +++ b/_data/toc.yaml @@ -222,6 +222,8 @@ guides: title: Get started with macvlan network driver - path: /engine/userguide/networking/overlay-security-model/ title: Swarm mode overlay network security model + - path: /engine/userguide/networking/network-debug/ + title: Debug overlay or swarm networking issues - path: /engine/userguide/networking/configure-dns/ title: Configure container DNS in user-defined networks - sectiontitle: Default bridge network diff --git a/engine/userguide/networking/network-debug.md b/engine/userguide/networking/network-debug.md new file mode 100644 index 000000000000..c3a426c4d6ee --- /dev/null +++ b/engine/userguide/networking/network-debug.md @@ -0,0 +1,212 @@ +--- +description: Learn to use the built-in network debugger to debug overlay networking problems +keywords: network, troubleshooting, debug +title: Debug overlay or swarm networking issues +--- + +Docker CE 17.12 and higher introduce a network debugging tool designed to help +debug issues with overlay networks and swarm services running on Linux hosts. +When enabled, a network diagnostic server listens on the specified port and +provides diagnostic information. The network debugging tool should only be +started to debug specific issues, and should not be left running all the time. + +Information about networks is stored in a database, which can be examined using +the API. The database consists of one table per network. + +The Docker API exposes endpoints to query and control the network debugging +tool. CLI integration is provided as a preview, but the implementation is not +yet considered stable and commands and options may change without notice. + +## Enable the diagnostic tool + +The tool currently only works on Docker hosts running on Linux. Repeat these +steps for each node participating in the swarm. + +1. Set the `network-diagnostic-port` to a port which is free on the Docker + host, in the `/etc/docker/daemon.json` configuration file. + + ```json + “network-diagnostic-port”: + ``` + +2. Get the process ID (PID) of the `dockerd` process. It is the second field in + the output, and is typically a number from 2 to 6 digits long. + + ```bash + $ ps aux |grep dockerd | grep -v grep + ``` + +3. Reload the Docker configuration without restarting Docker, by sending the + `HUP` signal to the PID you found in the previous step. + + ```bash + kill -HUP + ``` + +A message like the following will appear in the Docker host logs: + +```none +Starting the diagnose server listening on for commands +``` + +## Disable the diagnostic tool + +Repeat these steps for each node participating in the swarm. + +1. Remove the `network-diagnostic-port` key from the `/etc/docker/daemon.json` + configuration file. + +2. Get the process ID (PID) of the `dockerd` process. It is the second field in + the output, and is typically a number from 2 to 6 digits long. + + ```bash + $ ps aux |grep dockerd | grep -v grep + ``` + +3. Reload the Docker configuration without restarting Docker, by sending the + `HUP` signal to the PID you found in the previous step. + + ```bash + kill -HUP + ``` + +A message like the following will appear in the Docker host logs: + +```none +Disabling the diagnose server +``` + +## Access the diagnostic tool's API + +The network diagnostic tool exposes its own RESTful API. To access the API, +send a HTTP request to the port where the tool is listening. The following +commands assume the tool is listening on port 2000. + +Examples are not given for every endpoint. + +### Get help + +```bash +$ curl localhost:2000/help + +OK +/updateentry +/getentry +/gettable +/leavenetwork +/createentry +/help +/clusterpeers +/ready +/joinnetwork +/deleteentry +/networkpeers +/ +/join +``` + +### Join or leave the swarm + +```bash +$ curl localhost:2000/join?members=ip1,ip2,... +``` + +```bash +$ curl localhost:2000/leave?members=ip1,ip2,... +``` + +### Join or leave a network + +```bash +$ curl localhost:2000/joinnetwork?nid= +``` + +```bash +$ curl localhost:2000/leavenetwork?nid= +``` + +### List cluster peers + +```bash +$ curl localhost:2000/clusterpeers +``` + +### List nodes connected to a given network + +```bash +$ curl localhost:2000/networkpeers?nid= +``` + +### Dump database tables + +```bash +$ curl localhost:2000/gettable?nid=&tname= +``` + +### Interact with a specific database table + +```bash +$ curl localhost:2000/?nid=&tname=
&key=[&value=] +``` + +## Access the diagnostic tool's CLI + +The CLI is provided as a preview and is not yet stable. Commands or options may +change at any time. + +The CLI executable is called `debugClient` and is made available using a +standalone container. + +The following flags are supported: + +| Flag | Description | +|---------------|-------------------------------------------------| +| -c | Command to run. One of `sd` or `overlay`. | +| -ip | The IP address to query. Defaults to 127.0.0.1. | +| -net | The target network ID. | +| -port | The target port. | +| -v | Enable verbose output. | + +### Access the CLI + +The CLI is provided as a container that needs to run using privileged mode. + +1. To run the container, use a command like the following: + + ```bash + $ docker container run --name net-diagnostic -d --privileged --network host fcrisciani/network-diagnostic + ``` + +2. Connect to the container using `docker container attach `, + and start the server using the following command: + + ```bash + $ kill -HUP 1 + ``` + +3. If you have not already done so, join the Docker host to the swarm, then + run the diagnostic CLI within the container. + + ```bash + $ ./diagnosticClient ... + ``` + +4. When finished debugging, stop the container. + +### Examples + +#### Dump the Service discovery table and verify the node ownership + +**Standalone network:** + +```bash +$ debugClient -c sd -v -net n8a8ie6tb3wr2e260vxj8ncy4 +``` + +**Overlay network:** + +```bash +$ debugClient -port 2001 -c overlay -v -net n8a8ie6tb3wr2e260vxj8ncy4 +``` + + From 3af18970321b6a1a59e41e13427f2bd196493b4d Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Mon, 18 Dec 2017 15:42:23 -0800 Subject: [PATCH 2/3] Address feedback --- engine/userguide/networking/network-debug.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/engine/userguide/networking/network-debug.md b/engine/userguide/networking/network-debug.md index c3a426c4d6ee..54b84e55eeec 100644 --- a/engine/userguide/networking/network-debug.md +++ b/engine/userguide/networking/network-debug.md @@ -11,7 +11,7 @@ provides diagnostic information. The network debugging tool should only be started to debug specific issues, and should not be left running all the time. Information about networks is stored in a database, which can be examined using -the API. The database consists of one table per network. +the API. The Docker API exposes endpoints to query and control the network debugging tool. CLI integration is provided as a preview, but the implementation is not @@ -105,7 +105,7 @@ OK /join ``` -### Join or leave the swarm +### Join or leave the network database cluster ```bash $ curl localhost:2000/join?members=ip1,ip2,... @@ -154,7 +154,7 @@ $ curl localhost:2000/?nid=&tname=
&key=[&va The CLI is provided as a preview and is not yet stable. Commands or options may change at any time. -The CLI executable is called `debugClient` and is made available using a +The CLI executable is called `diagnosticClient` and is made available using a standalone container. The following flags are supported: From af416952c29c64a08eabe07d3160fc2146f0c6ce Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Mon, 18 Dec 2017 16:39:50 -0800 Subject: [PATCH 3/3] More feedback --- engine/userguide/networking/network-debug.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/engine/userguide/networking/network-debug.md b/engine/userguide/networking/network-debug.md index 54b84e55eeec..dc720ccff1dc 100644 --- a/engine/userguide/networking/network-debug.md +++ b/engine/userguide/networking/network-debug.md @@ -139,12 +139,18 @@ $ curl localhost:2000/networkpeers?nid= ### Dump database tables +The tables are called `endpoint_table` and `overlay_peer_table`. These names may +change. + ```bash $ curl localhost:2000/gettable?nid=&tname=
``` ### Interact with a specific database table +The tables are called `endpoint_table` and `overlay_peer_table`. These names may +change. + ```bash $ curl localhost:2000/?nid=&tname=
&key=[&value=] ``` @@ -195,7 +201,8 @@ The CLI is provided as a container that needs to run using privileged mode. ### Examples -#### Dump the Service discovery table and verify the node ownership +The following commands dump the service discovery table and verify node +ownership. **Standalone network:**