Building the sphinx docs fails with the latest sphinx version

[bneradt]

With the latest Sphinx version (mine was 4.0.2), make html from the docs directory fails with this error:

# Sphinx version: 4.0.2
# Python version: 3.8.3 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 3.0.1
# Last messages:
#   writing output... [  2%] admin-guide/configuration/transparent-proxy/wccp-service-config.en
#   writing output... [  2%] admin-guide/configuring-traffic-server.en
#   writing output... [  2%] admin-guide/files/cache.config.en
#   writing output... [  3%] admin-guide/files/hosting.config.en
#   writing output... [  3%] admin-guide/files/index.en
#   writing output... [  3%] admin-guide/files/ip_allow.yaml.en
#   writing output... [  3%] admin-guide/files/logging.yaml.en
#   writing output... [  3%] admin-guide/files/parent.config.en
#   writing output... [  4%] admin-guide/files/plugin.config.en
#   writing output... [  4%] admin-guide/files/records.config.en
# Loaded extensions:
#   sphinx.ext.mathjax (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (2.0.0) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.5) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/alabaster/__init__.py
#   sphinx.ext.graphviz (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/graphviz.py
#   sphinx.ext.intersphinx (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.autodoc.preserve_defaults (1.0) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/autodoc/preserve_defaults.py
#   sphinx.ext.autodoc.type_comment (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.todo (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/todo.py
#   sphinx.ext.coverage (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.viewcode (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/viewcode.py
#   sphinxcontrib.plantuml (unknown version) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinxcontrib/plantuml.py
#   traffic-server (unknown version) from /var/tmp/trafficserver/doc/ext/traffic-server.py
#   sphinx.ext.imgmath (4.0.2) from /root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/ext/imgmath.py
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/application.py", line 350, in build
    self.builder.build_update()
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 292, in build_update
    self.build(to_build,
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 356, in build
    self.write(docnames, list(updated_docnames), method)
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 530, in write
    self._write_serial(sorted(docnames))
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 537, in _write_serial
    doctree = self.env.get_and_resolve_doctree(docname, self)
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 527, in get_and_resolve_doctree
    self.apply_post_transforms(doctree, docname)
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 573, in apply_post_transforms
    transformer.apply_transforms()
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/transforms/__init__.py", line 86, in apply_transforms
    super().apply_transforms()
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/docutils/transforms/__init__.py", line 171, in apply_transforms
    transform.apply(**kwargs)
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/transforms/post_transforms/__init__.py", line 42, in apply
    self.run(**kwargs)
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/sphinx/transforms/post_transforms/__init__.py", line 252, in run
    node['classes'].append(node.parent['domain'])
  File "/root/.local/share/virtualenvs/doc-2zq9zEzg/lib/python3.8/site-packages/docutils/nodes.py", line 625, in __getitem__
    return self.attributes[key]
KeyError: 'domain'

This can be reproduced via our CI's centos:8 docker image like so:

docker pull controller.trafficserver.org/ats/centos:8
docker run --name build_ats_docs --init --cap-add=SYS_PTRACE --network=host -d -t controller.trafficserver.org/ats/centos:8
docker exec -it build_ats_docs /bin/bash
cd /var/tmp
git clone https://github.com/apache/trafficserver.git
cd trafficserver
source /opt/rh/gcc-toolset-9/enable
yum install -y java
cd doc

#-------------------------------------------------------
# Note, at the present time, this will install Sphinx 4.0.2:
#-------------------------------------------------------
pipenv install


pipenv shell
cd ..
autoreconf -fi
./configure --enable-docs
cd doc
make html

[bneradt]

@SolidWallOfCode shared his Python virtualenv with me in which he is able to successfully build. I've verified that pinning the sphinx package to 2.3.1 instead of the latest 4.0.2 causes the build to succeed:

(doc) [root@docker-desktop doc]# git diff
diff --git a/doc/Pipfile b/doc/Pipfile
index 78be762b7..b342fa4b7 100644
--- a/doc/Pipfile
+++ b/doc/Pipfile
@@ -22,7 +22,7 @@ verify_ssl = true
 [dev-packages]
 
 [packages]
-sphinx = "*"
+sphinx = "==2.3.1"
 sphinx-rtd-theme = "*"
 sphinxcontrib-plantuml = "*"
 # i18n
(doc) [root@docker-desktop doc]# 

[bneradt]

I've verified that the latest 3.x sphinx release (3.5.4) from two months ago builds our docs fine:

(doc) [root@docker-desktop doc]# git diff
diff --git a/doc/Pipfile b/doc/Pipfile
index 78be762b7..50e2208a2 100644
--- a/doc/Pipfile
+++ b/doc/Pipfile
@@ -22,7 +22,7 @@ verify_ssl = true
 [dev-packages]
 
 [packages]
-sphinx = "*"
+sphinx = "==3.5.4"
 sphinx-rtd-theme = "*"
 sphinxcontrib-plantuml = "*"
 # i18n
(doc) [root@docker-desktop doc]# 

Thus this issue is only experience for those building with the latest 4.x release (their master branch). To unblock people building the docs I'll create a PR with that pin and in the meantime investigate whether this is an issue with our docs not working well with sphinx 4.x or a bug with sphinx itself.

[bneradt]

I filed an issue dowht doc/sphinx about this build failure. We'll see what they say:
sphinx-doc/sphinx#9322