Move envoy.api.v2 core protos to envoy.api.v2.core and organize filters

.circleci/config.yml

@@ -1,16 +1,20 @@
+references:
+  envoy-build-image: &envoy-build-image
+    envoyproxy/envoy-build:220e5cb537b5185c953de1aac7d0613f8cf155ac
+
 version: 2
 jobs:
    test:
      docker:
-       - image: envoyproxy/envoy-build:52f6880ffbf761c9b809fc3ac208900956ff16b4
+       - image: *envoy-build-image
      resource_class: xlarge
      working_directory: /source
      steps:
        - checkout
        - run: ci/do_ci.sh bazel.test
    docs:
      docker:
-       - image: lyft/envoy-build:114e24c6fd05fc026492e9d2ca5608694e5ea59d
+       - image: *envoy-build-image
      resource_class: xlarge
      working_directory: /source
      steps:
@@ -22,7 +26,7 @@ jobs:
            path: generated/docs
    format:
      docker:
-       - image: lyft/envoy-build:114e24c6fd05fc026492e9d2ca5608694e5ea59d
+       - image: *envoy-build-image
      resource_class: xlarge
      working_directory: /source
      steps:

STYLE.md

@@ -92,3 +92,23 @@ In addition, the following conventions should be followed:
   value like `TYPE_NAME_UNSPECIFIED = 0`, and treat it as an error. This design
   pattern forces developers to explicitly choose the correct enum value for
   their use case, and avoid misunderstanding of the default behavior.
+
+## Package organization
+
+API definitions are layered hierarchically in packages from top-to-bottom:
+
+- `envoy.service` contains gRPC definitions of supporting services;
+- `envoy.config` contains definitions for service configuration, filter
+configuration, and bootstrap;
+- `envoy.api.v2` contains definitions for EDS, CDS, RDS, LDS, and top-level
+resources such as `Cluster`;
+- `envoy.api.v2.endpoint`, `envoy.api.v2.cluster`, `envoy.api.v2.route`,
+`envoy.api.v2.listener`, `envoy.api.v2.ratelimit` define sub-messages of the top-level resources;
+- `envoy.api.v2.core` and `envoy.api.v2.auth` hold core definitions consumed
+throughout the API.
+
+Dependencies are enforced from top-to-bottom using visibility constraints in
+the build system to prevent circular dependency formation. Package group
+`//envoy/api/v2:friends` selects consumers of the core API package (services and configs)
+and is the default visibility for the core API packages. The default visibility
+for services and configs should be `//docs` (proto documentation tool).

bazel/api_build_system.bzl

@@ -77,13 +77,19 @@ def api_go_grpc_library(name, proto, deps = []):
 # gRPC stub generation.
 # TODO(htuch): Automatically generate go_proto_library and go_grpc_library
 # from api_proto_library.
-def api_proto_library(name, srcs = [], deps = [], has_services = 0, require_py = 1):
+def api_proto_library(name, visibility = ["//visibility:private"], srcs = [], deps = [], has_services = 0, require_py = 1):
     # This is now vestigial, since there are no direct consumers in
     # data-plane-api. However, we want to maintain native proto_library support
     # in the proto graph to (1) support future C++ use of native rules with
     # cc_proto_library (or some Bazel aspect that works on proto_library) when
     # it can play well with the PGV plugin and (2) other language support that
     # can make use of native proto_library.
+
+    if visibility == ["//visibility:private"]:
+      visibility = ["//docs"]
+    elif visibility != ["//visibility:public"]:
+      visibility = visibility + ["//docs"]
+
     native.proto_library(
         name = name,
         srcs = srcs,
@@ -99,7 +105,7 @@ def api_proto_library(name, srcs = [], deps = [], has_services = 0, require_py =
             "@com_github_gogo_protobuf//:gogo_proto",
             "@com_lyft_protoc_gen_validate//validate:validate_proto",
         ],
-        visibility = ["//visibility:public"],
+        visibility = visibility,
     )
     # Under the hood, this is just an extension of the Protobuf library's
     # bespoke cc_proto_library. It doesn't consume proto_library as a proto

bazel/repositories.bzl

@@ -198,6 +198,7 @@ api_proto_library(
     srcs = [
         "metrics.proto",
     ],
+    visibility = ["//visibility:public"],
 )
         """,
     )
@@ -214,6 +215,7 @@ api_proto_library(
     srcs = [
         "trace.proto",
     ],
+    visibility = ["//visibility:public"],
 )
         """,
     )

docs/BUILD

@@ -0,0 +1,53 @@
+licenses(["notice"])  # Apache 2
+
+package_group(
+    name = "docs",
+    packages = [
+        "//docs",
+    ],
+)
+
+# TODO(htuch): Grow this to cover everything we want to generate docs for, so we can just invoke
+# bazel build //docs:protos --aspects tools/protodoc/protodoc.bzl%proto_doc_aspect  --output_groups=rst
+proto_library(
+    name = "protos",
+    deps = [
+        "//envoy/api/v2:cds",
+        "//envoy/api/v2:discovery",
+        "//envoy/api/v2:eds",
+        "//envoy/api/v2:lds",
+        "//envoy/api/v2:rds",
+        "//envoy/api/v2/cluster:circuit_breaker",
+        "//envoy/api/v2/cluster:outlier_detection",
+        "//envoy/api/v2/core:protocol",
+        "//envoy/api/v2/listener",
+        "//envoy/api/v2/ratelimit",
+        "//envoy/api/v2/route",
+        "//envoy/config/bootstrap/v2:bootstrap",
+        "//envoy/config/filter/accesslog/v2:accesslog",
+        "//envoy/config/filter/http/buffer/v2:buffer",
+        "//envoy/config/filter/http/ext_authz/v2:ext_authz",
+        "//envoy/config/filter/http/fault/v2:fault",
+        "//envoy/config/filter/http/gzip/v2:gzip",
+        "//envoy/config/filter/http/health_check/v2:health_check",
+        "//envoy/config/filter/http/lua/v2:lua",
+        "//envoy/config/filter/http/rate_limit/v2:rate_limit",
+        "//envoy/config/filter/http/router/v2:router",
+        "//envoy/config/filter/http/squash/v2:squash",
+        "//envoy/config/filter/http/transcoder/v2:transcoder",
+        "//envoy/config/filter/network/client_ssl_auth/v2:client_ssl_auth",
+        "//envoy/config/filter/network/ext_authz/v2:ext_authz",
+        "//envoy/config/filter/network/http_connection_manager/v2:http_connection_manager",
+        "//envoy/config/filter/network/mongo_proxy/v2:mongo_proxy",
+        "//envoy/config/filter/network/rate_limit/v2:rate_limit",
+        "//envoy/config/filter/network/redis_proxy/v2:redis_proxy",
+        "//envoy/config/filter/network/tcp_proxy/v2:tcp_proxy",
+        "//envoy/config/metrics/v2:metrics_service",
+        "//envoy/config/metrics/v2:stats",
+        "//envoy/config/ratelimit/v2:rls",
+        "//envoy/config/trace/v2:trace",
+        "//envoy/service/discovery/v2:ads",
+        "//envoy/service/load_stats/v2:lrs",
+        "//envoy/service/metrics/v2:metrics_service",
+    ],
+)

docs/build.sh

@@ -20,14 +20,18 @@ fi
 
 source "${BUILD_DIR}"/venv/bin/activate
 
-bazel --batch build ${BAZEL_BUILD_OPTIONS} //envoy --aspects \
+bazel --batch build ${BAZEL_BUILD_OPTIONS} //docs:protos --aspects \
   tools/protodoc/protodoc.bzl%proto_doc_aspect --output_groups=rst --action_env=CPROFILE_ENABLED
 
 # These are the protos we want to put in docs, this list will grow.
 # TODO(htuch): Factor this out of this script.
 PROTO_RST="
-  /envoy/api/v2/address/envoy/api/v2/address.proto.rst
-  /envoy/api/v2/base/envoy/api/v2/base.proto.rst
+  /envoy/api/v2/core/address/envoy/api/v2/core/address.proto.rst
+  /envoy/api/v2/core/base/envoy/api/v2/core/base.proto.rst
+  /envoy/api/v2/core/config_source/envoy/api/v2/core/config_source.proto.rst
+  /envoy/api/v2/core/grpc_service/envoy/api/v2/core/grpc_service.proto.rst
+  /envoy/api/v2/core/health_check/envoy/api/v2/core/health_check.proto.rst
+  /envoy/api/v2/core/protocol/envoy/api/v2/core/protocol.proto.rst
   /envoy/api/v2/auth/cert/envoy/api/v2/auth/cert.proto.rst
   /envoy/api/v2/eds/envoy/api/v2/eds.proto.rst
   /envoy/api/v2/endpoint/endpoint/envoy/api/v2/endpoint/endpoint.proto.rst
@@ -38,34 +42,30 @@ PROTO_RST="
   /envoy/api/v2/route/route/envoy/api/v2/route/route.proto.rst
   /envoy/api/v2/lds/envoy/api/v2/lds.proto.rst
   /envoy/api/v2/listener/listener/envoy/api/v2/listener/listener.proto.rst
-  /envoy/api/v2/config_source/envoy/api/v2/config_source.proto.rst
-  /envoy/api/v2/grpc_service/envoy/api/v2/grpc_service.proto.rst
-  /envoy/api/v2/health_check/envoy/api/v2/health_check.proto.rst
-  /envoy/api/v2/protocol/envoy/api/v2/protocol.proto.rst
   /envoy/api/v2/ratelimit/ratelimit/envoy/api/v2/ratelimit/ratelimit.proto.rst
   /envoy/config/bootstrap/v2/bootstrap/envoy/config/bootstrap/v2/bootstrap.proto.rst
   /envoy/api/v2/discovery/envoy/api/v2/discovery.proto.rst
   /envoy/config/ratelimit/v2/rls/envoy/config/ratelimit/v2/rls.proto.rst
   /envoy/config/metrics/v2/metrics_service/envoy/config/metrics/v2/metrics_service.proto.rst
   /envoy/config/metrics/v2/stats/envoy/config/metrics/v2/stats.proto.rst
   /envoy/config/trace/v2/trace/envoy/config/trace/v2/trace.proto.rst
-  /envoy/api/v2/filter/accesslog/accesslog/envoy/api/v2/filter/accesslog/accesslog.proto.rst
-  /envoy/api/v2/filter/fault/envoy/api/v2/filter/fault.proto.rst
-  /envoy/api/v2/filter/http/buffer/envoy/api/v2/filter/http/buffer.proto.rst
-  /envoy/api/v2/filter/http/fault/envoy/api/v2/filter/http/fault.proto.rst
-  /envoy/api/v2/filter/http/gzip/envoy/api/v2/filter/http/gzip.proto.rst
-  /envoy/api/v2/filter/http/health_check/envoy/api/v2/filter/http/health_check.proto.rst
-  /envoy/api/v2/filter/http/lua/envoy/api/v2/filter/http/lua.proto.rst
-  /envoy/api/v2/filter/http/rate_limit/envoy/api/v2/filter/http/rate_limit.proto.rst
-  /envoy/api/v2/filter/http/router/envoy/api/v2/filter/http/router.proto.rst
-  /envoy/api/v2/filter/http/squash/envoy/api/v2/filter/http/squash.proto.rst
-  /envoy/api/v2/filter/http/transcoder/envoy/api/v2/filter/http/transcoder.proto.rst
-  /envoy/api/v2/filter/network/client_ssl_auth/envoy/api/v2/filter/network/client_ssl_auth.proto.rst
-  /envoy/api/v2/filter/network/http_connection_manager/envoy/api/v2/filter/network/http_connection_manager.proto.rst
-  /envoy/api/v2/filter/network/mongo_proxy/envoy/api/v2/filter/network/mongo_proxy.proto.rst
-  /envoy/api/v2/filter/network/rate_limit/envoy/api/v2/filter/network/rate_limit.proto.rst
-  /envoy/api/v2/filter/network/redis_proxy/envoy/api/v2/filter/network/redis_proxy.proto.rst
-  /envoy/api/v2/filter/network/tcp_proxy/envoy/api/v2/filter/network/tcp_proxy.proto.rst
+  /envoy/config/filter/accesslog/v2/accesslog/envoy/config/filter/accesslog/v2/accesslog.proto.rst
+  /envoy/config/filter/fault/v2/fault/envoy/config/filter/fault/v2/fault.proto.rst
+  /envoy/config/filter/http/buffer/v2/buffer/envoy/config/filter/http/buffer/v2/buffer.proto.rst
+  /envoy/config/filter/http/fault/v2/fault/envoy/config/filter/http/fault/v2/fault.proto.rst
+  /envoy/config/filter/http/gzip/v2/gzip/envoy/config/filter/http/gzip/v2/gzip.proto.rst
+  /envoy/config/filter/http/health_check/v2/health_check/envoy/config/filter/http/health_check/v2/health_check.proto.rst
+  /envoy/config/filter/http/lua/v2/lua/envoy/config/filter/http/lua/v2/lua.proto.rst
+  /envoy/config/filter/http/rate_limit/v2/rate_limit/envoy/config/filter/http/rate_limit/v2/rate_limit.proto.rst
+  /envoy/config/filter/http/router/v2/router/envoy/config/filter/http/router/v2/router.proto.rst
+  /envoy/config/filter/http/squash/v2/squash/envoy/config/filter/http/squash/v2/squash.proto.rst
+  /envoy/config/filter/http/transcoder/v2/transcoder/envoy/config/filter/http/transcoder/v2/transcoder.proto.rst
+  /envoy/config/filter/network/client_ssl_auth/v2/client_ssl_auth/envoy/config/filter/network/client_ssl_auth/v2/client_ssl_auth.proto.rst
+  /envoy/config/filter/network/http_connection_manager/v2/http_connection_manager/envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto.rst
+  /envoy/config/filter/network/mongo_proxy/v2/mongo_proxy/envoy/config/filter/network/mongo_proxy/v2/mongo_proxy.proto.rst
+  /envoy/config/filter/network/rate_limit/v2/rate_limit/envoy/config/filter/network/rate_limit/v2/rate_limit.proto.rst
+  /envoy/config/filter/network/redis_proxy/v2/redis_proxy/envoy/config/filter/network/redis_proxy/v2/redis_proxy.proto.rst
+  /envoy/config/filter/network/tcp_proxy/v2/tcp_proxy/envoy/config/filter/network/tcp_proxy/v2/tcp_proxy.proto.rst
 "
 
 # Dump all the generated RST so they can be added to PROTO_RST easily.

docs/root/api-v2/api.rst

@@ -8,26 +8,26 @@ v2 API reference
   :maxdepth: 2
 
   config/bootstrap/v2/bootstrap.proto
-  api/v2/config_source.proto
-  api/v2/grpc_service.proto
+  api/v2/core/config_source.proto
+  api/v2/core/grpc_service.proto
   api/v2/lds.proto
   api/v2/listener/listener.proto
   api/v2/cds.proto
   api/v2/cluster/outlier_detection.proto
   api/v2/cluster/circuit_breaker.proto
   api/v2/endpoint/endpoint.proto
   api/v2/eds.proto
-  api/v2/health_check.proto
+  api/v2/core/health_check.proto
   api/v2/route/route.proto
   api/v2/rds.proto
   api/v2/auth/cert.proto
   config/metrics/v2/stats.proto
   config/metrics/v2/metrics_service.proto
   config/trace/v2/trace.proto
-  api/v2/base.proto
-  api/v2/address.proto
-  api/v2/protocol.proto
+  api/v2/core/base.proto
+  api/v2/core/address.proto
+  api/v2/core/protocol.proto
   api/v2/discovery.proto
   api/v2/ratelimit/ratelimit.proto
   config/ratelimit/v2/rls.proto
-  api/v2/filter/filter
+  config/filter/filter

docs/root/api-v2/api/v2/filter/filter.rst → docs/root/api-v2/config/filter/filter.rst

@@ -7,5 +7,5 @@ Filters
 
   network/network
   http/http
-  accesslog/accesslog.proto
-  fault.proto
+  accesslog/v2/accesslog.proto
+  fault/v2/fault.proto

docs/root/api-v2/api/v2/filter/http/http.rst → docs/root/api-v2/config/filter/http/http.rst

@@ -5,4 +5,4 @@ HTTP filters
   :glob:
   :maxdepth: 2
 
-  *
+  */v2/*

docs/root/api-v2/api/v2/filter/network/network.rst → docs/root/api-v2/config/filter/network/network.rst

@@ -5,4 +5,4 @@ Network filters
   :glob:
   :maxdepth: 2
 
-  *
+  */v2/*

docs/root/configuration/access_log.rst

@@ -10,7 +10,7 @@ Access logs are configured as part of the :ref:`HTTP connection manager config
 <config_http_conn_man>` or :ref:`TCP Proxy <config_network_filters_tcp_proxy>`.
 
 * :ref:`v1 API reference <config_access_log_v1>`
-* :ref:`v2 API reference <envoy_api_msg_filter.accesslog.AccessLog>`
+* :ref:`v2 API reference <envoy_api_msg_config.filter.accesslog.v2.AccessLog>`
 
 .. _config_access_log_format:
 

docs/root/configuration/cluster_manager/cluster_hc.rst

@@ -7,7 +7,7 @@ Health checking
 * If health checking is configured for a cluster, additional statistics are emitted. They are
   documented :ref:`here <config_cluster_manager_cluster_stats>`.
 * :ref:`v1 API documentation <config_cluster_manager_cluster_hc_v1>`.
-* :ref:`v2 API documentation <envoy_api_msg_HealthCheck>`.
+* :ref:`v2 API documentation <envoy_api_msg_core.HealthCheck>`.
 
 .. _config_cluster_manager_cluster_hc_tcp_health_checking:
 

docs/root/configuration/http_conn_man/headers.rst

@@ -205,7 +205,7 @@ following features are available:
 
 * Stable :ref:`access logging <config_access_log>` via the
   :ref:`v1 API runtime filter<config_http_con_manager_access_log_filters_runtime_v1>` or the
-  :ref:`v2 API runtime filter<envoy_api_field_filter.accesslog.AccessLogFilter.runtime_filter>`.
+  :ref:`v2 API runtime filter<envoy_api_field_config.filter.accesslog.v2.AccessLogFilter.runtime_filter>`.
 * Stable tracing when performing random sampling via the :ref:`tracing.random_sampling
   <config_http_conn_man_runtime_random_sampling>` runtime setting or via forced tracing using the
   :ref:`config_http_conn_man_headers_x-envoy-force-trace` and

docs/root/configuration/http_conn_man/http_conn_man.rst

@@ -6,7 +6,7 @@ HTTP connection manager
 * HTTP connection manager :ref:`architecture overview <arch_overview_http_conn_man>`
 * HTTP protocols :ref:`architecture overview <arch_overview_http_protocols>`
 * :ref:`v1 API reference <config_network_filters_http_conn_man_v1>`
-* :ref:`v2 API reference <envoy_api_msg_filter.network.HttpConnectionManager>`
+* :ref:`v2 API reference <envoy_api_msg_config.filter.network.http_connection_manager.v2.HttpConnectionManager>`
 
 .. toctree::
   :hidden:

docs/root/configuration/http_filters/buffer_filter.rst

@@ -8,7 +8,7 @@ This is useful in different situations including protecting some applications fr
 with partial requests and high network latency.
 
 * :ref:`v1 API reference <config_http_filters_buffer_v1>`
-* :ref:`v2 API reference <envoy_api_msg_filter.http.Buffer>`
+* :ref:`v2 API reference <envoy_api_msg_config.filter.http.buffer.v2.Buffer>`
 
 Statistics
 ----------

docs/root/configuration/http_filters/cors_filter.rst

@@ -9,4 +9,4 @@ For the meaning of the headers please refer to the pages below.
 - https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
 - https://www.w3.org/TR/cors/
 - :ref:`v1 API reference <config_http_filters_cors_v1>`
-- :ref:`v2 API reference <envoy_api_field_filter.network.HttpFilter.name>`
+- :ref:`v2 API reference <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpFilter.name>`

docs/root/configuration/http_filters/dynamodb_filter.rst

@@ -5,7 +5,7 @@ DynamoDB
 
 * DynamoDB :ref:`architecture overview <arch_overview_dynamo>`
 * :ref:`v1 API reference <config_http_filters_dynamo_v1>`
-* :ref:`v2 API reference <envoy_api_field_filter.network.HttpFilter.name>`
+* :ref:`v2 API reference <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpFilter.name>`
 
 Statistics
 ----------

docs/root/configuration/http_filters/fault_filter.rst

@@ -34,7 +34,7 @@ Configuration
   including the router filter.
 
 * :ref:`v1 API reference <config_http_filters_fault_injection_v1>`
-* :ref:`v2 API reference <envoy_api_msg_filter.http.HTTPFault>`
+* :ref:`v2 API reference <envoy_api_msg_config.filter.http.fault.v2.HTTPFault>`
 
 Runtime
 -------

docs/root/configuration/http_filters/grpc_http1_bridge_filter.rst

@@ -5,7 +5,7 @@ gRPC HTTP/1.1 bridge
 
 * gRPC :ref:`architecture overview <arch_overview_grpc>`
 * :ref:`v1 API reference <config_http_filters_grpc_bridge_v1>`
-* :ref:`v2 API reference <envoy_api_field_filter.network.HttpFilter.name>`
+* :ref:`v2 API reference <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpFilter.name>`
 
 This is a simple filter which enables the bridging of an HTTP/1.1 client which does not support
 response trailers to a compliant gRPC server. It works by doing the following:

docs/root/configuration/http_filters/grpc_json_transcoder_filter.rst

@@ -5,7 +5,7 @@ gRPC-JSON transcoder
 
 * gRPC :ref:`architecture overview <arch_overview_grpc>`
 * :ref:`v1 API reference <config_http_filters_grpc_json_transcoder_v1>`
-* :ref:`v2 API reference <envoy_api_msg_filter.http.GrpcJsonTranscoder>`
+* :ref:`v2 API reference <envoy_api_msg_config.filter.http.transcoder.v2.GrpcJsonTranscoder>`
 
 This is a filter which allows a RESTful JSON API client to send requests to Envoy over HTTP
 and get proxied to a gRPC service. The HTTP mapping for the gRPC service has to be defined by

docs/root/configuration/http_filters/grpc_web_filter.rst

@@ -5,7 +5,7 @@ gRPC-Web
 
 * gRPC :ref:`architecture overview <arch_overview_grpc>`
 * :ref:`v1 API reference <config_http_filters_grpc_web_v1>`
-* :ref:`v2 API reference <envoy_api_field_filter.network.HttpFilter.name>`
+* :ref:`v2 API reference <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpFilter.name>`
 
 This is a filter which enables the bridging of a gRPC-Web client to a compliant gRPC server by
 following https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-WEB.md.

docs/root/configuration/http_filters/gzip_filter.rst

@@ -9,7 +9,7 @@ compromising the response time.
 
 Configuration
 -------------
-* :ref:`v2 API reference <envoy_api_msg_filter.http.Gzip>`
+* :ref:`v2 API reference <envoy_api_msg_config.filter.http.gzip.v2.Gzip>`
 
 .. attention::
 

docs/root/configuration/http_filters/health_check_filter.rst

@@ -5,7 +5,7 @@ Health check
 
 * Health check filter :ref:`architecture overview <arch_overview_health_checking_filter>`
 * :ref:`v1 API reference <config_http_filters_health_check_v1>`
-* :ref:`v2 API reference <envoy_api_msg_filter.http.HealthCheck>`
+* :ref:`v2 API reference <envoy_api_msg_config.filter.http.health_check.v2.HealthCheck>`
 
 .. note::