From 46b7e95390716cab4a3a71cb4351d4faa09f41ef Mon Sep 17 00:00:00 2001 From: Doug Davis Date: Wed, 26 Jun 2019 19:42:22 -0700 Subject: [PATCH 1/3] Clean-up the use of RFC2119 keywords Please check each change carefully. I tried to not change the semantics meaning of anything, rather just wanted to make sure each use of an RFC2119 keyword was actually used correctly - meaing it's "normative". RFC2119 does not distinguish between lower-case and upper-case use of the words, so "may" and "MAY" have the same meaning. However, in most cases people use upper-case versions of the words so the 1) they stand-out, and 2) it's more obvious that the author used the RFC2119 keyword on purpose and didn't just use it by mistake as part of the descriptive english text. We should add an RFC2119 keyword checker to docs like this to catch mis-uses of the keywords going forward (meaning using lower-case by mistake). Signed-off-by: Doug Davis --- docs/runtime-contract.md | 76 ++++++++++++++++++++-------------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/docs/runtime-contract.md b/docs/runtime-contract.md index 49741a7acbb9..1b5844993fa8 100644 --- a/docs/runtime-contract.md +++ b/docs/runtime-contract.md @@ -9,10 +9,10 @@ workloads. In contrast to general-purpose containers, stateless request-triggered (i.e. on-demand) autoscaled containers have the following properties: -- Little or no long-term runtime state (especially in cases where code may be +- Little or no long-term runtime state (especially in cases where code might be scaled to zero in the absence of request traffic). - Logging and monitoring aggregation (telemetry) is important for understanding - and debugging the system, as containers may be created or deleted at any time + and debugging the system, as containers might be created or deleted at any time in response to autoscaling. - Multitenancy is highly desirable to allow cost sharing for bursty applications on relatively stable underlying hardware resources. @@ -35,15 +35,15 @@ The ([v1.0.1](https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md)) is the basis for this document. When this document and the OCI specification conflict, this document is assumed to override the general OCI recommendations. -Where this document does not specify behavior, developers should assume an OCI -compliant underlying implementation. Additionally, the core Knative definition +Where this document does not specify behavior, runtime implementations SHOULD +be OCI compliant with respect to those features. Additionally, the core Knative definition assumes the [Linux Container Configuration](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md). In particular, the default Knative implementation relies on Kubernetes behavior to implement container operation. In some cases, current Kubernetes behavior in -2018 is not as performant as recommended in this documentation. The goal of the -Knative authors is to push as much of the required functionality into Kubernetes +2018 is not as performant as specified in this documentation. The goal of the +Knative authors is to push as much of the needed functionality into Kubernetes and/or Istio as possible, rather than implementing reach-around layers. This document considers two users of a given Knative environment, and is @@ -62,7 +62,7 @@ tooling developers_, by extension) running code in the environment. ## Runtime and Lifecycle Knative aims to minimize the amount of tuning and production configuration -required to run a service. Some of these production-friendly features include: +needed to run a service. Some of these production-friendly features include: 1. Stateless computation at request-scale or event-scale granularity. 1. Automatic scaling between 0 and many instances (the process scale-out @@ -82,9 +82,9 @@ serverless platform SHOULD observe the following properties: ### State -The general OCI state may not be available for introspection within the -container, and may only be visible to the system operator or platform provider. -In a highly-shared environment, containers may experience the following: +The general OCI state might not be available for introspection within the +container, and might only be visible to the system operator or platform provider. +In a highly-shared environment, containers might experience the following: - Containers with `status` of `stopped` MAY be immediately reclaimed by the system. @@ -103,12 +103,12 @@ In a highly-shared environment, containers may experience the following: - The environment MAY restrict the use of `prestart`, `poststart`, and `poststop` hooks to platform operators rather than developers. All of these hooks are defined in the context of the runtime namespace, rather than the - container namespace, and may expose system-level information (and are + container namespace, and might expose system-level information (and are non-portable). - Failures of the developer-specified process MUST be logged to a developer-visible logging system. -In addition, some serverless environments may use an execution model other than +In addition, some serverless environments MAY use an execution model other than docker in linux (for example, [runv](https://github.com/hyperhq/runv) or [Kata Containers](https://katacontainers.io/)). Implementations using an execution model beyond docker in linux MAY alter the lifecycle contract beyond @@ -126,9 +126,9 @@ the OCI specification as long as: contents from a particular execution. Because containers (particularly failing containers) can experience frequent starts, operators or platform providers SHOULD limit the total space consumed by these failures. -- A container should write its own termination message to `/dev/termination-log` +- A container SHOULD write its own termination message to `/dev/termination-log` by default. If no message is written by the container, the last few lines of - log output should be reported as the execution error (i.e. by + log output SHOULD be reported as the execution error (i.e. by [setting the `terminationMessagePolicy` to `FallbackToLogsOnError`](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/#customizing-the-termination-message)) on Kubernetes. @@ -165,7 +165,7 @@ always result in `EOF`. The `stdout` and `stderr` file descriptors on the container SHOULD be collected and retained in a developer-accessible logging repository. (TODO:[docs#902](https://github.com/knative/docs/issues/902)). -Within the container, pipes and file descriptors may be used to communicate +Within the container, pipes and file descriptors MAY be used to communicate between processes running in the same container. #### Dev symbolic links @@ -204,7 +204,7 @@ available in the `PORT` environment variable. The platform provider SHOULD configure the platform to perform HTTPS termination and protocol transformation e.g. between QUIC or HTTP/2 and HTTP/1.1. Developers -should not need to implement multiple transports between the platform and their +ought not need to implement multiple transports between the platform and their code. Unless overridden by setting the [`name`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#containerport-v1-core) field on the inbound port, the platform will perform automatic detection as @@ -217,9 +217,9 @@ following protocol will be used: - `h2c`: HTTP/2 transport, as described in [section 3.4 of the HTTP2 spec (Starting HTTP/2 with Prior Knowledge)](https://http2.github.io/http2-spec/#known-http) -Developers SHOULD prefer to use automatic content negotiation where available, +Developers SHOULD use automatic content negotiation where available, and MUST NOT set the `name` field to arbitrary values, as additional transports -may be defined in the future. Developers MUST assume all traffic is +might be defined in the future. Developers MUST assume all traffic is intermediated by an L7 proxy. Developers MUST NOT assume a direct network connection between their server process and client processes. @@ -282,7 +282,7 @@ target the inbound container port; platform providers SHOULD disallow other probe methods. Because serverless platforms automatically scale instances based on inbound -requests, and because noncompliant (or even failing) containers may be provided +requests, and because noncompliant (or even failing) containers might be provided by developers, the following defaults SHOULD be applied by the platform provider if not set by the developer. The probes are intended to be trivially supportable by naive conforming containers while preventing interference with developer @@ -293,7 +293,7 @@ code. These settings apply to both `livenessProbe` and `readinessProbe`: - `periodSeconds` set to platform-specific value In order to enable scaling in response to load, developers SHOULD NOT set the -`initialDelaySeconds` to a value greater than 0, and should aim to minimize +`initialDelaySeconds` to a value greater than 0, and SHOULD aim to minimize container startup time (aka cold start time). ##### Deployment probe @@ -321,7 +321,7 @@ be run at least once despite scale-to-zero guarantees. OCI does not specify any properties of the network environment in which a container runs. The following items are OPTIONAL additions to the runtime -contract which describe services which may be of particular value to platform +contract which describe services which might be of particular value to platform providers. #### DNS @@ -355,9 +355,9 @@ serverless workloads. Containers MUST use the provided temporary storage areas ### Mounts -In general, stateless applications should package their dependencies within the -container and not rely on mutable external state for templates, logging -configuration, etc. In some cases, it may be necessary for certain application +In general, stateless applications package their dependencies within the +container and do not rely on mutable external state for templates, logging +configuration, etc. In some cases, it might be necessary for certain application settings to be overridden at deploy time (for example, database backends or authentication credentials). When these settings need to be loaded via a file, read-only mounts of application configuration and secrets are supported by @@ -406,7 +406,7 @@ such variables will follow demonstrated usage and utility. ### User -Developers MAY specify that containers should be run as a specific user or group +Developers MAY specify that containers be run as a specific user or group ID using the `runAsUser` container property. If specified, the runtime [MUST](https://github.com/knative/serving/blob/master/test/conformance/runtime/user_test.go) run the container as the specified user ID if allowed by the platform (see @@ -421,12 +421,12 @@ be failed. ### Default Filesystems -The OCI specification describes a default container environment which may be +The OCI specification describes a default container environment which might be used for many different purposes, including containerization of existing legacy -or stateful processes which may store substantial amounts of on-disk state. In a +or stateful processes which might store substantial amounts of on-disk state. In a scaled-out, stateless environment, container startup and teardown is accelerated -when on-disk resources are kept to a minimum. Additionally, developers may not -have access to the container filesystems (or the containers may be rapidly +when on-disk resources are kept to a minimum. Additionally, developers might not +have access to the container filesystems (or the containers might be rapidly recycled), so log aggregation SHOULD be provided. In addition to the filesystems recommended in the OCI, the following filesystems @@ -438,13 +438,13 @@ be provided: | `/tmp` | MUST be Read-write.

SHOULD be backed by tmpfs if disk load is a concern. | | `/var/log` | MUST be a directory with write permissions for logs storage. Implementations MAY permit the creation of additional subdirectories and log rotation and renaming. | -In addition, the following files may be overridden by the runtime environment to +In addition, the following files MAY be overridden by the runtime environment to enable DNS resolution: | File | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/etc/hosts` | MAY be overridden to provide host mappings for well-known or provider-specific resources. | -| `/etc/hostname` | OPTIONAL: some environments may set this to a different value for each container, but other environments may use the same value for all containers. | +| `/etc/hostname` | OPTIONAL: some environments might set this to a different value for each container, but other environments might use the same value for all containers. | | `/etc/resolv.conf` | SHOULD be set to a valid cluster-specific recursive resolver. Providers MAY provide additional default search domains to improve customer experience in the cluster. | Platform providers MAY provide additional platform-specific mount points @@ -503,7 +503,7 @@ Seccomp provides a mechanism for further restricting the set of linux syscalls permitted to the processes running inside the container environment. A seccomp sandbox MAY be enforced by the platform operator; any such application profiles SHOULD be configured and applied in a consistent mechanism outside of the -container specification. As the seccomp policy may be part of the platform +container specification. As the seccomp policy might be part of the platform security hardening, operators MAY tune this over time as the threat environment changes. @@ -517,16 +517,16 @@ From the OCI spec: > article in the kernel documentation has more information about mount > propagation. -This option should only be set by the operator or platform provider, and MUST -NOT be configurable by the developer. As mount propagation may be part of the -platform security hardening, operators MAY tune this over time as the threat +This option SHOULD only be set by the operator or platform provider, and MUST +NOT be configurable by the developer. As mount propagation might be part of the +platform security hardening, operators might tune this over time as the threat environment changes. ### Masked Paths -This option MAY only be set by the operator or platform provider, and MUST NOT -be configurable by the developer. As masked paths may be part of the platform -security hardening, operators may tune this from time to time as the threat +This option SHOULD only be set by the operator or platform provider, and MUST NOT +be configurable by the developer. As masked paths might be part of the platform +security hardening, operators might tune this from time to time as the threat environment changes. ### Readonly Paths From 11449e94beaedb6edf5e7f1b73a1a15184259764 Mon Sep 17 00:00:00 2001 From: Doug Davis Date: Mon, 1 Jul 2019 17:55:01 -0700 Subject: [PATCH 2/3] more edits from evan and dan Signed-off-by: Doug Davis --- docs/runtime-contract.md | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/docs/runtime-contract.md b/docs/runtime-contract.md index 1b5844993fa8..31a6b59b4f75 100644 --- a/docs/runtime-contract.md +++ b/docs/runtime-contract.md @@ -42,9 +42,9 @@ assumes the In particular, the default Knative implementation relies on Kubernetes behavior to implement container operation. In some cases, current Kubernetes behavior in -2018 is not as performant as specified in this documentation. The goal of the +2018 is not as performant as envisioned in this documentation. The goal of the Knative authors is to push as much of the needed functionality into Kubernetes -and/or Istio as possible, rather than implementing reach-around layers. +and/or HTTP routers as possible, rather than implementing reach-around layers. This document considers two users of a given Knative environment, and is particularly concerned with the expectations of _developers_ (and _language and @@ -71,7 +71,7 @@ needed to run a service. Some of these production-friendly features include: where possible. In order to achieve these properties, containers which are operated as part of a -serverless platform SHOULD observe the following properties: +serverless platform are expected to observe the following properties: - Fast startup time (<1s until a request or event can be processed, given container image layer caching), @@ -82,8 +82,6 @@ serverless platform SHOULD observe the following properties: ### State -The general OCI state might not be available for introspection within the -container, and might only be visible to the system operator or platform provider. In a highly-shared environment, containers might experience the following: - Containers with `status` of `stopped` MAY be immediately reclaimed by the @@ -126,7 +124,7 @@ the OCI specification as long as: contents from a particular execution. Because containers (particularly failing containers) can experience frequent starts, operators or platform providers SHOULD limit the total space consumed by these failures. -- A container SHOULD write its own termination message to `/dev/termination-log` +- A container ought to write its own termination message to `/dev/termination-log` by default. If no message is written by the container, the last few lines of log output SHOULD be reported as the execution error (i.e. by [setting the `terminationMessagePolicy` to `FallbackToLogsOnError`](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/#customizing-the-termination-message)) @@ -165,7 +163,7 @@ always result in `EOF`. The `stdout` and `stderr` file descriptors on the container SHOULD be collected and retained in a developer-accessible logging repository. (TODO:[docs#902](https://github.com/knative/docs/issues/902)). -Within the container, pipes and file descriptors MAY be used to communicate +Within the container, pipes and file descriptors can be used to communicate between processes running in the same container. #### Dev symbolic links @@ -217,10 +215,10 @@ following protocol will be used: - `h2c`: HTTP/2 transport, as described in [section 3.4 of the HTTP2 spec (Starting HTTP/2 with Prior Knowledge)](https://http2.github.io/http2-spec/#known-http) -Developers SHOULD use automatic content negotiation where available, +Developers ought to use automatic content negotiation where available, and MUST NOT set the `name` field to arbitrary values, as additional transports -might be defined in the future. Developers MUST assume all traffic is -intermediated by an L7 proxy. Developers MUST NOT assume a direct network +might be defined in the future. Developers can assume all traffic is +intermediated by an L7 proxy. Developers can not assume a direct network connection between their server process and client processes. #### Headers @@ -292,9 +290,9 @@ code. These settings apply to both `livenessProbe` and `readinessProbe`: - `initialDelaySeconds` set to 0 - `periodSeconds` set to platform-specific value -In order to enable scaling in response to load, developers SHOULD NOT set the -`initialDelaySeconds` to a value greater than 0, and SHOULD aim to minimize -container startup time (aka cold start time). +In order to enable scaling in response to load, setting `initialDelaySeconds` +to a value greater than 0 can be used, while striving to minimize container +startup time (aka cold start time). ##### Deployment probe @@ -421,7 +419,7 @@ be failed. ### Default Filesystems -The OCI specification describes a default container environment which might be +The OCI specification describes a default container environment which can be used for many different purposes, including containerization of existing legacy or stateful processes which might store substantial amounts of on-disk state. In a scaled-out, stateless environment, container startup and teardown is accelerated @@ -438,13 +436,13 @@ be provided: | `/tmp` | MUST be Read-write.

SHOULD be backed by tmpfs if disk load is a concern. | | `/var/log` | MUST be a directory with write permissions for logs storage. Implementations MAY permit the creation of additional subdirectories and log rotation and renaming. | -In addition, the following files MAY be overridden by the runtime environment to -enable DNS resolution: +In addition, the following constraints are specified for the overridden files +indicated: | File | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/etc/hosts` | MAY be overridden to provide host mappings for well-known or provider-specific resources. | -| `/etc/hostname` | OPTIONAL: some environments might set this to a different value for each container, but other environments might use the same value for all containers. | +| `/etc/hostname` | some environments MAY set this to a different value for each container, but other environments might use the same value for all containers. | | `/etc/resolv.conf` | SHOULD be set to a valid cluster-specific recursive resolver. Providers MAY provide additional default search domains to improve customer experience in the cluster. | Platform providers MAY provide additional platform-specific mount points @@ -517,14 +515,14 @@ From the OCI spec: > article in the kernel documentation has more information about mount > propagation. -This option SHOULD only be set by the operator or platform provider, and MUST +This option MAY be set by the operator or platform provider, and MUST NOT be configurable by the developer. As mount propagation might be part of the platform security hardening, operators might tune this over time as the threat environment changes. ### Masked Paths -This option SHOULD only be set by the operator or platform provider, and MUST NOT +This option MAY be set by the operator or platform provider, and MUST NOT be configurable by the developer. As masked paths might be part of the platform security hardening, operators might tune this from time to time as the threat environment changes. From bfe62b0336150bc6b53e261a0c30212a770816cc Mon Sep 17 00:00:00 2001 From: Doug Davis Date: Tue, 2 Jul 2019 13:49:28 -0700 Subject: [PATCH 3/3] more tweaks Signed-off-by: Doug Davis --- docs/runtime-contract.md | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/docs/runtime-contract.md b/docs/runtime-contract.md index 31a6b59b4f75..f3139ef1bffe 100644 --- a/docs/runtime-contract.md +++ b/docs/runtime-contract.md @@ -124,11 +124,6 @@ the OCI specification as long as: contents from a particular execution. Because containers (particularly failing containers) can experience frequent starts, operators or platform providers SHOULD limit the total space consumed by these failures. -- A container ought to write its own termination message to `/dev/termination-log` - by default. If no message is written by the container, the last few lines of - log output SHOULD be reported as the execution error (i.e. by - [setting the `terminationMessagePolicy` to `FallbackToLogsOnError`](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/#customizing-the-termination-message)) - on Kubernetes. ### Warnings @@ -290,9 +285,9 @@ code. These settings apply to both `livenessProbe` and `readinessProbe`: - `initialDelaySeconds` set to 0 - `periodSeconds` set to platform-specific value -In order to enable scaling in response to load, setting `initialDelaySeconds` -to a value greater than 0 can be used, while striving to minimize container -startup time (aka cold start time). +Setting `initialDelaySeconds` to a value greater than 0 impacts container +startup time (aka cold start time) as a container will not serve traffic until +the probe succeeds. ##### Deployment probe @@ -424,7 +419,7 @@ used for many different purposes, including containerization of existing legacy or stateful processes which might store substantial amounts of on-disk state. In a scaled-out, stateless environment, container startup and teardown is accelerated when on-disk resources are kept to a minimum. Additionally, developers might not -have access to the container filesystems (or the containers might be rapidly +have access to the container's filesystems (or the containers might be rapidly recycled), so log aggregation SHOULD be provided. In addition to the filesystems recommended in the OCI, the following filesystems @@ -436,8 +431,7 @@ be provided: | `/tmp` | MUST be Read-write.

SHOULD be backed by tmpfs if disk load is a concern. | | `/var/log` | MUST be a directory with write permissions for logs storage. Implementations MAY permit the creation of additional subdirectories and log rotation and renaming. | -In addition, the following constraints are specified for the overridden files -indicated: +To enable DNS resolution, the following files might be overwritten at runtime: | File | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -501,9 +495,9 @@ Seccomp provides a mechanism for further restricting the set of linux syscalls permitted to the processes running inside the container environment. A seccomp sandbox MAY be enforced by the platform operator; any such application profiles SHOULD be configured and applied in a consistent mechanism outside of the -container specification. As the seccomp policy might be part of the platform -security hardening, operators MAY tune this over time as the threat environment -changes. +container specification. A seccomp policy MAY be part of the platform +security configuration that operators can tune over time as the +threat environment changes. ### Rootfs Mount Propagation @@ -516,16 +510,16 @@ From the OCI spec: > propagation. This option MAY be set by the operator or platform provider, and MUST -NOT be configurable by the developer. As mount propagation might be part of the -platform security hardening, operators might tune this over time as the threat -environment changes. +NOT be configurable by the developer. Mount propagation MAY be part of the +platform security configuration that operators can tune over time +as the threat environment changes. ### Masked Paths This option MAY be set by the operator or platform provider, and MUST NOT -be configurable by the developer. As masked paths might be part of the platform -security hardening, operators might tune this from time to time as the threat -environment changes. +be configurable by the developer. Masked paths MAY be part of the platform +security configuration that operators can tune over time as the +threat environment changes. ### Readonly Paths