diff --git a/docs/connectors/io-crd-config/sink-crd-config.md b/docs/connectors/io-crd-config/sink-crd-config.md index 67853542..d7d94508 100644 --- a/docs/connectors/io-crd-config/sink-crd-config.md +++ b/docs/connectors/io-crd-config/sink-crd-config.md @@ -14,9 +14,10 @@ This table lists sink configurations. | `name` | The name of a sink connector. | | `classname` | The class name of a sink connector. | | `tenant` | The tenant of a sink connector. | +| `namespace` | The Pulsar namespace of a sink connector. | | `Replicas`| The number of instances that you want to run this sink connector. By default, the `Replicas` is set to `1`. | | `MaxReplicas`| The maximum number of Pulsar instances that you want to run for this sink connector. When the value of the `maxReplicas` parameter is greater than the value of `replicas`, it indicates that the sink controller automatically scales the sink connector based on the CPU usage. By default, `maxReplicas` is set to 0, which indicates that auto-scaling is disabled. | -| `SinkConfig` | The map to a ConfigMap specifying the configuration of a sink connector. | +| `SinkConfig` | The sink connector configurations in YAML format.| | `Timeout` | The message timeout in milliseconds. | | `NegativeAckRedeliveryDelayMs`| The number of redelivered messages due to negative acknowledgement. | | `AutoAck` | Whether or not the framework acknowledges messages automatically. This field is required. You can set it to `true` or `false`.| @@ -122,3 +123,6 @@ Function Mesh supports customizing the Pod running Pulsar connectors. This table | `ServiceAccountName` | Specify the name of the service account which is used to run Pulsar Functions or connectors in the Function Mesh Worker service.| | `InitContainers` | The initialization containers belonging to a Pod. A typical use case could be using an initialization container to download a remote JAR to a local path. | | `Sidecars` | Sidecar containers run together with the main function container in a Pod. | +| `BuiltinAutoscaler` | Specify the built-in autoscaling rules.
- CPU-based autoscaling: auto-scale the number of Pods based on the CPU usage (80%, 50%, or 20%).
- Memory-based autoscaling: auto-scale the number of Pods based on the memory usage (80%, 50%, or 20%).
If you configure the `BuiltinAutoscaler` field, you do not need to configure the `AutoScalingMetrics` and `AutoScalingBehavior` options and vice versa.| +| `AutoScalingMetrics` | Specify how to scale based on customized metrics defined in connectors. For details, see [MetricSpec v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#metricspec-v2beta2-autoscaling). | +| `AutoScalingBehavior` | Configure the scaling behavior of the target in both up and down directions (`scaleUp` and `scaleDown` fields respectively). If not specified, the default Kubernetes scaling behaviors are adopted. For details, see [HorizontalPodAutoscalerBehavior v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#horizontalpodautoscalerbehavior-v2beta2-autoscaling). | diff --git a/docs/connectors/io-crd-config/source-crd-config.md b/docs/connectors/io-crd-config/source-crd-config.md index 891be031..eff7da57 100644 --- a/docs/connectors/io-crd-config/source-crd-config.md +++ b/docs/connectors/io-crd-config/source-crd-config.md @@ -14,10 +14,12 @@ This table lists source configurations. | `name` | The name of a source connector. | | `classname` | The class name of a source connector. | | `tenant` | The tenant of a source connector. | +| `namespace` | The Pulsar namespace of a source connector. | | `Replicas`| The number of instances that you want to run this source connector. | | `MaxReplicas`| The maximum number of Pulsar instances that you want to run for this source connector. When the value of the `maxReplicas` parameter is greater than the value of `replicas`, it indicates that the source controller automatically scales the source connector based on the CPU usage. By default, `maxReplicas` is set to 0, which indicates that auto-scaling is disabled. | -| `SourceConfig` | The map to a ConfigMap specifying the configuration of a source connector. | +| `SourceConfig` | The source connector configurations in YAML format. | | `ProcessingGuarantee` | The processing guarantees (delivery semantics) applied to the source connector. Available values: `ATLEAST_ONCE`, `ATMOST_ONCE`, `EFFECTIVELY_ONCE`.| +| `ForwardSourceMessageProperty` | Configure whether to pass message properties to a target topic. | ## Images @@ -114,3 +116,6 @@ Function Mesh supports customizing the Pod running connectors. This table lists | `ServiceAccountName` | Specify the name of the service account which is used to run Pulsar Functions or connectors in the Function Mesh Worker service.| | `InitContainers` | Initialization containers belonging to a Pod. A typical use case could be using an Initialization container to download a remote JAR to a local path. | | `Sidecars` | Sidecar containers run together with the main function container in a Pod. | +| `BuiltinAutoscaler` | Specify the built-in autoscaling rules.
- CPU-based autoscaling: auto-scale the number of Pods based on the CPU usage (80%, 50%, or 20%).
- Memory-based autoscaling: auto-scale the number of Pods based on the memory usage (80%, 50%, or 20%).
If you configure the `BuiltinAutoscaler` field, you do not need to configure the `AutoScalingMetrics` and `AutoScalingBehavior` options and vice versa.| +| `AutoScalingMetrics` | Specify how to scale based on customized metrics defined in connectors. For details, see [MetricSpec v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#metricspec-v2beta2-autoscaling). | +| `AutoScalingBehavior` | Configure the scaling behavior of the target in both up and down directions (`scaleUp` and `scaleDown` fields respectively). If not specified, the default Kubernetes scaling behaviors are adopted. For details, see [HorizontalPodAutoscalerBehavior v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#horizontalpodautoscalerbehavior-v2beta2-autoscaling). | diff --git a/docs/connectors/pulsar-io-debug.md b/docs/connectors/pulsar-io-debug.md index ec1998a7..fdcecf1a 100644 --- a/docs/connectors/pulsar-io-debug.md +++ b/docs/connectors/pulsar-io-debug.md @@ -18,7 +18,7 @@ In addition, you can use the following command to check the specific Pod. - `kubectl describe pod POD_NAME -n NAMESPACE_NAME`: check the current state of the Pod and recent events. -For the use of `kubectl` commands, see [here](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands). +For the use of `kubectl` commands, see [kubectl command reference](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands). ## Use log topics diff --git a/docs/connectors/run-connector.md b/docs/connectors/run-connector.md index 10982eac..d60e9702 100644 --- a/docs/connectors/run-connector.md +++ b/docs/connectors/run-connector.md @@ -148,6 +148,10 @@ This section describes how to package a Pulsar connector to a NAR or JAR package Use the `pulsar-admin` CLI tool to upload the NAR or uber JAR package to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). +> **Note** +> +> Before uploading the package to Pulsar package management service, you need to enable the package management service in the `broker.config` file. + This example shows how to upload the NAR package of the `my-sink` connector to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). ```bash diff --git a/docs/functions/function-crd.md b/docs/functions/function-crd.md index 5b4f5a94..6b6e96a1 100644 --- a/docs/functions/function-crd.md +++ b/docs/functions/function-crd.md @@ -15,16 +15,17 @@ This table lists Pulsar Function configurations. | `name` | The name of a Pulsar Function. | | `classname` | The class name of a Pulsar Function. | | `tenant` | The tenant of a Pulsar Function. | -| `namespace` | The namespace of a Pulsar Function. | +| `namespace` | The Pulsar namespace of a Pulsar Function. | | `Replicas`| The number of instances that you want to run this Pulsar Function. By default, the `Replicas` is set to `1`. | | `MaxReplicas`| The maximum number of Pulsar instances that you want to run for this Pulsar Function. When the value of the `maxReplicas` parameter is greater than the value of `replicas`, it indicates that the Functions controller automatically scales the Pulsar Functions based on the CPU usage. By default, `maxReplicas` is set to 0, which indicates that auto-scaling is disabled. | | `Timeout` | The message timeout in milliseconds. | | `DeadLetterTopic` | The topic where all messages that were not processed successfully are sent. This parameter is not supported in Python Functions. | -| `FuncConfig` | The map to a ConfigMap specifying the configuration of a Pulsar function. | +| `FuncConfig` | Pulsar Functions configurations in YAML format. | | `LogTopic` | The topic to which the logs of a Pulsar Function are produced. | | `AutoAck` | Whether or not the framework acknowledges messages automatically. This field is required. You can set it to `true` or `false`.| | `MaxMessageRetry` | How many times to process a message before giving up. | | `ProcessingGuarantee` | The processing guarantees (delivery semantics) applied to the function. Available values: `ATLEAST_ONCE`, `ATMOST_ONCE`, `EFFECTIVELY_ONCE`.| +| `ForwardSourceMessageProperty` | Configure whether to pass message properties to a target topic. | | `RetainOrdering` | Function consumes and processes messages in order. | | `RetainKeyOrdering`| Configure whether to retain the key order of messages. | | `SubscriptionName` | Pulsar Functions’ subscription name if you want a specific subscription-name for the input-topic consumer. | @@ -147,3 +148,6 @@ Function Mesh supports customizing the Pod running function instance. This table | `ServiceAccountName` | Specify the name of the service account which is used to run Pulsar Functions or connectors in the Function Mesh Worker service.| | `InitContainers` | Initialization containers belonging to a Pod. A typical use case could be using an Initialization container to download a remote JAR to a local path. | | `Sidecars` | Sidecar containers run together with the main function container in a Pod. | +| `BuiltinAutoscaler` | Specify the built-in autoscaling rules.
- CPU-based autoscaling: auto-scale the number of Pods based on the CPU usage (80%, 50%, or 20%).
- Memory-based autoscaling: auto-scale the number of Pods based on the memory usage (80%, 50%, or 20%).
If you configure the `BuiltinAutoscaler` field, you do not need to configure the `AutoScalingMetrics` and `AutoScalingBehavior` options and vice versa.| +| `AutoScalingMetrics` | Specify how to scale based on customized metrics defined in Pulsar Functions. For details, see [MetricSpec v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#metricspec-v2beta2-autoscaling). | +| `AutoScalingBehavior` | Configure the scaling behavior of the target in both up and down directions (`scaleUp` and `scaleDown` fields respectively). If not specified, the default Kubernetes scaling behaviors are adopted. For details, see [HorizontalPodAutoscalerBehavior v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#horizontalpodautoscalerbehavior-v2beta2-autoscaling). | diff --git a/docs/functions/function-debug.md b/docs/functions/function-debug.md index de2be26f..8101968e 100644 --- a/docs/functions/function-debug.md +++ b/docs/functions/function-debug.md @@ -18,7 +18,7 @@ In addition, you can use the following command to check the specific Pod. - `kubectl describe pod POD_NAME -n NAMESPACE_NAME`: check the current state of the Pod and recent events. -For the use of `kubectl` commands, see [here](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands). +For the use of `kubectl` commands, see [kubectl command reference](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands). ## Use log topic diff --git a/docs/functions/run-function/run-go-function.md b/docs/functions/run-function/run-go-function.md index 8009313a..2bc4d1dd 100644 --- a/docs/functions/run-function/run-go-function.md +++ b/docs/functions/run-function/run-go-function.md @@ -86,6 +86,10 @@ To package Go Functions in Go, follow these steps. Use the `pulsar-admin` CLI tool to upload the package to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). +> **Note** +> +> Before uploading the package to Pulsar package management service, you need to enable the package management service in the `broker.config` file. + This example shows how to upload the package of the `my-function@0.1` Functions to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). ```bash diff --git a/docs/functions/run-function/run-java-function.md b/docs/functions/run-function/run-java-function.md index 81e9f5e2..61d0fa31 100644 --- a/docs/functions/run-function/run-java-function.md +++ b/docs/functions/run-function/run-java-function.md @@ -116,6 +116,10 @@ To package a Functions in Java, follow these steps. Use the `pulsar-admin` CLI tool to upload the package to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). +> **Note** +> +> Before uploading the package to Pulsar package management service, you need to enable the package management service in the `broker.config` file. + This example shows how to upload the package of the `my-function@0.1` Functions to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). ```bash diff --git a/docs/functions/run-function/run-python-function.md b/docs/functions/run-function/run-python-function.md index 9d729c35..4e641922 100644 --- a/docs/functions/run-function/run-python-function.md +++ b/docs/functions/run-function/run-python-function.md @@ -79,6 +79,10 @@ Python Function supports One Python file or ZIP file. Use the `pulsar-admin` CLI tool to upload the package to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). +> **Note** +> +> Before uploading the package to Pulsar package management service, you need to enable the package management service in the `broker.config` file. + This example shows how to upload the package of the `my-function@0.1` Functions to the [Pulsar package management service](http://pulsar.apache.org/docs/en/next/admin-api-packages/). ```bash diff --git a/docs/install-function-mesh.md b/docs/install-function-mesh.md index afb2b1a4..68db49c3 100644 --- a/docs/install-function-mesh.md +++ b/docs/install-function-mesh.md @@ -41,7 +41,7 @@ This example shows how to install Function Mesh through [Helm](https://helm.sh/) > **Note** > > - Before installation, ensure that Helm v3 is installed properly. -> - For the use of `kubectl` commands, see [here](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands). +> - For the use of `kubectl` commands, see [kubectl command reference](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands). 1. Clone the StreamNative Function Mesh repository. diff --git a/docs/scaling.md b/docs/scaling.md index cae6725e..0df7a5aa 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -30,81 +30,144 @@ In CRDs, the `replicas` parameter is used to specify the number of Pods (Pulsar ## Autoscaling -Function Mesh supports scaling Pods (Pulsar instances) based on the CPU utilization automatically. By default, autoscaling is disabled (The value of the `maxReplicas` parameter is set to `0`). To enable autoscaling, you can specify the `maxReplicas` parameter and set a value for it in the CRD. This value should be greater than the value of the `replicas` parameter. - -### Auto-scale Pulsar Functions - -This example shows how to auto-scale the number of Pods running Pulsar Functions to `8`. - -1. Specify the `maxReplicas` to `8` in the Pulsar Functions CRD. The `maxReplicas` refers to the maximum number of Pods that are required for running the Pulsar Functions. - - ```yaml - apiVersion: cloud.streamnative.io/v1alpha1 - kind: Function - metadata: - name: java-function-sample - namespace: default - spec: - className: org.apache.pulsar.functions.api.examples.ExclamationFunction - forwardSourceMessageProperty: true - MaxPendingAsyncRequests: 1000 - replicas: 1 - maxReplicas: 8 - logTopic: persistent://public/default/logging-function-logs - input: - topics: - - persistent://public/default/java-function-input-topic - typeClassName: java.lang.String - output: - topic: persistent://public/default/java-function-output-topic - typeClassName: java.lang.String - # Other function configs - ``` - -2. Apply the configurations. - - ```bash - kubectl apply -f path/to/source-sample.yaml - ``` - -### Auto-scale Pulsar connectors - -This example shows how to auto-scale the number of Pods for running a Pulsar source connector to `5`. - -1. Specify the the `maxReplicas` to `5` in the Pulsar source CRD. The `maxReplicas` refers to the maximum number of Pods that are required for running the Pulsar source connector. - - **Example** - - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Source - metadata: - name: source-sample - spec: - className: org.apache.pulsar.io.debezium.mongodb.DebeziumMongoDbSource - replicas: 1 - maxReplicas: 5 - replicas: 1 - maxReplicas: 1 - output: - producerConf: - maxPendingMessages: 1000 - maxPendingMessagesAcrossPartitions: 50000 - useThreadLocalProducers: true - topic: persistent://public/default/destination - typeClassName: org.apache.pulsar.common.schema.KeyValue - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - # Other configurations - ``` - -2. Apply the configurations. - - ```bash - kubectl apply -f path/to/source-sample.yaml - ``` \ No newline at end of file +Function Mesh auto-scales the number of Pods based on the CPU usage, memory usage, customized metrics. + +- CPU usage: auto-scale the number of Pods based on CPU utilization. + + This table lists built-in CPU-based autoscaling metrics. If these metrics cannot meet your requirements, you can auto-scale the number of Pods based on customized metrics defined in Pulsar Functions or connectors. For details, see [MetricSpec v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#metricspec-v2beta2-autoscaling). + + | Option | Description | + | --- | --- | + | AverageUtilizationCPUPercent80 | Auto-scale the number of Pods if 80% CPU is utilized.| + | AverageUtilizationCPUPercent50 | Auto-scale the number of Pods if 50% CPU is utilized.| + | AverageUtilizationCPUPercent20 | Auto-scale the number of Pods if 20% CPU is utilized. | + +- Memory usage: auto-scale the number of Pods based on memory utilization. + + This table lists built-in CPU-based autoscaling metrics. If these metrics cannot meet your requirements, you can auto-scale the number of Pods based on customized metrics defined in Pulsar Functions or connectors. For details, see [MetricSpec v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#metricspec-v2beta2-autoscaling). + + | Option | Description | + | --- | --- | + | AverageUtilizationMemoryPercent80 | Auto-scale the number of Pods if 80% memory is utilized. | + | AverageUtilizationMemoryPercent50 | Auto-scale the number of Pods if 50% memory is utilized. | + | AverageUtilizationMemoryPercent20 | Auto-scale the number of Pods if 20% memory is utilized. | + +- metrics: auto-scale the number of Pods based on customized metrics defined in Pulsar Functions or connectors. For details, see [MetricSpec v2beta2 autoscaling](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#metricspec-v2beta2-autoscaling). + +> **Note** +> +> If you have configured autoscaling based on the CPU usage, memory usage, or both of them, you do not need to configure autoscaling based on customized metrics defined in Pulsar Functions or connectors and vice versa. + +By default, autoscaling is disabled (the value of the `maxReplicas` parameter is set to `0`). To enable autoscaling, you can specify the `maxReplicas` parameter and set a value for it in the CRD. This value should be greater than the value of the `replicas` parameter. Then, the number of Pods is automatically scaled when 80% CPU is utilized. + +### Prerequisites + +Deploy the metrics server in the cluster. The Metrics server provides metrics through the Metrics API. The Horizontal Pod Autoscaler (HPA) uses this API to collect metrics. To learn how to deploy the metrics-server, see the [metrics-server documentation](https://github.com/kubernetes-sigs/metrics-server#deployment). + +### Examples + +These examples describe how to auto-scale the number of Pods running Pulsar Functions. + +- Function Mesh supports automatically scaling up the number of Pods by updating the `maxReplica` parameter. In this case, the number of Pods is updated to `8` when 80% CPU is utilized. + + 1. Specify the `maxReplicas` to `8` in the Pulsar Functions CRD. The `maxReplicas` refers to the maximum number of Pods that are required for running the Pulsar Functions. + + ```yaml + apiVersion: cloud.streamnative.io/v1alpha1 + kind: Function + metadata: + name: java-function-sample + namespace: default + spec: + className: org.apache.pulsar.functions.api.examples.ExclamationFunction + forwardSourceMessageProperty: true + MaxPendingAsyncRequests: 1000 + replicas: 1 + maxReplicas: 8 + logTopic: persistent://public/default/logging-function-logs + input: + topics: + - persistent://public/default/java-function-input-topic + typeClassName: java.lang.String + output: + topic: persistent://public/default/java-function-output-topic + typeClassName: java.lang.String + # Other function configs + ``` + + 2. Apply the configurations. + + ```bash + kubectl apply -f path/to/source-sample.yaml + ``` + +- Function Mesh supports automatically scaling up the number of Pods based on a built-in autoscaling metric. This example auto-scales the number of Pods if 20% CPU is utilized. + + 1. Specify the CPU-based autoscaling metric under `pod.builtinAutoscaler` in the Pulsar Functions CRD. + + ```yaml + apiVersion: cloud.streamnative.io/v1alpha1 + kind: Function + metadata: + name: java-function-sample + namespace: default + spec: + className: org.apache.pulsar.functions.api.examples.ExclamationFunction + forwardSourceMessageProperty: true + MaxPendingAsyncRequests: 1000 + replicas: 1 + maxReplicas: 4 + logTopic: persistent://public/default/logging-function-logs + input: + topics: + - persistent://public/default/java-function-input-topic + typeClassName: java.lang.String + pod: + builtinAutoscaler: + - AverageUtilizationCPUPercent20 + # Other function configs + ``` + + 2. Apply the configurations. + + ```bash + kubectl apply -f path/to/source-sample.yaml + ``` + + >**Note** + > + > If you specify multiple metrics for the HPA to scale on, the HPA controller evaluates each metric, and proposes a new scale based on that metric. The largest of the proposed scales will be used as the new scale. + +- Function Mesh supports automatically scaling up the number of Pods based on a customized autoscaling metric. This example auto-scales the number of Pods if 45% CPU is utilized. + + 1. Specify the customized autoscaling metric under `pod.autoScalingMetrics` in the Pulsar Functions CRD. + + ```yaml + apiVersion: cloud.streamnative.io/v1alpha1 + kind: Function + metadata: + name: java-function-sample + namespace: default + spec: + className: org.apache.pulsar.functions.api.examples.ExclamationFunction + forwardSourceMessageProperty: true + MaxPendingAsyncRequests: 1000 + replicas: 1 + maxReplicas: 4 + logTopic: persistent://public/default/logging-function-logs + pod: + autoScalingMetrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 45 + # Other function configs + ``` + + 2. Apply the configurations. + + ```bash + kubectl apply -f path/to/source-sample.yaml + ``` \ No newline at end of file