diff --git a/site2/website/versioned_docs/version-2.10.1/admin-api-brokers.md b/site2/website/versioned_docs/version-2.10.1/admin-api-brokers.md index 89255b7f85e75..9aeeea1a904a6 100644 --- a/site2/website/versioned_docs/version-2.10.1/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.10.1/admin-api-brokers.md @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.10.1/admin-api-clusters.md b/site2/website/versioned_docs/version-2.10.1/admin-api-clusters.md index 7e4f7651ffb69..53cd43187e069 100644 --- a/site2/website/versioned_docs/version-2.10.1/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.10.1/admin-api-clusters.md @@ -89,7 +89,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -113,11 +113,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -122,7 +122,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-cgo.md index 72db5ae7e8aca..feee2cac3bafb 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-cgo.md @@ -5,7 +5,7 @@ sidebar_label: "CGo(deprecated)" original_id: client-libraries-cgo --- -> The CGo client has been deprecated since version 2.7.0. If possible, use the [Go client](client-libraries-go) instead. +> The CGo client has been deprecated since version 2.7.0. If possible, use the [Go client](client-libraries-go.md) instead. You can use Pulsar Go client to create Pulsar [producers](#producers), [consumers](#consumers), and [readers](#readers) in Go (aka Golang). @@ -26,7 +26,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -59,7 +59,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -77,7 +77,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -238,14 +238,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -514,7 +514,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-cpp.md index 8a2d4977a9448..75b35ec97bbff 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-cpp.md @@ -658,7 +658,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-dotnet.md index 0663beb23520e..52b6200c478af 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-dotnet.md @@ -189,7 +189,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-go.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-go.md index 45cec7e677fe0..d2f5dd5a13d0d 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-go.md @@ -56,7 +56,7 @@ $ ./test_example ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -82,7 +82,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -1023,7 +1023,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -1043,7 +1043,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-java.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-java.md index 8cc2e0644810f..e126f3fb24840 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-java.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [readers](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [readers](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), [readers](#reader) and [TableView](#tableview) of a Java client are thread-safe. @@ -20,10 +20,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | [The producer and consumer API](/api/client/) | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -70,7 +70,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -96,7 +96,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -127,7 +127,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -301,7 +301,7 @@ Currently, cluster-level failover can perform probes to prevent data loss, but i > #### What are the relationships between cluster-level failover and geo-replication? -The cluster-level failover is an extension of [geo-replication](concepts-replication) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. +The cluster-level failover is an extension of [geo-replication](concepts-replication.md) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. Influence |Cluster-level failover|Geo-replication |---|---|--- @@ -598,7 +598,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -1331,7 +1331,7 @@ tv.forEach((key, value) -> /*operations on all existing messages*/) ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -1449,11 +1449,11 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication -To use [TLS](security-tls-authentication), you need to set TLS to `true` using the `setUseTls` method, point your Pulsar client to a TLS cert path, and provide paths to cert and key files. +To use [TLS](security-tls-authentication.md), you need to set TLS to `true` using the `setUseTls` method, point your Pulsar client to a TLS cert path, and provide paths to cert and key files. The following is an example. @@ -1477,7 +1477,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1515,7 +1515,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-node.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-node.md index a098e7c10fafd..a023b51d8ceb0 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-python.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-python.md index bd39c32660885..1000023799044 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-python.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.10.1/client-libraries-websocket.md index 51b9da4dfd70f..145866e41644b 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.10.1/client-libraries.md b/site2/website/versioned_docs/version-2.10.1/client-libraries.md index 4899a1ace788d..6cdc1e615c81f 100644 --- a/site2/website/versioned_docs/version-2.10.1/client-libraries.md +++ b/site2/website/versioned_docs/version-2.10.1/client-libraries.md @@ -9,12 +9,12 @@ Pulsar supports the following client libraries: |Language|Documentation|Release note|Code repo |---|---|---|--- -Java |- [User doc](client-libraries-java)

- [API doc](/api/client/)|[Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) -C++ | - [User doc](client-libraries-cpp)

- [API doc](/api/cpp/)|[Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) -Python | - [User doc](client-libraries-python)

- [API doc](/api/python/)|[Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) -WebSocket| [User doc](client-libraries-websocket) | [Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) +Java |- [User doc](client-libraries-java.md)

- [API doc](/api/client/)|[Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) +C++ | - [User doc](client-libraries-cpp.md)

- [API doc](/api/cpp/)|[Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) +Python | - [User doc](client-libraries-python.md)

- [API doc](/api/python/)|[Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) +WebSocket| [User doc](client-libraries-websocket.md) | [Here](/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) Go client|[User doc](client-libraries-go.md)|[Here](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG) |[Here](https://github.com/apache/pulsar-client-go) -Node.js|[User doc](client-libraries-node)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) +Node.js|[User doc](client-libraries-node.md)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) C# |[User doc](client-libraries-dotnet.md)| [Here](https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG)|[Here](https://github.com/apache/pulsar-dotpulsar) :::note diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.10.1/concepts-architecture-overview.md index a9e5141dd130e..5f2fb2ea99167 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -140,17 +140,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -158,7 +158,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-authentication.md b/site2/website/versioned_docs/version-2.10.1/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-clients.md b/site2/website/versioned_docs/version-2.10.1/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-messaging.md b/site2/website/versioned_docs/version-2.10.1/concepts-messaging.md index 55073f9ce9fed..b1d1483232ebd 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-messaging.md @@ -23,8 +23,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Topic name | The name of the topic that the message is published to. @@ -54,7 +54,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -172,7 +172,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -657,7 +657,7 @@ A [subscription](#concepts-messaging.md/#subscriptions) can have one or more con #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -761,7 +761,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -773,7 +773,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -811,7 +811,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -914,7 +914,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -937,12 +937,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.10.1/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-overview.md b/site2/website/versioned_docs/version-2.10.1/concepts-overview.md index e13e691bbda79..67e9c4024aa53 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/longterm storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/longterm storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-replication.md b/site2/website/versioned_docs/version-2.10.1/concepts-replication.md index 7a4a42f979981..1ac455c702832 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-replication.md @@ -9,7 +9,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.10.1/concepts-tiered-storage.md index f91d4fea48dec..b45ccea5888bf 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-tiered-storage.md @@ -13,6 +13,6 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol > Data written to BookKeeper is replicated to 3 physical machines by default. However, once a segment is sealed in BookKeeper it becomes immutable and can be copied to long term storage. Long term storage can achieve cost savings by using mechanisms such as [Reed-Solomon error correction](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction) to require fewer physical copies of data. -Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](cookbooks-tiered-storage). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). +Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](cookbooks-tiered-storage.md). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.10.1/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.10.1/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.10.1/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.10.1/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.10.1/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.10.1/cookbooks-compaction.md index 0a3623381db79..82ded8081e400 100644 --- a/site2/website/versioned_docs/version-2.10.1/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.10.1/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.10.1/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.10.1/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.10.1/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.10.1/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.10.1/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.10.1/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.10.1/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.10.1/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.10.1/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.10.1/cookbooks-retention-expiry.md index 2b9c0d28aedbb..bb268ecf67166 100644 --- a/site2/website/versioned_docs/version-2.10.1/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.10.1/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.10.1/deploy-aws.md b/site2/website/versioned_docs/version-2.10.1/deploy-aws.md index 9378188425769..5497aadd7865f 100644 --- a/site2/website/versioned_docs/version-2.10.1/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.10.1/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal-multi-cluster.md index d6d8956795ee2..9ac1a85580ffa 100644 --- a/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal-multi-cluster.md @@ -7,13 +7,13 @@ original_id: deploy-bare-metal-multi-cluster :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -22,7 +22,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal.md index f01144a8dcadd..25dc04458613b 100644 --- a/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.10.1/deploy-bare-metal.md @@ -7,9 +7,9 @@ original_id: deploy-bare-metal :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -123,7 +123,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -288,9 +288,9 @@ Flag | Description `--metadata-store` | A "local" metadata store connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-metadata-store` | The configuration metadata store connection string for the entire instance. As with the `--metadata-store` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -430,7 +430,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -448,7 +448,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.10.1/deploy-dcos.md b/site2/website/versioned_docs/version-2.10.1/deploy-dcos.md index 49cf624ea24f1..35a0a83d716ad 100644 --- a/site2/website/versioned_docs/version-2.10.1/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.10.1/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.10.1/deploy-docker.md b/site2/website/versioned_docs/version-2.10.1/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.10.1/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.10.1/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.10.1/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.10.1/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.10.1/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.10.1/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.10.1/deploy-monitoring.md b/site2/website/versioned_docs/version-2.10.1/deploy-monitoring.md index 07b59b59a7560..69d994b7d586f 100644 --- a/site2/website/versioned_docs/version-2.10.1/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.10.1/deploy-monitoring.md @@ -101,7 +101,7 @@ pulsar-admin functions-worker function-stats ``` -The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get [`FUNCTIONS_WORKER_ADDRESS`](functions-worker) and `WORKER_PORT` from the `functions_worker.yml` file. +The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get [`FUNCTIONS_WORKER_ADDRESS`](functions-worker.md) and `WORKER_PORT` from the `functions_worker.yml` file. ``` @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.10.1/develop-schema.md b/site2/website/versioned_docs/version-2.10.1/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.10.1/develop-schema.md +++ b/site2/website/versioned_docs/version-2.10.1/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.10.1/functions-deploy.md b/site2/website/versioned_docs/version-2.10.1/functions-deploy.md index 482185c5840a9..826804db6bbb7 100644 --- a/site2/website/versioned_docs/version-2.10.1/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.10.1/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), DC/OS, and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), DC/OS, and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](client-libraries-python) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](client-libraries-python.md) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.10.1/functions-develop.md b/site2/website/versioned_docs/version-2.10.1/functions-develop.md index ac6523ff1efbe..c32199517cfcc 100644 --- a/site2/website/versioned_docs/version-2.10.1/functions-develop.md +++ b/site2/website/versioned_docs/version-2.10.1/functions-develop.md @@ -1205,7 +1205,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1292,7 +1292,7 @@ func metricRecorderFunction(ctx context.Context, in []byte) error { ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.10.1/functions-overview.md b/site2/website/versioned_docs/version-2.10.1/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.10.1/functions-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.10.1/functions-package.md b/site2/website/versioned_docs/version-2.10.1/functions-package.md index 82346aa8c5098..a995d5c158877 100644 --- a/site2/website/versioned_docs/version-2.10.1/functions-package.md +++ b/site2/website/versioned_docs/version-2.10.1/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.10.1/functions-worker.md b/site2/website/versioned_docs/version-2.10.1/functions-worker.md index 5a97d61340f64..60eb84657919b 100644 --- a/site2/website/versioned_docs/version-2.10.1/functions-worker.md +++ b/site2/website/versioned_docs/version-2.10.1/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -199,7 +199,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -219,7 +219,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -243,7 +243,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -282,7 +282,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: @@ -337,7 +337,7 @@ Hence you need to configure your `pulsar-admin` to use the right service URL acc In order to address this inconvenience, you can start a proxy cluster for routing the admin rest requests accordingly. Hence you will have one central entry point for your admin service. -If you already have a proxy cluster, continue reading. If you haven't setup a proxy cluster before, you can follow the [instructions](administration-proxy) to start proxies. +If you already have a proxy cluster, continue reading. If you haven't setup a proxy cluster before, you can follow the [instructions](administration-proxy.md) to start proxies. ![assets/functions-worker-separated.png](/assets/functions-worker-separated-proxy.png) diff --git a/site2/website/versioned_docs/version-2.10.1/getting-started-helm.md b/site2/website/versioned_docs/version-2.10.1/getting-started-helm.md index 9e32b3315a1ea..5d5401cc86a08 100644 --- a/site2/website/versioned_docs/version-2.10.1/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.10.1/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -409,7 +409,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.10.1/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.10.1/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.10.1/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.10.1/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.10.1/getting-started-standalone.md b/site2/website/versioned_docs/version-2.10.1/getting-started-standalone.md index d81e25519203b..f3688c496d58d 100644 --- a/site2/website/versioned_docs/version-2.10.1/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.10.1/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary [RocksDB](http://rocksdb.org/) and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -122,9 +122,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](/tools/pulsar-admin/). -`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
**Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
**Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration.md) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -180,7 +180,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -193,7 +193,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -226,12 +226,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -263,7 +263,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `bin/pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](reference-cli-tools.md#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.10.1/helm-install.md b/site2/website/versioned_docs/version-2.10.1/helm-install.md index 80fb7f8d6c8d6..9f81f52e0dab1 100644 --- a/site2/website/versioned_docs/version-2.10.1/helm-install.md +++ b/site2/website/versioned_docs/version-2.10.1/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -31,8 +31,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.10.1/helm-overview.md b/site2/website/versioned_docs/version-2.10.1/helm-overview.md index fc4cbfe426a93..125f595cbe68a 100644 --- a/site2/website/versioned_docs/version-2.10.1/helm-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/helm-overview.md @@ -52,9 +52,9 @@ Moreover, Helm chart supports: ## Quick start -To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install). +This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ Although we have done our best to make these charts as seamless as possible, tro The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.10.1/helm-tools.md b/site2/website/versioned_docs/version-2.10.1/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.10.1/helm-tools.md +++ b/site2/website/versioned_docs/version-2.10.1/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.10.1/helm-upgrade.md b/site2/website/versioned_docs/version-2.10.1/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.10.1/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.10.1/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.10.1/io-cdc.md b/site2/website/versioned_docs/version-2.10.1/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.10.1/io-cdc.md +++ b/site2/website/versioned_docs/version-2.10.1/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.10.1/io-develop.md b/site2/website/versioned_docs/version-2.10.1/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.10.1/io-develop.md +++ b/site2/website/versioned_docs/version-2.10.1/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.10.1/io-kafka-source.md b/site2/website/versioned_docs/version-2.10.1/io-kafka-source.md index c160112cb0877..dd6000aa0bd35 100644 --- a/site2/website/versioned_docs/version-2.10.1/io-kafka-source.md +++ b/site2/website/versioned_docs/version-2.10.1/io-kafka-source.md @@ -213,7 +213,7 @@ This example explains how to create a Kafka source connector in an on-premises c ``` -2. Reload all [built-in connectors](io-connectors). +2. Reload all [built-in connectors](io-connectors.md). ``` diff --git a/site2/website/versioned_docs/version-2.10.1/io-overview.md b/site2/website/versioned_docs/version-2.10.1/io-overview.md index 0554e927e4fcc..82d0cd04a31d7 100644 --- a/site2/website/versioned_docs/version-2.10.1/io-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/io-overview.md @@ -160,4 +160,4 @@ For more information about the options of `pulsar-admin sinks update`, see [here You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the `Connector Admin CLI` with sources and sinks subcommands. For the latest and complete information, see [Pulsar admin docs](/tools/pulsar-admin/). -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via the `Connector Admin CLI` or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via the `Connector Admin CLI` or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.10.1/io-quickstart.md b/site2/website/versioned_docs/version-2.10.1/io-quickstart.md index b976296bccbf2..fd2ce9fbb0667 100644 --- a/site2/website/versioned_docs/version-2.10.1/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.10.1/io-quickstart.md @@ -7,17 +7,17 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: -- [Connect Pulsar to Cassandra](#Connect-Pulsar-to-Cassandra) +- [Connect Pulsar to Cassandra](#connect-pulsar-to-cassandra) -- [Connect Pulsar to PostgreSQL](#Connect-Pulsar-to-PostgreSQL) +- [Connect Pulsar to PostgreSQL](#connect-pulsar-to-postgreSQL) :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-node Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,7 +236,7 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink @@ -455,7 +455,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster diff --git a/site2/website/versioned_docs/version-2.10.1/io-use.md b/site2/website/versioned_docs/version-2.10.1/io-use.md index 8b0c1676607e1..6fbb26b4704bd 100644 --- a/site2/website/versioned_docs/version-2.10.1/io-use.md +++ b/site2/website/versioned_docs/version-2.10.1/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.10.1/performance-pulsar-perf.md b/site2/website/versioned_docs/version-2.10.1/performance-pulsar-perf.md index 6d493f2e7618f..d2ffc575ab892 100644 --- a/site2/website/versioned_docs/version-2.10.1/performance-pulsar-perf.md +++ b/site2/website/versioned_docs/version-2.10.1/performance-pulsar-perf.md @@ -153,7 +153,7 @@ For the latest and complete information about `pulsar-perf`, including commands, ## Transactions -This section shows how Pulsar Perf runs transactions. For more information, see [Pulsar transactions](txn-why). +This section shows how Pulsar Perf runs transactions. For more information, see [Pulsar transactions](txn-why.md). ### Use transaction diff --git a/site2/website/versioned_docs/version-2.10.1/reference-cli-tools.md b/site2/website/versioned_docs/version-2.10.1/reference-cli-tools.md index 72b646f848477..1e426501e23a3 100644 --- a/site2/website/versioned_docs/version-2.10.1/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.10.1/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) @@ -826,7 +826,7 @@ Options ### `transaction` -Run a transaction. For more information, see [Pulsar transactions](txn-why). +Run a transaction. For more information, see [Pulsar transactions](txn-why.md). **Usage** diff --git a/site2/website/versioned_docs/version-2.10.1/reference-configuration.md b/site2/website/versioned_docs/version-2.10.1/reference-configuration.md index 1dcc973cc452c..4df15fdf78215 100644 --- a/site2/website/versioned_docs/version-2.10.1/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.10.1/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) @@ -810,6 +810,8 @@ The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) can be config |tokenAudienceClaim| The token audience "claim" name, e.g. "aud". It is used to get the audience from token. If it is not set, the audience is not verified. || | tokenAudience | The token audience stands for this broker. The field `tokenAudienceClaim` of a valid token need contains this parameter.| | |haProxyProtocolEnabled | Enable or disable the [HAProxy](http://www.haproxy.org/) protocol. |false| +| numIOThreads | Number of threads used for Netty IO. | 2 * Runtime.getRuntime().availableProcessors() | +| numAcceptorThreads | Number of threads used for Netty Acceptor. | 1 | #### Deprecated parameters of Pulsar proxy The following parameters have been deprecated in the `conf/proxy.conf` file. diff --git a/site2/website/versioned_docs/version-2.10.1/schema-get-started.md b/site2/website/versioned_docs/version-2.10.1/schema-get-started.md index f25ff68083f84..73a05d96d7f10 100644 --- a/site2/website/versioned_docs/version-2.10.1/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.10.1/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [Go client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [Go client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.10.1/schema-manage.md b/site2/website/versioned_docs/version-2.10.1/schema-manage.md index 40b5ab06c6b6a..56addb7b55a59 100644 --- a/site2/website/versioned_docs/version-2.10.1/schema-manage.md +++ b/site2/website/versioned_docs/version-2.10.1/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.10.1/schema-understand.md b/site2/website/versioned_docs/version-2.10.1/schema-understand.md index 4fe190bd58bdb..55bc662c66633 100644 --- a/site2/website/versioned_docs/version-2.10.1/schema-understand.md +++ b/site2/website/versioned_docs/version-2.10.1/schema-understand.md @@ -479,7 +479,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java @@ -517,7 +517,7 @@ This diagram illustrates how does schema work on the Producer side. The schema instance defines the schema for the data being produced using the producer instance. - Take AVRO as an example, Pulsar extract schema definition from the POJO class and construct the `SchemaInfo` that the producer needs to pass to a broker when it connects. + Take AVRO as an example, Pulsar extracts schema definition from the POJO class and constructs the `SchemaInfo` that the producer needs to pass to a broker when it connects. 2. The producer connects to the broker with the `SchemaInfo` extracted from the passed-in schema instance. diff --git a/site2/website/versioned_docs/version-2.10.1/security-athenz.md b/site2/website/versioned_docs/version-2.10.1/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-athenz.md +++ b/site2/website/versioned_docs/version-2.10.1/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.10.1/security-authorization.md b/site2/website/versioned_docs/version-2.10.1/security-authorization.md index cc9c03d5fc1dd..9cfd7c8c203f6 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-authorization.md +++ b/site2/website/versioned_docs/version-2.10.1/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -117,7 +117,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.10.1/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.10.1/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.10.1/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.10.1/security-jwt.md b/site2/website/versioned_docs/version-2.10.1/security-jwt.md index b69c7bcb7b20a..fcee5c0ce21da 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-jwt.md +++ b/site2/website/versioned_docs/version-2.10.1/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.10.1/security-kerberos.md b/site2/website/versioned_docs/version-2.10.1/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.10.1/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.10.1/security-overview.md b/site2/website/versioned_docs/version-2.10.1/security-overview.md index ee12a589fcc48..d03b8c85c38f1 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.10.1/security-tls-authentication.md b/site2/website/versioned_docs/version-2.10.1/security-tls-authentication.md index 93f566939bfc2..17da5aab34d47 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.10.1/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.10.1/security-tls-keystore.md b/site2/website/versioned_docs/version-2.10.1/security-tls-keystore.md index 0216edd6d00b9..8a4654a0c33ae 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.10.1/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -184,7 +184,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a minimal configuration, you need to provide the TrustStore information. For example: -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -279,7 +279,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. For example: -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.10.1/security-tls-transport.md b/site2/website/versioned_docs/version-2.10.1/security-tls-transport.md index 670aff54d9c83..72bd0ae422cbf 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.10.1/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -147,7 +147,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -220,7 +220,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.10.1/security-token-admin.md b/site2/website/versioned_docs/version-2.10.1/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.10.1/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.10.1/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.10.1/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.10.1/sql-deployment-configurations.md index d8a7dc6ea06bc..0178ca41e017c 100644 --- a/site2/website/versioned_docs/version-2.10.1/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.10.1/sql-deployment-configurations.md @@ -142,7 +142,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.10.1/sql-overview.md b/site2/website/versioned_docs/version-2.10.1/sql-overview.md index 4a4d5f01b0bc5..8ba19d053003d 100644 --- a/site2/website/versioned_docs/version-2.10.1/sql-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.10.1/standalone.md b/site2/website/versioned_docs/version-2.10.1/standalone.md index c9a30801d4976..0a10f3ae21369 100644 --- a/site2/website/versioned_docs/version-2.10.1/standalone.md +++ b/site2/website/versioned_docs/version-2.10.1/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary [RocksDB](http://rocksdb.org/) and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -64,9 +64,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](/tools/pulsar-admin/). -`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
    **Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
    **Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration.md) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `bin/pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](reference-cli-tools.md#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.10.1/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.10.1/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.10.1/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.10.1/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.10.1/txn-monitor.md b/site2/website/versioned_docs/version-2.10.1/txn-monitor.md index 2ca2c887afb88..08e4a1be32036 100644 --- a/site2/website/versioned_docs/version-2.10.1/txn-monitor.md +++ b/site2/website/versioned_docs/version-2.10.1/txn-monitor.md @@ -7,4 +7,4 @@ original_id: txn-monitor You can monitor the status of the transactions in Prometheus and Grafana using the [transaction metrics](reference-metrics.md#pulsar-transaction). -For how to configure Prometheus and Grafana, see [here](deploy-monitoring). +For how to configure Prometheus and Grafana, see [here](deploy-monitoring.md).