Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature][doc]PIP-190: Generate 2.10.x/2.9.x/2.8.x docs #16925

Closed
wants to merge 18 commits into from
Closed
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
f5627ff
rename versioned_docs/version-2.10.0 to version-2.10.x
momo-jun Aug 3, 2022
487e320
Merge 2.10.1-specific docs from PR#15340 and #16410
momo-jun Aug 3, 2022
cba9c97
rename versioned_docs/version-2.9.0 to version-2.9.x
momo-jun Aug 4, 2022
b034618
Merge 2.9.1-specific docs from #11988 and 2.9.2-specific docs from #1…
momo-jun Aug 4, 2022
86571b5
Merge 2.9.3-specific docs from #15340 and #15818
momo-jun Aug 4, 2022
53dac7e
rename versioned_docs/version-2.8.0 to version-2.8.x and merge 2.8.3-…
momo-jun Aug 4, 2022
890f7c3
Merge 2.8.2-specific docs from #12641, #12504, #12448, and #12350
momo-jun Aug 4, 2022
8869ffe
Merge 2.8.2-specific docs from #12175, #11988, #11828, and #11681
momo-jun Aug 4, 2022
e18ce57
Merge 2.8.1-specific docs from #11509, #11512, #11739, #11810, and #1…
momo-jun Aug 4, 2022
c8cc7dc
Merge 2.8.1-specific docs from #11430, #11443, #11472, #11495, #11500
momo-jun Aug 4, 2022
2194d7a
Update versions.json
momo-jun Aug 4, 2022
1f02239
Add `.md` to fix 200+ broken links in 2.8.x docs
momo-jun Aug 5, 2022
260a554
Add `.md` to fix 200+ broken links in 2.9.x docs
momo-jun Aug 5, 2022
fe33323
Merge 2.8.4-specific docs from #15340, #16616, and #15818 to 2.8.x docs
momo-jun Aug 5, 2022
eb12bb4
Merge branch 'master' into PIP-190-2.10.x
momo-jun Aug 8, 2022
d4cb371
Merge recent docs changes in #16891, #16941, and #17004
momo-jun Aug 9, 2022
0800cdb
Merge running doc changes in #16746
momo-jun Aug 9, 2022
61c2b5a
update versioned_sidebars
momo-jun Aug 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ If you’re an experienced coder, review this page to learn the easiest way to a

## Get Started Now
<BlockLinks>
<BlockLink title="About Pulsar" url="/docs/2.10.0/concepts-overview/" />
<BlockLink title="Get Started" url="/docs/2.10.0/getting-started-standalone/" />
<BlockLink title="Install, Deploy, Upgrade" url="/docs/2.10.0/deploy-aws/" />
<BlockLink title="Pulsar for Developers" url="/docs/2.10.0/develop-tools/" />
<BlockLink title="How To" url="/docs/2.10.0/functions-develop/" />
<BlockLink title="References" url="/docs/2.10.0/reference-terminology/" />
<BlockLink title="About Pulsar" url="/docs/2.10.x/concepts-overview/" />
<BlockLink title="Get Started" url="/docs/2.10.x/getting-started-standalone/" />
<BlockLink title="Install, Deploy, Upgrade" url="/docs/2.10.x/deploy-aws/" />
<BlockLink title="Pulsar for Developers" url="/docs/2.10.x/develop-tools/" />
<BlockLink title="How To" url="/docs/2.10.x/functions-develop/" />
<BlockLink title="References" url="/docs/2.10.x/reference-terminology/" />
</BlockLinks>

## Navigation
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ Pulsar brokers are responsible for handling incoming messages from producers, di
|allowAutoTopicCreationType| The type of topic that is allowed to be automatically created.(partitioned/non-partitioned) |non-partitioned|
|allowAutoSubscriptionCreation| Enable subscription auto creation if a new consumer connected |true|
|defaultNumPartitions| The number of partitioned topics that is allowed to be automatically created if `allowAutoTopicCreationType` is partitioned |1|
|brokerDeleteInactiveTopicsEnabled| Enable the deletion of inactive topics. If topics are not consumed for some while, these inactive topics might be cleaned up. Deleting inactive topics is enabled by default. The default period is 1 minute. |true|
|brokerDeleteInactiveTopicsEnabled| Enable the deletion of inactive topics. If topics are not consumed for some while, these inactive topics might be cleaned up. Deleting inactive topics is enabled by default. The default period is 1 minute. <br />**Note:** When `brokerDeleteInactiveTopicsEnabled` is set to `true`, you need to ensure that `allowAutoTopicCreation` is also set to `true`. |true|
|brokerDeleteInactiveTopicsFrequencySeconds| How often to check for inactive topics |60|
| brokerDeleteInactiveTopicsMode | Set the mode to delete inactive topics. <li> `delete_when_no_subscriptions`: delete the topic which has no subscriptions or active producers. </li><li> `delete_when_subscriptions_caught_up`: delete the topic whose subscriptions have no backlogs and which has no active producers or consumers. </li>| `delete_when_no_subscriptions` |
| brokerDeleteInactiveTopicsMaxInactiveDurationSeconds | Set the maximum duration for inactive topics. If it is not specified, the `brokerDeleteInactiveTopicsFrequencySeconds` parameter is adopted. | N/A |
Expand Down Expand Up @@ -379,6 +379,20 @@ brokerServiceCompactionThresholdInBytes|If the estimated backlog size is greater
| strictBookieAffinityEnabled | Enable or disable the strict bookie isolation strategy. If enabled, <br /> - `bookie-ensemble` first tries to choose bookies that belong to a namespace's affinity group. If the number of bookies is not enough, then the rest bookies are chosen. <br /> - If namespace has no affinity group, `bookie-ensemble` only chooses bookies that belong to no region. If the number of bookies is not enough, `BKNotEnoughBookiesException` is thrown.| false |
|narExtractionDirectory | The extraction directory of the nar package. <br />Available for Protocol Handler, Additional Servlets, Entry Filter, Offloaders, Broker Interceptor. | System.getProperty("java.io.tmpdir") |

#### Configuration override for clients internal to broker

In 2.10.1 and later versions, you can configure some clients by using the appropriate prefix.

|Prefix|Description|
|---|---|
|brokerClient_| Configure **all** the broker's Pulsar Clients and Pulsar Admin Clients. These configurations are applied after hard coded configuration and before the above broker client configurations named above.|
|bookkeeper_| Configure the broker's BookKeeper clients used by managed ledgers and the BookkeeperPackagesStorage bookkeeper client. Takes precedence over most other configuration values.|

:::note

When running the function worker within the broker, these prefixed configurations do not apply to any of those clients. You must configure those clients using the `functions_worker.yml` file.

:::

#### Deprecated parameters of Broker
The following parameters have been deprecated in the `conf/broker.conf` file.
Expand Down Expand Up @@ -445,9 +459,13 @@ You can set the log level and configuration in the [log4j2.yaml](https://github
|log4j.appender.TRACEFILE.layout| org.apache.log4j.PatternLayout|
|log4j.appender.TRACEFILE.layout.ConversionPattern| %d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n|

> Note: 'topic' in log4j2.appender is configurable.
> - If you want to append all logs to a single topic, set the same topic name.
> - If you want to append logs to different topics, you can set different topic names.
:::note

'topic' in log4j2.appender is configurable.
- If you want to append all logs to a single topic, set the same topic name.
- If you want to append logs to different topics, you can set different topic names.

:::

## Log4j shell

Expand Down Expand Up @@ -491,7 +509,7 @@ You can set the log level and configuration in the [log4j2.yaml](https://github
|backlogQuotaCheckIntervalInSeconds| How often to check for topics that have reached the backlog quota. |60|
|backlogQuotaDefaultLimitBytes| The default per-topic backlog quota limit. Being less than 0 means no limitation. By default, it is -1. |-1|
|ttlDurationDefaultInSeconds|The default Time to Live (TTL) for namespaces if the TTL is not configured at namespace policies. When the value is set to `0`, TTL is disabled. By default, TTL is disabled. |0|
|brokerDeleteInactiveTopicsEnabled| Enable the deletion of inactive topics. If topics are not consumed for some while, these inactive topics might be cleaned up. Deleting inactive topics is enabled by default. The default period is 1 minute. |true|
|brokerDeleteInactiveTopicsEnabled| Enable the deletion of inactive topics. If topics are not consumed for some while, these inactive topics might be cleaned up. Deleting inactive topics is enabled by default. The default period is 1 minute. <br />**Note:** When `brokerDeleteInactiveTopicsEnabled` is set to `true`, you need to ensure that `allowAutoTopicCreation` is also set to `true`. |true|
|brokerDeleteInactiveTopicsFrequencySeconds| How often to check for inactive topics, in seconds. |60|
| maxPendingPublishRequestsPerConnection | Maximum pending publish requests per connection to avoid keeping large number of pending requests in memory | 1000|
|messageExpiryCheckIntervalInMinutes| How often to proactively check and purged expired messages. |5|
Expand Down Expand Up @@ -748,6 +766,14 @@ The following parameters have been deprecated in the `conf/standalone.conf` file
|tlsKeyFilePath |||
|tlsTrustCertsFilePath|||

#### Configuration Override For Clients Internal to WebSocket

In 2.10.1 and later versions, you can configure some clients by using the appropriate prefix.

|Prefix|Description|
|---|---|
|brokerClient_| Configure **all** the broker's Pulsar Clients. These configurations are applied after hard coded configuration and before the above brokerClient configurations named above.|

#### Deprecated parameters of WebSocket
The following parameters have been deprecated in the `conf/websocket.conf` file.

Expand Down Expand Up @@ -811,6 +837,16 @@ The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) can be config
|tokenAudienceClaim| The token audience "claim" name, e.g. "aud". It is used to get the audience from token. If it is not set, the audience is not verified. ||
| tokenAudience | The token audience stands for this broker. The field `tokenAudienceClaim` of a valid token need contains this parameter.| |
|haProxyProtocolEnabled | Enable or disable the [HAProxy](http://www.haproxy.org/) protocol. |false|
| numIOThreads | Number of threads used for Netty IO. <br />**Note:** This configuration is only available for 2.10.1 and later versions. | 2 * Runtime.getRuntime().availableProcessors() |
| numAcceptorThreads | Number of threads used for Netty Acceptor. <br />**Note:** This configuration is only available for 2.10.1 and later versions. | 1 |

#### Configuration Override For Clients Internal to Proxy

In 2.10.1 and later versions, you can configure some clients by using the appropriate prefix.

|Prefix|Description|
|---|---|
|brokerClient_| Configure **all** the proxy's Pulsar Clients. These configurations are applied after hard coded configuration and before the above brokerClient configurations named above.|

#### Deprecated parameters of Pulsar proxy
The following parameters have been deprecated in the `conf/proxy.conf` file.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ If you’re an experienced coder, review this page to learn the easiest way to a

## Get Started Now
<BlockLinks>
<BlockLink title="About Pulsar" url="/docs/2.8.0/concepts-overview/" />
<BlockLink title="Get Started" url="/docs/2.8.0/getting-started-standalone/" />
<BlockLink title="Install, Deploy, Upgrade" url="/docs/2.8.0/deploy-aws/" />
<BlockLink title="Pulsar for Developers" url="/docs/2.8.0/develop-tools/" />
<BlockLink title="How To" url="/docs/2.8.0/functions-develop/" />
<BlockLink title="References" url="/docs/2.8.0/reference-terminology/" />
<BlockLink title="About Pulsar" url="/docs/2.8.x/concepts-overview/" />
<BlockLink title="Get Started" url="/docs/2.8.x/getting-started-standalone/" />
<BlockLink title="Install, Deploy, Upgrade" url="/docs/2.8.x/deploy-aws/" />
<BlockLink title="Pulsar for Developers" url="/docs/2.8.x/develop-tools/" />
<BlockLink title="How To" url="/docs/2.8.x/functions-develop/" />
<BlockLink title="References" url="/docs/2.8.x/reference-terminology/" />
</BlockLinks>

## Navigation
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Pulsar brokers consist of two components:

[Brokers](reference-terminology.md#broker) can be managed via:

* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool
* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool
* The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API
* The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md)

Expand Down Expand Up @@ -166,7 +166,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su

But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values.

* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more.
* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more.
* In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint.

### Update dynamic configuration
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http

Clusters can be managed via:

* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool
* The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API
* The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md)

Expand Down Expand Up @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c
* The web service URL for the cluster
* A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster

You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster.
You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster.

> **No cluster metadata initialization through the REST API or the Java admin API**
>
Expand All @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \

```

You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance.
You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance.

### Get configuration

You can fetch the [configuration](reference-configuration) for an existing cluster at any time.
You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time.

````mdx-code-block
<Tabs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ Functions can be managed via the following methods.

Method | Description
---|---
**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin) tool.
**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool.
**REST API** |The `/admin/v3/functions` endpoint of the admin {@inject: rest:REST:/} API.
**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java).
**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md).

## Function resources

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of

Namespaces can be managed via:

* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool
* The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API
* The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md)

Expand Down Expand Up @@ -1273,7 +1273,7 @@ admin.namespaces().removeDeduplicationSnapshotInterval(namespace)

### Namespace isolation

You can use the [Pulsar isolation policy](administration-isolation) to allocate resources (broker and bookie) for a namespace.
You can use the [Pulsar isolation policy](administration-isolation.md) to allocate resources (broker and bookie) for a namespace.

### Unload namespaces from a broker

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sidebar_label: "Non-partitioned topics"
original_id: admin-api-non-partitioned-topics
---

For details of the content, refer to [manage topics](admin-api-topics).
For details of the content, refer to [manage topics](admin-api-topics.md).
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sidebar_label: "Non-Persistent topics"
original_id: admin-api-non-persistent-topics
---

For details of the content, refer to [manage topics](admin-api-topics).
For details of the content, refer to [manage topics](admin-api-topics.md).
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sidebar_label: "Partitioned topics"
original_id: admin-api-partitioned-topics
---

For details of the content, refer to [manage topics](admin-api-topics).
For details of the content, refer to [manage topics](admin-api-topics.md).
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sidebar_label: "Persistent topics"
original_id: admin-api-persistent-topics
---

For details of the content, refer to [manage topics](admin-api-topics).
For details of the content, refer to [manage topics](admin-api-topics.md).
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem';
````


Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants:
Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants:

* Admin roles
* Allowed clusters
Expand Down Expand Up @@ -108,7 +108,7 @@ admin.tenants().createTenant(tenantName, tenantInfo);

### Get configuration

You can fetch the [configuration](reference-configuration) for an existing tenant at any time.
You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time.

````mdx-code-block
<Tabs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -709,6 +709,49 @@ admin.topics().getMessageById(topic, ledgerId, entryId);
</Tabs>
````

### Examine messages
You can examine a specific message on a topic by position relative to the earliest or the latest message.

:::note

This REST API is only available in 2.8.1 and later versions.

:::

````mdx-code-block
<Tabs
defaultValue="pulsar-admin"
values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
<TabItem value="pulsar-admin">

```shell

$ ./bin/pulsar-admin topics examine-messages \
persistent://public/default/my-topic \
-i latest -m 1

```

</TabItem>
<TabItem value="REST API">

{@inject: endpoint|GET|/admin/v2/:schema/:tenant/:namespace/:topic|operation/examineMessage?version=@pulsar:version_number@}

</TabItem>
<TabItem value="Java">

```java

String topic = "persistent://my-tenant/my-namespace/my-topic";
admin.topics().examineMessage(topic, "latest", 1);

```

</TabItem>

</Tabs>
````

### Skip messages

You can skip a number of messages for a specific subscription of a given topic in the following ways.
Expand Down Expand Up @@ -1309,7 +1352,7 @@ admin.topics().removeOffloadPolicies(topic)


## Manage non-partitioned topics
You can use Pulsar [admin API](admin-api-overview) to create, delete and check status of non-partitioned topics.
You can use Pulsar [admin API](admin-api-overview.md) to create, delete and check status of non-partitioned topics.

### Create
Non-partitioned topics must be explicitly created. When creating a new non-partitioned topic, you need to provide a name for the topic.
Expand Down Expand Up @@ -1504,7 +1547,7 @@ admin.topics().getStats(topic, false /* is precise backlog */);
````

## Manage partitioned topics
You can use Pulsar [admin API](admin-api-overview) to create, update, delete and check status of partitioned topics.
You can use Pulsar [admin API](admin-api-overview.md) to create, update, delete and check status of partitioned topics.

### Create

Expand Down Expand Up @@ -2022,7 +2065,7 @@ If a message has a key, it supersedes the round robin routing policy. The follow
```

## Manage subscriptions
You can use [Pulsar admin API](admin-api-overview) to create, check, and delete subscriptions.
You can use [Pulsar admin API](admin-api-overview.md) to create, check, and delete subscriptions.
### Create subscription
You can create a subscription for a topic using one of the following methods.
````mdx-code-block
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ original_id: administration-dashboard

:::note

Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager).
Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager.md).

:::

Expand Down
Loading