PIP-209: Separate C++/Python clients to own repositories

[merlimat]

Motivation

Pulsar C++ code base is in the same main repository for the Pulsar project.

While the decision was the right one at the time, there is a considerable overhead
in keeping the C++ client in its current position.

Issues with the current approach

The Pulsar repository has grown a lot in size and number of active developers.

1. The frequency of changes in various parts of the codebase has increased to a
   point where the amount of resources dedicated to CI is very significant.

   Every change in Java code will trigger the CI jobs for the C++ client and every
   change in the C++ client will do the same.

   During a CI job we are building the C++ client multiple times:

   1. For C++ and Python client tests
   2. To build Python wheels to be included in the pulsar Docker images (for supporting
      Pulsar functions)

2. The release process for Pulsar has become very complex and requires building a
   large number of binaries for C++ and Python clients. This has become too much
   of a burden during the course of a Pulsar release.

Goal

Decouple the development of C++ and Python client libraries from the development
of the core components of Pulsar in Java.

API Changes

No response

Implementation

Changes

Repositories

1. Move the C++ client code to a new repository github.com/apache/pulsar-client-c++
2. Move the Python client code to a new repository github.com/apache/pulsar-client-python

The change will be done without losing any history, extracting a sub-directory into
a new Git repository.

git filter-repo --subdirectory-filter  pulsar-client-cpp

Release process

The release process will be split in multiple parts:

1. the main Pulsar release will only contain the Java parts (server distribution
   and Java client library)
2. The C++ client will have its own release schedule and versioning
3. The Python client will have its own release schedule and versioning

Versioning

Both C++ and Python clients will continue with their own individual versioning.

In order to not break anything or cause more confusion, we would need to use
a new version that is bigger than the current version (2.11.x).

The suggestion is to start the new releases for both C++ and Python from 3.0.0.

Existing branches

Existing branches of Pulsar, where the C++ client will still be in the same main
the repository and will be receiving bug fixes in their current location.

The different location of the new C++ code will make the cherry-picking process
slightly more painful in the short term, though it will even out in long term.

Projects dependencies

C++/Python --> Pulsar

Both C++ and Python unit/integration tests are designed to run against a standalone
instance of Pulsar broker. In the current form, they're using the master code
that is built to run the tests.

After the split, the unit tests will use a Docker image of Pulsar. We can use a few
different images to test for compatibility

1. Latest stable (eg: 2.10.1)
2. Nightly (Pulsar Docker image published every day from master branch)

Pulsar --> Python

To create a Pulsar image, we are now building the Python client wheel file and then
installing it at build time.

Instead, we are going to include a wheel file for a version of the Python client
that has been already released.

Python --> C++

The Python client library is just a wrapper on top of the C++ client. Today these
are built together, with Python wrapper code residing in a sub-directory of C++ client
code, and compiled using the same CMake build script.

By separating the Python client into a different repository, we are going to
depend on an already released version of the C++ client.

Automated documentation in the website

On the Pulsar website we are auto-generating C++ documentation with the Doxygen
tool and the Python one with Pdoc.

Instead of just fetching the main repo code, the website build job should be
also fetching the new repos to run the tooling.

Alternatives

No response

Anything else?

No response

[tisonkun]

There're still several works to complete PIP-209. For example, dangling references, release processes, and doc gen processes. I will create issues for all of them and try to pick up some.

[tisonkun]

@merlimat please add these issues to the description as a checklist so that we can properly track subtasks.

• Cleanup or migrate all reference to pulsar-client-cpp and the python client #17916
• [Doc] Migrate C++ and Python client API docs generate process #17917
• [Doc] Update the release process about how to release C++ and Python client #17918

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[tisonkun]

Closed as resolved. Doc updates is tracked by #17918.

Ping once more @BewareMyPower :)