Skip to content

Commit

Permalink
Merge with master of aiven/karapace to our repository (#8)
Browse files Browse the repository at this point in the history 
* tests/test_schema.py: splitting test_schema()

Split test_schema() to multiple single-purpose tests
No essential functional changes in the tests

* Added information how to run integration tests against Confluence stack

Instructions in README.rst
Docker Compose file to start the Confluence stack

* Kafka REST fixed version to 6.1.1 to match Schema Registry

* README.rst: clarified compatibility

Changed the claim that Karapace is compatible to that aims to be compatible with 6.1.1 and added a list of known incompabilities.

* Configuration Keys as table

* fixed content table

* Fixed small spelling bugs

* test_schema.py removed assert_schema_versions from test_schema_repost, unrelated

* test_schema.py added -> None to all test method signatures.

* test_schema.py: added annotations to all functions

* test_schema.py duplicate code removal

* test_schema.py moved a comment to a an assert message

* test_schema.py removed unneeded f-string wrappings

* utils.py AVRO name compatible (http://avro.apache.org/docs/current/spec.html#names). Must not have '-'.

* test_schema.py test_schema_version_numbering uses 'name' in the Avro to make the schema unique

* test_schema.py: str format() -> f-strings

* test_schema.py no more JSONs as strings, instead dicts that are dumped as JSON strings

* utils.py add create_schema_name_factory, create safer names

For example in Avro field names '-' is not allowed. Using underscore instead.

* test_schema.py: split test_schema_versions into two tests

New ones: test_schema_versions_multiple_subjects_same_schema and test_schema_versions_deleting

The tests use unique schema names

* test_schema.py: test_schema_remains_constant fixes

Wasn't using a unique schema id.
Added doc

* test_schema.py removed test_enum_schema_compatibility

Essentially a duplicate of test_enum_schema

* test_schema.py: fix test_schema_repost

Compares JSONs now, not strings.

* test_schema.py test_compatibility_endpoint fix

Now uses a dynamic unique schema name. Was clashing before.
Added documentation on what the test does.

* test_schema.py test_record_schema_compatibility_backward split into two

The new ones: test_record_schema_compatibility_backward and test_record_schema_compatibility_forward

* test_schema_version_number_existing_schema takes version ids from response

Now compatible with SR

* test_schema.py: test_schema_subject_version_schema fix

Changed to use a proper Avro schema

* test_schema.py: test_schema_same_subject fix

No longer expects the exact same string schema to be returned.
The str parsed as JSON needs to match.

* Handle gracefully if no node is master eligible

Karapace configuration allows configuring node to not be eligible for
master.  Handle gracefully ie. read-only mode if all nodes are
configured non-eligible for master.

* schema_registry: breaking change in an error message

The error message in POST to /subject/<subject> when schema is not specified in the request changed.

Fixes test_schema_subject_post_invalid to run in Karapace and against Schema Registry

* schema_registry: breaking change in subjects/{}/versions/{}

Fixed the error message in subjects/{}/versions/{} to match Schema Registry

Now test_schema_subject_invalid_id works against SR

* test_schema.py test_version_number_validation fix

Error message check matches the error from SR (was breaking the test)
Dynamically fetches the version number
Added description for the test

* Add some typing, rename eligible master flag for clarification

* schema_registry: breaking change in POST subjects/{subject}/versions

In the case the endpoint is submitted without body, changed
the HTTP status code, error_code and message match the ones in Schema Registry.
Made the necessary changes so that Karapace also returns correct values.

test_schema.py: test_schema_missing_body fixed accordingly.

* schema_registry: breaking changes in some HTTP error messages

Now HTTP error messages match with the ones coming from Schema Registry.
Adjusted test_http_headers in test_schema.py to correctly check the messages.

* schema_registry: breaking change in /schemas/ids/<>/versions

/schemas/ids/<schema_id:path>/versions now returns empty list in case nothing is found.
This is the behaviour of SR. Karapace used to fail in this case before this change.

The tests test_schema_lifecycle and test_schema_versions_deleting now works against Schema Registry (in addition to Karapace)

* test_schema.py: test_schema_versions_deleting: No unique field

Unique field name not needed, schema name is enough. Using a fixed one.

* readme: clarified and separated readme

moved documentation about development to the CONTRIBUTING.md file, and
tried to make the README.rst a bit more concise.

* Remove explicit master eligibility flag and utilize optional master_url

* CONTRIBUTING.md small fixes

Only minor changes, no essential content change:

Changed some rst formattings to md
Some typos fixed such as karapace -> Karapace
A few small tweaks

* doc: fixed grammar

* KarapaceAll: startup fix

When started from KarapaceAll, the __init__ of KarapaceSchemaRegistry is not called.
schema_lock is initialized in __init__. Thus it's not called when using KarapaceAll.

Fix is to move schema_lock init to _init() which gets called also when using KarapaceAll.

* docs: locahost -> localhost

Co-authored-by: Juha Mynttinen <[email protected]>
Co-authored-by: Francesco <[email protected]>
Co-authored-by: Tommi Vainikainen <[email protected]>
Co-authored-by: Augusto Hack <[email protected]>
  • Loading branch information
@libretto @juha-aiven
@ftisiot @tvainika
5 people authored Jun 3, 2021
1 parent a9abaee commit 9ff4580
Show file tree
Hide file tree
Showing 10 changed files with 1,029 additions and 742 deletions.
101 changes: 93 additions & 8 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,102 @@
# Welcome!

Contributions are very welcome on Karapace. Please follow the guidelines:
Contributions are very welcome on Karapace. When contributing please keep this in mind:

- It's recommended to open an issue to discuss a feature before putting in a lot of effort.
- Open an issue to discuss new bigger features.
- Write code consistent with the project style and make sure the tests are passing.
- Stay in touch with us if we have follow up questions or requests for further changes.

- We use [GitHub Flow](https://guides.github.com/introduction/flow/), check that your main branch is up to date, and create a new branch for changes.
# Development

- Commit messages should describe the changes, not the filenames. Win our admiration by following the [excellent advice from Chris Beams](https://chris.beams.io/posts/git-commit/) when composing commit messages.
## Local Environment

- Choose a meaningful title for your pull request.
There is very little you need to get started coding for Karapace:

- The pull request description should focus on what changed and why.
- Use [one of the supported python versions](https://github.com/aiven/karapace/blob/master/setup.py)
documented in the `setup.py` classifiers.
- Create [a virtual environment](https://docs.python.org/3/tutorial/venv.html) and install the dev dependencies in it:

- Check that the tests pass (and add test coverage for your changes if appropriate).
```python
python -m venv <path_to_venv>
source <path_to_venv>/bin/activate
pip install -r ./requirements-dev.txt
pip install -e .
```

- Stay in touch with us if we have follow up questions or requests for further changes.
## Tests

Tests are written with the [pytest](https://docs.pytest.org/) framework, and All PRs are tested for
each supported Python version using [GitHub Flow](https://guides.github.com/introduction/flow/).

There are two flavors of tests, unit tests and integration tests:

- Unit: These are faster and very useful for quick iterations. They are usually testing pure
functions.
- Integration: Are slower but more complete. These tests run Karapace, ZooKeeper, and Kafka servers,
pytest's fixtures are used to start/stop these for you.

Both flavors run in parallel using [pytest-xdist](https://github.com/pytest-dev/pytest-xdist). New
tests should be engineered with this in mind:

- Don't reuse schema/subject/topic names
- Expect other clients to be interacting with the servers at the same time.

To run the tests use `make`. It will download Kafka to be used in the tests for you:

```sh
make unittest
make integrationtest
```

### PyCharm

If you want to run the tests from within the IDE, first download Kafka using `make fetch-kafka`, and
use the project root as the working directory.

### Compatibility tests

The integration tests can be configured to use an external REST (`--rest-url`), Registry
(`--registry-url`) and Kafka (`--kafka-bootstrap-servers`). These can be used to make sure the
tests conform to the Kafka REST or Schema Registry APIs, and then that Karapace conform to the
tests:

```sh
docker-compose -f ./tests/integration/confluent-docker-compose.yml up -d
pytest --kafka-bootstrap-servers localhost:9092 --registry-url http://localhost:8081 --rest-url http://localhost:8082/ tests/integration
```

## Static checking and Linting

The code is statically checked and formatted using [a few
tools](https://github.com/aiven/karapace/blob/master/requirements-dev.txt). To run these
automatically on each commit please enable the [pre-commit](https://pre-commit.com) hooks.
Alternatively you can run it manually with `make pre-commit`.

## Manual testing

To use your development code, you just need to set up a Kafka server and run Karapace from you
virtual environment:

```
docker-compose -f ./container/docker-compose.yml up -d kafka
karapace karapace.config.json
```

### Configuration

To see descriptions of configuration keys see our
[README](https://github.com/aiven/karapace#configuration-keys).

Each configuration key can be overridden with an environment variable prefixed with `KARAPACE_`,
exception being configuration keys that actually start with the `karapace` string. For example, to
override the `bootstrap_uri` config value, one would use the environment variable
`KARAPACE_BOOTSTRAP_URI`.

# Opening a PR

- Commit messages should describe the changes, not the filenames. Win our admiration by following
the [excellent advice from Chris Beams](https://chris.beams.io/posts/git-commit/) when composing
commit messages.
- Choose a meaningful title for your pull request.
- The pull request description should focus on what changed and why.
- Check that the tests pass (and add test coverage for your changes if appropriate).
Loading

0 comments on commit 9ff4580

Please sign in to comment.