Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(domain-events): create env variable to control number of partitions for topics #947

Merged
merged 1 commit into from
Nov 2, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
### Purpose
_Describe the purpose of this pull request. Why is this change necessary? What problem does it solve?_

### Approach
_How does this change fulfill the purpose? Provide a high-level overview of the technical approach taken to address the problem._

### Changes Checklist
- [ ] **API Changes**: List any API paths, methods, request or response bodies changed, added, or removed.
- [ ] **Database Schema Changes**: Specify any database schema changes and their impact.
- [ ] **Interface Version Changes**: Indicate any changes to interface versions.
- [ ] **Interface Dependencies**: Note added or removed dependencies.
- [ ] **Permissions**: Document any changes to permissions.
- [ ] **Logging**: Confirm that logging is appropriately handled.
- [ ] **NEWS**: Ensure that the `NEWS` file is updated with relevant information about the changes made in this pull request.

### Related Issues
_List any Jira issues related to this pull request._

### Learning and Resources (if applicable)
_Discuss any research conducted during the development of this pull request. Include links to relevant blog posts, patterns, libraries, or addons that were used to solve the problem._

### Screenshots (if applicable)
_If this pull request involves any visual changes or new features, consider including screenshots or GIFs to illustrate the changes._
54 changes: 50 additions & 4 deletions README.MD
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,51 @@ Copyright (C) 2016-2023 The Open Library Foundation
This software is distributed under the terms of the Apache License,
Version 2.0. See the file "[LICENSE](LICENSE)" for more information.

<!-- TOC -->
* [mod-inventory-storage](#mod-inventory-storage)
* [Goal](#goal)
* [Prerequisites](#prerequisites)
* [Preparation](#preparation)
* [Git Submodules](#git-submodules)
* [Postgres](#postgres)
* [Kafka](#kafka)
* [Building](#building)
* [Environment Variables](#environment-variables)
* [Local Deployment using Docker](#local-deployment-using-docker)
* [Preparation](#preparation-1)
* [Start the infrastructure](#start-the-infrastructure)
* [Start the Module](#start-the-module)
* [Stop the Module](#stop-the-module)
* [Stop the infrastructure](#stop-the-infrastructure)
* [Local Deployment using Homebrew on macOS](#local-deployment-using-homebrew-on-macos)
* [Running](#running)
* [Preparation](#preparation-2)
* [Running Okapi](#running-okapi)
* [Registration](#registration)
* [Tenant Initialization](#tenant-initialization)
* [Sample Data](#sample-data)
* [Making Requests](#making-requests)
* [Okapi Root Address](#okapi-root-address)
* [Operating System Support](#operating-system-support)
* [Additional Information](#additional-information)
* [Issue tracker](#issue-tracker)
* [ModuleDescriptor](#moduledescriptor)
* [API documentation](#api-documentation)
* [Code analysis](#code-analysis)
* [Download and configuration](#download-and-configuration)
* [Appendix 1 - Docker Information](#appendix-1---docker-information)
* [When Using the Modules as Docker Containers](#when-using-the-modules-as-docker-containers)
* [Finding a Routable Address](#finding-a-routable-address)
* [Batch interface](#batch-interface)
* [HRID Management](#hrid-management)
* [Inventory view endpoint](#inventory-view-endpoint)
* [Domain event pattern](#domain-event-pattern)
* [Domain events for items](#domain-events-for-items)
* [Domain events for delete all APIs](#domain-events-for-delete-all-apis)
* [Reindex of instances](#reindex-of-instances)
* [Iteration of instances](#iteration-of-instances)
<!-- TOC -->

# Goal

FOLIO compatible inventory storage module.
Expand Down Expand Up @@ -46,18 +91,18 @@ Mod-inventory-storage implements domain event pattern and requires kafka to be l
`KAFKA_HOST` environment variables.

For production deployments it is also required to set `REPLICATION_FACTOR`
env variable, this property has following description:
env variable, this property has the following description:
> The replication factor controls how many servers will replicate each message that is written.
> If replication factor set to 3 then up to 2 servers can fail before access to the data will be lost.

The default configuration for this property is `1` for production environments
it is usually `3`.

There is another important property - `number of partitions` for a topic - it has following
There is another important property - `number of partitions` for a topic - it has the following
description:
> The partition count controls how many logs the topic will be sharded into.

This property has fixed value - `50`.
This property can be changed by setting `KAFKA_DOMAIN_TOPIC_NUM_PARTITIONS` env variable. Default value - `50`.

# Building

Expand All @@ -71,6 +116,7 @@ These environment variables configure Kafka, for details see [Kafka](#kafka):
* `KAFKA_PORT`
* `KAFKA_HOST`
* `REPLICATION_FACTOR`
* `KAFKA_DOMAIN_TOPIC_NUM_PARTITIONS`

mod-inventory-storage also supports all Raml Module Builder (RMB) environment variables,
for details see [RMB](https://github.com/folio-org/raml-module-builder#environment-variables):
Expand Down Expand Up @@ -288,7 +334,7 @@ Changing the start number to a number lower than the current number is not suppo
likely lead to generation of duplicate HRIDs. If an inventory type is added that contains a
duplicate HRID, the module will reject the submission.

# /inventory-view/instances
# Inventory view endpoint

Running a query against the `/inventory-view/instances` API writes this log message:

Expand Down
9 changes: 9 additions & 0 deletions src/main/java/org/folio/InventoryKafkaTopic.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package org.folio;

import org.apache.commons.lang3.StringUtils;
import org.folio.kafka.services.KafkaTopic;

public enum InventoryKafkaTopic implements KafkaTopic {
Expand All @@ -26,4 +27,12 @@ public String moduleName() {
public String topicName() {
return topic;
}

@Override
public int numPartitions() {
return Integer.parseInt(StringUtils.firstNonBlank(
System.getenv("KAFKA_DOMAIN_TOPIC_NUM_PARTITIONS"),
System.getProperty("KAFKA_DOMAIN_TOPIC_NUM_PARTITIONS"),
System.getProperty("kafka-domain-topic-num-partitions"), "50"));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,6 @@ public void canGetAllInstancesReindexJobs() {

var jobs = instanceReindex.getReindexJobs();

assertThat(jobs.getReindexJobs().get(0).getPublished(), is(0));
assertThat(jobs.getReindexJobs().get(0).getJobStatus(), is(IDS_PUBLISHED));
assertThat(jobs.getTotalRecords(), notNullValue());

Expand Down
1 change: 1 addition & 0 deletions src/test/java/org/folio/rest/api/StorageTestSuite.java
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ public static void before() {

// tests expect English error messages only, no Danish/German/...
Locale.setDefault(Locale.US);
System.setProperty("KAFKA_DOMAIN_TOPIC_NUM_PARTITIONS", "1");

PostgresClient.setPostgresTester(new PostgresTesterContainer());
startKafka();
Expand Down