Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add shared support for MongoDB Dev Services container #39787

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ public class GlobalDevServicesConfig {
boolean enabled;

/**
* Global flag that can be used to force the attachmment of Dev Services to shared netxork. Default is false.
* Global flag that can be used to force the attachmment of Dev Services to shared network. Default is false.
*/
@ConfigItem(defaultValue = "false")
public boolean launchOnSharedNetwork;
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/dev-services.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ include::{generated-dir}/config/quarkus-kubernetes-client-config-group-kubernete

The MongoDB Dev Service will be enabled when the `quarkus-mongodb-client` extension is present in your application, and
the server address has not been explicitly configured. More information can be found in the
xref:mongodb.adoc#dev-services[MongoDB Guide].
xref:mongodb-dev-services.adoc[MongoDB Guide].

include::{generated-dir}/config/quarkus-mongodb-config-group-dev-services-build-time-config.adoc[opts=optional, leveloffset=+1]

Expand Down
30 changes: 30 additions & 0 deletions docs/src/main/asciidoc/mongodb-dev-services.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
////
This guide is maintained in the main Quarkus repository
and pull requests should be submitted there:
https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
////
= Dev Services for MongoDB

Check warning on line 6 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Headings] Use sentence-style capitalization in 'Dev Services for MongoDB'. Raw Output: {"message": "[Quarkus.Headings] Use sentence-style capitalization in 'Dev Services for MongoDB'.", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 6, "column": 3}}}, "severity": "INFO"}

Quarkus supports a feature called Dev Services that allows you to create various datasources without any config. In the case of MongoDB this support extends to the default MongoDB connection.
What that means practically, is that if you have not configured `quarkus.mongodb.connection-string`, Quarkus will automatically start a MongoDB container when running tests or in dev mode,
and automatically configure the connection.

MongoDB Dev Services is based on link:https://www.testcontainers.org/modules/databases/mongodb/[Testcontainers MongoDB module] that will start a single node replicaset.

Check warning on line 12 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Spelling] Use correct American English spelling. Did you really mean 'replicaset'? Raw Output: {"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'replicaset'?", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 12, "column": 158}}}, "severity": "WARNING"}

When running the production version of the application, the MongoDB connection need to be configured as normal, so if you want to include a production database config in your

Check warning on line 14 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer. Raw Output: {"message": "[Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer.", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 14, "column": 1}}}, "severity": "INFO"}

Check warning on line 14 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Fluff] Depending on the context, consider using 'Rewrite the sentence, or use 'must', instead of' rather than 'need to'. Raw Output: {"message": "[Quarkus.Fluff] Depending on the context, consider using 'Rewrite the sentence, or use 'must', instead of' rather than 'need to'.", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 14, "column": 80}}}, "severity": "INFO"}

Check warning on line 14 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.TermsSuggestions] Depending on the context, consider using 'because' or 'while' rather than 'as'. Raw Output: {"message": "[Quarkus.TermsSuggestions] Depending on the context, consider using 'because' or 'while' rather than 'as'.", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 14, "column": 102}}}, "severity": "INFO"}
`application.properties` and continue to use Dev Services we recommend that you use the `%prod.` profile to define your MongoDB settings.

Check warning on line 15 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Fluff] Depending on the context, consider using 'using more direct instructions' rather than 'recommend'. Raw Output: {"message": "[Quarkus.Fluff] Depending on the context, consider using 'using more direct instructions' rather than 'recommend'.", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 15, "column": 62}}}, "severity": "INFO"}


== Shared server

Most of the time you need to share the server between applications.

Check warning on line 20 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Fluff] Depending on the context, consider using 'Rewrite the sentence, or use 'must', instead of' rather than 'need to'. Raw Output: {"message": "[Quarkus.Fluff] Depending on the context, consider using 'Rewrite the sentence, or use 'must', instead of' rather than 'need to'.", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 20, "column": 22}}}, "severity": "INFO"}
Dev Services for MongoDB implements a _service discovery_ mechanism for your multiple Quarkus applications running in _dev_ mode to share a single server.

NOTE: Dev Services for MongoDB starts the container with the `quarkus-dev-service-mongodb` label which is used to identify the container.

Check warning on line 23 in docs/src/main/asciidoc/mongodb-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.TermsSuggestions] Depending on the context, consider using ', which (non restrictive clause preceded by a comma)' or 'that (restrictive clause without a comma)' rather than 'which'. Raw Output: {"message": "[Quarkus.TermsSuggestions] Depending on the context, consider using ', which (non restrictive clause preceded by a comma)' or 'that (restrictive clause without a comma)' rather than 'which'.", "location": {"path": "docs/src/main/asciidoc/mongodb-dev-services.adoc", "range": {"start": {"line": 23, "column": 97}}}, "severity": "INFO"}

If you need multiple (shared) servers, you can configure the `quarkus.mongodb.devservices.service-name` attribute and indicate the server name.
It looks for a container with the same value, or starts a new one if none can be found.
The default service name is `mongodb`.

Sharing is enabled by default in dev mode, but disabled in test mode.
You can disable the sharing with `quarkus.mongodb.devservices.shared=false`.
13 changes: 2 additions & 11 deletions docs/src/main/asciidoc/mongodb.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -238,23 +238,14 @@
If you need more configuration properties, there is a full list at the end of this guide.

WARNING: By default, Quarkus will restrict the use of JNDI within an application, as a precaution to try and mitigate any future vulnerabilities similar to Log4Shell.
Because the `mongo+srv` protocol often used to connect to MongoDB requires JNDI, this protection is automatically disabled when using the MongoDB client extension.

Check warning on line 241 in docs/src/main/asciidoc/mongodb.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.TermsSuggestions] Depending on the context, consider using 'by using' or 'that uses' rather than 'using'. Raw Output: {"message": "[Quarkus.TermsSuggestions] Depending on the context, consider using 'by using' or 'that uses' rather than 'using'.", "location": {"path": "docs/src/main/asciidoc/mongodb.adoc", "range": {"start": {"line": 241, "column": 128}}}, "severity": "INFO"}

[[dev-services]]
=== Dev Services (Configuration Free Databases)
=== Use the MongoDB Dev Services

Quarkus supports a feature called Dev Services that allows you to create various datasources without any config. In the case of MongoDB this support extends to the default MongoDB connection.
What that means practically, is that if you have not configured `quarkus.mongodb.connection-string`, Quarkus will automatically start a MongoDB container when running tests or in dev mode,
and automatically configure the connection.

MongoDB Dev Services is based on link:https://www.testcontainers.org/modules/databases/mongodb/[Testcontainers MongoDB module] that will start a single node replicaset.

When running the production version of the application, the MongoDB connection need to be configured as normal, so if you want to include a production database config in your
`application.properties` and continue to use Dev Services we recommend that you use the `%prod.` profile to define your MongoDB settings.

include::{generated-dir}/config/quarkus-mongodb-config-group-dev-services-build-time-config.adoc[opts=optional, leveloffset=+1]
See xref:mongodb-dev-services.adoc[MongoDB Dev Services].

== Multiple MongoDB Clients

Check warning on line 248 in docs/src/main/asciidoc/mongodb.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Headings] Use sentence-style capitalization in 'Multiple MongoDB Clients'. Raw Output: {"message": "[Quarkus.Headings] Use sentence-style capitalization in 'Multiple MongoDB Clients'.", "location": {"path": "docs/src/main/asciidoc/mongodb.adoc", "range": {"start": {"line": 248, "column": 4}}}, "severity": "INFO"}

MongoDB allows you to configure multiple clients.
Using several clients works the same way as having a single client.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ public class DevServicesBuildTimeConfig {
/**
* If DevServices has been explicitly enabled or disabled. DevServices is generally enabled
* by default, unless there is an existing configuration present.
*
* <p>
* When DevServices is enabled Quarkus will attempt to automatically configure and start
* a database when running in Dev or Test mode.
*/
Expand Down Expand Up @@ -48,4 +48,30 @@ public class DevServicesBuildTimeConfig {
@ConfigDocMapKey("environment-variable-name")
public Map<String, String> containerEnv;

/**
* Indicates if the MongoDB server managed by Quarkus Dev Services is shared.
* When shared, Quarkus looks for running containers using label-based service discovery.
* If a matching container is found, it is used, and so a second one is not started.
* Otherwise, Dev Services for MongoDB starts a new container.
* <p>
* The discovery uses the {@code quarkus-dev-service-mongodb} label.
* The value is configured using the {@code service-name} property.
* <p>
* Container sharing is only used in dev mode.
*/
@ConfigItem(defaultValue = "true")
public boolean shared;

/**
* The value of the {@code quarkus-dev-service-mongodb} label attached to the started container.
* This property is used when {@code shared} is set to {@code true}.
* In this case, before starting a container, Dev Services for MongoDB looks for a container with the
* {@code quarkus-dev-service-mongodb} label
* set to the configured value. If found, it will use this container instead of starting a new one. Otherwise it
* starts a new container with the {@code quarkus-dev-service-mongodb} label set to the specified value.
* <p>
*/
@ConfigItem(defaultValue = "mongodb")
public String serviceName;

}
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.function.Supplier;
import java.util.stream.Collectors;

import org.eclipse.microprofile.config.ConfigProvider;
Expand All @@ -36,7 +36,9 @@
import io.quarkus.deployment.dev.devservices.GlobalDevServicesConfig;
import io.quarkus.deployment.logging.LoggingSetupBuildItem;
import io.quarkus.devservices.common.ConfigureUtil;
import io.quarkus.devservices.common.ContainerLocator;
import io.quarkus.mongodb.runtime.MongodbConfig;
import io.quarkus.runtime.LaunchMode;
import io.quarkus.runtime.configuration.ConfigUtils;

@BuildSteps(onlyIfNot = IsNormal.class, onlyIf = GlobalDevServicesConfig.Enabled.class)
Expand All @@ -48,6 +50,18 @@ public class DevServicesMongoProcessor {
static volatile Map<String, CapturedProperties> capturedProperties;
static volatile boolean first = true;

private static final String MONGO_SCHEME = "mongodb://";

private static final int MONGO_EXPOSED_PORT = 27017;

/**
* Label to add to shared Dev Service for Mongo running in containers.
* This allows other applications to discover the running service and use it instead of starting a new instance.
*/
private static final String DEV_SERVICE_LABEL = "quarkus-dev-service-mongodb";

private static final ContainerLocator MONGO_CONTAINER_LOCATOR = new ContainerLocator(DEV_SERVICE_LABEL, MONGO_EXPOSED_PORT);

@BuildStep
public List<DevServicesResultBuildItem> startMongo(List<MongoConnectionNameBuildItem> mongoConnections,
DockerStatusBuildItem dockerStatusBuildItem,
Expand Down Expand Up @@ -96,7 +110,7 @@ public List<DevServicesResultBuildItem> startMongo(List<MongoConnectionNameBuild
boolean useSharedNetwork = DevServicesSharedNetworkBuildItem.isSharedNetworkRequired(globalDevServicesConfig,
devServicesSharedNetworkBuildItem);
devService = startMongo(dockerStatusBuildItem, connectionName, currentCapturedProperties.get(connectionName),
useSharedNetwork, globalDevServicesConfig.timeout);
useSharedNetwork, globalDevServicesConfig.timeout, launchMode.getLaunchMode());
if (devService == null) {
compressor.closeAndDumpCaptured();
} else {
Expand Down Expand Up @@ -136,7 +150,8 @@ public void run() {
}

private RunningDevService startMongo(DockerStatusBuildItem dockerStatusBuildItem, String connectionName,
CapturedProperties capturedProperties, boolean useSharedNetwork, Optional<Duration> timeout) {
CapturedProperties capturedProperties, boolean useSharedNetwork, Optional<Duration> timeout,
LaunchMode launchMode) {
if (!capturedProperties.devServicesEnabled) {
// explicitly disabled
log.debug("Not starting devservices for " + (isDefault(connectionName) ? "default datasource" : connectionName)
Expand All @@ -161,28 +176,51 @@ private RunningDevService startMongo(DockerStatusBuildItem dockerStatusBuildItem
+ " or get a working docker instance");
return null;
}
MongoDBContainer mongoDBContainer;
if (capturedProperties.imageName != null) {
mongoDBContainer = new QuarkusMongoDBContainer(
DockerImageName.parse(capturedProperties.imageName).asCompatibleSubstituteFor("mongo"),
capturedProperties.fixedExposedPort, useSharedNetwork);
} else {
mongoDBContainer = new QuarkusMongoDBContainer(capturedProperties.fixedExposedPort, useSharedNetwork);
}
timeout.ifPresent(mongoDBContainer::withStartupTimeout);
mongoDBContainer.withEnv(capturedProperties.containerEnv);
mongoDBContainer.start();
Optional<String> databaseName = ConfigProvider.getConfig().getOptionalValue(configPrefix + "database", String.class);
String effectiveURL = databaseName.map(mongoDBContainer::getReplicaSetUrl).orElse(mongoDBContainer.getReplicaSetUrl());

Supplier<RunningDevService> defaultMongoServerSupplier = () -> {
MongoDBContainer mongoDBContainer;
if (capturedProperties.imageName != null) {
mongoDBContainer = new QuarkusMongoDBContainer(
DockerImageName.parse(capturedProperties.imageName).asCompatibleSubstituteFor("mongo"),
capturedProperties.fixedExposedPort, useSharedNetwork);
} else {
mongoDBContainer = new QuarkusMongoDBContainer(capturedProperties.fixedExposedPort, useSharedNetwork);
}
timeout.ifPresent(mongoDBContainer::withStartupTimeout);
mongoDBContainer.withEnv(capturedProperties.containerEnv);
mongoDBContainer.start();
final String effectiveUrl = getEffectiveUrl(configPrefix, mongoDBContainer.getHost(),
mongoDBContainer.getMappedPort(MONGO_EXPOSED_PORT), capturedProperties);
return new RunningDevService(Feature.MONGODB_CLIENT.getName(), mongoDBContainer.getContainerId(),
mongoDBContainer::close, getConfigPrefix(connectionName) + "connection-string", effectiveUrl);
};

return MONGO_CONTAINER_LOCATOR
.locateContainer(capturedProperties.serviceName(), capturedProperties.shared(), launchMode)
.map(containerAddress -> {
final String effectiveUrl = getEffectiveUrl(configPrefix, containerAddress.getHost(),
containerAddress.getPort(), capturedProperties);

return new RunningDevService(Feature.MONGODB_CLIENT.getName(), containerAddress.getId(),
null, getConfigPrefix(connectionName) + "connection-string", effectiveUrl);
})
.orElseGet(defaultMongoServerSupplier);
}

private String getEffectiveUrl(String configPrefix, String host, int port, CapturedProperties capturedProperties) {
final String databaseName = ConfigProvider.getConfig()
.getOptionalValue(configPrefix + "database", String.class)
.orElse("test");
String effectiveUrl = String.format("%s%s:%d/%s", MONGO_SCHEME, host, port, databaseName);
if ((capturedProperties.connectionProperties != null) && !capturedProperties.connectionProperties.isEmpty()) {
effectiveURL = effectiveURL + "?"
effectiveUrl = effectiveUrl + "?"
+ URLEncodedUtils.format(
capturedProperties.connectionProperties.entrySet().stream()
.map(e -> new BasicNameValuePair(e.getKey(), e.getValue())).collect(Collectors.toList()),
.map(e -> new BasicNameValuePair(e.getKey(), e.getValue()))
.collect(Collectors.toList()),
StandardCharsets.UTF_8);
}
return new RunningDevService(Feature.MONGODB_CLIENT.getName(), mongoDBContainer.getContainerId(),
mongoDBContainer::close, getConfigPrefix(connectionName) + "connection-string", effectiveURL);
return effectiveUrl;
}

private String getConfigPrefix(String connectionName) {
Expand Down Expand Up @@ -212,48 +250,15 @@ private CapturedProperties captureProperties(String connectionName, MongoClientB
boolean devServicesEnabled = devServicesConfig.enabled.orElse(true);
return new CapturedProperties(databaseName, connectionString, devServicesEnabled,
devServicesConfig.imageName.orElseGet(() -> ConfigureUtil.getDefaultImageNameFor("mongo")),
devServicesConfig.port.orElse(null), devServicesConfig.properties, devServicesConfig.containerEnv);
devServicesConfig.port.orElse(null), devServicesConfig.properties, devServicesConfig.containerEnv,
devServicesConfig.shared, devServicesConfig.serviceName);
}

private static final class CapturedProperties {
private final String database;
private final String connectionString;
private final boolean devServicesEnabled;
private final String imageName;
private final Integer fixedExposedPort;
private final Map<String, String> connectionProperties;
private final Map<String, String> containerEnv;

public CapturedProperties(String database, String connectionString, boolean devServicesEnabled, String imageName,
Integer fixedExposedPort, Map<String, String> connectionProperties, Map<String, String> containerEnv) {
this.database = database;
this.connectionString = connectionString;
this.devServicesEnabled = devServicesEnabled;
this.imageName = imageName;
this.fixedExposedPort = fixedExposedPort;
this.connectionProperties = connectionProperties;
this.containerEnv = containerEnv;
}
private record CapturedProperties(String database, String connectionString, boolean devServicesEnabled,
String imageName, Integer fixedExposedPort,
Map<String, String> connectionProperties, Map<String, String> containerEnv,
boolean shared, String serviceName) {

@Override
public boolean equals(Object o) {
if (this == o)
return true;
if (o == null || getClass() != o.getClass())
return false;
CapturedProperties that = (CapturedProperties) o;
return devServicesEnabled == that.devServicesEnabled && Objects.equals(database, that.database)
&& Objects.equals(connectionString, that.connectionString) && Objects.equals(imageName, that.imageName)
&& Objects.equals(fixedExposedPort, that.fixedExposedPort)
&& Objects.equals(connectionProperties, that.connectionProperties)
&& Objects.equals(containerEnv, that.containerEnv);
}

@Override
public int hashCode() {
return Objects.hash(database, connectionString, devServicesEnabled, imageName, fixedExposedPort,
connectionProperties, containerEnv);
}
}

private static final class QuarkusMongoDBContainer extends MongoDBContainer {
Expand Down
Loading