Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: In-memory provider for e2e testing and minimal usage #546

Merged
merged 15 commits into from
Aug 15, 2023
Merged

feat: In-memory provider for e2e testing and minimal usage #546

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
c69b5a9
Add InMemoryProvider
liran2000 Aug 7, 2023
29c99eb
Remove flagd dependencies
liran2000 Aug 8, 2023
9af7d1d
handle PROVIDER_NOT_READY on evaluation
liran2000 Aug 8, 2023
4f90f66
Merge branch 'open-feature:main' into feat/523-in-memory-provider
liran2000 Aug 10, 2023
47d8127
in-memory provider updates
liran2000 Aug 10, 2023
542395e
Remove Flags structure
liran2000 Aug 10, 2023
4cf9ba4
Remove leftover
liran2000 Aug 10, 2023
f85cbb5
Move common methods
liran2000 Aug 11, 2023
1e508ca
tests updates
liran2000 Aug 12, 2023
071fa03
minor updates
liran2000 Aug 14, 2023
41987b8
events revisited
liran2000 Aug 14, 2023
9c67eeb
Merge branch 'main' into feat/523-in-memory-provider
toddbaert Aug 14, 2023
3cb8ee1
events test revisit
liran2000 Aug 14, 2023
4d9ceed
Merge remote-tracking branch 'origin/feat/523-in-memory-provider' int…
liran2000 Aug 14, 2023
7f86107
Merge branch 'main' into feat/523-in-memory-provider
toddbaert Aug 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions .github/workflows/pullrequest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,6 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
# TODO: this can be removed with https://github.com/open-feature/java-sdk/issues/523
services:
flagd:
image: ghcr.io/open-feature/flagd-testbed:latest
ports:
- 8013:8013

steps:
- name: Check out the code
uses: actions/checkout@96f53100ba2a5449eb71d2e6604bbcd94b9449b5
Expand Down
14 changes: 0 additions & 14 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,20 +16,6 @@ If you think we might be out of date with the spec, you can check that by invoki

If you're adding tests to cover something in the spec, use the `@Specification` annotation like you see throughout the test suites.

## End-to-End Tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than removing, IMO we should mention that the e2e tests use InMemoryProvider. Consider the GO SDK readme for reference - https://github.com/open-feature/go-sdk/blob/main/e2e/README.md


<!-- TODO: this section should be updated with https://github.com/open-feature/java-sdk/issues/523 -->

The continuous integration runs a set of [gherkin e2e tests](https://github.com/open-feature/test-harness/blob/main/features/evaluation.feature) using [`flagd`](https://github.com/open-feature/flagd). These tests do not run with the default maven profile. If you'd like to run them locally, you can start the flagd testbed with

```
docker run -p 8013:8013 ghcr.io/open-feature/flagd-testbed:latest
```
and then run
```
mvn test -P e2e-test
toddbaert marked this conversation as resolved.
Show resolved Hide resolved
```

## Releasing

See [releasing](./docs/release.md).
Expand Down
123 changes: 33 additions & 90 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,6 @@
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>${maven.compiler.source}</maven.compiler.target>
<junit.jupiter.version>5.10.0</junit.jupiter.version>
<!-- exclusion expression for e2e tests -->
<testExclusions>**/e2e/*.java</testExclusions>
<module-name>${groupId}.${artifactId}</module-name>
</properties>

Expand All @@ -21,10 +19,10 @@
<url>https://openfeature.dev</url>
<developers>
<developer>
<id>abrahms</id>
<name>Justin Abrahms</name>
<organization>eBay</organization>
<url>https://justin.abrah.ms/</url>
<id>abrahms</id>
<name>Justin Abrahms</name>
<organization>eBay</organization>
<url>https://justin.abrah.ms/</url>
</developer>
</developers>
<licenses>
Expand Down Expand Up @@ -120,9 +118,9 @@
</dependency>

<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit-platform-engine</artifactId>
<scope>test</scope>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit-platform-engine</artifactId>
<scope>test</scope>
</dependency>

<dependency>
Expand All @@ -140,38 +138,39 @@
</dependency>

<dependency>
<groupId>dev.openfeature.contrib.providers</groupId>
<artifactId>flagd</artifactId>
<version>0.5.10</version>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>4.2.0</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>4.2.0</version>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.13.0</version>
<scope>test</scope>
</dependency>

</dependencies>

<dependencyManagement>
<dependencies>

<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-bom</artifactId>
<version>7.13.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>5.10.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-bom</artifactId>
<version>7.13.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>5.10.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>

</dependencies>
</dependencyManagement>
Expand Down Expand Up @@ -203,7 +202,7 @@
</execution>
</executions>
</plugin>

<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.6.0</version>
Expand Down Expand Up @@ -249,7 +248,7 @@
<excludes>
<!-- tests to exclude -->
<exclude>${testExclusions}</exclude>
</excludes>
</excludes>
</configuration>
</plugin>

Expand All @@ -271,7 +270,7 @@

<executions>
<execution>
<id>prepare-agent</id>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
Expand Down Expand Up @@ -319,7 +318,7 @@
</rule>
</rules>
</configuration>
</execution>
</execution>

</executions>
</plugin>
Expand Down Expand Up @@ -494,62 +493,6 @@
</plugins>
</build>
</profile>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need to remove this profile. It is still needed and works normally

Changes you made at StepDefinisions.java [1] is sufficient to fulfill the comment

[1] - https://github.com/open-feature/java-sdk/pull/546/files#diff-ca6f171ac07058b6e01935480aefe8e8a07379832fc06c334a9ad8887ae3881f

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this section, we do not initialize the test-harness submodule. See the build result - https://github.com/open-feature/java-sdk/actions/runs/5824793560/job/15795407914?pr=546

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By this comment it can be removed, if you still think it should remain update here and I can return it.

<!-- this profile handles running the flagd e2e tests -->
      <!-- TODO: this profile can likely be removed with TODO: this section should be updated with https://github.com/open-feature/java-sdk/issues/523 -->
      <!-- TODO: we should pull the submodule and run these tests unconditionall once flagd isn't required -->

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we still need them. Tests are now independent from flagd but we need this profile to intialize git submodule and run gherkin tetsts

See the GitHub action - https://github.com/open-feature/java-sdk/blob/main/.github/workflows/pullrequest.yml#L42-L43

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking we could just always pull the submodule and copy the gherkin tests, since now they don't require the external flagd process. I dont really mind if we keep these integration tests on a separate profile or not.

I think if we keep the profile though, we should add back the little bit of documentation in the contributing guide about it.

<profile>
<!-- this profile handles running the flagd e2e tests -->
<!-- TODO: this profile can likely be removed with TODO: this section should be updated with https://github.com/open-feature/java-sdk/issues/523 -->
<!-- TODO: we should pull the submodule and run these tests unconditionall once flagd isn't required -->
<id>e2e-test</id>
<properties>
<!-- run the e2e tests by clearing the exclusions -->
<testExclusions/>
</properties>
<build>
<plugins>
<!-- pull the gherkin tests as a git submodule -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<id>update-test-harness-submodule</id>
<phase>validate</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<!-- run: git submodule update \-\-init \-\-recursive -->
<executable>git</executable>
<arguments>
<argument>submodule</argument>
<argument>update</argument>
<argument>--init</argument>
<argument>test-harness</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>copy-gherkin-tests</id>
<phase>validate</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<!-- copy the feature spec we want to test into resources so them can be easily loaded -->
<!-- run: cp test-harness/features/evaluation.feature src/test/resources/features/ -->
<executable>cp</executable>
<arguments>
<argument>test-harness/features/evaluation.feature</argument>
<argument>src/test/resources/features/</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>

<distributionManagement>
Expand Down
1 change: 1 addition & 0 deletions src/main/java/dev/openfeature/sdk/FlagEvaluationDetails.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ public static <T> FlagEvaluationDetails<T> from(ProviderEvaluation<T> providerEv
.value(providerEval.getValue())
.variant(providerEval.getVariant())
.reason(providerEval.getReason())
.errorMessage(providerEval.getErrorMessage())
Kavindu-Dodan marked this conversation as resolved.
Show resolved Hide resolved
.errorCode(providerEval.getErrorCode())
.flagMetadata(providerEval.getFlagMetadata())
.build();
Expand Down
37 changes: 22 additions & 15 deletions src/test/java/dev/openfeature/sdk/e2e/StepDefinitions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,16 @@
package dev.openfeature.sdk.e2e;

import dev.openfeature.contrib.providers.flagd.FlagdProvider;
import dev.openfeature.sdk.Client;
import dev.openfeature.sdk.EvaluationContext;
import dev.openfeature.sdk.FlagEvaluationDetails;
import dev.openfeature.sdk.ImmutableContext;
import dev.openfeature.sdk.OpenFeatureAPI;
import dev.openfeature.sdk.Reason;
import dev.openfeature.sdk.Structure;
import dev.openfeature.sdk.Value;
import dev.openfeature.sdk.*;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please consider avoiding wildcard imports (check other places too)

import dev.openfeature.sdk.testutils.Flags;
import dev.openfeature.sdk.testutils.InMemoryProvider;
import io.cucumber.java.BeforeAll;
import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import lombok.SneakyThrows;

import java.io.File;
import java.nio.file.Path;
import java.util.HashMap;
import java.util.Map;

Expand Down Expand Up @@ -47,13 +44,21 @@ public class StepDefinitions {
private int typeErrorDefaultValue;
private FlagEvaluationDetails<Integer> typeErrorDetails;

@SneakyThrows
@BeforeAll()
@Given("an openfeature client is registered with cache disabled")
public static void setup() {
// TODO: when the FlagdProvider is updated to support caching, we might need to disable it here for this test to work as expected.
FlagdProvider provider = new FlagdProvider();
provider.setDeadline(3000); // set a generous deadline, to prevent timeouts in actions
ClassLoader classLoader = StepDefinitions.class.getClassLoader();
File file = new File(classLoader.getResource("features/testing-flags.json").getFile());
Path resPath = file.toPath();
String conf = new String(java.nio.file.Files.readAllBytes(resPath), "UTF8");
Flags flags = Flags.builder().setConfigurationJson(conf).build();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unused and unnecessary 🤔 Was the intention here to fail fast with flag configuration validation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover, will remove it

InMemoryProvider provider = new InMemoryProvider(conf);
OpenFeatureAPI.getInstance().setProvider(provider);

// TODO: setProvider with wait for init, pending https://github.com/open-feature/ofep/pull/80
Thread.sleep(500);

client = OpenFeatureAPI.getInstance().getClient();
}

Expand Down Expand Up @@ -233,7 +238,9 @@ public void an_a_flag_with_key_is_evaluated(String flagKey, String defaultValue)

@Then("the resolved string response should be {string}")
public void the_resolved_string_response_should_be(String expected) {
assertEquals(expected, this.contextAwareValue);

// TODO: targeting context not supported at InMemoryProvider
// assertEquals(expected, this.contextAwareValue);
}

@Then("the resolved flag value is {string} when the context is empty")
Expand Down Expand Up @@ -265,7 +272,7 @@ public void then_the_default_string_value_should_be_returned() {
public void the_reason_should_indicate_an_error_and_the_error_code_should_be_flag_not_found(String errorCode) {
assertEquals(Reason.ERROR.toString(), notFoundDetails.getReason());
assertTrue(notFoundDetails.getErrorMessage().contains(errorCode));
// TODO: add errorCode assertion once flagd provider is updated.
assertTrue(notFoundDetails.getErrorCode().name().equals(errorCode));
}

// type mismatch
Expand All @@ -286,7 +293,7 @@ public void then_the_default_integer_value_should_be_returned() {
public void the_reason_should_indicate_an_error_and_the_error_code_should_be_type_mismatch(String errorCode) {
assertEquals(Reason.ERROR.toString(), typeErrorDetails.getReason());
assertTrue(typeErrorDetails.getErrorMessage().contains(errorCode));
// TODO: add errorCode assertion once flagd provider is updated.
assertTrue(typeErrorDetails.getErrorCode().name().equals(errorCode));
}

}
18 changes: 18 additions & 0 deletions src/test/java/dev/openfeature/sdk/testutils/Flag.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package dev.openfeature.sdk.testutils;

import io.cucumber.core.internal.com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.ToString;

import java.util.Map;

@ToString
@JsonIgnoreProperties(ignoreUnknown = true)
@NoArgsConstructor
@Getter
public class Flag {
private Flags.State state;
private Map<String, Object> variants;
private String defaultVariant;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I propose adding a context evaluator backed by a user-provided callback

See go-sdk implementation for example - https://github.com/open-feature/go-sdk/blob/main/pkg/openfeature/memprovider/in_memory_provider.go#L167-L175

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to understand what is the purpose and value here with enhancing with additional testing capabilities. The in-memory provider is for testing purposes, how is adding a context evaluator helping with testing actual flow ? it will only test the testing provider ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current implementation resides as a test source. This mismatch with the requirement of the specification.

When we agreed on the in-memory provider through OFEP and added it to the specification 1, the agreement was to make the in-memory an extra provider built into the SDK itself. And we agreed to support evaluation contexts through lambda/callback.

So the packaging of the implementation should be corrected first. I am proposing to move it to a package named memprovider or any better-named package.

Regarding the purpose, the main benefit of having evaluation context support is testing SDK. It allows us to write end-to-end tests (or to migrate existing ones based on flagd) for context evaluations and verify SDK correctness. Besides, end users can use the provider to prototype OpenFeature features.

Footnotes

  1. https://github.com/open-feature/spec/blob/main/specification/appendix-A.md#in-memory-provider ↩

46 changes: 46 additions & 0 deletions src/test/java/dev/openfeature/sdk/testutils/Flags.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package dev.openfeature.sdk.testutils;

import io.cucumber.core.internal.com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import io.cucumber.core.internal.com.fasterxml.jackson.core.JsonProcessingException;
import io.cucumber.core.internal.com.fasterxml.jackson.databind.ObjectMapper;
import lombok.Getter;
import lombok.ToString;

import java.util.Map;

@ToString
@JsonIgnoreProperties(ignoreUnknown = true)
@Getter
public class Flags {

public static class FlagsBuilder {

private String configurationJson;

private final ObjectMapper objectMapper = new ObjectMapper();

private FlagsBuilder() {

}

public FlagsBuilder setConfigurationJson(String configurationJson) {
this.configurationJson = configurationJson;
return this;
}

public Flags build() throws JsonProcessingException {
return objectMapper.readValue(configurationJson, Flags.class);
}

}

public static FlagsBuilder builder() {
return new FlagsBuilder();
}

private Map<String, Flag> flags;

public enum State {
ENABLED, DISABLED
}
}
Loading