Skip to content
This repository has been archived by the owner on Apr 10, 2024. It is now read-only.

Existing kubeconfig file causes tests to fail #116

Closed
BramMeerten opened this issue Oct 16, 2023 · 2 comments · Fixed by #133
Closed

Existing kubeconfig file causes tests to fail #116

BramMeerten opened this issue Oct 16, 2023 · 2 comments · Fixed by #133

Comments

@BramMeerten
Copy link
Contributor

BramMeerten commented Oct 16, 2023 •
edited
Loading

When running a simple test, the waitUntilDefaultNamespaceAvailable check fails:

@EnableKubeAPIServer
class OutputPortCredentialsReconcilerTest {

    @KubeConfig
    static String kubeConfigYaml;

    @Test
    void testCommunication() {
        var client = new KubernetesClientBuilder()
                .withConfig(Config.fromKubeconfig(kubeConfigYaml))
                .build();
        client.resource(configMap()).createOrReplace();
        var cm = client.resource(configMap()).get();

        assertThat(cm).isNotNull();
    }

    private ConfigMap configMap() {
        return new ConfigMapBuilder()
                .withMetadata(new ObjectMetaBuilder()
                        .withName("test1")
                        .withNamespace("default")
                        .build())
                .withData(Map.of("key", "data"))
                .build();
    }
}
io.javaoperatorsdk.jenvtest.JenvtestException: Kube API Server did not start properly

	at io.javaoperatorsdk.jenvtest.process.ProcessReadinessChecker.pollWithTimeout(ProcessReadinessChecker.java:81)
	at io.javaoperatorsdk.jenvtest.process.ProcessReadinessChecker.waitUntilDefaultNamespaceAvailable(ProcessReadinessChecker.java:42)
	at io.javaoperatorsdk.jenvtest.process.KubeAPIServerProcess.waitUntilReady(KubeAPIServerProcess.java:94)
	at io.javaoperatorsdk.jenvtest.KubeAPIServer.start(KubeAPIServer.java:48)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.startApiServer(KubeAPIServerExtension.java:85)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.lambda$startIfAnnotationPresent$2(KubeAPIServerExtension.java:78)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.lambda$startIfAnnotationPresent$3(KubeAPIServerExtension.java:71)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.startIfAnnotationPresent(KubeAPIServerExtension.java:68)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.initialize(KubeAPIServerExtension.java:61)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.beforeAll(KubeAPIServerExtension.java:37)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$12(ClassBasedTestDescriptor.java:395)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:395)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:211)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:84)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

By debugging and reading the errorStream of the process in ProcessReadinessChecker::defaultNamespaceExists, I find the following reason:

authority=/Users/brammeerten/.jenvtest/apiserver.crt --server=https://127.0.0.1:51775 --request-timeout=5s get ns default
Error in configuration:
* client-cert-data and client-cert are both specified for rancher-desktop. client-cert-data will override.
* client-key-data and client-key are both specified for rancher-desktop; client-key-data will override

rancher-desktop refers to the context defined in my ~/.kube/config file. If I remove this file, the test works.
Or if I add @EnableKubeAPIServer(updateKubeConfigFile = true), then my test works without removing the kube config file.

But the documentation saids this flag is not advised. And it's unclear when and why I should use this flag.

In general, it is not advised but if instructed kube config file (~/kube/config) is updated by the framework. See related property in @EnableKubeAPIServer annotation. The config file is automatically cleaned up on stop..

So it's unclear if to me if this is a bug, or is this as designed and should I use the updateKubeConfigFile flag?
@jamesfarlow jamesfarlow mentioned this issue Nov 6, 2023
Closed
8 tasks
@scholzj
Copy link
Contributor

scholzj commented Nov 22, 2023

I run into this issue as well. What seems to help is when I override the KUBECONFIG variable and point it somewhere else when running the test -> for example to ~/.jenvtest/.kube/config. But it is quite complicated to set it in all places where I run the tests ... in the IDE, in the terminals when running from through Maven directly etc.

@csviri I wonder what would you think about adding to the code a custom KUBECONFIG environment variable like I described above when updateKubeConfig is set to false. If that sounds like a reasonable workaround/fix, I can try to open a PR.

@csviri
Copy link
Collaborator

csviri commented Nov 22, 2023

thx @BramMeerten and @scholzj for the issue, so if I understand when checking if the api server is up and ready, the kubectl picks up also certificates data from standard config file to query the API server.

@scholzj yes, that might be a viable workaround. Feel free to create a PR for this.

scholzj added a commit to scholzj/jenvtest that referenced this issue Nov 22, 2023
@scholzj
Work around conflicts with existing kubeconfig file - Closes java-ope…
20c7c91 
…rator-sdk#116

Signed-off-by: Jakub Scholz <[email protected]>
@scholzj scholzj mentioned this issue Nov 22, 2023
Merged
csviri pushed a commit that referenced this issue Nov 22, 2023
@scholzj
Work around conflicts with existing kubeconfig file (#133)
8e5f87f 
In some environments, where the user has an existing `kubeconfig` file
with an active context using the `client-key-data` or
`client-certificate-data` fields, the `kubectl` client used in the Kube
API server readiness check does not know what certificates to use and
the API Server startup fails with the exception `JenvtestException: Kube
API Server did not start properly`.

This PR works around it by specifying the `KUBECONFIG` environment
variable to an non-existent config file when calling `kubectl` in the
readiness check. That makes the `kubectl` process not use the
pre-existing `kubeconfig` with its certificates and makes everything
work.

This is not needed is the user asks for updating the `kubeconfig` file
as in such case, the `kubeconfig` already contains the correct
configuration and no special handling is needed.

This should resolve #116

Signed-off-by: Jakub Scholz <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Work around conflicts with existing kubeconfig file scholzj/jenvtest
3 participants
@BramMeerten @csviri @scholzj