Introduce /etc/slurm/slurm-dws.kubeconfig (#49)

Adjust the burst buffer plugin to expect a kubeconfig file in /etc/slurm/slurm-dws.kubeconfig. Change the integration test environment to move the kubeconfig file to /etc/slurm/slurm-dws.kubeconfig. Signed-off-by: Dean Roehrich <[email protected]>
DataWorkflowServices · Sep 29, 2023 · 487d4f2 · 487d4f2
1 parent 1df083c
commit 487d4f2
Show file tree

Hide file tree

Showing 4 changed files with 15 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -88,12 +88,18 @@ In the container, from inside the `/jobs` directory, submit this new batch job:
 bash-4.4$ sbatch test-bb.sh
 ```
 
-You can watch the Workflow resource appear and proceed through the states to DataOut, where it will pause with a state of DriverWait:
+You can watch the Workflow resource appear and proceed through the states to DataOut, where it will pause with a state of DriverWait.  Run all `kubectl` commands from your host, not from inside the slurmctld container:
 
 ```console
 kubectl get workflow -wA
 ```
 
+If you want to run the `kubectl` commands from inside the slumctld container, then set the KUBECONFIG environment variable in the container.  The kubectl command will then use this kubeconfig file.
+
+```console
+export KUBECONFIG=/etc/slurm/slurm-dws.kubeconfig
+```
+
 When the Workflow is in DriverWait, you can release it by marking it as completed.  In this case, my job ID is `12`, so the Workflow resource we're editing is `bb12`.  The paths specified in this patch refer to index 0, the first (and only) `#DW` directive in our job script.
 
 ```console

diff --git a/src/burst_buffer/burst_buffer.lua b/src/burst_buffer/burst_buffer.lua
@@ -39,6 +39,8 @@ local WORKFLOW_CRD = "workflows.dataworkflowservices.github.io"
 
 KUBECTL_CACHE_DIR = "/tmp/burst_buffer_kubectl_cache"
 
+KUBECONFIG = "/etc/slurm/slurm-dws.kubeconfig"
+
 math.randomseed(os.time())
 
 -- Used by the tests when stubbing io.popen.  These values will be
@@ -445,7 +447,8 @@ function DWS:kubectl(cmd)
 	if done ~= true then
 		return false, homedir_msg
 	end
-	local kcmd = homedir_msg .. " kubectl " .. self:token() .. " " .. cmd
+	local kubeconfig_arg = "--kubeconfig " .. KUBECONFIG
+	local kcmd = homedir_msg .. " kubectl " .. kubeconfig_arg .. " " .. self:token() .. " " .. cmd
 	return self:io_popen(kcmd)
 end
 

diff --git a/testsuite/integration/slurm/docker-compose.yml b/testsuite/integration/slurm/docker-compose.yml
@@ -74,7 +74,9 @@ services:
       - type: bind
         source: ../../../src/burst_buffer/burst_buffer.conf
         target: /etc/slurm/burst_buffer.conf
-      - ../kubeconfig:/home/slurm/.kube/config
+      - type: bind
+        source: ../kubeconfig
+        target: /etc/slurm/slurm-dws.kubeconfig
     expose:
       - "6817"
     # depends_on:

diff --git a/testsuite/integration/src/tests/environment/test_environment.py b/testsuite/integration/src/tests/environment/test_environment.py
@@ -54,7 +54,7 @@ def _(k8s):
 @when('the kube-system UID is queried from slurmctld', target_fixture="kube_system_uid_from_slurmctld")
 def _(slurmctld):
     """the kube-system UID is queried from slurmctld."""
-    rc,out = slurmctld.exec_run("kubectl get namespace -o=json kube-system")
+    rc,out = slurmctld.exec_run("kubectl --kubeconfig /etc/slurm/slurm-dws.kubeconfig get namespace -o=json kube-system")
     assert rc==0, "non-zero return code: \n" + out
     return json.loads(out)["metadata"]["uid"]