Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

minikube testing #6

Closed
jlewi opened this issue Feb 2, 2018 · 9 comments
Closed

minikube testing #6

jlewi opened this issue Feb 2, 2018 · 9 comments
Labels
priority/p1

Comments

@jlewi
Copy link
Contributor

jlewi commented Feb 2, 2018 •
edited
Loading

We should run E2E tests on minikube.
kube_test is supposed to help with this.
@jlewi
Copy link
Contributor Author

jlewi commented Mar 7, 2018

Some thoughts on possible approaches.

  • Investigate kube_test and see what it does
  • Run minikube on a VM and run everything via ssh

This should map pretty well to our existing E2E tests. Our existing E2E tests are Argo workflows that just run series of scripts. So the only change we'd need to make is to execute those scripts via ssh on the remote VM rather than in the pod itself.

@mhausenblas
Copy link
Member

@jlewi can you link to the Argo workflows pls? I might be interested in taking care of this …

@jlewi
Copy link
Contributor Author

jlewi commented Mar 8, 2018

Thanks.

Every repo should have a prow_config.yaml at the root. If you look at the contents you can find links to our Argo workflows.

This is our main workflow
https://github.com/kubeflow/kubeflow/blob/master/testing/workflows/components/workflows.libsonnet

@jlewi
Copy link
Contributor Author

jlewi commented Mar 19, 2018

@mhausenblas Are you still thinking of taking this on?

@jlewi
Copy link
Contributor Author

jlewi commented Mar 19, 2018

I did some initial experimentation this morning.

  • I looked at kubetest but it didn't seem to offer any support for minikube.

  • I setup minikube on a GCE VM using driver=None.

  • I was able to connect to that minikube deployment from a pod running in an actual GKE cluster by

    • Copying the .kube directory to the pod from the VM
    • Copying the .minikube directory to the pod from the VM
    • Modifying .kube/config to change the paths of the .minikube files (certs)

  • It looks like the K8s master is running on the GCE internal IP

  • So E2E tests running on minikube should work nearly identical to our existing E2E tests on GKE. The only difference would be that we would need to deploy a K8s cluster via minikube.

@jlewi jlewi mentioned this issue Mar 20, 2018
Merged
@mhausenblas
Copy link
Member

Thank you @jlewi and yes, I plan to work on it next week (on travel now) but if you want to have it earlier, feel free to start the work and I chime in when I'm back home?

@jlewi
Copy link
Contributor Author

jlewi commented Mar 20, 2018

Sounds great. Let me see how far I can get this week.

jlewi added a commit to jlewi/kubeflow that referenced this issue Mar 21, 2018
@jlewi
Run E2E tests on minikube.
8bef5f1 
* Parameterize the E2E workflow with a parameter that controls the platform
  we are running on.

* Related to kubeflow/testing#6
This was referenced Mar 21, 2018
Closed
Merged
jlewi added a commit to jlewi/k8s that referenced this issue Mar 21, 2018
@jlewi
Support testing on minikube.
0b01844 
* When using minikube we don't want call configure_kubectl which calls
  gcloud container clusters get-credentials.

* Instead we just need to call load_kube_config to load the kube config file.

* Related to kubeflow/testing#6
This was referenced Mar 21, 2018
Merged
Closed
Closed
k8s-ci-robot pushed a commit to kubeflow/training-operator that referenced this issue Mar 26, 2018
@jlewi @k8s-ci-robot
Support testing on minikube. (#485)
6214e56 
* Support testing on minikube.

* When using minikube we don't want call configure_kubectl which calls
  gcloud container clusters get-credentials.

* Instead we just need to call load_kube_config to load the kube config file.

* Related to kubeflow/testing#6

* Add a TODO.

* Fix lint issue.
jlewi pushed a commit to jlewi/kubeflow that referenced this issue Mar 28, 2018
@willb @jlewi
Changes to support running E2E tests on minikube.
d3aa8f6 
* Parameterize the E2E workflow with a parameter that controls the platform
  we are running on.

* Related to kubeflow/testing#6

* Add environment variable to configure KUBECONFIG.

* Split test_deploy into separate functions to deploy the kubeflow APP
  and get GKE credentials
* This allows us to handle credentials and the kubeconfig file consistently
  for GKE and minikube; on subsequent steps we just set KUBECONFIG to
  point to the config.

* Refactor the workflow to be a DAG.

* Update the TFServing workflow to use the new version of test_deploy
  which uses a separate step to get credentials and store them in .kube_config.

* The name for the test_runner step needs to be unique so that we will
  create different test cases in gubernator for the different platforms.

* Parameterize the E2E workflow by a platform parameter that controls whether we use minikube or gke.

* Always store kube config on an NFS share.

* Fix lint and add a style .yapf file to autoformat files copied for tf-operator.
jlewi pushed a commit to jlewi/kubeflow that referenced this issue Mar 28, 2018
@willb @jlewi
Changes to support running E2E tests on minikube.
174710d 
* Parameterize the E2E workflow with a parameter that controls the platform
  we are running on.

* Related to kubeflow/testing#6

* Add environment variable to configure KUBECONFIG.

* Split test_deploy into separate functions to deploy the kubeflow APP
  and get GKE credentials
* This allows us to handle credentials and the kubeconfig file consistently
  for GKE and minikube; on subsequent steps we just set KUBECONFIG to
  point to the config.

* Refactor the workflow to be a DAG.

* Update the TFServing workflow to use the new version of test_deploy
  which uses a separate step to get credentials and store them in .kube_config.

* The name for the test_runner step needs to be unique so that we will
  create different test cases in gubernator for the different platforms.

* Parameterize the E2E workflow by a platform parameter that controls whether we use minikube or gke.

* Always store kube config on an NFS share.

* Fix lint and add a style .yapf file to autoformat files copied for tf-operator.
jlewi added a commit to jlewi/kubeflow that referenced this issue Mar 28, 2018
@jlewi
Changes to support running E2E tests on minikube.
56335d5 
 * Parameterize the E2E workflow with a parameter that controls the platform
   we are running on.

* Related to kubeflow/testing#6

* Add environment variable to configure KUBECONFIG.

* Split test_deploy into separate functions to deploy the kubeflow APP
      and get GKE credentials

* This allows us to handle credentials and the kubeconfig file consistently
      for GKE and minikube; on subsequent steps we just set KUBECONFIG to
      point to the config.

* Refactor the workflow to be a DAG.

* Update the TFServing workflow to use the new version of test_deploy
      which uses a separate step to get credentials and store them in .kube_config.

* The name for the test_runner step needs to be unique so that we will
      create different test cases in gubernator for the different platforms.

* Parameterize the E2E workflow by a platform parameter that controls whether we use minikube or gke.

* Always store kube config on an NFS share.

* Fix lint and add a style .yapf file to autoformat files copied for tf-operator.
@jlewi jlewi mentioned this issue Mar 28, 2018
Merged
@dlorenc
Copy link

dlorenc commented Apr 1, 2018

Note that you can also use nested virtualization and the kvm2 driver on gce for a more traditional minikube testing setup.

k8s-ci-robot pushed a commit to kubeflow/kubeflow that referenced this issue Apr 2, 2018
@jlewi @k8s-ci-robot
Changes to support running E2E tests on minikube. (#523)
0823a40 
*  Changes to support running E2E tests on minikube.

 * Parameterize the E2E workflow with a parameter that controls the platform
   we are running on.

* Related to kubeflow/testing#6

* Add environment variable to configure KUBECONFIG.

* Split test_deploy into separate functions to deploy the kubeflow APP
      and get GKE credentials

* This allows us to handle credentials and the kubeconfig file consistently
      for GKE and minikube; on subsequent steps we just set KUBECONFIG to
      point to the config.

* Refactor the workflow to be a DAG.

* Update the TFServing workflow to use the new version of test_deploy
      which uses a separate step to get credentials and store them in .kube_config.

* The name for the test_runner step needs to be unique so that we will
      create different test cases in gubernator for the different platforms.

* Parameterize the E2E workflow by a platform parameter that controls whether we use minikube or gke.

* Always store kube config on an NFS share.

* Fix lint and add a style .yapf file to autoformat files copied for tf-operator.

* Fix merge issue.

* Autoformat jsonnet file.

* * Address reviewer comments.

* Fix bad merge in workflow.libsonnet.
@jlewi
Copy link
Contributor Author

jlewi commented Apr 2, 2018

Basic testing is done.
We should consider whether we want to run some of our other tests on minikube as well.

@jlewi jlewi closed this as completed Apr 2, 2018
@jlewi jlewi mentioned this issue Apr 2, 2018
Closed
swiftdiaries pushed a commit to swiftdiaries/kubeflow that referenced this issue Apr 4, 2018
@jlewi @swiftdiaries
Changes to support running E2E tests on minikube. (kubeflow#523)
a594c72 
*  Changes to support running E2E tests on minikube.

 * Parameterize the E2E workflow with a parameter that controls the platform
   we are running on.

* Related to kubeflow/testing#6

* Add environment variable to configure KUBECONFIG.

* Split test_deploy into separate functions to deploy the kubeflow APP
      and get GKE credentials

* This allows us to handle credentials and the kubeconfig file consistently
      for GKE and minikube; on subsequent steps we just set KUBECONFIG to
      point to the config.

* Refactor the workflow to be a DAG.

* Update the TFServing workflow to use the new version of test_deploy
      which uses a separate step to get credentials and store them in .kube_config.

* The name for the test_runner step needs to be unique so that we will
      create different test cases in gubernator for the different platforms.

* Parameterize the E2E workflow by a platform parameter that controls whether we use minikube or gke.

* Always store kube config on an NFS share.

* Fix lint and add a style .yapf file to autoformat files copied for tf-operator.

* Fix merge issue.

* Autoformat jsonnet file.

* * Address reviewer comments.

* Fix bad merge in workflow.libsonnet.
jetmuffin pushed a commit to jetmuffin/tf-operator that referenced this issue Jul 9, 2018
@jlewi @k8s-ci-robot
Support testing on minikube. (kubeflow#485)
babaec5 
* Support testing on minikube.

* When using minikube we don't want call configure_kubectl which calls
  gcloud container clusters get-credentials.

* Instead we just need to call load_kube_config to load the kube config file.

* Related to kubeflow/testing#6

* Add a TODO.

* Fix lint issue.
saffaalvi pushed a commit to StatCan/kubeflow that referenced this issue Feb 11, 2021
@jlewi @k8s-ci-robot
Changes to support running E2E tests on minikube. (kubeflow#523)
70c4b63 
*  Changes to support running E2E tests on minikube.

 * Parameterize the E2E workflow with a parameter that controls the platform
   we are running on.

* Related to kubeflow/testing#6

* Add environment variable to configure KUBECONFIG.

* Split test_deploy into separate functions to deploy the kubeflow APP
      and get GKE credentials

* This allows us to handle credentials and the kubeconfig file consistently
      for GKE and minikube; on subsequent steps we just set KUBECONFIG to
      point to the config.

* Refactor the workflow to be a DAG.

* Update the TFServing workflow to use the new version of test_deploy
      which uses a separate step to get credentials and store them in .kube_config.

* The name for the test_runner step needs to be unique so that we will
      create different test cases in gubernator for the different platforms.

* Parameterize the E2E workflow by a platform parameter that controls whether we use minikube or gke.

* Always store kube config on an NFS share.

* Fix lint and add a style .yapf file to autoformat files copied for tf-operator.

* Fix merge issue.

* Autoformat jsonnet file.

* * Address reviewer comments.

* Fix bad merge in workflow.libsonnet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority/p1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mhausenblas @jlewi @dlorenc