Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CrashLoopBackOff of pods - GCP deployment #791

Closed
MatteoFelici opened this issue Jun 11, 2020 · 3 comments
Closed

CrashLoopBackOff of pods - GCP deployment #791

MatteoFelici opened this issue Jun 11, 2020 · 3 comments
Labels

Comments

@MatteoFelici
Copy link

Expected Behavior

When installing feast with helm on a GKE cluster, following the doc, I expect all pods to be up and running when kubectl get pods.

Current Behavior

I get both feast-feast-core-869f6d6bcb-6fh7j and feast-feast-online-serving-7f99959884-zhqg7 in "CrashLoopBackOff" status after 7 restarts.

image

After kubectl describe pods I get a type: Warning / Reason: FailedMount / From: kubelet, gke-feast-default-pool-ac080818-0m18 / Message: MountVolume.SetUp failed for volume "feast-feast-core-config" : couldn't propagate object cache: timed out waiting for the condition for the first pod.

image

Steps to reproduce

Follow steps from 0 to 4 on the doc for GCP deployment.

Specifications

  • Version: latest
  • Platform: Mac OS
@woop woop added the kind/bug label Jun 19, 2020
@woop
Copy link
Member

woop commented Jun 19, 2020

I think this might be related to #806

Will try to run through the guide today to debug it.

@Xia-Minghong
Copy link

Xia-Minghong commented Jun 20, 2020

I solved this issue by overriding the default gcp credential values in the yaml config. The default key file names for "feast-gcp-service-account" for both feast-core and feast-batch-serving is "credentials.json". If your secret uses a different file name (e.g. key.json), make sure to override the config by doing the following:

feast-core:
  gcpServiceAccount:
    enabled: true
    existingSecret:
      name: feast-gcp-service-account
      key: key.json
...
feast-batch-serving:
  enabled: true
  gcpServiceAccount:
    enabled: true 
    name: feast-gcp-service-account
    key: key.json

@woop
Copy link
Member

woop commented Jun 21, 2020

I just ran into this problem as well. I have updated all the documentation and ensured that the guide works end to end. Please have a look at the guide here @MatteoFelici. https://docs.feast.dev/getting-started/deploying-feast/kubernetes

Closing the issue for now, please let me know if it isn't resolved.

@woop woop closed this as completed Jun 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants