Skip to content
This repository has been archived by the owner on May 9, 2019. It is now read-only.

Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items' #78

Open
lejoow opened this issue May 9, 2018 · 21 comments
Assignees

Comments

@lejoow
Copy link

lejoow commented May 9, 2018

I am not 100% sure if this is the issue specifics to my local settings or not. But every time I try to deploy the item-svc or bidding-svc to my Google KBE, I am getting this exception.

Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

screen shot 2018-05-09 at 7 23 54 am

@TimMoore
Copy link

I think I've found the problem. GCP enables Role-Based Access Control (aka RBAC) by default in new Kubernetes clusters. This requires configuring roles to allow access to resources within the Kubernetes API. Lightbend Orchestration (specifically, the Akka Cluster Bootstrap component) uses the Kubernetes API to automatically form a cluster. Without binding your pods' service account to a role that grants access to the required resources, these API calls fail.

To correct this, you'll need to follow the instructions on this page for configuring role-based access control: https://developer.lightbend.com/docs/akka-management/current/discovery.html#discovery-method-kubernetes-api

For example, you can supply these Kubernetes resources to kubectl apply:

---
#
# Create a role, `pod-reader`, that can list pods and
# bind the default service account in the `default` namespace
# to that role.
#

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
subjects:
# Note the `name` line below. The first default refers to the namespace. The second refers to the service account name.
# For instance, `name: system:serviceaccount:myns:default` would refer to the default service account in namespace `myns`
- kind: User
  name: system:serviceaccount:default:default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Once you've created this role and role binding, your Lagom services should start correctly. You might want to tailor this to your specific security needs.

@lejoow please let us know if that fixes the problem for you. Even if it does, we can leave this issue open until we update the documentation.

@lejoow
Copy link
Author

lejoow commented May 10, 2018

@TimMoore fantastic. Let me try this fix today and come back to you on this post. Thanks a lot!

@lejoow
Copy link
Author

lejoow commented May 10, 2018

hmm... I don't get that error anymore, but I am getting the different errors now:

2018-05-10T17:54:27.363Z [�[37minfo�[0m] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-24, akkaTimestamp=17:54:27.363UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Exceeded stable margins without locating seed-nodes, however this node is NOT the lowest address out of the discovered IPs in this deployment, thus NOT joining self. Expecting node Some(ResolvedTarget(10.16.0.13,Some(10002))) (out of List(ResolvedTarget(10.16.0.13,Some(10002)), ResolvedTarget(10.16.1.48,Some(10002)))) to perform the self-join and initiate the cluster.

I am still investigating..

@lejoow
Copy link
Author

lejoow commented May 10, 2018

It worked! I just had to re-deploy. Thanks!
Btw, when I tried to create the role by applying that file you pasted, I ran into "pod-reader forbidden" error message, which I resolved by following the instructions here:

kubectl create clusterrolebinding cluster-admin-binding
--clusterrole=cluster-admin
--user=$primary_account
https://github.com/jcbsmpsn/gke-rbac-walkthrough/blob/master/create-roles.md

@lejoow lejoow closed this as completed May 10, 2018
@TimMoore TimMoore reopened this May 11, 2018
@TimMoore
Copy link

Let's leave this open until the documentation is updated. Thanks!

@TimMoore
Copy link

Added an issue for the Akka Management documentation to note that Google Cloud users will need to grant themselves the cluster-admin ClusterRole: akka/akka-management#199

@Canislupax
Copy link

That solved my problem! Thanx a lot Tim! You`re the best!

@idoshamun
Copy link

@TimMoore I followed your suggestion but now I receive:

14:47:46.025 [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'ports'

For what it's worth, I am trying to run Lagom with Istio

@idoshamun
Copy link

Disabling Istio resolves this issue, I will try to look further and share results

@normenmueller
Copy link

normenmueller commented May 31, 2018

@TimMoore @lejoow I have the same issue running on Minikube. Here are my deployment steps:

$ sbt docker:publishLocal

$ minikube delete
$ minikube start --memory 8192
$ eval $(minikube docker-env)          <<< Should this command come before docker:publishLocal?
$ minikube addons enable ingress
$ kubectl apply -f RBAC.yml            <<< The file posted by @TimMoore

$ sbt "deploy minikube"

$ kubectl cluster-info
$ kubectl get deployments
$ kubectl get pods
$ echo "http://$(minikube ip)"

All Pods have status Running.

Then I had a look at the minikube dashboard and all my services are marked with

Readiness probe failed: Get http://172.17.0.8:10002/platform-tooling/ready: dial tcp 172.17.0.8:10002: getsockopt: connection refused

Then I had a look in the logs at the minikube dashboard and get:

[error] akka.actor.ActorSystemImpl [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=16:32:22.971UTC, akkaSource=akka.actor.ActorSystemImpl(application), sourceActorSystem=application] - Failed to unmarshal Kubernetes API response status [400 Bad Request]; check RBAC settings

@Bunyod
Copy link

Bunyod commented May 31, 2018

@TimMoore @lejoow @normenmueller I'm also facing with the same getsockopt: connection refused issue for Minikube. I follow this link.

Does anybody have an idea how to fix it?

@OguzhanSaltik
Copy link

We have the same error, due to Lightbend Orchestration documentation.

@TimMoore
Copy link

The getsockopt: connection refused message itself only indicates that the service is not running. Of course, there could be many reasons for that. You'll need to look at the logs for the pod to understand more. I'd like to keep this issue thread focused on the original issue reported, which has a solution described in a comment above: #78 (comment)

If you're having another problem, the best place to start is by posting an issue on the Discourse forum: https://discuss.lightbend.com/

@bahtiyarerden
Copy link

bahtiyarerden commented Jun 21, 2018

We are trying to deploy our lagom application with kubernetes. But we could not fix this issue. We clone this repository (chirper) and deploy with minikube to be sure the problem does not about our configuration. It is the result:

screen shot 2018-06-21 at 15 29 29

And this is the log file:

2018-06-21T12:36:25.703Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:25.703UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.756Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:26.756UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:26.883UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:27.802Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:27.801UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.839Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:28.839UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:28.882UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:29.863Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:29.863UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:30.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.882UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:30.936Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.935UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.051Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:32.051UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.893Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:32.892UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:33.089Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:33.088UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.116Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:34.115UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:34.872UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:35.159Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:35.157UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.257Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:36.257UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:36.873UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:37.304Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:37.301UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.332Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:38.332UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.885Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-2, akkaTimestamp=12:36:38.885UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:39.378Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:39.376UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.420Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:40.420UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:40.882UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:41.468Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:41.468UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

I think the problem is about configuration that given below. My microservices that do not have this configuration are working without any error. When i look the logs then i see the akka management port is 10002. The readiness prob also check this port and getting getsockopt: connection refused error.

The application.conf file configuration:

akka.actor.serialization-bindings {
  "akka.Done"                 = akka-misc
  "akka.actor.Address"        = akka-misc
  "akka.remote.UniqueAddress" = akka-misc
}

@ale222
Copy link

ale222 commented Jun 23, 2018

Trying to deploy online-auction, getting:

akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=12:27:31.874UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Discovered [1] observation, which is less than the required [2], retrying (interval: 1.000 s)

any ideas?

@zhangzhonglai
Copy link

same error because of akka/akka-management#222 when using sbt-reactive-app >= 1.2.0
if someone encounter the error like me, simply back to use sbt-reactive-app 1.1.0 will temporary fix the error now.

@ktoso
Copy link

ktoso commented Jun 27, 2018

The related issue #222 was resolved, 0.15 should fix it then. Thanks for digging into this!

@OguzhanSaltik
Copy link

Great news @ktoso , when will be 0.15 version applied to the Lagom?

@ktoso
Copy link

ktoso commented Jun 27, 2018

I'll have to rely this to @TimMoore, not sure on release schedule of intermediate deps etc.

@TimMoore
Copy link

Lagom has no dependency on akka-management. That comes in by way of Lightbend Orchestration, so I think @mitkus is the right one to ask.

@OguzhanSaltik
Copy link

You are right, the related fix is part of the sbt-reactive-app plugin. We look forward to the version updates, thanks to everyone.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests