About nebula-spark-connector v3.0.0

[porscheme]

Nebula: v3.1.0
Spark Connector: 3.0.0
Spark: 3.2.1

1. Our spark and nebula are in two different K8S clusters. Therefore, what how can spark jobs talk to Nebula cluster using nebula-spark-connector?
2. Has anyone successfully used nebula-spark-connector (written in Scala?) in pySpark? If, example will be very help full.

[wey-gu]

1. To connect to nebula from a spark running in a different cluster, we need to expose all meta/storage out of the cluster, and explicitly set them to the reader rather than only specifying the meta(then it will fetch storage from meta only), I will look into this later(now on my phone replying you) now I remember that we need this to be implemented to enable it withStorageAddress introduced nebula-spark-connector#39
2. Scala/Pyspark runnable examples: https://www.siwei.io/en/spark-on-nebula-graph/

[porscheme]

• @wey-gu we used Helm Charts to deploy Nebula
• Unfortunately, it does have any method to expose metad/storaged out of the cluster
• AKS has something called internal Loadbalancer (not exposed to the internet). We use this feature to expose Elasticsearch to the applications, for example

Can Nebula engineering team add ability to expose metad/storaged service outside the cluster?

[porscheme]

Even if I make were to follow @MegaByte875 suggestion to expose graphd example here https://github.com/vesoft-inc/nebula-operator/tree/master/config/samples, how can expose meta?

Looks like gap in the solution, confirm?

We are blocked!

Any alternate solution is really appreciated.

[wey-gu]

expose them should be common experiences in k8s, I could provide poc for that, before that we need spark-connector to support specifying storaged addresses instead of fetching from meta only, which need changes in nebula-Java and nebula-spark-connector. And I had been working on pr of this since this morning for you:).

Hopefully I could make them done shortly, I got you back, no worries 😆

[porscheme]

Awesome, thanks looking forward to your message.

[wey-gu]

progress

• Nebula-Java support specifying storageD addresses setStorageAddresses support to optionally override storage addresses nebula-java#451
• Nebula-Spark-Connecotr support specifying storageD addresses with storage address nebula-spark-connector#47
• PoC of exposing meta, storage instance addresses

[wey-gu]

Update, after my PR on nebula-java and nebula-spark-c to support specifying exposed storaged addresses being discussed in the review, it turned out this is not easily feasible due to the map of storage-hosts to their address was used when seeking data distribution among storage hosts, thus simply specify list of storage hosts isn't enough, their map should be configurable, too, this design is more complex, thus need further discussion.

Thus, I created a PoC to only expose them and use TCP proxy and DNS to make spark-connector connect to nebula with exactly the same addresses as it was inside the nebula cluster namespace:

a. create services per metad and storaged pod with LoadBalancer type to expose outside the cluster b. Use TCP Proxy(b.1) or/and DNS(b.2) to resolve the endpoint as their: inside the cluster

In this example, in b. I give a demo where both TCP Proxy and DNS are used, actually, only DNS is enough if the exposed port remains the same of src and target.

details refer to: https://gist.github.com/wey-gu/699b9a2ef5dff5f0fb5f288d692ddfd5

[sunkararp]

Update, after my PR on nebula-java and nebula-spark-c to support specifying exposed storaged addresses being discussed in the review, it turned out this is not easily feasible due to the map of storage-hosts to their address was used when seeking data distribution among storage hosts, thus simply specify list of storage hosts isn't enough, their map should be configurable, too, this design is more complex, thus need further discussion.

Thus, I created a PoC to only expose them and use TCP proxy and DNS to make spark-connector connect to nebula with exactly the same addresses as it was inside the nebula cluster namespace:

a. create services per metad and storaged pod with LoadBalancer type to expose outside the cluster b. Use TCP Proxy(b.1) or/and DNS(b.2) to resolve the endpoint as their: inside the cluster

In this example, in b. I give a demo where both TCP Proxy and DNS are used, actually, only DNS is enough if the exposed port remains the same of src and target.

details refer to: https://gist.github.com/wey-gu/699b9a2ef5dff5f0fb5f288d692ddfd5

Any update on this work item?

[porscheme]

• I created graphd-service.yaml and metad-service.yaml YAML files
• Placed above two YAML in /nebula-cluster/templates folder (see attached)
• Using this helm charts values file I was able to register internal loadbalance for both graphd and metad.
• I was able to connect and query using nebula3-python python package from Jupyter notebook, I have not tried spark-connector yet
• Please review attached files and provide any feedback, if nay

[wey-gu]

I don't think this would work, in GraphD is stateless, thus exposing it with LB should be fine. While Meta/Storage are stateful instances for meta client and storage client, putting them behind one load balancer round-robin-ly is not supported. For spark-connector, the normal pattern was to(not like every other clients) scan data via storage client, it requires to get the data distribution from meta and then scanning storage instances based on that distribution.

[porscheme]

@wey-gu thanks for the reply.

• I did not change Storage service
• Even though meta is exposed as LB there is only replica. Do we need more than one meta replica?
• Also, load balancing doesn't need be round-robin. check this

[wey-gu]

• I did not change Storage service

Not like other clients, the spark connector of Nebula Graph was designed for analytical purposes, thus it's fetching data with StorageClient (instead of querying with nGQL through GraphD), the storage client was designed to be aware of each of the storage instances, this means all storage instances' addresses should be exposed separately.

• Even though meta is exposed as LB there is only replica. Do we need more than one meta replica?

If Meta replication is set 1, the LB model would work perfectly, in this case, it's recommended to have its data replicated with PV.

• Also, load balancing doesn't need be round-robin. check this

Sorry, I had put it wrongly, I didn't mean it's the problem of the round-robin(which should be applied to all LB mechanisms). As spark client was designed to scan data from storaged, it needs to know each instance of them instead of a load-balanced single point.