Storage Error: part: 82, error: E_RPC_FAILURE(-3) while inserting edges

[gauravsingh-nagarro-scm-oms]

Nebula version used->3.1.0
3 storage nodes -> each node has 5GB RAM and 300 GB disk space(120GB in use)
1 metad node and 3 graphd nodes

Current data stored in Nebula as per statistics is as below ->
Space | vertices | 43509
Space | edges | 2063624772

Now We are trying to insert 32 crore edges where we could see now RAM has reached already near to 4.9GB and giving error as Storage Error: part: 82, error: E_RPC_FAILURE(-3) .

We have restarted pods of storaged but RAM is not released. Can someone help me to understand how RAM is filled up full? Is it because of Cache or the total vertex data and edge data will take this much of RAM and i have to increase RAM to add more edge data.

Is there a way to know total memory and storage requirement for number of edges to make better decision on infra requirements for our case

[xiajingchun]

For the memory requirement, you may refer to https://docs.nebula-graph.io/3.1.0/4.deployment-and-installation/1.resource-preparations/#capacity_requirements_for_running_a_nebula_graph_cluster and calculate.

5G per node seems a little too low to me for the scale of your dataset.

Having said that, you could still better control the memory consumed by storage service, take a look at these two parameters: rocksdb_block_cache and enable_partitioned_index_filter in your storage conf, refer to https://docs.nebula-graph.io/3.1.0/5.configurations-and-logs/1.configurations/4.storage-config/

[gauravsingh-nagarro-scm-oms]

Thanks for the information, it is really helpful.
We did calculation as per the provided link and with total number of vertex + edges for our dataset, Memory required is coming out to be 37GB and we have 15Gb as with our cluster.
Will insertions in Nebula would not happen as we are below the required memory capacity of 37GB?
Insertions were happening before but now has stopped completely and giving only this error : E_RPC_FAILURE(-3) storage for even one single edge insertion.

We are unable to change these properties(rocksdb_block_cache and enable_partitioned_index_filter) in nebula-storaged.conf file as file is readonly in storage service, is there a different way to allow updation?

[wey-gu]

Regarding readonly for conf files, are you using k8s operators, and you would like to change it in file system? If that is the case, you should go add options in NebulaCluster'CRD for storageD as this document: https://github.com/vesoft-inc/nebula-operator/blob/master/doc/user/custom_config.md

[gauravsingh-nagarro-scm-oms]

One more query regarding the Memory used by storage service->
This is our current snapshot of 3 storaged nodes each having 5GB RAM allocated

As you could see from above screenshot, all the nodes are always consuming near to 5GB RAM which is the allocated RAM, Is it because of RocksDb Cache filled up?
Is there a way we can do some configuration to evict items from cache if it has reached 80% of capacity or we can clear the complete cache itself to only allow items at disk.

[xiajingchun]

Have you checked the storaged log? It might be the case that the service got restarted due to out of memory.
Please try what wey suggested above to first bring the memory usage down.