Setup persistent volumes for kubernetes database manifests

[Tonterias]

Overview of the feature request

Hi,

I am trying to deploy a Jhipster app in Google Cloud Platform using the automatic kubernetes YAML (files that are created by jhipster kuberbernetes as explained here: https://www.jhipster.tech/kubernetes/#enter-the-admin-password-used-to-secure-the-jhipster-registry-admin).

Right now, it deploys but as a STATELESS app. I cannot think of a majority of users who want to deploy Jhipster in GCP (considering any database) as a Stateless App. Once the database pod is dead, your data is lost and you cannot login in your app again. I think that the default behavior should be STATEFUL with a persistent disk (bucket?) that saves the database files outside the VM (pods).

Do you have an example (YAML files) where a Statefull database works in GCP k8s? An example for the ecommerce app should be fine. I would love to see it. May be this can be solved with that example or explained in a tip or included in the documentation.

Thanks

Motivation for or Use Case

Because every Jhipster App uses a database and if the data is not persisted there is no use in having Kubernetes deployments.

Related issues or PR

#9576

• Checking this box is mandatory (this is just to show you read everything)

[PierreBesson]

So basically this feature request is to setup persistent volumes for databases in Kubernetes.

[PierreBesson]

@Tonterias If you want to give a shot to contribute this feature yourself. Have a look at the documentation here: https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/
The Kubernetes docs about volumes and persistent will be relevant as well : https://kubernetes.io/docs/concepts/storage/volumes/

[Tonterias]

Hi @PierreBesson,

I have tried for the last 3 weeks. I've even spotted a mistake on the Google documentation for mysql: https://stackoverflow.com/questions/55637769/why-am-i-getting-a-crashloopbackoff-on-this-kubernetes-example/55843997#55843997 that is now solved, but I can not figure out how to do it for Postgres (or even mysql when the files are automatically created by jhipster kubernetes).

I opened this question: https://stackoverflow.com/questions/55620417/kubernetes-storageclass-does-not-retain-existing-data/55622803#55622803 that is not solved yet.

My question still is: Do you have an example of the ecommerce demo project (or any other demo project) where jhipster kubernetes has created the YAML files of a Statefull database for deployment? Please, show it to me!

Thanks a lot, Pierre

PD: I would have done a tip if I knew how to do it and Spingular would be running in GCP (not in Heroku slow free tier). I do not know, that is why I ask for help. I do not have a computer science degree, but if I can help you, please let me know how! I feel I'm contributing with Spingular open source project to help others who know as little as me, on how to use Jhipster. That's all I can do now.

[PierreBesson]

@Tonterias I understand why you have problems being new to the space and trying to deploy to such a complex platform like kubernetes. I really admire your motivation. Don't worry I have seen people with over 10 years of experience struggle with Kubernetes, so you are not the only one 😄.

I think you have the mistaken assumption that Kubernetes can dynamically creates volumes for you inside the cluster. Actually it can only plug-in to the underlying platform or cloud provider storage capabilities (and those work differently from one cloud to another). This is why you needed the StorageClass object to configure your cluster to plug into the GCP persistent disk API. I think the problem you had before was that it was expecting a manually provisioned disk.

Do you have an example of the ecommerce demo project (or any other demo project) where jhipster kubernetes has created the YAML files of a Statefull database for deployment?

Sorry but as I have told you before, this is currently not supported by JHipspter, you will have to write your own configuration.

[PierreBesson]

A big problem to develop this feature is that I have no idea how to generate a config that will work out of the box for all Kubernetes clusters because the PersistentVolume configuration (Storageclass, CSI driver, etc.) depends on the cloud provider.
@saturnism What do you think could be done here ?

[ruddell]

Related feature request for the same thing, closed due to inactivity #5173

[Tonterias]

Hi @PierreBesson,

I know what could be done. Let's solve the problem for one of the cloud providers (GCP or AWS) and people will use it and promote it, so the others cloud providers will have no other choice but do it too if they want users to use their cloud. That is how it works, it is competition. I know about that!!!

Do not forget that you do not need to automate it, you just need to explain how it should be modified form the files that are created by jhipster kubernetes to what it is need for it to work! It is the knowledge what is missing here! Who knows how to do it?

I created the Storage Class in the storage-google.yaml

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
zone: us-east4-a

May it is something else.

I think that if it is not supported it shouldn't be in the Jhipster webpage like any other supported technology or it should be explained in the limitations https://www.jhipster.tech/kubernetes/#deploying-to-kubernetes

Seriously, how do you want people to use Jhipster for microservices if there is no knowledge about how to persist the database in any cloud provider?

Thanks a lot Pierre.

@jdubois Is there a chance to give a $Bounty? I'll pay $50. Did I hear $100?

PD: Until I finish it, it will appreciate your help more than your admiration ;-) I'm on my knees now, but I can't stop after all the time I have put in this. I wouldn't got to this point without Jhipster, it is a great tool, I can say that. I get quite passionate Pierre, nothing personal, that's how I am.

[jdubois]

I wouldn't recommend this setup, as you'll need to manage the database yourself and that's really not the goal when you move to the cloud. I always use a managed database service instead.

[Tonterias]

Hi @jdubois,

3 questions:

• So you recommend to use the jhipster gae (for monolith), right ? (App Engine flex plus SQL Cloud or Heroku or AWS)
• How would you do that with a microservice? (meaning the managed databases for a Spingular blog microsvc and a mongodb Spingular chat microsvc together, for example). Can jhipster gae do that?
• I miss something: I do not understand the need for jhipster kubernetes then (why google and aws are using it) or why is it covered in the Full Stack Dev Book if that is not the goal when you move to the cloud. In my ignorance, I love k8s. Wannabes, like me, will get lost into how to deploy. The best practice is not clear to me.

So, where & how should I deploy Spingular's new code features and not changing everything once we do a microservice?

Thanks Julien,

[jdubois]

Hi @Tonterias there are 2 different things here in fact:

• For hosting your applications (a microservice being just one type of application), you can use either a VM, a managed service (like Heroku, GAE, Azure app service), or Kubernetes (typically: GCP, EKS or Azure). All of those options work, but provide different levels of price/options/features. For something simple, I would rather use a managed service (so the option with "Heroku, GAE, Azure app service"), that's less work. If I have several microservices, then Kubernetes probably makes more sense. And I only use VMs when I have something very specific that I want to do, like on start.jhipster.tech.
• For databases, it's the same story for all those options: I would rather use a managed service. All cloud providers have several very good options, and they'll take care of upgrades, security issues, backups... All those things you wouldn't want to do in production. Now for Kubernetes we do generate images for those, like we also do with Docker Compose: I believe this is really good for development and test, but I wouldn't use those personally in production. This may look simpler and cheaper on first sight, but I don't want to handle database production myself.

[Tonterias]

Hi @jdubois,

Can you help me? Casue I feel pretty exhausted, at this point.

Now you have an idea of what I want to do with Spingular. I can promote Spingular as a website for programmers who want to learn Jhipster with a real examples. Chaitanya, a programmer from India, has started working on Spingular Chat because he wants to learn websockets in Jhipster.

As we are learning, in the process, we will find issues or have lack of knowledge and we will need your help (from jhipster team) to move forward.

In the end, and with your help, we may end up with a showcase of open source solutions that work with Jhipster tool, programmers that have learned doing them and a site where they can talk about their problems (wannabes problems, but still problems) when using Jhipster.

I can take care of Spingular and curate the site putting time and effort but I need help to have a stable deployment (that does not take 1 minute to wake up like Heroku), where I can easily deploy the new functionalities with the new Spingular programmers and I do not feel like paying for the hosting of Spingular myself.

I was going to use free $300 GoogleCP and ask Google in 6 months to sponsor Springular, but I need to have a stable version to start working and have something to show. Can you help me to stabilize Spingular in GCP? or can you talk to Heroku and sponsor us with a free account?

Without your help, this will end in any whole like the one I am in, right now.

Regards,

[jdubois]

Yes, if money is an issue, I don't think Kubernetes is the best choice for you. I think a mix of Google SQL (for the database) and GAE would cost less.
If you really want something cheap you can use Google Cloud Datastore, that's what I used to have for JHipster Online. There is some extra work to use it with JHipster, but it removes most of the database cost.

[Tonterias]

Hi @jdubois,

Surely, I'll follow your advice. Thanks!!!

I'll wait for GAE issue to be fixed #9605. I have never used it, so I have no idea how difficult it will be. Not covered in the Full Stack book either. If I can't make it work, I will have to ask you for your help because the https://www.youtube.com/watch?v=J9_MW3HOj5w is not clear. If I can, I will make some documentation about how to do it.

I prefer the SQL version so it can be migrated later on, but is there any example of your Google Cloud Datastore option?

But, leaving $ considerations aside, I think that the kubernetes stateful persistent problem is something you will have to solve sooner (microservice) or later (fashion).

Thanks for your suppont,

[SudharakaP]

@Tonterias : I stumbled upon this thread today and I just want to point out there's a pending pull request against #9605 as well as I've created a separate pull request for supporting GAE with Java 11 (#10196); if you are in a hurry you might be able try them out by building the generator yourself. 😄

[pascalgrimaud]

How can we advance on this ticket ?

• do you see some improvement which can be done on the generator-jhipster ? If yes, PR are welcome
• what can you propose for our Kubernetes support, GAE, Openshift ?

Anyway, I think persistent volumes is specific to the cloud you'll choose, so not sure we can do something on our side.

I'm closing this ticket as there are a lot of questions, and few proposals.
But feel free to discuss if you think we can do something here.