Remove current shared memory pattern

[diegodelemos]

About

Since we are facing the initial release it would be nice to remove one of the ugliest things inside this component, which is the fact that we are using shared memory across different threads.

1. Add a cache

While trying to solve #18 I have faced some existential questions related with sharing information across the different threads, (the Flask one and the 2 additional for Job and Pod watchers) it is not nice at all how it is done right now.

I think we should remove the current shared memory model and refactor it towards a more production like approach before doing any further change.

2. Split execution in different processes

Another proposal would be to have three different CLIs, one for the app itself and two more for each of the watchers so we can spawn different processes.

Proposals

• Shall we go for a cache? Redis maybe?
• Shall we go for a process splitting of the three different concerns?
• Shall we give to this the higher priority since it is a crucial component (also the weakest right now) for the current system?

Let's discuss the above mentioned proposals or new ones, so we have a configuration as clean and scalable as possible.

CC @reanahub/developers

[tiborsimko]

Some quick reflections: (1) using Redis looks good, although if we take into account some further needs (say client job quota accounting and friends), we may think of future PostgreSQL already perhaps; (2) not sure about pros/cons of multiprocessing, let's see IRL; (3) yes, I fully agree, the job controller is at core of the REANA base system, so that we should ideally clear its API, implementation, and docs first, before going higher up to the REANA workflow components.

[diegodelemos]

It seems fair enough, as long as we do something like #28 it doesn't look that bad, at least we wont be sharing the HTTPConnections which was my main concern.
Regarding to caching, as you mention and as we talked yesterday, the PostgreSQL option looks like the right thing to do in the long term. 👍

[lukasheinrich]

yes we should definitely use either redis or a proper DB (redis can run into memory problems if the dataset exceeds the memory of the node it runs on, so maybe I'd opt for mongo or something similar). Another reason for this is that this allows us to scale the job-controller as a replica set. Right now if we attempt to scale, each replica will have its own DB with ambiguous information. Better to have a one soure of truth.

[diegodelemos]

Closing since this will be reborn with job caching and persistence in Client-server milestone.