feat(task-runner): New task runner package and dashboard

[JumpLink]

Work

New Services in docker-compose.yml

• A redis service
• A simple task runner dashboard
• A task runner worker

New apps

• apps/cli - A cli to generate index files, build the packges, deploy a new instance and more
• apps/task-runner-dashboard - The task runner dashboard
• apps/task-runner-worker - The task runner worker

New packages

• packages/config - Contains the app configuration code
• packages/core - Contains the core
• packages/locales - Moved the i18n related code into this packages
• packages/models - Contains the TypeORM models
• packages/task-runner - Contains shared code between the task runner dashboard, task runner worker and backend.

New docker images

• A docker/deno-node image to be able to use both: Node.js and Deno
• A docker/workspace image to build the complete workspace. This is used to copy the result into the application images.

GitHub

• Merged the actions
• When the docker images are built and pushed, the deno-node and workspace image are now always built first and published with the same version, the remaining images then use this version as a basis.

docker-compose.yml

• Instead of referring to the Dockerfile, the locally built docker image is now referenced. These can be built using yarn docker:build:local.

CLI

• Outsourced the CLI from the common module as a standalone app so that it can also be used for the other packages, e.g. to generate the index.ts files. An LLM also ported the new-instanze script here for me, so it wasn't a big effort.

Task Runner Worker

A simple application that waits for new jobs and processes them. Currently, everything is simply executed in the same process of the Node.js application. Since Node.js can only use one CPU core, this could lead to problems with a large number of jobs. With BullMQ this could also be outsourced to separate processes, but I don't think we need that at the moment.

Task Runner Dashboard

Also a simple application that uses bull-board, so far without much customization, but would be possible. The password protection is currently a fixed user, but the authentication of beabee could perhaps be used here if this is to become public.

Docker Deno and Node.js Image

Our workspace currently requires Node.js and Deno. Unfortunately, I could not get both to run with alpine, so this image is based on debian, but is currently only used for building and not for runtime.

Docker Workspace Image

Because we now have many apps that use the same packages (which themselves also have dependencies among each other), the same packages always had to be built in every image. That's why I introduced the workspace image, which builds everything once and the other images copy out what they need. I have also customized the GitHub Actions so that the deno-node and workspace image are built and published first and the others then use them.

Models

I created the models package because they have to be built with the typescript compiler. esbuild is lightning fast in contrast to the TypeScript compiler, but has no information about the types used. Since TypeORM uses the experimental decorators of TypeScript and not the new ones of the JavaScript standard, the types information is needed for this, which esbuild cannot do. For this reason, we build the models with the typescript compiler. This way, the models in the apps can still be used with esbuild.

Version

Since everything is now always released and deployed together, we should bring all apps and packages to the same version number. beabee-common is currently the only outlier here with version 0.21.0, so I suggest bringing everything to version 0.22.0. Alternatively, we could rename @beabee/beabee-common to @beabee/common (here, common is also the only outlier with the name) and then set everything to version 0.16.0.

Why so many changes?

This was not planned and has the following reasons:

• I needed Deno in its current version, unfortunately I didn't manage to get Node.js and Deno running together with Alpine because they both need core libraries that are incompatible with each other. Therefore I switched to Debian for the build and created a common "deno-node" and "workspace" image. The apps still start with Alpine.
• I could not simply remove the core module from the backiend because there were some circular dependencies between the folders. For example, the API needed the config but also the core module, both also needed the translations and the models. I therefore also moved these to separate modules. Some of this can certainly still be improved, but it requires some code restructuring, but my goal was to have to adapt as little of the code as possible.
• This inevitably led to some problems, for example with TypeORM, which I had to fix.
• We are currently still mixing CJS and ESM (backend, frontend), I wanted to rely on ESM for new apps and not introduce anything outdated. However, I have to build the core for both, which at least has the advantage that we can migrate the backend to ESM more easily in the future, since the core works with the task-runner-worker with ESM and uses the core.
• Since we generate the index.ts files for the backend and comman, it was obvious to outsource this to avoid duplicate code, so I moved this to a separate CLI with which I outsourced the scripts from the common module.

Potential for improvement

• Currently the core package requires the config package and loads the config from the corresponding environment variables. It would be better if the config object could be passed to the core module.
• Other parts could be outsourced in packages, for example everything that has to do with the database and typeorm could be a separate database package. Also everything related to express could be a separate package, so not every application needs express, the task runner worker for example does not need express dependencies which are currently delivered by the core package.
• Currently we use two different package formats ESM and CJS which leads to increased complexity. To improve this, the first step would be to remove CJS, which would also simplify the builds of many packages we currently have to build for ESM and CJS.
• We also use two JavaScript runtimes: Node.js and Deno. Here, too, it would be desirable to only need one. In addition, the mixture of both runtimes in one workspace leads to various minor problems in the IDE (although Deno is working on improving this and has already done so). Whether we can rely completely on Deno or should stick with Node.js would have to be evaluated through further experiments.
  • If we use Deno, the advantage would be that we could also use their formatter and linter, thus avoiding further dependencies and configuration files and no longer needing JavaScript builds (except for the browser and packages that we want to make available to the general public)
  • If we use Node.js, the advantage would be that we would retain the best possible compatibility with NPM packages
• The Task Runner Dashboard currently uses a fixed authentication, we could link this to the authentication of beabee so that the admins can automatically log in to the dashboard, which shouldn't be too difficult since we can also use express for this, so it should be possible to move the corresponding code to a separate package and then use it
• Currently, the backend still contains several apps that should also be converted into their own packages: API and webhooks

[JumpLink]

@wpf500 I still have to fix a few small bugs and update the documentation, but otherwise the PR is ready for review. But let's take another look at it together

[wpf500]

Closing this PR, it will remain as a reference as we bring in the relevant changes