feat(openapi): update reana.yaml schema to include dask field #467

Alputer · 2024-08-13T14:08:03Z

This pull request contains the ongoing work of dask integration into REANA.

codecov · 2024-08-13T14:13:26Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 41.79%. Comparing base (3b80320) to head (d0becac).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #467   +/-   ##
=======================================
  Coverage   41.79%   41.79%           
=======================================
  Files          28       28           
  Lines        2010     2010           
=======================================
  Hits          840      840           
  Misses       1170     1170

tiborsimko · 2024-09-09T08:17:23Z

AUTHORS.md

@@ -4,6 +4,7 @@ The list of contributors in alphabetical order:

 - [Adelina Lintuluoto](https://orcid.org/0000-0002-0726-1452)
 - [Agisilaos Kounelis](https://orcid.org/0000-0001-9312-3189)
+- [Alp Tuna](https://orcid.org/0009-0001-1915-3993)


You can add yourself in the same commit that introduced functionality. It is not necessary to split this into two commits.

tiborsimko · 2024-09-09T11:27:06Z

reana_commons/validation/schemas/reana_analysis_schema.json

+              "type": "object",
+              "title": "Dask cluster to be used by the workflow",
+              "properties": {
+                "image": {


Please introduce also help for the various properties such as image. (By means of description fields, see .)

tiborsimko · 2024-09-23T13:09:54Z

reana_commons/openapi_specifications/reana_server.json

+                "maximum_kubernetes_jobs_timeout",
+                "maximum_workspace_retention_period",
+                "workspaces_available",
+                "dask_enabled"


Note that testing the reana-client with this reana-commons PR does not work against "older" REANA clusters that do not contain Dask support, I mean even with non-Dask workflows, for example:

$ cd reana-demo-root6-roofit $ export REANA_SERVER_URL=https://reana.cern.ch $ read -s REANA_ACCESS_TOKEN $ reana-client create -w test ==> ERROR: Cannot create workflow test: 'dask_enabled' is a required property

It would be good to make the client workable against both "old" (Dask-unaware) and "new" (Dask-aware) clusters.

Basically, we have four cases in the test matrix to consider: old client <-> old cluster, old client <-> new cluster, new client <-> old cluster, new client <-> new cluster. The Dask support is "optional" and so ideally all these combinations should keep working.

I removed the required part from the openapi spec and now it should be working fine with all combinations. I tested with several examples and all should be working fine.

…eanahub#467)

…ub#467)

tiborsimko · 2024-10-30T10:28:07Z

reana_commons/validation/schemas/reana_analysis_schema.json

+                },
+                "single_worker_memory": {
+                  "type": "string",
+                  "description": "Memory for one Dask worker.",


Requested memory for one Dask worker

tiborsimko · 2024-10-30T10:28:22Z

reana_commons/validation/schemas/reana_analysis_schema.json

+                },
+                "number_of_workers": {
+                  "type": "number",
+                  "description": "Number of workers in the Dask cluster."


Requested number of Dask workers

tiborsimko · 2024-10-30T10:28:45Z

reana_commons/validation/schemas/reana_analysis_schema.json

+              "properties": {
+                "image": {
+                  "type": "string",
+                  "description": "Image to be used by Dask workers and scheduler"


Container image to be used by Dask scheduler and workers

tiborsimko · 2024-10-30T10:29:13Z

reana_commons/validation/schemas/reana_analysis_schema.json

+            "dask": {
+              "$id": "/properties/workflow/properties/resources/properties/dask",
+              "type": "object",
+              "title": "Dask cluster to be used by the workflow",


Information about Dask cluster requested for the analysis.

tiborsimko · 2024-10-30T10:30:56Z

reana_commons/openapi_specifications/reana_server.json

@@ -429,6 +429,26 @@
                    "slurmcern"
                  ]
                },
+                "dask_cluster_default_number_of_workers": {
+                  "title": "Default number of workers for Dask clusters",


For the server-side variables, please see notes about documentation lines in https://github.com/reanahub/reana/pull/821/files

tiborsimko · 2024-10-30T10:44:44Z

reana_commons/openapi_specifications/reana_server.json

+                  "value": "8Gi"
+                },
+                "dask_enabled": {
+                  "title": "Dask workflows allowed in the cluster",


Boolean to inform whether cluster supports running Dask workflows.

(BTW some wild thoughts on the necessity of this variable. I'm wondering whether we could use some of the other existing variables to decide whether Dask is enabled or disabled on the given deployment. OT1H, if the total memory limit is 0, and the number of workers is 0, then this could in theory function as a sign that Dask is not supported. OTOH, we may want to start offering more enabled/disabled options for various functionalities in the future, such as is CWL enabled, is Snakemake enabled, etc. So I kind of like having it 😉 since we could generalise it towards enabling/disabling various functionalities in general. We can muse more IRL.)

…ub#467)

tiborsimko

Works nicely 👍

Alputer self-assigned this Aug 13, 2024

Alputer changed the title ~~Dask integration~~ feat(dask): Dask integration (#427) Aug 13, 2024

Alputer changed the title ~~feat(dask): Dask integration (#427)~~ feat(dask): Dask integration (#467) Aug 13, 2024

Alputer force-pushed the dask-integration branch from 3aa482c to 3aa9dfd Compare August 26, 2024 15:34

Alputer added a commit to Alputer/reana-commons that referenced this pull request Aug 28, 2024

feat(dask): update reana.yaml schema (reanahub#467)

50952aa

Alputer force-pushed the dask-integration branch from 3aa9dfd to 50952aa Compare August 28, 2024 15:33

Alputer added a commit to Alputer/reana-commons that referenced this pull request Aug 29, 2024

feat(dask): update reana.yaml schema to include dask (reanahub#467)

c868b01

Alputer added a commit to Alputer/reana-commons that referenced this pull request Aug 29, 2024

docs(authors): add Alp Tuna (reanahub#467)

c67d08a

Alputer force-pushed the dask-integration branch from 50952aa to c67d08a Compare August 29, 2024 11:38

Alputer marked this pull request as ready for review August 29, 2024 11:38

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 5, 2024

feat(dask): update reana.yaml schema to include dask (reanahub#467)

6670137

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 5, 2024

docs(authors): add Alp Tuna (reanahub#467)

8caeb77

Alputer force-pushed the dask-integration branch from c67d08a to 8caeb77 Compare September 5, 2024 19:29

tiborsimko reviewed Sep 9, 2024

View reviewed changes

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 9, 2024

feat(dask): Add descriptions to dask properties (reanahub#467)

f3206b5

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 9, 2024

feat(dask): add descriptions to dask properties (reanahub#467)

b3e522b

Alputer force-pushed the dask-integration branch from f3206b5 to b3e522b Compare September 9, 2024 13:00

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 13, 2024

feat(dask): update reana.yaml schema (reanahub#467)

b05a124

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 15, 2024

feat(dask): update reana.yaml schema (reanahub#467)

b8bb2a9

Alputer force-pushed the dask-integration branch from b05a124 to b8bb2a9 Compare September 15, 2024 21:13

This was referenced Sep 20, 2024

Clean up resources if dask workflow fails reanahub/reana#829

Closed

Use CPU and RAM requirements specified by the user for dask workers reanahub/reana#830

Closed

tiborsimko reviewed Sep 23, 2024

View reviewed changes

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 25, 2024

feat(dask): update reana.yaml (reanahub#467)

851cc93

Alputer added a commit to Alputer/reana-commons that referenced this pull request Sep 25, 2024

feat(dask): update openapi (reanahub#467)

973dc63

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 16, 2024

feat(dask): update reana.yaml schema to include dask (reanahub#467)

e476354

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 16, 2024

docs(authors): add Alp Tuna (reanahub#467)

cbfb71d

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 16, 2024

feat(dask): add descriptions to dask properties (reanahub#467)

f520dc3

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 16, 2024

feat(dask): update reana.yaml schema (reanahub#467)

fc58884

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 16, 2024

feat(dask): update reana.yaml (reanahub#467)

212d742

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 16, 2024

feat(dask): update openapi (reanahub#467)

923930b

Alputer force-pushed the dask-integration branch from 973dc63 to 923930b Compare October 16, 2024 09:19

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

feat(dask): update reana.yaml schema to include dask (reanahub#467)

3ee54a6

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

docs(authors): add Alp Tuna (reanahub#467)

0f2b5c1

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

feat(dask): add descriptions to dask properties (reanahub#467)

60f3299

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

feat(dask): update reana.yaml schema (reanahub#467)

0245bca

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

feat(dask): update reana.yaml (reanahub#467)

3ac1661

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

feat(dask): update openapi (reanahub#467)

4d39acf

Alputer force-pushed the dask-integration branch from 923930b to 4d39acf Compare October 24, 2024 11:00

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

feat(openapi): update reana.yaml schema to include dask properties (r…

2ea5d7a

…eanahub#467)

Alputer force-pushed the dask-integration branch from 4d39acf to 2ea5d7a Compare October 24, 2024 11:45

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 24, 2024

feat(openapi): update reana.yaml schema to include dask field (reanah…

84b92f6

…ub#467)

Alputer force-pushed the dask-integration branch from 2ea5d7a to 84b92f6 Compare October 24, 2024 11:49

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 29, 2024

feat(openapi): update reana.yaml schema to include dask field (reanah…

bbac7ca

…ub#467)

Alputer force-pushed the dask-integration branch from 84b92f6 to bbac7ca Compare October 29, 2024 15:36

tiborsimko reviewed Oct 30, 2024

View reviewed changes

feat(openapi): update reana.yaml schema to include dask field (reanah…

d0becac

…ub#467)

Alputer force-pushed the dask-integration branch from bbac7ca to d0becac Compare October 30, 2024 12:49

tiborsimko reviewed Oct 31, 2024

View reviewed changes

tiborsimko merged commit d0becac into reanahub:master Oct 31, 2024
16 checks passed

github-actions bot mentioned this pull request Oct 28, 2024

chore(master): release 0.95.0 #450

Open

Alputer changed the title ~~feat(dask): Dask integration (#467)~~ feat(openapi): update reana.yaml schema to include dask field Oct 31, 2024

Alputer mentioned this pull request Oct 31, 2024

feat(dask): inform users when they request too few resources reanahub/reana-server#703

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(openapi): update reana.yaml schema to include dask field #467

feat(openapi): update reana.yaml schema to include dask field #467

Alputer commented Aug 13, 2024

codecov bot commented Aug 13, 2024 •

edited

Loading

tiborsimko Sep 9, 2024

tiborsimko Sep 9, 2024

tiborsimko Sep 23, 2024

Alputer Sep 25, 2024

tiborsimko Oct 30, 2024

tiborsimko Oct 30, 2024

tiborsimko Oct 30, 2024

tiborsimko Oct 30, 2024

tiborsimko Oct 30, 2024

tiborsimko Oct 30, 2024

tiborsimko left a comment

feat(openapi): update reana.yaml schema to include dask field #467

feat(openapi): update reana.yaml schema to include dask field #467

Conversation

Alputer commented Aug 13, 2024

codecov bot commented Aug 13, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiborsimko left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 13, 2024 •

edited

Loading