Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Launcher + S3 overhaul #827

Draft
wants to merge 162 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
162 commits
Select commit Hold shift + click to select a range
6510c56
Launcher overhaul checkpoint
garronej Jul 2, 2024
f6a4448
Dumb down the Onyxia API port
garronej Jul 3, 2024
8276f76
Lazily download AWS S3 Client
garronej Jul 3, 2024
e744a3c
Don't include s3 STS in the restorable configuration
garronej Jul 4, 2024
4bb89cb
No special case for ONYXIA_API_URL
garronej Jul 4, 2024
feca217
Better initialization of sqlOlap for s3
garronej Jul 4, 2024
f6068ae
Remove useless mocks
garronej Jul 4, 2024
f2467e1
Refactor S3 config management
garronej Jul 11, 2024
d110a7e
Fix readability issues
garronej Jul 11, 2024
35eaf1c
Initialize default s3 config for xOnyxia and explorer
garronej Jul 15, 2024
716d488
Reimplementation of the thunks of the launcher
garronej Jul 23, 2024
087008b
Update stufs related to duckdb wasm
garronej Jul 24, 2024
0e60aca
Update file explorer to reflect recent change on s3 config managment
garronej Jul 24, 2024
b8ec5c5
Update s3 config creation usecase
garronej Jul 29, 2024
c3edecc
update project updateConfigValue() thunk
garronej Jul 29, 2024
7efb6dc
Simplify resolveXOnyxiaValueReference
garronej Aug 5, 2024
d30a7fc
Only reference xOnyxia context values in overwriteDefaultWith and ove…
garronej Aug 5, 2024
f666c95
Implement getHelmValues_default
garronej Aug 6, 2024
b3bb81f
Add unit tests for values resolution from schema
garronej Aug 7, 2024
65c94ad
Add minimal unit test setup
garronej Aug 7, 2024
1bdca5d
Complete launcher thunks
garronej Aug 8, 2024
c57424b
Validation of individual fields is up to the UI, the values should al…
garronej Aug 13, 2024
628ee15
Add tests for getValuesAtPath in object and type so that it supports …
garronej Aug 14, 2024
66255e4
Correctly initialize arrays
garronej Aug 16, 2024
2e22e9e
Complete mergeRangeSliders
garronej Aug 16, 2024
b7d472f
Complete mergeRangeSliders
garronej Aug 16, 2024
5d806ca
Introduce Vitest
garronej Aug 16, 2024
4530c4d
Add test for removeFormFieldGroupWithNoChildren
garronej Aug 19, 2024
84626bc
Add test for mergeTemporaryRangeSliders
garronej Aug 19, 2024
015a8cd
Add test case for helmValuesPathToFormFieldPath
garronej Aug 20, 2024
10e051b
Make function easyly testable by using subset of types
garronej Aug 20, 2024
c850926
Add test case for getFormFieldPath
garronej Aug 20, 2024
7e220e1
Add test case for getFormFieldAtPath
garronej Aug 20, 2024
2b17b1a
Add test case for getHelmValuesPathDeeperCommonSubpath
garronej Aug 20, 2024
d1badb9
checkpoint
garronej Aug 20, 2024
dfddf07
Fix helmValuesPathToFormFieldPath
garronej Aug 20, 2024
5921937
Fix test case for insertRangeSliderFormField
garronej Aug 21, 2024
b6bd688
Checkpoint before fixing all tests
garronej Aug 21, 2024
9d31ba4
Fix insertRangeSliderFormField test
garronej Aug 21, 2024
3add85d
Complete mergeRangeSliders
garronej Aug 21, 2024
bde3241
Checkpoint getRootFormFieldGroup
garronej Aug 22, 2024
671d714
checkpoint
garronej Aug 23, 2024
23fb7cf
Check validity of JSONSchema at the edge
garronej Aug 23, 2024
20cd40d
Move getHelmValues_default to a directory
garronej Aug 23, 2024
d7e3915
Checkpoint refactor
garronej Aug 23, 2024
3248ad4
Checkpoint
garronej Aug 24, 2024
75af421
checkpoint
garronej Aug 26, 2024
a921872
checkpoint
garronej Aug 27, 2024
252e529
Checkpoint
garronej Aug 27, 2024
1678b4e
Big checkpoint (lots of unit test to write)
garronej Aug 27, 2024
6b8af51
Fix some type errors
garronej Aug 28, 2024
96a9184
Fix tests
garronej Aug 28, 2024
f9e7c45
Fix bug in resolveEnum and use toStrictEqual instead of toEqual
garronej Aug 28, 2024
6a784a0
Add unit tests for listEnums
garronej Aug 28, 2024
b770174
Add unit tests for resolveEnum and validateValueAgainstJSONSchema
garronej Aug 28, 2024
77b786f
Add tests for gtRootFromFieldGroup
garronej Aug 30, 2024
b136d5c
Add unit test for getRootFrom
garronej Aug 30, 2024
7095b0e
Fix and finish testing validateValueAgainstJSONSchema
garronej Sep 2, 2024
377f8d8
Rename bestApproximation to reasonableApproximation
garronej Sep 2, 2024
1524ff4
Add updateHelmValues method and connect it (need to implement it)
garronej Sep 2, 2024
fa56753
Put full helmValuesPath in the form fields groups
garronej Sep 2, 2024
035075b
Refactor launcher decoupled business logic
garronej Sep 3, 2024
742ca83
Rename getHelmValues_default to coputeHelmValue
garronej Sep 3, 2024
ffff0bf
Rename getRootForm to computeRootForm
garronej Sep 3, 2024
25dfd28
Do not wrap in a directory when we don't need to
garronej Sep 3, 2024
099eb58
Decouple diff object computation
garronej Sep 3, 2024
ee79c2d
Fix small compile error
garronej Sep 4, 2024
f0891ae
Connect conditional dependenciy enabling
garronej Sep 5, 2024
2b92d8c
Fix compile error
garronej Sep 5, 2024
5760545
Make the launcher initialization easyer to understand
garronej Sep 5, 2024
576ab63
Fix the url parser so it supports array deletions
garronej Sep 5, 2024
ce5ebf7
Rename decoupledBusinessLogic to decoupledLogic
garronej Sep 6, 2024
a536c27
Make type declaration more explicit
garronej Sep 6, 2024
a959ade
Fill with default items in compute helm value (arrays)
garronej Sep 6, 2024
948d85e
Implemet all helm value mutation for adding and removing array values
garronej Sep 9, 2024
263c029
Add tests for mutateHelmValues_addArrayItem
garronej Sep 9, 2024
654d4dc
Add test for mutateHelmValues_removeArrayItem
garronej Sep 9, 2024
1d8c17e
Remove unused state in launcher$
garronej Sep 10, 2024
1bf6664
Implement mutateHelmValues_update
garronej Sep 11, 2024
783a56e
findInRootForm
garronej Sep 12, 2024
0837f7c
Add test for mutateHelmValues_update
garronej Sep 12, 2024
13d046e
Complete tests for mutateHelmValues_update
garronej Sep 12, 2024
b9c1c33
Externalize createObjectWithSomePropertiesThatThrowIfAccessed
garronej Sep 12, 2024
b61b102
Refactor s3CodeSnippets
garronej Sep 12, 2024
7f39b8b
Fix k8sCodeSnippets
garronej Sep 12, 2024
e971033
Fix viewQuotas
garronej Sep 12, 2024
5a4f3a8
Fix serviceManagement
garronej Sep 12, 2024
5e825c5
current project now in protected selectors
garronej Sep 12, 2024
44ac0c2
Rename utils -> decoupledLogic
garronej Sep 12, 2024
774703f
Fix restorableConfigManagement
garronej Sep 12, 2024
87bf495
Fix las compile error in the core
garronej Sep 12, 2024
7d7b73c
Fix merge conflicts
garronej Sep 14, 2024
fc4be60
Refactor #838
garronej Sep 16, 2024
0425a81
Update launcher link from MyServices page
garronej Sep 16, 2024
9af7e7b
Update ProjectSettings according to refactor
garronej Sep 16, 2024
6bdfae4
Add testS3Connection and deleteS3Config thunks
garronej Sep 17, 2024
04cea08
Rework s3config creating and deletion
garronej Sep 18, 2024
2c5dd27
Add changeIsDefault to s3ConfigManagment
garronej Sep 18, 2024
b7a04c3
Complete S3 project settings migration
garronej Sep 19, 2024
1876782
Setup framework for new form fields
garronej Sep 20, 2024
dd57886
Refactor
garronej Sep 20, 2024
a97dfe4
Add YamlCodeBlockFormField
garronej Sep 21, 2024
0077294
Merge branch 'main' into launcher_overhaul
garronej Sep 27, 2024
ec36d9a
Merge branch 'main' into launcher_overhaul
garronej Sep 28, 2024
94e7647
style FormFieldWrapper
garronej Sep 30, 2024
58b36ab
implement TextFormField
garronej Oct 1, 2024
40f613b
Change form fields title to label 1 typo
garronej Oct 1, 2024
7fafc03
Add CheckboxFormField
garronej Oct 1, 2024
2759a55
Add tooltip for the reset button in form field wrapper
garronej Oct 1, 2024
2c1ea95
Implement select input
garronej Oct 1, 2024
d1a31b7
Simple slider text field
garronej Oct 2, 2024
bdad535
Add RangeSliderFormField
garronej Oct 3, 2024
75c9e9e
Change 'inter field' by 'number field'
garronej Oct 3, 2024
1b70825
Add NumberFormField
garronej Oct 3, 2024
4fe0a81
Make throttling cancelable in form
garronej Oct 4, 2024
d66201d
Add onRemove callback on form fields
garronej Oct 4, 2024
04ccb5c
Set FormFieldGroup up
garronej Oct 4, 2024
946412c
Implement FormFieldGroup
garronej Oct 7, 2024
bbd3a8f
Compleate the implementation of the React logic for the FormFieldGrou…
garronej Oct 7, 2024
e004fd3
Refactor to avoid confustion between the data type and the component
garronej Oct 7, 2024
f439181
Rename FormFieldGroup children to nodes to avoid confustion with Reac…
garronej Oct 7, 2024
85a718c
Small tweaks
garronej Oct 7, 2024
e713bc3
Overengeenered hook for recursively getting the background color
garronej Oct 7, 2024
f35f21b
Everything set up in the new luncher except the configuration card
garronej Oct 9, 2024
ad3ddc3
Fix build
garronej Oct 9, 2024
953ab8c
Implement setup for migration
garronej Oct 9, 2024
9a9b2b8
Fix not crash when not logged in
garronej Oct 10, 2024
30d5cd8
Disable Screen Scaller for Dylan
garronej Oct 10, 2024
1971b53
Fix dual mode with/without screen scaler
garronej Oct 10, 2024
c5000b1
Merge branch 'main' into launcher_overhaul
garronej Oct 10, 2024
63f7e6c
Stable column ref
garronej Oct 10, 2024
b0f8fb1
Stable reference of colone even when data changes
garronej Oct 11, 2024
85ef7a1
Update oidc-spa
garronej Oct 14, 2024
87e1f46
Make the inner component wrapper accessible
garronej Oct 14, 2024
8684318
Add AccordionGroupComponent
garronej Oct 14, 2024
d3d9593
Add AccordionFormComponent
garronej Oct 14, 2024
a778608
Add ConfigurationTopLevelGroup component
garronej Oct 15, 2024
f536d66
Add DependenciesTab component
garronej Oct 15, 2024
b23cb9b
Move all form related component to RootFormComponent
garronej Oct 15, 2024
30a4edb
Link up RootFormComponent
garronej Oct 15, 2024
11038b1
Connect RootFromComponent in the launcher
garronej Oct 15, 2024
34d495f
Do not display accordion if no global config
garronej Oct 15, 2024
0013cef
CSS fixes
garronej Oct 15, 2024
8890899
Fix route query param serialization
garronej Oct 15, 2024
ead8eb0
CSS adjustments
garronej Oct 15, 2024
a3a8b5c
capitalize title and descriptions
garronej Oct 15, 2024
7680328
add unit test
garronej Oct 16, 2024
31b51ba
Update submodules on v10 branch
garronej Oct 16, 2024
d88a74b
Merge branch 'main' into launcher_overhaul
garronej Oct 16, 2024
535e452
Fix relative hidden file resolution
garronej Oct 17, 2024
4bdb8bf
implement throttle on a case by case basis
garronej Oct 17, 2024
f7e6ca6
Fix ofset jumps in launcher
garronej Oct 17, 2024
770c022
Fix empty arrays being removed
garronej Oct 17, 2024
2ad5c1f
Enable root arrays
garronej Oct 17, 2024
11c359f
Make sonarcloud ignore test files
garronej Oct 17, 2024
713514b
Fix bug in getValueAtPath
garronej Oct 18, 2024
9a5c77d
Fix removable array item when items are object
garronej Oct 18, 2024
7e754d1
Fix the launcher query params parsing
garronej Oct 18, 2024
5eaaf29
Fix yarn.lock
garronej Oct 19, 2024
3a14ee3
fix build
garronej Oct 19, 2024
e94c90d
Fix wrong parsing of unitless sliders
garronej Oct 20, 2024
2f80c54
Display top level yaml code block as group
garronej Oct 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ jobs:
- uses: bahmutov/[email protected]
env:
XDG_CACHE_HOME: "/home/runner/.cache/yarn"
- run: yarn test
- run: yarn build
- run: npx keycloakify

Expand Down
2 changes: 1 addition & 1 deletion .sonarcloud.properties
Original file line number Diff line number Diff line change
@@ -1 +1 @@
sonar.cpd.exclusions=/web/src/ui/i18n/resources/*
sonar.cpd.exclusions=/web/src/ui/i18n/resources/*,**/*.test.ts
12 changes: 7 additions & 5 deletions web/.env
Original file line number Diff line number Diff line change
Expand Up @@ -801,6 +801,11 @@ S3_DOCUMENTATION_LINK=https://docs.sspcloud.fr/content/storage.html
#
VAULT_DOCUMENTATION_LINK=https://docs.sspcloud.fr/content/secrets.html

# For devloppement only, to use if you want to run onyxia-web locally and connect to a remote
# onyxia-api instance.
# You would set for example ONYXIA_API_URL=https://datalab.sspcloud.fr/api
ONYXIA_API_URL=/api

# ==================================================================================
# Private parameters - Not expected to be configured by the instance administrator
# ==================================================================================
Expand All @@ -812,9 +817,6 @@ VAULT_DOCUMENTATION_LINK=https://docs.sspcloud.fr/content/secrets.html
ONYXIA_VERSION=
ONYXIA_VERSION_URL=

# This is to be configured if you want to spin up a local instance of onyxia-web
# and connect to a remote instance of onyxia-api.
# You would set for example ONYXIA_API_URL=https://datalab.sspcloud.fr/api
ONYXIA_API_URL=/api

ALLOW_THEME_TESTING_VIA_URL=false

SCREEN_SCALER=true
26 changes: 21 additions & 5 deletions web/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,28 +14,43 @@
"format": "yarn _format --write",
"format:check": "yarn _format --list-different",
"emails-domain-accept-list-helper": "tsx scripts/emails-domain-accept-list-helper.ts",
"test": "vitest --run",
"storybook": "storybook dev -p 6006",
"build-storybook": "storybook build",
"prestorybook": "tsx scripts/unyamlify-env-local.ts"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.513.0",
"@aws-sdk/lib-storage": "^3.513.0",
"@aws-sdk/s3-request-presigner": "^3.513.0",
"@babel/runtime": "7.25.6",
"@codemirror/autocomplete": "6.18.1",
"@codemirror/lang-yaml": "6.1.1",
"@codemirror/language": "6.10.3",
"@codemirror/lint": "6.8.1",
"@codemirror/search": "6.5.6",
"@codemirror/state": "6.4.1",
"@codemirror/theme-one-dark": "6.1.2",
"@codemirror/view": "6.33.0",
"@duckdb/duckdb-wasm": "^1.13.1-dev230.0",
"@duckdb/duckdb-wasm-shell": "^1.13.1-dev230.0",
"@emotion/cache": "^11.11.0",
"@emotion/react": "^11.11.3",
"@emotion/styled": "^11.11.0",
"@lezer/common": "1.2.1",
"@lezer/highlight": "1.2.1",
"@mui/base": "^5.0.0-beta.58",
"@mui/icons-material": "5.14.15",
"@mui/material": "^5.15.10",
"@mui/system": "^5.15.9",
"@mui/x-data-grid": "^6.19.4",
"@uiw/codemirror-themes": "4.23.2",
"@uiw/react-codemirror": "4.23.2",
"@ungap/structured-clone": "^1.2.0",
"async-mutex": "^0.4.0",
"axios": "^0.26.0",
"bytes": "^3.1.2",
"clean-architecture": "^4.3.7",
"clean-architecture": "^4.3.8",
"codemirror": "6.0.1",
"color": "^4.2.3",
"compare-versions": "^5.0.1",
"evt": "^2.5.7",
Expand All @@ -47,8 +62,8 @@
"memoizee": "^0.4.15",
"minimal-polyfills": "^2.2.3",
"moment": "^2.29.1",
"oidc-spa": "^5.4.0",
"onyxia-ui": "^5.3.6",
"oidc-spa": "^5.4.3",
"onyxia-ui": "^5.4.0",
"pathe": "^1.1.2",
"powerhooks": "^1.0.12",
"react": "^18.2.0",
Expand Down Expand Up @@ -108,7 +123,8 @@
"vite": "^5.0.8",
"vite-envs": "^4.4.5",
"vite-plugin-commonjs": "^0.10.1",
"vite-tsconfig-paths": "^4.3.1"
"vite-tsconfig-paths": "^4.3.1",
"vitest": "^2.0.5"
},
"lint-staged": {
"*.{ts,tsx,json,md}": [
Expand Down
1 change: 1 addition & 0 deletions web/src/core/adapters/oidc/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from "./oidc";
5 changes: 1 addition & 4 deletions web/src/core/adapters/oidc/mock.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,7 @@ export async function createOidc(params: {
const { isUserInitiallyLoggedIn } = params;

const oidc = await createMockOidc({
isUserInitiallyLoggedIn,
"mockedTokens": {
"decodedIdToken": {}
}
isUserInitiallyLoggedIn
});

return oidc;
Expand Down
12 changes: 3 additions & 9 deletions web/src/core/adapters/oidc/utils/createOidcOrFallback.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
import type { Oidc } from "core/ports/Oidc";
import { assert, type Equals } from "tsafe/assert";
import { assert } from "tsafe/assert";
import { noUndefined } from "tsafe/noUndefined";

export async function createOidcOrFallback(params: {
oidcAdapterImplementationToUseIfNotFallingBack: "default";
oidcParams:
| {
issuerUri?: string;
Expand All @@ -12,8 +11,7 @@ export async function createOidcOrFallback(params: {
| undefined;
fallbackOidc: Oidc.LoggedIn | undefined;
}): Promise<Oidc.LoggedIn> {
const { oidcAdapterImplementationToUseIfNotFallingBack, oidcParams, fallbackOidc } =
params;
const { oidcParams, fallbackOidc } = params;

const wrap = (() => {
const { issuerUri, clientId } = {
Expand Down Expand Up @@ -47,11 +45,7 @@ export async function createOidcOrFallback(params: {
case "oidc client":
return wrap.oidc;
case "oidc params": {
assert<
Equals<typeof oidcAdapterImplementationToUseIfNotFallingBack, "default">
>();

const { createOidc } = await import("../default");
const { createOidc } = await import("../oidc");

const oidc = await createOidc({
"issuerUri": wrap.oidcParams.issuerUri,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/* eslint-disable no-template-curly-in-string */
import type { LocalizedString, JSONSchemaObject } from "core/ports/OnyxiaApi";
import type { LocalizedString, JSONSchema } from "core/ports/OnyxiaApi";
import type { ArrayOrNot } from "core/tools/ArrayOrNot";

export type ApiTypes = {
"/public/ip": {
Expand Down Expand Up @@ -83,7 +84,7 @@ export type ApiTypes = {
};
};
data?: {
S3?: {
S3?: ArrayOrNot<{
URL: string;
pathStyleAccess?: true;

Expand Down Expand Up @@ -116,7 +117,7 @@ export type ApiTypes = {
bucketNamePrefix: string;
bucketNamePrefixGroup: string;
};
};
}>;
};
vault?: {
URL: string;
Expand Down Expand Up @@ -179,23 +180,22 @@ export type ApiTypes = {
}[];
};
"/public/catalogs/${catalogId}/charts/${chartName}/versions/${chartVersion}": {
config: JSONSchemaObject;
config: JSONSchema;
defaultValues: string;
sources?: string[];
dependencies?: {
name: string;
repository: string;
version: string;
condition?: string;
}[];
};
"/my-lab/services": {
apps: {
id: string;
status: "deployed" | "pending-install" | "failed";
urls: string[];
env: {
// Actually Record<string, string> but we prefer only listing known properties
// for type safety.
};
env: Record<string, string>;
owner: string;
startedAt: number;
tasks: {
Expand Down
1 change: 0 additions & 1 deletion web/src/core/adapters/onyxiaApi/default/index.ts

This file was deleted.

Loading
Loading