Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: central cache with symlinks #274

Merged
merged 59 commits into from
May 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
32cfcfc
feat: added central caching
fritzduchardt Mar 24, 2024
667822a
Merge branch 'main' into central-cache
fritzduchardt Mar 24, 2024
18b4e05
Formatting
fritzduchardt Mar 24, 2024
8c9e116
Merge branch 'central-cache' of github.com:mykso/myks into central-cache
fritzduchardt Mar 24, 2024
6cdf637
Merge branch 'main' into central-cache
Zebradil Mar 31, 2024
fd8eb38
Ensure that all service files are stored under .myks
fritzduchardt Mar 31, 2024
5907433
Ignore .myks and .vendor
fritzduchardt Mar 31, 2024
1b5c412
Remove obsolete service files
fritzduchardt Mar 31, 2024
0df9421
Adressed some linting issues
fritzduchardt Mar 31, 2024
7663d7b
Merge branch 'central-cache' of github.com:mykso/myks into central-cache
fritzduchardt Mar 31, 2024
d62def2
Remove obsolete service files
fritzduchardt Mar 31, 2024
c8bc474
Adressed some more linting issues
fritzduchardt Mar 31, 2024
873571a
Switch hash function for shorter safer hash value
fritzduchardt Mar 31, 2024
8db1af3
Fix test
fritzduchardt Mar 31, 2024
9014544
Move and rename cache dir
fritzduchardt Apr 7, 2024
b82632f
Made cached directory names more readable
fritzduchardt Apr 7, 2024
4b6648e
Added global flag to disable cache
fritzduchardt Apr 7, 2024
6845a1e
fix hash calculation
Zebradil Apr 7, 2024
fba69cb
Merge remote-tracking branch 'origin/main' into central-cache
Zebradil Apr 7, 2024
8677cf2
fix hash calculation stronger
Zebradil Apr 7, 2024
3eb64cc
small refactoring
Zebradil Apr 7, 2024
2517d3a
change variable name to avoid shadowing one from outer scope
Zebradil Apr 7, 2024
f56c255
fix hash in tests
Zebradil Apr 7, 2024
0421c9d
sync only one contents item per directory in vendir config
Zebradil Apr 7, 2024
0bded65
fix multi-chart vendir configs, simplify code
Zebradil Apr 8, 2024
8a8559e
linter warnings
Zebradil Apr 8, 2024
40e7baf
Merge branch 'main' into central-cache
Zebradil Apr 11, 2024
a9a3a3c
Ensure sync is always run after cache invalidation.
fritzduchardt Apr 14, 2024
b39e452
Merge remote-tracking branch 'origin/main' into central-cache
Zebradil Apr 20, 2024
e5dc74b
Merge remote-tracking branch 'origin/main' into central-cache
Zebradil Apr 20, 2024
46ca903
Merge remote-tracking branch 'origin/main' into central-cache
Zebradil Apr 20, 2024
c4934cc
move integration tests charts deeper into _lib
Zebradil Apr 21, 2024
63d4ff1
add currently failing test case with multiple sources
Zebradil Apr 21, 2024
e1553ba
remove unused import
Zebradil Apr 21, 2024
f5335c0
stashing work
Zebradil Apr 21, 2024
fd169ce
sync seems to work
Zebradil Apr 21, 2024
1ace817
helm-sync kinda works
Zebradil Apr 21, 2024
7c3f3a6
helm render works
Zebradil Apr 21, 2024
d0a2fa3
ytt render works
Zebradil Apr 22, 2024
0d7d78e
probably working ytt package rendering
Zebradil Apr 22, 2024
c7425bf
remove unused field
Zebradil Apr 22, 2024
5d2e288
fix integration tests
Zebradil Apr 22, 2024
6319cf6
Merge branch 'central-cache' into central-cache-symlinks
Zebradil Apr 22, 2024
55d83dc
fix relative dir source by removing chdir
Zebradil Apr 22, 2024
4eb84a5
fix integration tests
Zebradil Apr 22, 2024
837194a
Merge branch 'central-cache' into central-cache-symlinks
Zebradil Apr 22, 2024
a8ddd3e
remove unused field and enable lazy sync
Zebradil Apr 22, 2024
cc78051
fix linter warnings
Zebradil Apr 22, 2024
ea80327
fix error message when no helm charts in app
Zebradil Apr 26, 2024
e36f79c
Merge branch 'main' into central-cache-symlinks
Zebradil Apr 26, 2024
5b1f3a2
Merge branch 'main' into central-cache-symlinks
Zebradil May 6, 2024
025200d
support for multiple content items
Zebradil May 6, 2024
fe3aa71
Merge remote-tracking branch 'origin/main' into central-cache-symlinks
Zebradil May 10, 2024
d7d9988
add integration test for multiple contents
Zebradil May 10, 2024
adb21ef
cleanup vendor dir before creating any links in it
Zebradil May 10, 2024
f8bf183
remove stale comment
Zebradil May 10, 2024
5269af7
cleanup cache dir on sync failure
Zebradil May 10, 2024
94197bc
remove path from cache name generation
Zebradil May 10, 2024
fb94718
workaround race conditions by using locks
Zebradil May 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ bin/*
dist
.idea
.run
/myks
**/.myks/tmp
.myks
.vendor
coverage.txt
vendor/
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
kind: rendering
metadata:
name: render-test
name: helm-{{ .Release.Name }}
outputYaml:
{{ toYaml .Values.outputYaml | indent 2 }}
4 changes: 4 additions & 0 deletions examples/_lib/ytt/render-test-1/all.ytt.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
kind: rendering
metadata:
name: ytt-render-test-1
4 changes: 4 additions & 0 deletions examples/_lib/ytt/render-test-2/all.ytt.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
kind: rendering
metadata:
name: ytt-render-test-2

This file was deleted.

12 changes: 0 additions & 12 deletions examples/default/envs/mykso/dev/_apps/argocd/.myks/vendir.yaml

This file was deleted.

This file was deleted.

20 changes: 0 additions & 20 deletions examples/default/envs/mykso/dev/_apps/goldpinger/.myks/vendir.yaml

This file was deleted.

This file was deleted.

This file was deleted.

11 changes: 0 additions & 11 deletions examples/default/envs/mykso/dev/_apps/httpbingo/.myks/vendir.yaml

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

2 changes: 2 additions & 0 deletions examples/integration-tests/envs/dev/env-data.ytt.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ environment:
applications:
- proto: helm-render-test
name: helm-installation
- proto: multiple-sources
- proto: multiple-contents-sources
- proto: ytt-render-test
name: ytt-installation
- proto: static-test
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ directories:
contents:
- path: .
directory:
path: ../charts/render-test-chart
path: ../_lib/charts/render-test-chart
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: .
contents:
- path: charts/render-test-1
directory:
path: ../_lib/charts/render-test-chart
- path: charts/render-test-2
directory:
path: ../_lib/charts/render-test-chart
- path: ytt/render-test-1
directory:
path: ../_lib/ytt/render-test-1
- path: ytt/render-test-2
directory:
path: ../_lib/ytt/render-test-2
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: charts/render-test-1
contents:
- path: .
directory:
path: ../_lib/charts/render-test-chart
- path: charts/render-test-2
contents:
- path: .
directory:
path: ../_lib/charts/render-test-chart
- path: ytt/render-test-1
contents:
- path: .
directory:
path: ../_lib/ytt/render-test-1
- path: ytt/render-test-2
contents:
- path: .
directory:
path: ../_lib/ytt/render-test-2
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: app-mykso-dev-multiple-contents-sources
namespace: system-argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: env-mykso-dev
destination:
name: mykso-dev
namespace: multiple-contents-sources
source:
path: examples/integration-tests/rendered/envs/mykso-dev/multiple-contents-sources
plugin:
name: argocd-vault-plugin-v1.0.0
repoURL: [email protected]:mykso/myks.git
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: app-mykso-dev-multiple-sources
namespace: system-argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: env-mykso-dev
destination:
name: mykso-dev
namespace: multiple-sources
source:
path: examples/integration-tests/rendered/envs/mykso-dev/multiple-sources
plugin:
name: argocd-vault-plugin-v1.0.0
repoURL: [email protected]:mykso/myks.git
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
kind: rendering
metadata:
name: render-test
name: helm-render-test
outputYaml:
fromAppConfig: true
fromAppConfigAppData: true
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: rendering
metadata:
name: helm-render-test-1
outputYaml:
fromChartDefaultValues: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: rendering
metadata:
name: helm-render-test-2
outputYaml:
fromChartDefaultValues: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
kind: rendering
metadata:
name: ytt-render-test-1
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
kind: rendering
metadata:
name: ytt-render-test-2
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: rendering
metadata:
name: helm-render-test-1
outputYaml:
fromChartDefaultValues: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: rendering
metadata:
name: helm-render-test-2
outputYaml:
fromChartDefaultValues: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
kind: rendering
metadata:
name: ytt-render-test-1
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
kind: rendering
metadata:
name: ytt-render-test-2
10 changes: 0 additions & 10 deletions examples/simple/envs/dev/_apps/httpbingo/.myks/helm/httpbingo.yaml

This file was deleted.

This file was deleted.

11 changes: 0 additions & 11 deletions examples/simple/envs/dev/_apps/httpbingo/.myks/vendir.yaml

This file was deleted.

This file was deleted.

This file was deleted.

11 changes: 0 additions & 11 deletions examples/simple/envs/prod/_apps/httpbingo/.myks/vendir.yaml

This file was deleted.

4 changes: 0 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -415,15 +415,11 @@ golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220906165534-d0df966e6959/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.19.0 h1:q5f1RH2jigJ1MoAWp2KTp3gm5zAGFUTarQZ5U386+4o=
golang.org/x/sys v0.19.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0 h1:Od9JTbYCk261bKm4M/mw7AklTlFYIa0bIp9BgSm1S8Y=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.0.0-20220411215600-e5f449aeb171/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.19.0 h1:+ThwsDv+tYfnJFhF4L8jITxu1tdTWRTZpdsWgEgjL6Q=
golang.org/x/term v0.19.0/go.mod h1:2CuTdWZ7KHSQwUzKva0cbMg6q2DMI3Mmxp+gKJbskEk=
golang.org/x/term v0.20.0 h1:VnkxpohqXaOBYJtBmEppKUG6mXpi+4O6purfc2+sMhw=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down
2 changes: 1 addition & 1 deletion internal/integration/render_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ func findRepos(t *testing.T, basefolder string) []testRepo {
if !d.IsDir() {
continue
}
if d.Name() == "charts" {
if d.Name() == "_lib" {
continue
}
repos = append(repos, testRepo{
Expand Down
Loading
Loading