Skip to content

Commit

Permalink
Restructure ML Storage CR (#1508)
Browse files Browse the repository at this point in the history
  • Loading branch information
nikita-vanyasin authored Nov 28, 2023
1 parent 2c528fb commit b2986a0
Show file tree
Hide file tree
Showing 14 changed files with 757 additions and 177 deletions.
77 changes: 50 additions & 27 deletions docs/api/ArangoMLStorage.V1Alpha1.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,72 +2,95 @@

## Spec

### .spec.listenPort
### .spec.backend.s3.allowInsecure

Type: `integer` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec.go#L32)</sup>
Type: `boolean` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec_backend_s3.go#L43)</sup>

ListenPort defines on which port the sidecar container will be listening for connections
AllowInsecure if set to true, the Endpoint certificates won't be checked

Default Value: `9201`
Default Value: `false`

***

### .spec.resources
### .spec.backend.s3.bucketName

Type: `core.ResourceRequirements` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec.go#L37)</sup>

Resources holds resource requests & limits for container running the S3 proxy
Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec_backend_s3.go#L37)</sup>

Links:
* [Documentation of core.ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#resourcerequirements-v1-core)
BucketName specifies the name of the bucket
Required

***

### .spec.s3.bucketName
### .spec.backend.s3.caSecret.name

Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_s3_spec.go#L39)</sup>
Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/batchjob_status.go#L12)</sup>

BucketName specifies the name of the bucket
Required
Name of the object

***

### .spec.s3.credentialsSecret
### .spec.backend.s3.caSecret.namespace

Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_s3_spec.go#L42)</sup>
Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/batchjob_status.go#L13)</sup>

CredentialsSecretName specifies the name of the secret containing AccessKey and SecretKey for S3 API authorization
Required
Namespace of the object. Should default to the namespace of the parent object

***

### .spec.s3.disableSSL
### .spec.backend.s3.credentialsSecret.name

Type: `boolean` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_s3_spec.go#L33)</sup>
Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/batchjob_status.go#L12)</sup>

DisableSSL if set to true, no certificate checks will be performed for Endpoint
Name of the object

Default Value: `false`
***

### .spec.backend.s3.credentialsSecret.namespace

Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/batchjob_status.go#L13)</sup>

Namespace of the object. Should default to the namespace of the parent object

***

### .spec.s3.endpoint
### .spec.backend.s3.endpoint

Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_s3_spec.go#L30)</sup>
Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec_backend_s3.go#L34)</sup>

Endpoint specifies the S3 API-compatible endpoint which implements storage
Required

***

### .spec.s3.region
### .spec.backend.s3.region

Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_s3_spec.go#L36)</sup>
Type: `string` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec_backend_s3.go#L52)</sup>

Region defines the availability zone name. If empty, defaults to 'us-east-1'
Region defines the availability zone name.

Default Value: `""`

***

### .spec.mode.sidecar.listenPort

Type: `integer` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec_mode_sidecar.go#L41)</sup>

ListenPort defines on which port the sidecar container will be listening for connections

Default Value: `9201`

***

### .spec.mode.sidecar.resources

Type: `core.ResourceRequirements` <sup>[\[ref\]](https://github.com/arangodb/kube-arangodb/blob/1.2.35/pkg/apis/ml/v1alpha1/storage_spec_mode_sidecar.go#L46)</sup>

Resources holds resource requests & limits for container running the S3 proxy

Links:
* [Documentation of core.ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#resourcerequirements-v1-core)

## Status

### .status.conditions
Expand Down
15 changes: 13 additions & 2 deletions internal/docs_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ func Test_GenerateAPIDocs(t *testing.T) {
root := os.Getenv("ROOT")
require.NotEmpty(t, root)

sharedFields, sharedFilesSet := parseSourceFiles(t, fmt.Sprintf("%s/pkg/apis/shared/v1", root))

// package path -> result doc file name -> name of the top-level field to be described -> field instance for reflection
input := map[string]map[string]map[string]interface{}{
fmt.Sprintf("%s/pkg/apis/deployment/v1", root): {
Expand Down Expand Up @@ -180,8 +182,17 @@ func Test_GenerateAPIDocs(t *testing.T) {

resultPaths := make(map[string]string)
for apiDir, docs := range input {
fields, fileSets := parseSourceFiles(t, apiDir)
util.CopyMap(resultPaths, generateDocs(t, docs, fields, fileSets))
fields, fileSet := parseSourceFiles(t, apiDir)

for n, f := range sharedFields {
fields[n] = f
}
sharedFilesSet.Iterate(func(file *token.File) bool {
fileSet.AddFile(file.Name(), fileSet.Base()+file.Base(), file.Size())
return true
})

util.CopyMap(resultPaths, generateDocs(t, docs, fields, fileSet))
}
generateIndex(t, resultPaths)
}
Expand Down
58 changes: 0 additions & 58 deletions pkg/apis/ml/v1alpha1/storage_s3_spec.go

This file was deleted.

54 changes: 24 additions & 30 deletions pkg/apis/ml/v1alpha1/storage_spec.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,47 +21,41 @@
package v1alpha1

import (
"github.com/pkg/errors"
core "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
"github.com/arangodb/kube-arangodb/pkg/apis/shared"
)

type ArangoMLStorageSpec struct {
// ListenPort defines on which port the sidecar container will be listening for connections
// +doc/default: 9201
ListenPort *uint16 `json:"listenPort,omitempty"`

// Resources holds resource requests & limits for container running the S3 proxy
// +doc/type: core.ResourceRequirements
// +doc/link: Documentation of core.ResourceRequirements|https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#resourcerequirements-v1-core
Resources core.ResourceRequirements `json:"resources,omitempty"`

S3 *ArangoMLStorageS3Spec `json:"s3,omitempty"`
// Mode defines how storage implementation should be deployed
Mode *ArangoMLStorageSpecMode `json:"mode,omitempty"`
// Backend defines how storage is implemented
Backend *ArangoMLStorageSpecBackend `json:"backend,omitempty"`
}

func (s *ArangoMLStorageSpec) Validate() error {
if s.S3 == nil {
return errors.New("Currently only s3 storage type is supported")
func (s *ArangoMLStorageSpec) GetMode() *ArangoMLStorageSpecMode {
if s == nil || s.Mode == nil {
return &ArangoMLStorageSpecMode{}
}
return s.Mode
}

return s.S3.Validate()
func (s *ArangoMLStorageSpec) GetBackend() *ArangoMLStorageSpecBackend {
if s == nil || s.Backend == nil {
return &ArangoMLStorageSpecBackend{}
}
return s.Backend
}

// SetDefaults fills in missing defaults
func (s *ArangoMLStorageSpec) SetDefaults() {
func (s *ArangoMLStorageSpec) Validate() error {
if s == nil {
return
s = &ArangoMLStorageSpec{}
}

resources := s.Resources
if len(resources.Requests) == 0 {
resources.Requests = make(core.ResourceList)
resources.Requests[core.ResourceCPU] = resource.MustParse("100m")
resources.Requests[core.ResourceMemory] = resource.MustParse("100m")
}
if len(resources.Limits) == 0 {
resources.Limits = make(core.ResourceList)
resources.Limits[core.ResourceCPU] = resource.MustParse("250m")
resources.Limits[core.ResourceMemory] = resource.MustParse("250m")
if err := shared.WithErrors(shared.PrefixResourceErrors("spec",
shared.PrefixResourceError("backend", s.Backend.Validate()),
shared.PrefixResourceError("mode", s.Mode.Validate()),
)); err != nil {
return err
}

return nil
}
50 changes: 50 additions & 0 deletions pkg/apis/ml/v1alpha1/storage_spec_backend.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
//
// DISCLAIMER
//
// Copyright 2023 ArangoDB GmbH, Cologne, Germany
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// Copyright holder is ArangoDB GmbH, Cologne, Germany
//

package v1alpha1

import (
"github.com/arangodb/kube-arangodb/pkg/apis/shared"
"github.com/arangodb/kube-arangodb/pkg/util/errors"
)

type ArangoMLStorageSpecBackend struct {
// S3 backend implements storage as a proxy to the provided S3 API endpoint
S3 *ArangoMLStorageSpecBackendS3 `json:"s3,omitempty"`
}

func (s *ArangoMLStorageSpecBackend) GetS3() *ArangoMLStorageSpecBackendS3 {
if s == nil || s.S3 == nil {
return &ArangoMLStorageSpecBackendS3{}
}
return s.S3
}

func (s *ArangoMLStorageSpecBackend) Validate() error {
if s == nil {
return errors.Newf("Backend is not specified")
}

if s.S3 == nil {
return errors.Newf("At least one backend needs to be defined")
}

return shared.WithErrors(shared.PrefixResourceError("s3", s.S3.Validate()))
}
Loading

0 comments on commit b2986a0

Please sign in to comment.