Skip to content

Commit

Permalink
feat: User defined git repo for GitOps deployment (devtron-labs#4281)
Browse files Browse the repository at this point in the history
* wip: global gitops config updated

* feat: devtron app stage API

* validate custom git repo url helm apps

* wip

* updated: custom git repo url validate function

* wip: devtron app flow

* feat: handling for helm apps and edge cases of devtron apps

* updated migration

* fixed: app stage api query

* updated FetchAppStageStatus function

* fixed: import cycle error

* fix: validation handling

* fixed: pg no rows error

* fixed: nil pointer issue

* wire files added

* updated: validation function

* feat: updated validations and added IsGitOpsRepoNotConfigured

* feat: added IsGitOpsRepoNotConfigured to app-wf API

* fixed: create cd pipeline flow

* feat: handling for azure devops

* added GitOpsRepoMigrationRequired for devtron apps

* fixed: pg no row for app-wf API

* fixed: pg no row for app-wf API

* fixed: pg no row for app-wf API

* fixed: pg no row for app-wf API

* fixed: isGitOpsRepoNotConfigured in app-wf API

* fixed condition for cd pipeline create

* refactored conditions

* fixed: IsGitOpsRepoConfiguredForDevtronApps condition

* refactored: condition

* fixed: cd pipeline create issues

* fixed: GetGitOpsConfigurationOfApp

* fixed: GetGitOpsConfigurationOfApp

* fixed: GetGitOpsConfigurationOfApp

* fixed: RegisterInArgo logic

* wip

* handling for pre and post CD

* updated flag for app group

* fixed: error handling

* PR comments resolved

* removed unnecessary changes

* fixed: auto trigger

* fixed: releaseErr skipped

* fixed: gitops repo migration

* updated: migration number

* fixed: API response

* chore: fixed helm app default gitops url

* fixed: repo creation helm app

* added logger to AddDeploymentEnforcementConfig

* updated: down migration

* updated: comments

* chore: removed unnecessary statement

* removed unnecessary todo

* feat: Gitops repo refactor (devtron-labs#4492)

* refactored and backward compatibility handling for gitops repo name

* fixed: wire issue

* fixed: query condition

* beta: fix argocd register

* beta: fix argocd register

* beta: added log

* beta: added TODO

* beta: fixed missing payload

* beta: fixed missing payload

* added: ssh clone stage to validation

* fix: Error code changed for resource api (devtron-labs#4414)

* letting build abort even after starting stage of ci_workflow

* error 500 removed

* forceAbort query param in cancle stage

* minor fix

* minor fix

* error 500 removed

* some random change

* fix

* some error messaging changed

* if workflow already cancelled then return from ci success event

* helm uninstall issue fixed

* status code changes

* changed err code and msg in getWorkflowLogs in cihandler

* k8s issue fixed for 409

* when workflow not found to terminate then throw bad req instead of 5xx

* fix

* fix

* incorporated grpc code check in cd-pipeline/trigger api in createHelmAppForCdPipeline func

* small fix

* minor refactoring

* revert cd-pipeline/trigger 5xx fix

* bug fix

---------

Co-authored-by: Prakash Kumar <[email protected]>
Co-authored-by: ShashwatDadhich <[email protected]>

* updated API error

* removed empty repo in Custom Validation and added retry to RegisterInArgo

* removed empty repo in Custom Validation and added retry to RegisterInArgo

* fixed: clone validation

* added validation for unique repo

* fixed: missing payload issue

* fixed: RegisterInArgo func

* fixed: RegisterInArgo func

* fixed nil pointer issue

* fixed: nil pointer issue

* removed: custom validation

* removed: unused wire injections

* added: org git repo validation

* fixed: org git repo validation

* fixed api error

* fixed api error

* fixed api error

* feat: refactoring authorisation checks and support of super-admin in permissions groups. (devtron-labs#4433)

* refactored directories

* refactoring in enforcer code, removed enforceByEmail/InBatch usages

* removed isSuperAdmin method calls

* updated missed auth check

* replaced userService.GetById methods by lighter calls

* replaced getEmailFromToken by lighter calls

* removed remaining 2 usages of enforceByEmail

* self registration roles service renaming

* updated authenticator version for userVerifier changes

* added sql file for creating user group

* test: Origin/self register auth change (devtron-labs#4479)

* global-authorisation and resfactoring

* rbacfor deletematerial

* script number change

* role group superAdmin support

* script change

* sql script delete

* super-admin role group

* chart group duplication fix

* api-token

* check user roles for super admin group

* updated authenticator version

---------

Co-authored-by: Shivam Nagar <[email protected]>
Co-authored-by: Shivam-nagar23 <[email protected]>

* fixed: default git repo name

* fixed: default validation

---------

Co-authored-by: Aditya Ranjan <[email protected]>
Co-authored-by: Prakash Kumar <[email protected]>
Co-authored-by: ShashwatDadhich <[email protected]>
Co-authored-by: kartik-579 <[email protected]>
Co-authored-by: Shivam Nagar <[email protected]>
Co-authored-by: Shivam-nagar23 <[email protected]>

* sql script update (devtron-labs#4522)

* fix: panic in app clone service for [linked ci, external ci, linked cd] cases (devtron-labs#4526)

* fix for panic in app clone service for linked ci case

* updated fix

* dex-config namespace configurable (devtron-labs#4499)

* adding isHook and hook type in ResourceNode obj in proto file (devtron-labs#4472)

* feat: Adds support for gRPC in health probes (devtron-labs#4495)

* Adds support for gRPC in health probes

* grpc support in deployment, sts, rollout chart

* migration for chart ref 4.19

* adds hidden files .helmignore and ,image_descriptor_template.json in 4.19.0

* adds copacetic plugin migration

* Revert "adds copacetic plugin migration"

This reverts commit 60df5d1.

* Update statefulset.yaml

replaced liveness with readiness wherever required

* deleted ref 4.19

---------

Co-authored-by: Rupin Solanki <[email protected]>

* fix: show deployed time for artifacts those were deployed in past (devtron-labs#4446)

* set deployed and deployed at fields values in artifacts

* set deployed and deployed at fields values in artifacts

* remove unused ids slice

* review changes

* doc: Created Catalog Framework Doc (devtron-labs#4512)

* blank draft

* Catalog Framework Doc

* Incorporated SME Feedback

* Incorporated PM Feedback

* docs: Segregated SSO Docs + Added Auto-Assign Permissions Feature (devtron-labs#4493)

* Segregated SSO Docs + Added Auto-Assign Permissions Feature

* Incorporated Feedback

* Added Redirect URI snaps

* Incorporated PM Feedback

* fix: error handling for trigger release (devtron-labs#4488)

* fixed: error handling for trigger release

* removed: common return statements

* fix: resource tree err handling (devtron-labs#4530)

* improved error handling for resource tree

* added return to avoid multiple writes

* fix: CD stage trigger is not working for external CI (devtron-labs#4440)

* fix: CD stage trigger for external CI

* feat: handling for DataSource type backward compatibility

* migration for deprecated DataSourceType

* updated ArtifactsSourceType const

* updated: ci_artifact.data_source migration condition (devtron-labs#4548)

* fix: Image-Scanner status for failed request (devtron-labs#4513)

* retry envs added

* json naming

* migration for image scan plugin and retry env fix

* fix

* image scan migration script updated - added new input variables

* migration updated

* migration updated

* name changed

* refactor

* added default value and change script no

* chore: added sql-validator in git-hub action (devtron-labs#4255)

* added sql-validator in git-hub action

* removed exit commands

* edited the grep with whole word

* modified

* modified comments

---------

Co-authored-by: Badal Kumar Prusty <[email protected]>

* feat: Adds Copacetic plugin to patch vulnerable images  (devtron-labs#4566)

* Adds copacetic plugin

* copacetic plugin: adds plugin icon and updates url in migration file

* Update 211_copacetic_plugin_v1_0_0.up.sql

adds copa patch command exit condition

* Update 211_copacetic_plugin_v1_0_0.up.sql

adds exit 1 if patching multi architecture image

* dummy commit 211_copacetic_plugin_v1_0_0.up.sql

dummy commit

* chore: added sql-validator in git-hub action (devtron-labs#4255)

* added sql-validator in git-hub action

* removed exit commands

* edited the grep with whole word

* modified

* modified comments

---------

Co-authored-by: Badal Kumar Prusty <[email protected]>

* updates migration no.

---------

Co-authored-by: Badal Kumar <[email protected]>
Co-authored-by: Badal Kumar Prusty <[email protected]>

* feat: Create Dockerslim plugin and Create EKS Cluster plugin (devtron-labs#4525)

* Plugins icon

* eks and dockerslim plugin sql up

* eks and dockerslim plugin down script

* dockerslim plugin after review PR

* Changes after review dockerslim

* image changes

* new image dockerslim

* eks creation plugin

* Final PR commit, eks and dockerslim plugins

* feat: making pre,post,deploy triggers flows idempotent (devtron-labs#4486)

* update common-lib

* added idempotency code and common-lib version bump

* added the optionsal validations pubsub msg pre-processing-hooks logic and added duplicate trigger check as a validation for pre,post,deploy trigger flows

* bump common-lib

* added callback logger func

* sql scripts added

* query fix

* query fix

* fix

* refactoring

* remove dag exececutor dependency from ciEeventHalndler Sevice

* bump common-lib

* move context to trigger context

* add logs

* migration version fix

* remove test data

* update script number

---------

Co-authored-by: Kripansh <[email protected]>

* chore: refactoring branch merge

* fix: sql script added for putting Identifier type=0 for all null values present in generic_note (devtron-labs#4568)

* script for initializing null identifier type col to 0(cluster type)

* script fix

* empty script for 216

* 216 up script revert

* fix: 500 fixes second iteration (devtron-labs#4464)

* FetchChartNotes return 4xx when installed app or installed app versions not found

* GetDeploymentHistory and GetInstalledAppVersion to return 4xx when installed app or installed app versions not found

* fixes

* adding pg:",discard_unknown_columns" in tables where doesn't exist

* fix for Git Sensor response

* fix for inception

* fix for k8s resource delete

* fix for pipeline already exists

* fix for pipeline already exists

* fix

* fix

* fix

* fix

* go mod version changes

* merged

* common lib main version sync

* main merge

---------

Co-authored-by: ShashwatDadhich <[email protected]>

* chore: updated migration number

* chore: update stale data status code

* chore: update error status code

* updated: custom gitops repo handling

* added: API code for conflict state identification

* chore: reverted error condition

* code refactoring

* fixed: nil pointer

* fix: gitops auto fix repo issue

* fix: devtron-app gitops repo migration

* chore: added wire gen file

* fix: GitOps repo migration in devtron app

* fix: GitOps repo migration in devtron app

* chore: update migration number

* fix: org url validation

* fix: is gitops configured flag data in workflow response

* chore: fixed validation condition

* fix: gitops repo url for helm installation

* replaced handleCustomGitOpsRepoValidation func

* added default value for env variable

* fix: review comment fixes for user defined GitOps repo url (devtron-labs#4729)

* incorporated review comments

* updated import alias

* updated github host url validation

* fix: gitops repo url is visible for cli migrated helm apps

---------

Co-authored-by: Aditya Ranjan <[email protected]>
Co-authored-by: Prakash Kumar <[email protected]>
Co-authored-by: ShashwatDadhich <[email protected]>
Co-authored-by: kartik-579 <[email protected]>
Co-authored-by: Shivam Nagar <[email protected]>
Co-authored-by: Shivam-nagar23 <[email protected]>
Co-authored-by: Shashwat Dadhich <[email protected]>
Co-authored-by: tayalrishabh96 <[email protected]>
Co-authored-by: Rupin Solanki <[email protected]>
Co-authored-by: Gireesh Naidu <[email protected]>
Co-authored-by: ashokdevtron <[email protected]>
Co-authored-by: ashishdevtron <[email protected]>
Co-authored-by: Badal Kumar <[email protected]>
Co-authored-by: Badal Kumar Prusty <[email protected]>
Co-authored-by: Rupin Solanki <[email protected]>
Co-authored-by: Kripansh <[email protected]>
  • Loading branch information
17 people authored and raunit-verma committed Mar 11, 2024
1 parent 48c4486 commit 1037a85
Show file tree
Hide file tree
Showing 81 changed files with 2,513 additions and 1,781 deletions.
8 changes: 6 additions & 2 deletions Wire.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,12 +109,14 @@ import (
repository4 "github.com/devtron-labs/devtron/pkg/appStore/chartGroup/repository"
"github.com/devtron-labs/devtron/pkg/appStore/installedApp/service/FullMode"
deployment3 "github.com/devtron-labs/devtron/pkg/appStore/installedApp/service/FullMode/deployment"
"github.com/devtron-labs/devtron/pkg/appStore/installedApp/service/FullMode/deploymentTypeChange"
"github.com/devtron-labs/devtron/pkg/appStore/installedApp/service/FullMode/resource"
"github.com/devtron-labs/devtron/pkg/appWorkflow"
"github.com/devtron-labs/devtron/pkg/attributes"
"github.com/devtron-labs/devtron/pkg/build"
"github.com/devtron-labs/devtron/pkg/bulkAction"
"github.com/devtron-labs/devtron/pkg/chart"
"github.com/devtron-labs/devtron/pkg/chart/gitOpsConfig"
chartRepoRepository "github.com/devtron-labs/devtron/pkg/chartRepo/repository"
"github.com/devtron-labs/devtron/pkg/commonService"
delete2 "github.com/devtron-labs/devtron/pkg/delete"
Expand Down Expand Up @@ -344,6 +346,8 @@ func InitializeApp() (*App, error) {

// end

gitOpsConfig.NewDevtronAppGitOpConfigServiceImpl,
wire.Bind(new(gitOpsConfig.DevtronAppGitOpConfigService), new(*gitOpsConfig.DevtronAppGitOpConfigServiceImpl)),
chart.NewChartServiceImpl,
wire.Bind(new(chart.ChartService), new(*chart.ChartServiceImpl)),
bulkAction.NewBulkUpdateServiceImpl,
Expand Down Expand Up @@ -541,8 +545,8 @@ func InitializeApp() (*App, error) {
wire.Bind(new(FullMode.InstalledAppDBExtendedService), new(*FullMode.InstalledAppDBExtendedServiceImpl)),
resource.NewInstalledAppResourceServiceImpl,
wire.Bind(new(resource.InstalledAppResourceService), new(*resource.InstalledAppResourceServiceImpl)),
FullMode.NewInstalledAppDeploymentTypeChangeServiceImpl,
wire.Bind(new(FullMode.InstalledAppDeploymentTypeChangeService), new(*FullMode.InstalledAppDeploymentTypeChangeServiceImpl)),
deploymentTypeChange.NewInstalledAppDeploymentTypeChangeServiceImpl,
wire.Bind(new(deploymentTypeChange.InstalledAppDeploymentTypeChangeService), new(*deploymentTypeChange.InstalledAppDeploymentTypeChangeServiceImpl)),

appStoreRestHandler.NewAppStoreRouterImpl,
wire.Bind(new(appStoreRestHandler.AppStoreRouter), new(*appStoreRestHandler.AppStoreRouterImpl)),
Expand Down
2 changes: 1 addition & 1 deletion api/appStore/AppStoreRouter.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package appStore

import (
chartProvider "github.com/devtron-labs/devtron/api/appStore/chartProvider"
"github.com/devtron-labs/devtron/api/appStore/chartProvider"
appStoreDeployment "github.com/devtron-labs/devtron/api/appStore/deployment"
appStoreDiscover "github.com/devtron-labs/devtron/api/appStore/discover"
appStoreValues "github.com/devtron-labs/devtron/api/appStore/values"
Expand Down
5 changes: 3 additions & 2 deletions api/appStore/InstalledAppRestHandler.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"fmt"
client "github.com/devtron-labs/devtron/api/helm-app/gRPC"
"github.com/devtron-labs/devtron/pkg/appStore/installedApp/service/FullMode"
"github.com/devtron-labs/devtron/pkg/appStore/installedApp/service/FullMode/deploymentTypeChange"
"github.com/devtron-labs/devtron/pkg/appStore/installedApp/service/FullMode/resource"
"github.com/devtron-labs/devtron/pkg/bean"
"net/http"
Expand Down Expand Up @@ -93,7 +94,7 @@ type InstalledAppRestHandlerImpl struct {
installedAppRepository repository.InstalledAppRepository
K8sApplicationService application2.K8sApplicationService
appCrudOperationService app2.AppCrudOperationService
installedAppDeploymentTypeChangeService FullMode.InstalledAppDeploymentTypeChangeService
installedAppDeploymentTypeChangeService deploymentTypeChange.InstalledAppDeploymentTypeChangeService
}

func NewInstalledAppRestHandlerImpl(Logger *zap.SugaredLogger, userAuthService user.UserService,
Expand All @@ -105,7 +106,7 @@ func NewInstalledAppRestHandlerImpl(Logger *zap.SugaredLogger, userAuthService u
cdApplicationStatusUpdateHandler cron.CdApplicationStatusUpdateHandler,
installedAppRepository repository.InstalledAppRepository,
appCrudOperationService app2.AppCrudOperationService,
installedAppDeploymentTypeChangeService FullMode.InstalledAppDeploymentTypeChangeService) *InstalledAppRestHandlerImpl {
installedAppDeploymentTypeChangeService deploymentTypeChange.InstalledAppDeploymentTypeChangeService) *InstalledAppRestHandlerImpl {
return &InstalledAppRestHandlerImpl{
Logger: Logger,
userAuthService: userAuthService,
Expand Down
35 changes: 0 additions & 35 deletions api/bean/GitOpsConfig.go

This file was deleted.

55 changes: 55 additions & 0 deletions api/bean/gitOps/GitOpsConfig.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package gitOps

import "time"

type GitOpsConfigDto struct {
Id int `json:"id,omitempty"`
Provider string `json:"provider"`
Username string `json:"username"`
Token string `json:"token"`
GitLabGroupId string `json:"gitLabGroupId"`
GitHubOrgId string `json:"gitHubOrgId"`
Host string `json:"host"`
Active bool `json:"active"`
AzureProjectName string `json:"azureProjectName"`
BitBucketWorkspaceId string `json:"bitBucketWorkspaceId"`
BitBucketProjectKey string `json:"bitBucketProjectKey"`
AllowCustomRepository bool `json:"allowCustomRepository"`

// TODO refactoring: create different struct for internal fields
GitRepoName string `json:"-"`
UserEmailId string `json:"-"`
Description string `json:"-"`
UserId int32 `json:"-"`
}

type GitRepoRequestDto struct {
Host string `json:"host"`
Provider string `json:"provider"`
GitRepoName string `json:"gitRepoName"`
Username string `json:"username"`
UserEmailId string `json:"userEmailId"`
Token string `json:"token"`
GitLabGroupId string `json:"gitLabGroupId"`
GitHubOrgId string `json:"gitHubOrgId"`
AzureProjectName string `json:"azureProjectName"`
BitBucketWorkspaceId string `json:"bitBucketWorkspaceId"`
BitBucketProjectKey string `json:"bitBucketProjectKey"`
}

type DetailedErrorGitOpsConfigResponse struct {
SuccessfulStages []string `json:"successfulStages"`
StageErrorMap map[string]string `json:"stageErrorMap"`
ValidatedOn time.Time `json:"validatedOn"`
DeleteRepoFailed bool `json:"deleteRepoFailed"`
ValidationSkipped bool `json:"validationSkipped"`
}

const (
GIT_REPO_DEFAULT = "Default"
GIT_REPO_NOT_CONFIGURED = "NOT_CONFIGURED" // The value of the constant has been used in the migration script for `custom_gitops_repo_url`; Need to add another migration script if the value is updated.
)

func IsGitOpsRepoNotConfigured(gitRepoUrl string) bool {
return len(gitRepoUrl) == 0 || gitRepoUrl == GIT_REPO_NOT_CONFIGURED
}
26 changes: 16 additions & 10 deletions api/restHandler/GitOpsConfigRestHandler.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ package restHandler
import (
"encoding/json"
"errors"
bean2 "github.com/devtron-labs/devtron/api/bean/gitOps"
"net/http"
"strconv"

bean2 "github.com/devtron-labs/devtron/api/bean"
"github.com/devtron-labs/devtron/api/restHandler/common"
"github.com/devtron-labs/devtron/pkg/auth/authorisation/casbin"
"github.com/devtron-labs/devtron/pkg/auth/user"
Expand Down Expand Up @@ -84,25 +84,26 @@ func (impl GitOpsConfigRestHandlerImpl) CreateGitOpsConfig(w http.ResponseWriter
var bean bean2.GitOpsConfigDto
err = decoder.Decode(&bean)
if err != nil {
impl.logger.Errorw("request err, CreateGitOpsConfig", "err", err, "payload", bean)
impl.logger.Errorw("request err, createGitOpsConfig", "err", err, "payload", bean)
common.WriteJsonResp(w, err, nil, http.StatusBadRequest)
return
}
bean.UserId = userId
impl.logger.Infow("request payload, CreateGitOpsConfig", "err", err, "payload", bean)
impl.logger.Infow("request payload, createGitOpsConfig", "err", err, "payload", bean)
err = impl.validator.Struct(bean)
if err != nil {
impl.logger.Errorw("validation err, CreateGitOpsConfig", "err", err, "payload", bean)
impl.logger.Errorw("validation err, createGitOpsConfig", "err", err, "payload", bean)
common.WriteJsonResp(w, err, nil, http.StatusBadRequest)
return
}
detailedErrorGitOpsConfigResponse, err := impl.gitOpsConfigService.ValidateAndCreateGitOpsConfig(&bean)
if err != nil {
impl.logger.Errorw("service err, SaveGitRepoConfig", "err", err, "payload", bean)
common.WriteJsonResp(w, err, nil, http.StatusInternalServerError)
return
}
common.WriteJsonResp(w, nil, detailedErrorGitOpsConfigResponse, http.StatusOK)

return
}

func (impl GitOpsConfigRestHandlerImpl) UpdateGitOpsConfig(w http.ResponseWriter, r *http.Request) {
Expand Down Expand Up @@ -131,25 +132,26 @@ func (impl GitOpsConfigRestHandlerImpl) UpdateGitOpsConfig(w http.ResponseWriter
bean.Token = res.Token
}
if err != nil {
impl.logger.Errorw("request err, UpdateGitOpsConfig", "err", err, "payload", bean)
impl.logger.Errorw("request err, updateGitOpsConfig", "err", err, "payload", bean)
common.WriteJsonResp(w, err, nil, http.StatusBadRequest)
return
}
bean.UserId = userId
impl.logger.Infow("request payload, UpdateGitOpsConfig", "payload", bean)
impl.logger.Infow("request payload, updateGitOpsConfig", "payload", bean)
err = impl.validator.Struct(bean)
if err != nil {
impl.logger.Errorw("validation err, UpdateGitOpsConfig", "err", err, "payload", bean)
impl.logger.Errorw("validation err, updateGitOpsConfig", "err", err, "payload", bean)
common.WriteJsonResp(w, err, nil, http.StatusBadRequest)
return
}
detailedErrorGitOpsConfigResponse, err := impl.gitOpsConfigService.ValidateAndUpdateGitOpsConfig(&bean)
if err != nil {
impl.logger.Errorw("service err, UpdateGitOpsConfig", "err", err, "payload", bean)
impl.logger.Errorw("service err, updateGitOpsConfig", "err", err, "payload", bean)
common.WriteJsonResp(w, err, nil, http.StatusInternalServerError)
return
}
common.WriteJsonResp(w, nil, detailedErrorGitOpsConfigResponse, http.StatusOK)

return
}

func (impl GitOpsConfigRestHandlerImpl) GetGitOpsConfigById(w http.ResponseWriter, r *http.Request) {
Expand Down Expand Up @@ -196,15 +198,19 @@ func (impl GitOpsConfigRestHandlerImpl) GitOpsConfigured(w http.ResponseWriter,
return
}
gitopsConfigured := false
allowCustomRepository := false
if len(result) > 0 {
for _, gitopsConf := range result {
if gitopsConf.Active {
gitopsConfigured = true
allowCustomRepository = gitopsConf.AllowCustomRepository
break
}
}
}
res := make(map[string]bool)
res["exists"] = gitopsConfigured
res["allowCustomRepository"] = allowCustomRepository
common.WriteJsonResp(w, err, res, http.StatusOK)
}

Expand Down
Loading

0 comments on commit 1037a85

Please sign in to comment.