Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 #25984

tobiasbp · 2023-07-19T09:57:04Z

Description

Storage in s3 buckets (using minio) no longer works when using Gitea 1.20.0 with chart v9.0.0. Changing the Gitea version to 1.19.4 with chart v9.0.0 (same configuration) makes storage in s3 buckets work again as described in the issue here:
https://gitea.com/gitea/helm-chart/issues/469

The storage configuration is:

    storage:
      MINIO_ACCESS_KEY_ID: ***
      MINIO_BUCKET: my-existing-bucket
      MINIO_ENDPOINT: storage.googleapis.com
      MINIO_INSECURE_SKIP_VERIFY: false
      MINIO_SECRET_ACCESS_KEY: ***
      MINIO_USE_SSL: true
      SERVE_DIRECT: false
      STORAGE_TYPE: minio

With the above storage config, Gitea 1.20.0 says:

2023/07/18 16:33:54 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/18 16:33:54 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/18 16:33:54 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:my-existing-bucket with base path attachments/
2023/07/18 16:33:55 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again

Is Gitea/minio trying to create the existing bucket?

Gitea Version

1.20.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

In Kubernetes with chart v9.0.0.

Database

PostgreSQL

The text was updated successfully, but these errors were encountered:

tobiasbp · 2023-07-25T15:57:00Z

I have updated to Gitea 1.20.1 (Now using Gitea chart 9.0.4). If I configure MINIO_LOCATION to the location of my existong bucket, it sounds like minio tries to create the bucket??

Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
2023/07/25 15:51:00 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/07/25 15:51:00 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.1 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/07/25 15:51:00 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/07/25 15:51:00 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/07/25 15:51:00 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/07/25 15:51:00 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/07/25 15:51:00 cmd/web.go:153:serveInstalled() [I] Run mode: dev
2023/07/25 15:51:00 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/07/25 15:51:01 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/07/25 15:51:02 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/07/25 15:51:02 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/07/25 15:51:02 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/25 15:51:02 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/25 15:51:02 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/25 15:51:02 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: permission denied

lunny · 2023-07-25T16:05:44Z

I have updated to Gitea 1.20.1 (Now using Gitea chart 9.0.4). If I configure MINIO_LOCATION to the location of my existong bucket, it sounds like minio tries to create the bucket??

Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
2023/07/25 15:51:00 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/07/25 15:51:00 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.1 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/07/25 15:51:00 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/07/25 15:51:00 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/07/25 15:51:00 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/07/25 15:51:00 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/07/25 15:51:00 cmd/web.go:153:serveInstalled() [I] Run mode: dev
2023/07/25 15:51:00 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/07/25 15:51:01 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/07/25 15:51:02 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/07/25 15:51:02 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/07/25 15:51:02 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/25 15:51:02 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/25 15:51:02 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/25 15:51:02 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: permission denied

What's your workround? Maybe I can get some clues to fix it.

tobiasbp · 2023-07-25T16:14:03Z

What's your workround? Maybe I can get some clues to fix it.

I have no workaround? I can no longer use my S3 storage. It's just that the error i the log changes if I set MINIO_LOCATION.

I assume my problem is related to Rework storage settings -> #23911

But since I only have one section with storage config, I think my config should still work. I have a hard time following the changes being described in #23911

lunny · 2023-07-25T16:18:24Z

What's your workround? Maybe I can get some clues to fix it.

I have no workaround? I can no longer use my S3 storage.

I assume my problem is related to Rework storage settings -> #23911

Yes, I think so. Can you post your generated storage configuration in app.ini and remove the seurity informations. And I will figure it out tomorrow.

tobiasbp · 2023-07-25T16:43:01Z

Here is my app.ini. I have had to set storage.STORAGE_TYPE = local to get the pod to start up. The rest of the config in [storage] is untouched. I don't know if I should configure storage.MINIO_LOCATION or not? The error changes when I do (As described in earlier posts):

/var/lib/gitea $ cat /data/gitea/conf/app.ini


APP_NAME = Gitea (stg)
RUN_MODE = dev
WORK_PATH = /data

[storage]
MINIO_LOCATION = europe-north1
MINIO_USE_SSL = true
MINIO_ENDPOINT = storage.googleapis.com
STORAGE_TYPE = local
SERVE_DIRECT = false
MINIO_BUCKET = gitea-stg-aqrk
MINIO_SECRET_ACCESS_KEY = **REDACTED**
MINIO_ACCESS_KEY_ID = **REDACTED**
MINIO_INSECURE_SKIP_VERIFY = false

[indexer]
ISSUE_INDEXER_TYPE = db

[server]
LFS_JWT_SECRET = **REDACTED**
ROOT_URL = **REDACTED**
LFS_START_SERVER = true
ENABLE_PPROF = false
LANDING_PAGE = login
SSH_PORT = 22
APP_DATA_PATH = /data
SSH_LISTEN_PORT = 2222
PROTOCOL = http
HTTP_PORT = 3000
START_SSH_SERVER = true
DISABLE_SSH = true
SSH_DOMAIN = **REDACTED**
DOMAIN = **REDACTED**

[database]
NAME = gitea-db
SSL_MODE = disable
DB_TYPE = postgres
HOST = sql-proxy-gcloud-sqlproxy:5432
USER = gitea-user
PASSWD = **REDACTED**

[service]
SHOW_MILESTONES_DASHBOARD_PAGE = false
REQUIRE_SIGNIN_VIEW = true
DISABLE_REGISTRATION = true
EXPLORE = map[DISABLE_USERS_PAGE:true REQUIRE_SIGNIN_VIEW:true]
DEFAULT_ORG_VISIBILITY = private
DEFAULT_USER_VISIBILITY = private
DEFAULT_ALLOW_CREATE_ORGANIZATION = false

[repository]
DISABLE_STARS = true
DISABLE_MIGRATIONS = true
FORCE_PRIVATE = true
ROOT = /data/git/gitea-repositories
DISABLED_REPO_UNITS = repo.wiki, repo.ext_wiki, repo.packages, repo.projects, repo.issues, repo.ext_issues, repo.pulls
DEFAULT_PRIVATE = true

[oauth2_client]
REGISTER_EMAIL_CONFIRM = false
USERNAME = email
UPDATE_AVATAR = true
ENABLE_AUTO_REGISTRATION = true
OPENID_CONNECT_SCOPES = openid profile email

[security]
DISABLE_WEBHOOKS = true
INTERNAL_TOKEN = **REDACTED**
SECRET_KEY = **REDACTED**
INSTALL_LOCK = true

[session]
PROVIDER_CONFIG = redis://redis-1.source-access.private:6379/0?pool_size=100&idle_timeout=180s
PROVIDER = redis

[cache]
HOST = redis://redis-1.source-access.private:6379/0?pool_size=100&idle_timeout=180s
ENABLED = true
ADAPTER = redis

[admin]
DISABLE_REGULAR_ORG_CREATION = true
DEFAULT_EMAIL_NOTIFICATIONS = false

[metrics]
ENABLED = true

[oauth2]
JWT_SECRET = **REDACTED**

evanreichard · 2023-07-27T20:22:22Z

I'm assuming this is related, but since the update my [storage.packages] override no longer works. I don't think I violate the incompatibility detailed in #23911

For the time being I just set up a symbolic link.

My redacted configuration:

APP_NAME = Gitea: Git with a cup of tea
RUN_MODE = prod
RUN_USER = git
WORK_PATH = /data/gitea

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
SSH_DOMAIN = <REDACTED>
HTTP_PORT = 3000
ROOT_URL = <REDACTED>
DISABLE_SSH = false
SSH_PORT = 22
SSH_LISTEN_PORT = 22
LFS_START_SERVER = true
DOMAIN = <REDACTED>
LFS_JWT_SECRET = <REDACTED>
OFFLINE_MODE = false

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = <REDACTED>
PASSWD = <REDACTED>
SCHEMA = 
SSL_MODE = disable
CHARSET = utf8

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars
DISABLE_GRAVATAR = false
ENABLE_FEDERATED_AVATAR = true

[log]
ROOT_PATH = /data/gitea/log
MODE = file
LEVEL = info

[security]
INSTALL_LOCK = true
SECRET_KEY = <REDACTED>
INTERNAL_TOKEN = <REDACTED>

[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = false
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost
SHOW_REGISTRATION_BUTTON = false

[oauth2]
JWT_SECRET = <REDACTED>

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[ui]
DEFAULT_THEME = arc-green
THEMES = gitea,arc-green

[lfs]
PATH = /data/git/lfs

[attachment]
PATH = /data/gitea/attachments
MAX_SIZE = 50

[webhook]
ALLOWED_HOST_LIST = <REDACTED>

[storage.packages]
PATH = /packages

tobiasbp · 2023-08-01T11:47:18Z

Any idea as to what could be causing this issue? Is it a problem in Minio?

Lapotor · 2023-08-09T12:26:36Z

On v1.20.2 the error still appears.
I think that is a problem on the implementation because it tries to create the bucket on every storage.

…st and then create one if it doesn't exist (#26420) For some reason, the permission of the client_id and secret may cannot create bucket, so now we will check whether bucket does exist first and then try to create a bucket if it doesn't exist. Try to fix #25984 Co-authored-by: silverwind <[email protected]>

…st and then create one if it doesn't exist (go-gitea#26420) For some reason, the permission of the client_id and secret may cannot create bucket, so now we will check whether bucket does exist first and then try to create a bucket if it doesn't exist. Try to fix go-gitea#25984 Co-authored-by: silverwind <[email protected]>

@lunny

#26465) Backport #26420 by @lunny For some reason, the permission of the client_id and secret may cannot create bucket, so now we will check whether bucket does exist first and then try to create a bucket if it doesn't exist. Try to fix #25984 Co-authored-by: Lunny Xiao <[email protected]> Co-authored-by: silverwind <[email protected]>

tobiasbp added the type/bug label Jul 19, 2023

tobiasbp changed the title ~~Storage in S3 buckets using minio not working in 1.2.0~~ Storage in S3 buckets using minio not working in 1.20.0 Jul 19, 2023

tobiasbp changed the title ~~Storage in S3 buckets using minio not working in 1.20.0~~ Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 Jul 25, 2023

lunny added the issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP label Jul 27, 2023

lunny mentioned this issue Aug 10, 2023

Check first if minio bucket exists before trying to create it #26420

Merged

wxiaoguang closed this as completed in #26420 Aug 12, 2023

GiteaBot mentioned this issue Aug 12, 2023

Check first if minio bucket exists before trying to create it (#26420) #26465

Merged

github-actions bot locked as resolved and limited conversation to collaborators Sep 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 #25984

Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 #25984

tobiasbp commented Jul 19, 2023

tobiasbp commented Jul 25, 2023

lunny commented Jul 25, 2023 •

edited

Loading

tobiasbp commented Jul 25, 2023 •

edited

Loading

lunny commented Jul 25, 2023

tobiasbp commented Jul 25, 2023

evanreichard commented Jul 27, 2023 •

edited

Loading

tobiasbp commented Aug 1, 2023

Lapotor commented Aug 9, 2023

Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 #25984

Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 #25984

Comments

tobiasbp commented Jul 19, 2023

Description

Gitea Version

Can you reproduce the bug on the Gitea demo site?

Log Gist

Screenshots

Git Version

Operating System

How are you running Gitea?

Database

tobiasbp commented Jul 25, 2023

lunny commented Jul 25, 2023 • edited Loading

tobiasbp commented Jul 25, 2023 • edited Loading

lunny commented Jul 25, 2023

tobiasbp commented Jul 25, 2023

evanreichard commented Jul 27, 2023 • edited Loading

tobiasbp commented Aug 1, 2023

Lapotor commented Aug 9, 2023

lunny commented Jul 25, 2023 •

edited

Loading

tobiasbp commented Jul 25, 2023 •

edited

Loading

evanreichard commented Jul 27, 2023 •

edited

Loading