Fix Permission denied: '/var/lib/ckan/storage/uploads/group'

[brunopacheco1]

Expected Behavior

After editing the organization and click create, the organization is correctly persisted.

Current Behavior

After editing the organization, when create is clicked, 500 is thrown, and the following log is printed:

gdi-userportal-ckan-deployment-ckan-1  | 2023-12-27 20:11:46,577 INFO  [ckan.config.middleware.flask_app]  200 /api/i18n/en render time 0.010 seconds
gdi-userportal-ckan-deployment-ckan-1  | 2023-12-27 20:12:11,074 ERROR [ckan.config.middleware.flask_app] [Errno 13] Permission denied: '/var/lib/ckan/storage/uploads/group'
gdi-userportal-ckan-deployment-ckan-1  | Traceback (most recent call last):
gdi-userportal-ckan-deployment-ckan-1  |   File "/usr/lib/python3.10/site-packages/flask/app.py", line 1516, in full_dispatch_request
gdi-userportal-ckan-deployment-ckan-1  |     rv = self.dispatch_request()
gdi-userportal-ckan-deployment-ckan-1  |   File "/usr/lib/python3.10/site-packages/flask/app.py", line 1502, in dispatch_request
gdi-userportal-ckan-deployment-ckan-1  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
gdi-userportal-ckan-deployment-ckan-1  |   File "/usr/lib/python3.10/site-packages/flask/views.py", line 84, in view
gdi-userportal-ckan-deployment-ckan-1  |     return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
gdi-userportal-ckan-deployment-ckan-1  |   File "/usr/lib/python3.10/site-packages/flask/views.py", line 158, in dispatch_request
gdi-userportal-ckan-deployment-ckan-1  |     return current_app.ensure_sync(meth)(*args, **kwargs)
gdi-userportal-ckan-deployment-ckan-1  |   File "/srv/app/src/ckan/ckan/config/middleware/../../views/group.py", line 862, in post
gdi-userportal-ckan-deployment-ckan-1  |     group = _action(u'group_create')(context, data_dict)
gdi-userportal-ckan-deployment-ckan-1  |   File "/srv/app/src/ckan/ckan/logic/__init__.py", line 580, in wrapped
gdi-userportal-ckan-deployment-ckan-1  |     result = _action(context, data_dict, **kw)
gdi-userportal-ckan-deployment-ckan-1  |   File "/srv/app/src/ckan/ckan/logic/action/create.py", line 946, in organization_create
gdi-userportal-ckan-deployment-ckan-1  |     return _group_or_org_create(context, data_dict, is_org=True)
gdi-userportal-ckan-deployment-ckan-1  |   File "/srv/app/src/ckan/ckan/logic/action/create.py", line 739, in _group_or_org_create
gdi-userportal-ckan-deployment-ckan-1  |     upload = uploader.get_uploader('group')
gdi-userportal-ckan-deployment-ckan-1  |   File "/srv/app/src/ckan/ckan/lib/uploader.py", line 61, in get_uploader
gdi-userportal-ckan-deployment-ckan-1  |     upload = Upload(upload_to, old_filename)
gdi-userportal-ckan-deployment-ckan-1  |   File "/srv/app/src/ckan/ckan/lib/uploader.py", line 129, in __init__
gdi-userportal-ckan-deployment-ckan-1  |     os.makedirs(self.storage_path)
gdi-userportal-ckan-deployment-ckan-1  |   File "/usr/lib/python3.10/os.py", line 225, in makedirs
gdi-userportal-ckan-deployment-ckan-1  |     mkdir(name, mode)
gdi-userportal-ckan-deployment-ckan-1  | PermissionError: [Errno 13] Permission denied: '/var/lib/ckan/storage/uploads/group'
gdi-userportal-ckan-deployment-ckan-1  | 2023-12-27 20:12:11,111 INFO  [ckan.config.middleware.flask_app]  500 /organization/new render time 0.061 seconds

Possible Solution

The folder /var/lib/ckan/storage/uploads/group does not exist when the exception happens. So I tested two things:

• mkdir /var/lib/ckan/storage/uploads/group
• chmod u+rwx /var/lib/ckan

I believe there is some inconsistence with user permissions. The second command seems more reasonable to me, but I am not aware of other possible implications, that is why I oppened this bug.

Steps to Reproduce

1. Build a custom CKAN docker image (e.g. https://github.com/GenomicDataInfrastructure/gdi-userportal-ckan-docker).
2. Run docker image using docker compose v2 (e.g. https://github.com/GenomicDataInfrastructure/gdi-userportal-ckan-deployment).
3. Add organization
4. Click create

Context (Environment)

OS: Darwin 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64
Docker: Docker version 24.0.6, build ed223bc820
Colima: colima version 0.5.5

Detailed Description

On line 94, the user ckan is set as owner of the folder ${CKAN_STORAGE_PATH}. On top of it, we could ensure the owner has rights to read, write and execute files.

Possible Implementation

Replace line 93 and 94 from ckan-base Dockerfile by the following command?

RUN mkdir -p ${CKAN_STORAGE_PATH} && \
    chown -R ckan:ckan ${CKAN_STORAGE_PATH} && \
    chmod u+rwx ${CKAN_STORAGE_PATH}

[kowh-ai]

I have just tested your "Steps to Reproduce" and I don't get the same behaviour. I don't get an error.

What do you mean by a "custom image" I'm building and running the CKAN image from https://github.com/ckan/ckan-docker/tree/master/ckan which by default is using the CKAN base image from https://github.com/ckan/ckan-docker-base/tree/main/ckan-2.10/base

[dev-rke]

I have this issue, too, when creating new organisations.
Currently i am using

FROM ckan/ckan-base:2.10.4

in my Dockerfile to create a custom image, so it's the production image, not the DEV image.
System is Ubuntu 22.04, Docker 24.0.5.

Directory structure + permissions:

08a46e208f94:/var/lib/ckan# ls -lah
total 20K    
drwxr-xr-x    1 ckan     ckan        4.0K May 23 10:07 .
drwxr-xr-x    1 root     root        4.0K Mar 13 11:34 ..
drwxr-xr-x    3 root     root        4.0K May 23 10:07 storage
drwxr-xr-x    7 ckan     ckan        4.0K May 23 09:07 webassets
08a46e208f94:/var/lib/ckan# ls -lah storage
total 12K    
drwxr-xr-x    3 root     root        4.0K May 23 10:07 .
drwxr-xr-x    1 ckan     ckan        4.0K May 23 10:07 ..
drwxr-xr-x    3 root     root        4.0K May 23 10:07 uploads
08a46e208f94:/var/lib/ckan# ls -lah storage/uploads
total 12K    
drwxr-xr-x    3 root     root        4.0K May 23 10:07 .
drwxr-xr-x    3 root     root        4.0K May 23 10:07 ..
drwxr-xr-x    2 root     root        4.0K May 23 10:07 user

As you can see, the directory structure has been created by root user, instead of the ckan user.

The proposed approach, to change permissions via chmod u+rwx /var/lib/ckan did not solve the issue on my side - at least this needs to be recursive, but won't work, when the storage folder will be created later.
The main issue seems to be, that CKAN creates the storage folder as root user, instead applying the ckan user of the uswgi service.
I assume, the reason could be, that the CLI CKAN command is always executed as root user, instead of running it as regular user and creates the folder with root permissions.

Edit:
Investigated further.
I found out, that the wrong behaviour happens, when one creates manually a user via CLI

ckan user add test name=test [email protected]

BEFORE creating an organisation.

This creates a path /var/lib/ckan/storage/uploads/user with the permission issue described in my previous posting.

When i remove the storage directory manually and run ckan in ckan user context via

$ docker-compose exec -u ckan ckan bash
0e4980751199:/srv/app$ ckan user add test2 name=test2 [email protected]
[...]
0e4980751199:/var/lib/ckan$ ls -lah
total 20K    
drwxr-xr-x    1 ckan     ckan        4.0K May 23 11:54 .
drwxr-xr-x    1 root     root        4.0K Mar 13 11:34 ..
drwxr-xr-x    3 ckan     ckan        4.0K May 23 11:54 storage
drwxr-xr-x    7 ckan     ckan        4.0K May 23 11:39 webassets

Then permissions are fine and organisations can be created.

Therefore the bug seems to be within the user add command, do you agree?