Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

library envs share a namespace with filesystem envs and cannot have the same name #72

Closed
jaimergp opened this issue Jun 24, 2021 · 5 comments
Closed

library envs share a namespace with filesystem envs and cannot have the same name #72

jaimergp opened this issue Jun 24, 2021 · 5 comments
Labels
type: bug 🐛 Something isn't working
Milestone
Release 0.3.0

Comments

@jaimergp
Copy link
Member

With a fresh local deployment (no previous env created), creating an empty env with just python results in a second copy.

Env:

name: env-name-1
channels:
  - conda-forge
dependencies:
  - python

Relevant log lines:

conda-store-server_1  | [2021-06-24 14:47:53,120] {/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/werkzeug/_internal.py:225} INFO - 172.26.0.1 - - [24/Jun/2021 14:47:53] "GET /create/ HTTP/1.1" 200 -
conda-store-server_1  | [2021-06-24 14:48:29,630] {/opt/conda-store-server/conda_store_server/app.py:96} INFO - registering specification name=env-name-1 sha256=44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8
conda-store-server_1  | [2021-06-24 14:48:29,653] {/opt/conda-store-server/conda_store_server/app.py:102} INFO - scheduling specification for build name=env-name-1 sha256=44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8
conda-store-server_1  | [2021-06-24 14:48:29,672] {/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/werkzeug/_internal.py:225} INFO - 172.26.0.1 - - [24/Jun/2021 14:48:29] "POST /create/ HTTP/1.1" 302 -
conda-store-server_1  | [2021-06-24 14:48:29,836] {/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/werkzeug/_internal.py:225} INFO - 172.26.0.1 - - [24/Jun/2021 14:48:29] "GET / HTTP/1.1" 200 -

conda-store-build_1   | [2021-06-24 14:48:54,514] {/opt/conda-store-server/conda_store_server/conda.py:38} INFO - downloading channeldata url=https://repo.anaconda.com/pkgs/main/channeldata.json
conda-store-build_1   | [2021-06-24 14:48:54,900] {/opt/conda-store-server/conda_store_server/conda.py:46} INFO - downloading repodata url=https://repo.anaconda.com/pkgs/main/linux-64/repodata.json
conda-store-build_1   | [2021-06-24 14:48:57,214] {/opt/conda-store-server/conda_store_server/conda.py:46} INFO - downloading repodata url=https://repo.anaconda.com/pkgs/main/noarch/repodata.json
conda-store-build_1   | [2021-06-24 14:49:21,735] {/opt/conda-store-server/conda_store_server/build.py:111} INFO - polling interval set to 10 seconds
conda-store-build_1   | [2021-06-24 14:49:21,793] {/opt/conda-store-server/conda_store_server/app.py:96} INFO - registering specification name=env-name-1 sha256=b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e
conda-store-build_1   | [2021-06-24 14:49:21,815] {/opt/conda-store-server/conda_store_server/app.py:102} INFO - scheduling specification for build name=env-name-1 sha256=b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e
conda-store-build_1   | [2021-06-24 14:49:21,858] {/opt/conda-store-server/conda_store_server/app.py:96} INFO - registering specification name=env-name-2 sha256=45d17df515991ab4b0d9176c96cabe2a641da43fb8a8958b054c07f9c431cc70
conda-store-build_1   | [2021-06-24 14:49:21,866] {/opt/conda-store-server/conda_store_server/app.py:102} INFO - scheduling specification for build name=env-name-2 sha256=45d17df515991ab4b0d9176c96cabe2a641da43fb8a8958b054c07f9c431cc70
conda-store-build_1   | [2021-06-24 14:49:21,896] {/opt/conda-store-server/conda_store_server/build.py:119} INFO - number of queued conda builds 3
conda-store-build_1   | [2021-06-24 14:49:21,921] {/opt/conda-store-server/conda_store_server/build.py:129} INFO - number of schedulable conda builds 3
conda-store-build_1   | [2021-06-24 14:49:21,959] {/opt/conda-store-server/conda_store_server/build.py:157} INFO - building /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 symlinked to /data/envs/env-name-1
conda-store-build_1   | [2021-06-24 14:49:21,960] {/opt/conda-store-server/conda_store_server/build.py:159} INFO - previously unfinished build of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 cleaning directory

conda-store-build_1   | [2021-06-24 14:51:33,015] {/opt/conda-store-server/conda_store_server/utils.py:42} INFO - building /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 took 131.052 [s]
conda-store-build_1   | [2021-06-24 14:51:33,041] {/opt/conda-store-server/conda_store_server/build.py:189} INFO - modifying permissions of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 to permissions=775
conda-store-build_1   | [2021-06-24 14:51:45,527] {/opt/conda-store-server/conda_store_server/utils.py:42} INFO - chmod of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 took 12.486 [s]
conda-store-build_1   | [2021-06-24 14:51:45,528] {/opt/conda-store-server/conda_store_server/build.py:200} INFO - modifying permissions of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 to uid=1000 and gid=100
conda-store-build_1   | [2021-06-24 14:52:00,788] {/opt/conda-store-server/conda_store_server/utils.py:42} INFO - chown of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 took 15.260 [s]
conda-store-build_1   | [2021-06-24 14:52:02,518] {/opt/conda-store-server/conda_store_server/build.py:238} INFO - packaging archive of conda environment=/data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1
conda-store-build_1   | [2021-06-24 14:52:04,864] {/opt/conda-store-server/conda_store_server/build.py:214} ERROR - 
conda-store-build_1   | Files managed by conda were found to have been deleted/overwritten in the
conda-store-build_1   | following packages:
conda-store-build_1   | 
conda-store-build_1   | - ncurses 6.2:
conda-store-build_1   |     share/terminfo/2/2621A
conda-store-build_1   |     share/terminfo/E/Eterm
conda-store-build_1   |     share/terminfo/E/Eterm-color
conda-store-build_1   |     + 1034 others
conda-store-build_1   | 
conda-store-build_1   | This is usually due to `pip` uninstalling or clobbering conda managed files,
conda-store-build_1   | resulting in an inconsistent environment. Please check your environment for
conda-store-build_1   | conda/pip conflicts using `conda list`, and fix the environment by ensuring
conda-store-build_1   | only one version of each package is installed (conda preferred).
conda-store-build_1   | Traceback (most recent call last):
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 209, in conda_build
conda-store-build_1   |     build_conda_pack(conda_store, build_path, build)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 241, in build_conda_pack
conda-store-build_1   |     conda.conda_pack(prefix=conda_prefix, output=output_filename)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/conda.py", line 25, in conda_pack
conda-store-build_1   |     conda_pack.pack(prefix=str(prefix), output=str(output))
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 514, in pack
conda-store-build_1   |     env = CondaEnv.from_prefix(prefix,
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 163, in from_prefix
conda-store-build_1   |     files = load_environment(prefix, **kwargs)
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 856, in load_environment
conda-store-build_1   |     raise CondaPackException(_missing_files_error.format(packages))
conda-store-build_1   | conda_pack.core.CondaPackException: 
conda-store-build_1   | Files managed by conda were found to have been deleted/overwritten in the
conda-store-build_1   | following packages:
conda-store-build_1   | 
conda-store-build_1   | - ncurses 6.2:
conda-store-build_1   |     share/terminfo/2/2621A
conda-store-build_1   |     share/terminfo/E/Eterm
conda-store-build_1   |     share/terminfo/E/Eterm-color
conda-store-build_1   |     + 1034 others
conda-store-build_1   | 
conda-store-build_1   | This is usually due to `pip` uninstalling or clobbering conda managed files,
conda-store-build_1   | resulting in an inconsistent environment. Please check your environment for
conda-store-build_1   | conda/pip conflicts using `conda list`, and fix the environment by ensuring
conda-store-build_1   | only one version of each package is installed (conda preferred).
conda-store-build_1   | [2021-06-24 14:52:04,992] {/opt/conda-store-server/conda_store_server/build.py:57} INFO - specification name=env-name-1 build has failed=1 times
conda-store-build_1   | [2021-06-24 14:52:04,994] {/opt/conda-store-server/conda_store_server/build.py:68} INFO - rescheduling specification name=env-name-1 on 2021-06-24 14:52:24.992881
conda-store-build_1   | [2021-06-24 14:52:06,074] {/opt/conda-store-server/conda_store_server/build.py:119} INFO - number of queued conda builds 3
conda-store-build_1   | [2021-06-24 14:52:06,093] {/opt/conda-store-server/conda_store_server/build.py:129} INFO - number of schedulable conda builds 2
conda-store-build_1   | [2021-06-24 14:52:06,123] {/opt/conda-store-server/conda_store_server/build.py:157} INFO - building /data/store/b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e-env-name-1 symlinked to /data/envs/env-name-1
conda-store-build_1   | [2021-06-24 14:52:06,124] {/opt/conda-store-server/conda_store_server/build.py:159} INFO - previously unfinished build of /data/store/b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e-env-name-1 cleaning directory

Screenshot 2021-06-24 at 16 34 29

I am guessing the other errors I have seen might be due to mutual overwrites between the copies.

Any idea why this is happening and how can I prevent this?
@jaimergp
Copy link
Member Author

jaimergp commented Jun 24, 2021 •
edited
Loading

This error

conda-store-build_1   | Traceback (most recent call last):
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 209, in conda_build
conda-store-build_1   |     build_conda_pack(conda_store, build_path, build)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 241, in build_conda_pack
conda-store-build_1   |     conda.conda_pack(prefix=conda_prefix, output=output_filename)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/conda.py", line 25, in conda_pack
conda-store-build_1   |     conda_pack.pack(prefix=str(prefix), output=str(output))
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 514, in pack
conda-store-build_1   |     env = CondaEnv.from_prefix(prefix,
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 163, in from_prefix
conda-store-build_1   |     files = load_environment(prefix, **kwargs)
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 856, in load_environment
conda-store-build_1   |     raise CondaPackException(_missing_files_error.format(packages))
conda-store-build_1   | conda_pack.core.CondaPackException: 
conda-store-build_1   | Files managed by conda were found to have been deleted/overwritten in the
conda-store-build_1   | following packages:
conda-store-build_1   | 
conda-store-build_1   | - ncurses 6.2:
conda-store-build_1   |     share/terminfo/2/2621A
conda-store-build_1   |     share/terminfo/E/Eterm
conda-store-build_1   |     share/terminfo/E/Eterm-color
conda-store-build_1   |     + 1034 others

can be fixed by adding this kwarg to conda_pack:

def conda_pack(prefix, output):
    import conda_pack

    conda_pack.pack(prefix=str(prefix), output=str(output), ignore_missing_files=True)
#                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^

... but I don't think we want to do that.

@jaimergp
Copy link
Member Author

It then enters into a never ending loop of failed attempts to build the environment, one attempt per copy each, like a table tennis game... :)

@jaimergp
Copy link
Member Author

Chris and I talked about this. One of the issues discussed here is that library and filesystem triggers share the same namespace, so if you happen to add a library env with the same name as one of the filesystem envs (which was my case, given that I was using one of the example environments in tests/assets), the filesystem one is ignored.

This is is just a small issue. The missing files issue is a different one and has nothing to do with this. I'll open more issues to address them individually.

@jaimergp jaimergp changed the title Creating a library env triggers a filesystem env copy library envs share a namespace with filesystem` envs and cannot have the same name Jun 29, 2021
@jaimergp jaimergp changed the title library envs share a namespace with filesystem` envs and cannot have the same name library envs share a namespace with filesystem envs and cannot have the same name Jun 29, 2021
@costrouc costrouc added the type: bug 🐛 Something isn't working label Jul 13, 2021
@costrouc
Copy link
Member

This must be an error with conda-store not properly checking that the namespace + environment name is unique and instead is only checking environment name. The login in the environment name check need to be improved.

@costrouc costrouc added this to the Release 0.3.0 milestone Jul 13, 2021
@costrouc
Copy link
Member

This is now fixed with #96

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: bug 🐛 Something isn't working
Projects
None yet
Milestone
Release 0.3.0
Development

No branches or pull requests
2 participants
@costrouc @jaimergp