Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEV - Update dev environments #838

Merged
merged 47 commits into from
Aug 9, 2024
Merged

Conversation

trallard
Copy link
Collaborator

@trallard trallard commented Jul 1, 2024

Supersedes #801

Description

This PR should remove some unnecessary dependencies and environment file. This was originally done in #801 but with all the recent changes it was easier to clean this up in a separate PR

Pull request checklist

  • Did you test this change locally?
  • Did you update the documentation (if required)?
  • Did you add/update relevant tests for this change (if required)?

Additional information

How to test

Copy link

netlify bot commented Jul 1, 2024

Deploy Preview for conda-store canceled.

Name Link
🔨 Latest commit 43e93dd
🔍 Latest deploy log https://app.netlify.com/sites/conda-store/deploys/66b63fa1b3174000088ab632

@trallard trallard mentioned this pull request Jul 3, 2024
3 tasks
@peytondmurray peytondmurray force-pushed the trallard/update-dev-environments branch from 008810d to fdc5a05 Compare July 24, 2024 00:48
@peytondmurray
Copy link
Contributor

The last test is being stubborn. I ssh'd into the runner using tmate and ran it manually:

runner@fv-az665-524:~/work/conda-store/conda-store/conda-store-server$ pytest -svv tests/test_app_api.py::test_conda_store_register_environment_workflow
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.12.4, pytest-8.3.1, pluggy-1.5.0 -- /home/runner/miniconda3/envs/conda-store-server-dev/bin/python3.12
cachedir: .pytest_cache
rootdir: /home/runner/work/conda-store/conda-store/conda-store-server
configfile: pyproject.toml
plugins: base-url-2.1.0, celery-0.0.0a1, mock-3.14.0, anyio-4.4.0, playwright-0.5.1
collected 1 item                                                                                                                                                                                                                            

tests/test_app_api.py::test_conda_store_register_environment_workflow INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 48be4072fe58, initial schema
INFO  [alembic.runtime.migration] Running upgrade 48be4072fe58 -> 8d63a091aff8, Add Environment.description
INFO  [alembic.runtime.migration] Running upgrade 8d63a091aff8 -> 5ad723de2abd, Adding CONTAINER_REGISTRY value to enum
INFO  [alembic.runtime.migration] Running upgrade 5ad723de2abd -> 16f65805dc8f, split conda_package into conda_package and conda_package_build
INFO  [alembic.runtime.migration] Running upgrade 16f65805dc8f -> abd7248d5327, adding a settings table
INFO  [alembic.runtime.migration] Running upgrade abd7248d5327 -> b387747ca9b7, role mapping
INFO  [alembic.runtime.migration] Running upgrade b387747ca9b7 -> d78e9889566a, add status_info
INFO  [alembic.runtime.migration] Running upgrade d78e9889566a -> 30b37e725c32, add build_key_version
INFO  [alembic.runtime.migration] Running upgrade 30b37e725c32 -> 771180018e1b, add v2 role mappings
INFO  [alembic.runtime.migration] Running upgrade 771180018e1b -> 0f7e23ff24ee, add worker
INFO  [alembic.runtime.migration] Running upgrade 0f7e23ff24ee -> 57cd11b949d5, add installer
INFO  [alembic.runtime.migration] Running upgrade 57cd11b949d5 -> 03c839888c82, add canceled status
INFO  [alembic.runtime.migration] Running upgrade 03c839888c82 -> e17b4cc6e086, add build hash
INFO  [alembic.runtime.migration] Running upgrade e17b4cc6e086 -> bf065abf375b, lockfile spec
[CondaStoreWorker] building conda_prefix=/tmp/pytest-of-runner/pytest-0/test_conda_store_register_envi0/.conda-store/state/pytest-namespace/35f60418-1721784756-1-pytest-name took 21.338 [s]
[CondaStoreWorker] packaging archive of conda environment=/tmp/pytest-of-runner/pytest-0/test_conda_store_register_envi0/.conda-store/state/pytest-namespace/35f60418-1721784756-1-pytest-name took 5.453 [s]
[CondaStoreWorker] creating installer for conda environment=/tmp/pytest-of-runner/pytest-0/test_conda_store_register_envi0/.conda-store/state/pytest-namespace/35f60418-1721784756-1-pytest-name took 91.980 [s]
PASSEDRestoring 2 unacknowledged message(s)


============================================================================================================= warnings summary ==============================================================================================================
../../../../miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_lock/_vendor/conda/_vendor/boltons/timeutils.py:439
  /home/runner/miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_lock/_vendor/conda/_vendor/boltons/timeutils.py:439: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
    EPOCH_NAIVE = datetime.utcfromtimestamp(0)

../../../../miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_lock/_vendor/poetry/core/_vendor/lark/utils.py:176
  /home/runner/miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_lock/_vendor/poetry/core/_vendor/lark/utils.py:176: DeprecationWarning: module 'sre_parse' is deprecated
    import sre_parse

../../../../miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_lock/_vendor/poetry/core/_vendor/lark/utils.py:177
  /home/runner/miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_lock/_vendor/poetry/core/_vendor/lark/utils.py:177: DeprecationWarning: module 'sre_constants' is deprecated
    import sre_constants

../../../../miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/boltons/timeutils.py:426
  /home/runner/miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/boltons/timeutils.py:426: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
    EPOCH_NAIVE = datetime.utcfromtimestamp(0)

../../../../miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/traitlets/traitlets.py:3615
  /home/runner/miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/traitlets/traitlets.py:3615: DeprecationWarning: metadata {'description': 'Conda platforms to solve environments for via conda-lock. Must include current platform.'} was set from the constructor. With traitlets 4.1, metadata should be set using the .tag() method, e.g., Int().tag(key1='value1', key2='value2')
    super().__init__(trait=trait, default_value=default_value, **kwargs)

../../../../miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_pack/core.py:15
  /home/runner/miniconda3/envs/conda-store-server-dev/lib/python3.12/site-packages/conda_pack/core.py:15: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    import pkg_resources

conda_store_server/_internal/action/generate_constructor_installer.py:119
  /home/runner/work/conda-store/conda-store/conda-store-server/conda_store_server/_internal/action/generate_constructor_installer.py:119: SyntaxWarning: invalid escape sequence '\S'
    post_install = """\

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================================= 1 passed, 7 warnings in 124.06s (0:02:04) =================================================================================================
runner@fv-az665-524:~/work/conda-store/conda-store/conda-store-server$ 

@peytondmurray
Copy link
Contributor

Tried the following with no failures from the conda-store-server directory:

  • pytest tests/test_app_api.py
  • pytest -svv tests/test_app_api.py
  • python -m pytest -svv tests/test_app_api.py
  • python -m pytest tests/test_app_api.py

Will continue investigating later.

@trallard
Copy link
Collaborator Author

Yes that is the same that keeps failing...

Note that since you are using the s flag this is being picked up by pre-pended to the imports and other bits so that is the source of this particular fail.
So without flags it is

@peytondmurray
Copy link
Contributor

I changed the conda_store fixture so that -svv wouldn't propagate to test functions that use it. Is it still possible passing these flags is interfering somehow?

@trallard
Copy link
Collaborator Author

Per SyntaxWarning: invalid escape sequence '\S' post_install = """ in the logs there is definitely something odd

@peytondmurray
Copy link
Contributor

I think those are probably not to blame, it's a number of bad escape sequences in the post_install script for the action_generate_constructor_installer function. In any case I can fix this now.

@peytondmurray
Copy link
Contributor

peytondmurray commented Jul 24, 2024

@trallard I added a line that raises an exception with a bunch of information about the arguments to run_lock that is breaking this test. Here's what I did:

    # CONDA_FLAGS is used by conda-lock in conda_solver.solve_specs_for_arch
    try:
        conda_flags_name = "CONDA_FLAGS"
        # print(f"{conda_flags_name}={conda_flags}")
        os.environ[conda_flags_name] = conda_flags

        run_lock(
            environment_files=[environment_filename],
            platforms=platforms,
            lockfile_path=lockfile_filename,
            conda_exe=conda_command,
            with_cuda=cuda_version,
        )
    except KeyError as exc:
        with open(environment_filename, "r") as f:
            env_file_content = f.read()

        raise DebuggingException(
            "\n".join(
                [
                    f"environment_files={[environment_filename]}",
                    f"platforms={platforms}",
                    f"lockfile_path={lockfile_filename}",
                    f"conda_exe={conda_command}",
                    f"with_cuda={cuda_version}",
                    f"conda_flags_name={conda_flags_name}",
                    f"conda_flags={conda_flags}",
                    f"os.environ={os.environ}",
                    f"environment_files content={env_file_content}",
                ]
            )
        ) from exc
    finally:
        os.environ.pop(conda_flags_name, None)

Here's what I found:

environment_files=[PosixPath('/tmp/tmplmqca7a4/environment.yaml')]
platforms=['linux-64']
lockfile_path=/tmp/tmplmqca7a4/conda-lock.yaml
conda_exe=mamba
with_cuda=None
conda_flags_name=CONDA_FLAGS
conda_flags=--strict-channel-priority
os.environ=environ({'SELENIUM_JAR_PATH': '/usr/share/java/selenium-server.jar', 'CONDA': '/home/runner/miniconda3', 'GITHUB_WORKSPACE': '/home/runner/work/conda-store/conda-store', 'JAVA_HOME_11_X64': '/usr/lib/jvm/temurin-11-jdk-amd64', 'CONDA_EXE': '/home/runner/miniconda3/bin/conda', '_CE_M': '', 'GITHUB_PATH': '/home/runner/work/_temp/_runner_file_commands/add_path_faa0daee-1d8c-4853-867c-f49e042c66d2', 'GITHUB_ACTION': '__run_3', 'JAVA_HOME': '/usr/lib/jvm/temurin-11-jdk-amd64', 'GITHUB_RUN_NUMBER': '165', 'RUNNER_NAME': 'GitHub Actions 15', 'GRADLE_HOME': '/usr/share/gradle-8.9', 'GITHUB_REPOSITORY_OWNER_ID': '68608217', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', 'XDG_CONFIG_HOME': '/home/runner/.config', 'DOTNET_SKIP_FIRST_TIME_EXPERIENCE': '1', 'ANT_HOME': '/usr/share/ant', 'JAVA_HOME_8_X64': '/usr/lib/jvm/temurin-8-jdk-amd64', 'GITHUB_TRIGGERING_ACTOR': 'peytondmurray', 'GITHUB_REF_TYPE': 'branch', 'XML_CATALOG_FILES': 'file:///home/runner/miniconda3/envs/conda-store-server-dev
environment_files content={"name": "pytest-name", "channels": ["main"], "dependencies": ["python"], "variables": null, "prefix": null, "description": ""}

The only thing that looks strange here is that CONDA_FLAGS is not actually in os.environ as I would expect. Maybe that's related to the order in which the try-except-finally runs? In any case I tried to recreate this call with a separate file so that I can try to understand what the actual problem is:

import pathlib

from conda_lock.conda_lock import run_lock

tmpdir = pathlib.Path('/tmp/tmplmqca7a4')
tmpdir.mkdir(parents=True, exist_ok=True)
environment_filename = tmpdir / 'environment.yaml'

with open(environment_filename, 'w') as f:
    f.write('{"name": "pytest-name", "channels": ["main"], "dependencies": ["python"], "variables": null, "prefix": null, "description": ""}')

platforms = ['linux-x64']
lockfile_filename = tmpdir / 'conda-lock.yaml'
conda_command = 'conda'
cuda_version = None

run_lock(
    environment_files=[environment_filename],
    platforms=platforms,
    lockfile_path=lockfile_filename,
    conda_exe=conda_command,
    with_cuda=cuda_version,
)

Strangely I get a separate error, which is that python is not available as a package:

Locking dependencies for ['linux-x64']...
Could not lock the environment for platform linux-x64
The following packages are not available from current channels:

  - python

Current channels:

  - https://conda.anaconda.org/main
  - file:///tmp/tmpcnsg4row

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

    Command: ['/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/bin/conda', 'create', '--prefix', '/tmp/tmpglrrd2p_/prefix', '--dry-run', '--json', '--override-channels', '--channel', 'main', '--channel', 'file:///tmp/tmpcnsg4row', 'python']
    STDOUT:
{
  "allow_retry": false,
  "caused_by": "None",
  "channel_urls": [
    {
      "auth": null,
      "location": "conda.anaconda.org",
      "name": "main",
      "package_filename": null,
      "platform": null,
      "scheme": "https",
      "token": null
    },
    {
      "auth": null,
      "location": "/tmp",
      "name": "tmpcnsg4row",
      "package_filename": null,
      "platform": null,
      "scheme": "file",
      "token": null
    }
  ],
  "channels_formatted": "  - https://conda.anaconda.org/main\n  - file:///tmp/tmpcnsg4row",
  "error": "PackagesNotFoundError: The following packages are not available from current channels:\n\n  - python\n\nCurrent channels:\n\n  - https://conda.anaconda.org/main\n  - file:///tmp/tmpcnsg4row\n\nTo search for alternate channels that may provide the conda package you're\nlooking for, navigate to\n\n    https://anaconda.org\n\nand use the search bar at the top of the page.\n",
  "exception_name": "PackagesNotFoundError",
  "exception_type": "<class 'conda.exceptions.PackagesNotFoundError'>",
  "message": "The following packages are not available from current channels:\n\n  - python\n\nCurrent channels:\n\n  - https://conda.anaconda.org/main\n  - file:///tmp/tmpcnsg4row\n\nTo search for alternate channels that may provide the conda package you're\nlooking for, navigate to\n\n    https://anaconda.org\n\nand use the search bar at the top of the page.\n",
  "packages": [
    "python"
  ],
  "packages_formatted": "  - python"
}


Traceback (most recent call last):
  File "/home/pdmurray/Desktop/workspace/conda-store/conda-store-server/foo.py", line 29, in <module>
    run_lock(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1111, in run_lock
    make_lock_files(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 393, in make_lock_files
    fresh_lock_content = create_lockfile_from_spec(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 836, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 748, in _solve_for_arch
    conda_deps = solve_conda(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_solver.py", line 159, in solve_conda
    dry_run_install = solve_specs_for_arch(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/conda_solver.py", line 359, in solve_specs_for_arch
    proc.check_returncode()
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 168, in check_returncode
    raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/bin/conda', 'create', '--prefix', '/tmp/tmpglrrd2p_/prefix', '--dry-run', '--json', '--override-channels', '--channel', 'main', '--channel', 'file:///tmp/tmpcnsg4row', 'python']' returned non-zero exit status 1.

For now I'm just going to try using the simple_specification fixture with zlib to see if that works.

@trallard trallard marked this pull request as ready for review August 6, 2024 16:08
Copy link
Collaborator Author

@trallard trallard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 Thanks for seeing this over the finish line @peytondmurray
Merging now 🚀

I just cannot approve since I opened the PR but shall be ok

Copy link
Contributor

@peytondmurray peytondmurray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! 🚀

@peytondmurray peytondmurray merged commit 27106fd into main Aug 9, 2024
28 checks passed
@peytondmurray peytondmurray deleted the trallard/update-dev-environments branch August 9, 2024 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done 💪🏾
Development

Successfully merging this pull request may close these issues.

3 participants