Possible issue in _activate_current_env.sh or something upstream?

[Alex-ley-scrub]

Firstly, thanks for micromamba and micromamba-docker. These 2 combined with astral.sh's uv makes installing dependencies during Docker build much faster!

We've been using micromamba, micromamba-docker, and uv at my company for our production builds for a while now and it has been working well.

This morning, something changed (although our VM, our docker version, and our Dockerfile did not) - and I'm not 100% certain what caused it yet - but I've found a fix and wanted to share it here and also ask if this is something you've seen before and something you expect? And if it is not expected, does it need a fix from your side or from something upstream, or not?

Anyway let's get into it. When running a docker image I got this this morning (and didn't last night) and it fails to run:

$ docker run --rm -it workflows:5.44.0 /bin/bash
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-addr2line
+AR=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-ar
+AS=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-as
+CXXFILT=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-c++filt
+ELFEDIT=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-elfedit
+GPROF=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-gprof
+HOST=x86_64-conda_cos6-linux-gnu
+LD=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-ld
+LD_GOLD=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-ld.gold
+NM=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-nm
+OBJCOPY=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-objcopy
+OBJDUMP=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-objdump
+RANLIB=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-ranlib
+READELF=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-readelf
+SIZE=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-size
+STRINGS=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-strings
+STRIP=/opt/conda/bin/x86_64-conda_cos6-linux-gnu-strip
/usr/local/bin/_activate_current_env.sh: line 37: pop_var_context: head of shell_variables not a function context
/usr/local/bin/_activate_current_env.sh: line 35: pop_var_context: head of shell_variables not a function context
$

I found some related things I think such as:

• Excessive INFO output on activate/deactivate conda/conda#6945
• Micromamba not initializing correctly #359
• ENV_NAME defaults to 'base' in _activate_current_env.sh #366

my error does seem to come from the last few lines of that file:

micromamba-docker/_activate_current_env.sh

Lines 32 to 35 in e2bd4fe

	# For robustness, try all possible activate commands.
	conda activate "${ENV_NAME:-base}" 2>/dev/null \
	|| mamba activate "${ENV_NAME:-base}" 2>/dev/null \
	|| micromamba activate "${ENV_NAME:-base}"

but none of those are new, so why this started happening suddenly to me this morning has confused me. I'm suspecting some non-frozen or transient/peer dependency has updated overnight (maybe based on the first link some compiler package) or something or I am being dumb and overlooking something.

Either way, the fix was to add these 2 lines to my Dockerfile which basically just skips the last few lines of that _activate_current_env.sh file:

ENV ENV_NAME base
ENV MAMBA_SKIP_ACTIVATE 1

Then everything works as expected:

$ docker run --rm -it workflows:5.44.0 /bin/bash
root@dea5188b783c:/app# 

What do you think? Normal/Abnormal/Expected/Unexpected?

[maresb]

Firstly, thanks for micromamba and micromamba-docker. These 2 combined with astral.sh's uv makes installing dependencies during Docker build much faster!

For mixed conda/pip dependencies, I'd strongly recommend checking out pixi and pixi-docker.

Regarding the error message, the first thing that stands out to me is the line 37 although the _activate_current_env.sh only has 35 lines. Could you check to make sure you're on a recent version and that you're not doing some strange modification?

[Alex-ley-scrub]

@maresb we're actually evaluating pixi at the minute! pixi uses uv for pip dependencies as I'm sure you know :) I'll hopefully get us migrated over to pixi for prod in the near future! I love all the Rust tooling coming out for python (pyO3/maturin/ruff/uv/pixi), but micromamba rescued us from conda docker build times a year or two ago and has been a lifesaver since 🙏

Yeah I noticed that as well but when I did this they all came back 35 lines:

sudo find / -name "*activate_current_env.sh"
/var/lib/docker/overlay2/bbca529b748820796e8d8e462b15cade6bd5e1d369a12480451a2821c64150b7/diff/usr/local/bin/_activate_current_env.sh
... 4 other similar locations
sudo cat  /var/lib/docker/overlay2/bbca529b748820796e8d8e462b15cade6bd5e1d369a12480451a2821c64150b7/diff/usr/local/bin/_activate_current_env.sh
... 35 lines as shown in the repo

We don't manually manipulate that file _activate_current_env.sh if that's what you mean by strange modification? Or did you mean some strange modification of your base Docker image?
We use mambaorg/micromamba:1.5.9

Here are some of the main snippets of our Dockerfile (which apart from these 2 new lines added hasn't changed much recently and certainly didn't change overnight when I started experiencing this issue):

# Stage 1: Start with NVIDIA CUDA image as base
FROM nvidia/cuda:12.2.2-devel-ubuntu22.04 as cuda-base

# Stage 2: Micromamba Image
FROM mambaorg/micromamba:1.5.9 as micromamba-base
USER root

RUN apt-get update
RUN apt-get install -y --reinstall build-essential

# Copy over CUDA libraries from the CUDA base image
COPY --from=cuda-base /usr/local/cuda /usr/local/cuda

# Ensure CUDA libraries are in the correct path
ENV PATH /usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/cuda/lib64:${LD_LIBRARY_PATH}

WORKDIR /app

# new issue: https://github.com/mamba-org/micromamba-docker/issues/535
ENV ENV_NAME base
ENV MAMBA_SKIP_ACTIVATE 1

# copy over the conda env.yml file and update the conda dependencies only
COPY ./env.yml /app
RUN micromamba install -y -n base -f /app/env.yml
# Clean up the conda cached files to reduce the image size
RUN micromamba clean --all --yes

# copy over the conda pyproject.tom file and update the pip dependencies only
COPY ./pyproject.toml /app

# use the extremely quick pip drop-in replacement, uv (https://pypi.org/project/uv/):
RUN micromamba run python -m pip install uv==0.4.1

# update the pip dependencies only (so pip dependency changes don't invalidate expensive conda cache layer)
RUN micromamba run uv pip install -r pyproject.toml
# Clean up the pip cached files to reduce the image size
RUN micromamba run uv clean && micromamba run pip cache purge

[wholtz]

Hi @Alex-ley-scrub - thanks for the report.

First off, you appear to be getting some output that you didn't expect but it all appears to be INFO level logging. Maybe the last two lines are not not from the logger -- it is hard to tell, but my point is that nothing explicitly says there was an error. So without your "fix", was there functionality that was not working?

My guess is that the last time the micromamba images were built, a different version of bash was present in the base image. Perhaps related to this thread: https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00073.html ? It is a bit old, but sometimes these fixes take a while to propagate to distributions.

To see if the change was in your installed programs or the micromamba images you could use our git-<hash> tags to
go back to an older micromamba image. Then you test if you get the same message when installing the current programs and dependencies onto the older image.

For these reasons, we recommend using the image digest (sha256) to reference the image and not the tags when reproducibility is important.