Memoization doesn't work in templates that are steps #10769

moleskin-smile · 2023-03-28T18:49:42Z

Pre-requisites

I have double-checked my configuration
I can confirm the issues exists when I tested with :latest
I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

memoize configuration added to a steps template doesn't have any effect. No ConfigMap is created at all.

Memoization status is always like this after multiple submits:

      memoizationStatus:
        cacheName: cache-this-cm-doesnt-exist
        hit: false
        key: hello-10

It prevents us from caching larger reusable chunks of work exposed via Workflow Templates. For example, building images is a multiple step effort – we have to build an app, login to container registry, build Dockerfile and push image. Entire thing could be cached as a whole, but it doesn't work.

There are few workarounds we considered. We're not happy with neither of them.

Add memoize to all of the steps. It's hard to maintain and it's error-prone and sometimes incorrect. For example we we shouldn't cache "container registry login" step, because token it generates has short time to live, but also we don't need it at all if the image is already built (and it's waste of time and resources to login just for nothing).
Merge all steps into one, which is bad for code reusability and it's harder to maintain special images with all the tools installed (they're also bigger, so slower).
Add memoize to the slowest templates only and run these faster ones unnecessarily.

Image building is just an example. We have more use cases for this.

Is there a way to fix or work around this? Thanks!

Similar issue was reported for dag: #10426

Version

v3.4.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: testing-memoization-mini-
  annotations:
    workflows.argoproj.io/description: Test memoization
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: echo-and-wait-step
            template: echo-and-wait
            arguments:
              parameters:
                - name: time
                  value: 10
                - name: message
                  value: "hello"
      memoize:
        key: "hello-10"
        cache:
          configMap:
            name: cache-this-cm-doesnt-exist

    - name: echo-and-wait
      inputs:
        parameters:
          - name: message
          - name: time
      script:
        image: alpine:3.17.2
        command: [sh]
        source: |
          set -e
          date
          echo {{inputs.parameters.message}}
          sleep {{inputs.parameters.time}}

Logs from the workflow controller

time="2023-03-28T18:19:27.757Z" level=info msg="Processing workflow" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.762Z" level=info msg="Updated phase  -> Running" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.767Z" level=info msg="Steps node testing-memoization-mini-d2k2z initialized Pending" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.767Z" level=info msg="StepGroup node testing-memoization-mini-d2k2z-1462015324 initialized Running" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.767Z" level=info msg="Pod node testing-memoization-mini-d2k2z-2612586120 initialized Pending" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.792Z" level=info msg="Created pod: testing-memoization-mini-d2k2z[0].echo-and-wait-step (testing-memoization-mini-d2k2z-echo-and-wait-2612586120)" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.792Z" level=info msg="Workflow step group node testing-memoization-mini-d2k2z-1462015324 not yet completed" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.792Z" level=info msg="TaskSet Reconciliation" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.792Z" level=info msg=reconcileAgentPod namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:27.801Z" level=info msg="Workflow update successful" namespace=ci-workflows phase=Running resourceVersion=273753196 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:37.799Z" level=info msg="Processing workflow" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:37.799Z" level=info msg="Task-result reconciliation" namespace=ci-workflows numObjs=0 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:37.799Z" level=info msg="node changed" namespace=ci-workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=testing-memoization-mini-d2k2z-2612586120 old.message= old.phase=Pending old.progress=0/1 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:37.800Z" level=info msg="Workflow step group node testing-memoization-mini-d2k2z-1462015324 not yet completed" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:37.800Z" level=info msg="TaskSet Reconciliation" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:37.800Z" level=info msg=reconcileAgentPod namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:37.811Z" level=info msg="Workflow update successful" namespace=ci-workflows phase=Running resourceVersion=273753277 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:47.812Z" level=info msg="Processing workflow" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:47.813Z" level=info msg="Task-result reconciliation" namespace=ci-workflows numObjs=0 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:47.813Z" level=info msg="node unchanged" namespace=ci-workflows nodeID=testing-memoization-mini-d2k2z-2612586120 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:47.813Z" level=info msg="Workflow step group node testing-memoization-mini-d2k2z-1462015324 not yet completed" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:47.813Z" level=info msg="TaskSet Reconciliation" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:47.813Z" level=info msg=reconcileAgentPod namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:57.906Z" level=info msg="Processing workflow" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:57.906Z" level=info msg="Task-result reconciliation" namespace=ci-workflows numObjs=0 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:57.907Z" level=info msg="node changed" namespace=ci-workflows new.message= new.phase=Running new.progress=0/1 nodeID=testing-memoization-mini-d2k2z-2612586120 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:57.907Z" level=info msg="Workflow step group node testing-memoization-mini-d2k2z-1462015324 not yet completed" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:57.907Z" level=info msg="TaskSet Reconciliation" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:57.907Z" level=info msg=reconcileAgentPod namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:19:57.917Z" level=info msg="Workflow update successful" namespace=ci-workflows phase=Running resourceVersion=273753440 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.944Z" level=info msg="Processing workflow" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="Task-result reconciliation" namespace=ci-workflows numObjs=1 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="task-result changed" namespace=ci-workflows nodeID=testing-memoization-mini-d2k2z-2612586120 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="node changed" namespace=ci-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=testing-memoization-mini-d2k2z-2612586120 old.message= old.phase=Running old.progress=0/1 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="Step group node testing-memoization-mini-d2k2z-1462015324 successful" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="node testing-memoization-mini-d2k2z-1462015324 phase Running -> Succeeded" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="node testing-memoization-mini-d2k2z-1462015324 finished: 2023-03-28 18:20:08.945950054 +0000 UTC" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="Outbound nodes of testing-memoization-mini-d2k2z-2612586120 is [testing-memoization-mini-d2k2z-2612586120]" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="Outbound nodes of testing-memoization-mini-d2k2z is [testing-memoization-mini-d2k2z-2612586120]" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="node testing-memoization-mini-d2k2z phase Pending -> Succeeded" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.945Z" level=info msg="node testing-memoization-mini-d2k2z finished: 2023-03-28 18:20:08.945993045 +0000 UTC" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.946Z" level=info msg="Checking daemoned children of testing-memoization-mini-d2k2z" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.946Z" level=info msg="TaskSet Reconciliation" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.946Z" level=info msg=reconcileAgentPod namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.946Z" level=info msg="Updated phase Running -> Succeeded" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.946Z" level=info msg="Marking workflow completed" namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.946Z" level=info msg="Checking daemoned children of " namespace=ci-workflows workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.951Z" level=info msg="cleaning up pod" action=deletePod key=ci-workflows/testing-memoization-mini-d2k2z-1340600742-agent/deletePod
time="2023-03-28T18:20:08.957Z" level=info msg="Workflow update successful" namespace=ci-workflows phase=Succeeded resourceVersion=273753515 workflow=testing-memoization-mini-d2k2z
time="2023-03-28T18:20:08.958Z" level=info msg="Queueing Succeeded workflow ci-workflows/testing-memoization-mini-d2k2z for delete in 4h0m0s due to TTL"
time="2023-03-28T18:20:08.978Z" level=info msg="cleaning up pod" action=labelPodCompleted key=ci-workflows/testing-memoization-mini-d2k2z-echo-and-wait-2612586120/labelPodCompleted

Logs from in your workflow's wait container

time="2023-03-28T18:19:48.072Z" level=info msg="capturing logs" argo=true
Tue Mar 28 18:19:48 UTC 2023
hello
time="2023-03-28T18:19:58.078Z" level=info msg="sub-process exited" argo=true error="<nil>"

The text was updated successfully, but these errors were encountered:

JPZ13 · 2023-03-30T17:26:54Z

It seems like memoization is only declared on execution templates rather than orchestration templates currently. This and #10426 are requesting that it work for orchestration templates

moleskin-smile · 2023-04-02T22:56:52Z

@JPZ13: Thanks for looking into this. What needs to be done to implement this? I think it's an important missing feature that would allow us to cache more complicated steps that run on big monorepos.

JPZ13 · 2023-04-04T01:30:25Z

Not sure about what technical steps, but I'll bring it up at the next contributors meeting to see if anyone has capacity. Would you be interested in contributing it @moleskin-smile?

moleskin-smile · 2023-04-05T01:21:07Z

I'd love to, but I lack Go skills. :(

Thanks for taking care of this issue.

Despite argoproj#10769 and argoproj#10426 both having examples of memoization not working with the examples having no output, no-one has picked up on this. To address this improve the documentation for memoization and work-avoidance, linking the two ideas and pointing people who want to skip steps towards work-avoidance unless they are really doing what memoization was designed to do. Issue argoproj#10426 is problematic in that some steps get memoized when perhaps they should't, so this commit shouldn't close it. Fixes argoproj#10769 Signed-off-by: Alan Clucas <[email protected]>

Joibel · 2023-06-23T13:15:13Z

This is not a bug. Memoization should not memoize steps without outputs. Adding outputs to the above example does what is intended.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: memo-steps-
  annotations:
    workflows.argoproj.io/description: Test memoization
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: echo-and-wait-step
            template: echo-and-wait
            arguments:
              parameters:
                - name: time
                  value: 10
                - name: message
                  value: "hello"
      memoize:
        key: "hello-10"
        cache:
          configMap:
            name: cache-this-cm-doesnt-exist
      outputs:
        parameters:
          - name: message
            valueFrom:
              parameter: "{{ steps.echo-and-wait-step.outputs.parameters.message }}"

    - name: echo-and-wait
      inputs:
        parameters:
          - name: message
          - name: time
      script:
        image: alpine:3.17.2
        command: [sh]
        source: |
          set -e
          date
          echo {{inputs.parameters.message}} > /tmp/msg
          sleep {{inputs.parameters.time}}
      outputs:
        parameters:
          - name: message
            valueFrom:
              path: /tmp/msg

I have put up a PR to try to address this issue through documentation.

moleskin-smile · 2023-06-26T16:54:53Z

Thanks! That works indeed.

Please note: I am not totally in favour of this change, it makes memoization work without outputs and I'm not conviced this is actually healthy, but it's what people seem to expect. See the linked issues in the fixes list below. This change will cause memoization to work for all step and dag tasks even without outputs. Note: They were working semi-erroneously for some dag tasks due to implicit outputs before this change. Fixes argoproj#11280 (raised to cover this desire) Fixes argoproj#10769 (already closed by documentation) Partially addresses argoproj#10426: Dags will memoize now, but retries still won't Signed-off-by: Alan Clucas <[email protected]>

moleskin-smile added the type/bug label Mar 28, 2023

JPZ13 added type/feature Feature request and removed type/bug labels Mar 30, 2023

JPZ13 mentioned this issue Jun 8, 2023

Memoization isn't feature complete #11191

Open

3 tasks

Joibel mentioned this issue Jun 23, 2023

docs: Link memoization and work-avoidance #11257

Merged

JPZ13 closed this as completed Jun 28, 2023

Joibel mentioned this issue Jul 17, 2023

feat: Allow memoization without outputs #11379

Merged

agilgur5 added the area/memoization label Sep 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memoization doesn't work in templates that are steps #10769

Memoization doesn't work in templates that are steps #10769

moleskin-smile commented Mar 28, 2023 •

edited

Loading

JPZ13 commented Mar 30, 2023

moleskin-smile commented Apr 2, 2023

JPZ13 commented Apr 4, 2023

moleskin-smile commented Apr 5, 2023

Joibel commented Jun 23, 2023

moleskin-smile commented Jun 26, 2023

Memoization doesn't work in templates that are steps #10769

Memoization doesn't work in templates that are steps #10769

Comments

moleskin-smile commented Mar 28, 2023 • edited Loading

Pre-requisites

What happened/what you expected to happen?

Version

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Logs from the workflow controller

Logs from in your workflow's wait container

JPZ13 commented Mar 30, 2023

moleskin-smile commented Apr 2, 2023

JPZ13 commented Apr 4, 2023

moleskin-smile commented Apr 5, 2023

Joibel commented Jun 23, 2023

moleskin-smile commented Jun 26, 2023

moleskin-smile commented Mar 28, 2023 •

edited

Loading