Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core Feature] [flytekit] Map support for Pod Tasks #1051

Closed
wild-endeavor opened this issue May 25, 2021 · 1 comment
Closed

[Core Feature] [flytekit] Map support for Pod Tasks #1051

wild-endeavor opened this issue May 25, 2021 · 1 comment
Assignees
Labels
enhancement New feature or request flytekit FlyteKit Python related issue
Milestone

Comments

@wild-endeavor
Copy link
Contributor

wild-endeavor commented May 25, 2021

Map support for Pod Tasks

Motivation

Currently the map_task construct only works for regular PythonFunctionTasks. Pod tasks are close enough to regular function tasks however that they should be supported as well.

Goal

To be able to map over Pod Tasks and have the correct PodSpec applied to the run-time pod.

Alternatives

None

Additional Context

Currently the container object generated by the map task looks something like this, when mapping over a basic PythonFunctionTask.

      container {
        image: "flytecookbook:core-732e20036452a7798086bf5abee1ede985f25e51"
        args: "pyflyte-map-execute"
        args: "--inputs"
        args: "{{.input}}"
        args: "--output-prefix"
        args: "{{.outputPrefix}}"
        args: "--raw-output-data-prefix"
        args: "{{.rawOutputDataPrefix}}"
        args: "--resolver"
        args: "flytekit.core.python_auto_container.default_task_resolver"
        args: "--"
        args: "task-module"
        args: "core.control_flow.map_task"
        args: "task-name"
        args: "a_mappable_task"
        resources {
        }
        env {
          key: "FLYTE_INTERNAL_CONFIGURATION_PATH"
          value: "/root/sandbox.config"
        }
        env {
          key: "FLYTE_INTERNAL_IMAGE"
          value: "flytecookbook:core-732e20036452a7798086bf5abee1ede985f25e51"
        }
      }
      task_type_version: 1
    }

This ends up actually kinda working, since Pod tasks are closely related to regular function tasks, but all the Pod dependencies are missing.

The custom field in the map task TaskTemplate today is an ArrayJob. We'll have to figure out how to add the PodSpec in addition to the ArrayJob, and how to invoke the Pod plugin instead of just the container plugin.

This is related to #1050

@wild-endeavor wild-endeavor added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers flytekit FlyteKit Python related issue labels May 25, 2021
@jeevb
Copy link
Contributor

jeevb commented May 27, 2021

@apatel-fn: FYI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request flytekit FlyteKit Python related issue
Projects
None yet
Development

No branches or pull requests

3 participants