Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Op asset specificity #3841

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Op asset specificity #3841

wants to merge 13 commits into from

Conversation

jsnoble
Copy link
Member

@jsnoble jsnoble commented Nov 25, 2024

  • jobs will throw if there are name collisions and the op name has no asset identifier
  • can now specify op/reader/apis names as op-name@assetName:v1.4.0 or op-name@assetName

@godber
Copy link
Member

godber commented Nov 27, 2024

When you're back Jared, you'll need to rebase this and bump the version to 2.10.0 since we did a 2.9.0 release today.

@jsnoble
Copy link
Member Author

jsnoble commented Dec 3, 2024

To test this out, setup a running teraslice instance, and use teraslice-cli to upload assets

teraslice-cli assets deploy localhost terascope/[email protected]
teraslice-cli assets deploy localhost terascope/[email protected]

Sample Job that should fail because we don't know which asset to use for the op, please adjust index and any other settings for your setup

{
    "name": "test",
    "lifecycle": "once",
    "workers": 1,
    "analytics": true,
    "assets": [
        "elasticsearch:4.0.2",
        "elasticsearch:4.0.5"
    ],
    "operations": [
        {
            "_op": "elasticsearch_reader",
            "index": "ts_test_example-1000",
            "size": 10000,
            "date_field_name": "created",
            "preserve_id": true
        },
        {
            "_op": "elasticsearch_bulk",
            "index": "op_asset_version_test",
            "preserve_id": true,
            "size": 10000
        }
    ]
}

Sample job that should pass, please make the same updates to index and any other changes neccessary

{
    "name": "test",
    "lifecycle": "once",
    "workers": 1,
    "analytics": true,
    "assets": [
        "elasticsearch:4.0.2",
        "elasticsearch:4.0.5"
    ],
    "operations": [
        {
            "_op": "elasticsearch_reader@elasticsearch:4.0.2",
            "index": "ts_test_example-1000",
            "size": 10000,
            "date_field_name": "created",
            "preserve_id": true
        },
        {
            "_op": "elasticsearch_bulk@elasticsearch:4.0.5",
            "index": "op_asset_version_test",
            "preserve_id": true,
            "size": 10000
        }
    ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants