Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

COS filesystem high availability support #43

Open
wants to merge 2 commits into
base: python-udf-oss-cos
Choose a base branch
from
Open

COS filesystem high availability support #43

wants to merge 2 commits into from

Commits on Oct 18, 2024

  1. COS filesystem high availability support 

    If you need to use COS filesystem for the dagger, provide the cos
bucket/key configuration in the state.backend.fs.checkpointdir,
state.savepoints.dir, high-availability.storageDir to flinkdeployment
manifest.

If the filesystem protocol begins with cosn for the above
configurations, dagger uses the below configurations provided in the
flinkdeployment manifest file.

    fs.cosn.impl: org.apache.hadoop.fs.CosFileSystem
    fs.AbstractFileSystem.cosn.impl: org.apache.hadoop.fs.CosN
    fs.cosn.userinfo.secretId: <secretID>
    fs.cosn.userinfo.secretKey: <secretKey>
    fs.cosn.bucket.region: <region>
    fs.cosn.bucket.endpoint_suffix: <tencent-provided-prefix.xyz.com>
    rajuGT committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    a20f3c8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'python-udf-oss-cos' into cos-fs-high-availability

    rajuGT committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    b7a1179 View commit details
    Browse the repository at this point in the history