-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BeamRunJavaPipelineOperator fails without job_name set #40515
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Is there a difference in the version of |
@RNHTTR from the documentation I understand it was upgraded from 10.12.0 to 10.17.0, however I am unsure of it since I did not keep track of the precise version before upgrading (and neither GCP has a log of it). |
The parsing of Dataflow job_name from the I should also note that this feature, where the job name was copied from the task ID is somewhat unique to the Apache Beam operator, as normally task id and job name are kept separate. |
Apache Airflow Provider(s)
apache-beam
Versions of Apache Airflow Providers
Can't find the actual information in Google Cloud Composer.
Some info in GCP doc
Apache Airflow version
2.7.3
Operating System
Google Cloud Composer
Deployment
Google Cloud Composer
Deployment details
Google Cloud Composer 2.8.2
What happened
BeamRunJavaPipelineOperator
was running perfectly on Airflow 2.5.After upgrading Google Cloud Composer with Airflow 2.6:
Failed to execute job xxx for task yyyy (Invalid job_name ({{task.task-id}}); the name must consist of only the characters [-a-z0-9], starting with a letter and ending with a letter or number ; 497143)
It seems
{{task.task-id}}
is not resolved, to task-id.Upgrading then to Airflow 2.7 gives the same result.
Workaround: harcode
"job_name": "ZZZ"
indataflow_config
property.What you think should happen instead
job_name
should be automatically resolved fromtask_id
as it was in 2.5.How to reproduce
Run any
BeamRunJavaPipelineOperator
on 2.6+ without settingjob_name
.Anything else
I am not at all familiar with Airflow internals, so providing a PR would take a lot of time.
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: