[Improvement][Task] Improved way to collect yarn job's appIds #12197
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import aop way to collect yarn job's applicationId add new environment configuration for each type of yarn tasks to support aop add user property `appId.collect` for user to decide how to collect applicationId This closes apache#11262
Radeity
requested review from
kezhenxu94,
zhongjiajie,
Tianqi-Dotes,
EricGao888,
caishunfeng,
SbloodyS,
zhuangchong and
ruanwenjun
as code owners
fuchanghai
reviewed
Sep 29, 2022
dolphinscheduler-aop/src/main/java/org/apache/dolphinscheduler/aop/YarnClientAspect.java
Outdated
Show resolved
Hide resolved
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java
Outdated
Show resolved
Hide resolved
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java
Outdated
Show resolved
Hide resolved
fuchanghai
reviewed
Sep 29, 2022
dolphinscheduler-aop/src/main/java/org/apache/dolphinscheduler/aop/YarnClientAspect.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/dolphinscheduler/server/worker/utils/TaskExecutionCheckerUtils.java
Outdated
Show resolved
Hide resolved
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @AfterReturning(pointcut = "cflow(execution(ApplicationId org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(ApplicationSubmissionContext))) " | ||
| + | ||
| "&& !within(CfowAspect) && execution(ApplicationReport org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(ApplicationId)) && args(appId)", returning = "appReport", argNames = "appReport,appId") | ||
| public void registerApplicationReport(ApplicationReport appReport, ApplicationId appId) { |
Check notice
Code scanning / CodeQL
Useless parameter
Radeity
requested review from
caishunfeng,
fuchanghai,
ruanwenjun,
SbloodyS and
gabrywu
and removed request for
ruanwenjun,
SbloodyS,
EricGao888,
caishunfeng and
fuchanghai
|
SonarCloud Quality Gate failed. |
hstdream
pushed a commit
to hstdream/dolphinscheduler
that referenced
this pull request
Nov 2, 2022
…#12197) * Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code. * Add user property `appId.collect` for user to decide how to collect applicationId. * Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh` * Update docs to declare how to use aop way. * Update `LogUtils` to support fetch applicationId in different ways based on the user property. Co-authored-by: gabrywu <[email protected]>
hstdream
pushed a commit
to hstdream/dolphinscheduler
that referenced
this pull request
Nov 2, 2022
…apache#12197)" This reverts commit 0d50490.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of the pull request
Import
aopway as an alternative way to collect yarn job's applicationId.Also, this PR closes [Improvement][Task] Improved way to collect yarn job's appIds #11262
Brief change log
Create new module
dolphinscheduler-aopforaopcode.Add new environment configuration for each type of yarn tasks to support
aop, as follow:# way to collect applicationId: log(original regex match), aop appId.collect: logVerify this pull request
aopfor supported types of yarn job are simply tested on my local cluster.