Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement] DataQuality module improve #14463

Merged
merged 4 commits into from
Jul 10, 2023
Merged

[improvement] DataQuality module improve #14463

merged 4 commits into from
Jul 10, 2023

Conversation

boy-xiaozhang
Copy link
Contributor

Purpose of the pull request

To improve the usability of the data quality module, some users did not install Apahce Hive when using it,  
but the class org.apache.dolphinscheduler.data.quality.execution.SparkRuntimeEnvironment starts Hive loading by default,  
and users also need to manually specify the default quality class.

Brief change log

1. Fixed that if the user does not use Hive, Hive support is turned off by default
2. Fixed that the user uses the default class provided by the dolphinscheduler if there is no custom quality Jar

Verify this pull request

No major changes, the original unit test can be

@boy-xiaozhang boy-xiaozhang changed the title [Improve] DataQuality module improve [improvement] DataQuality module improve Jul 6, 2023
@zhongjiajie
Copy link
Member

Fixed that if the user does not use Hive, Hive support is turned off by default

I have run the dq task locally, it can success even though I do not have hive install, do you mean default dq task must ask hive install to run ?

@zhongjiajie
Copy link
Member

ci failed could you please task a look?

caishunfeng
caishunfeng previously approved these changes Jul 7, 2023
Copy link
Contributor

@caishunfeng caishunfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@caishunfeng caishunfeng added the improvement make more easy to user or prompt friendly label Jul 7, 2023
@zhongjiajie
Copy link
Member

rerun the failed ci

@zhongjiajie
Copy link
Member

could you rebase to latest dev branch? CI Backend / cluster-test-mysql (pull_request) keep failing

@zhongjiajie
Copy link
Member

re-run failed CI

@codecov-commenter
Copy link

Codecov Report

Merging #14463 (784a8c4) into dev (0b69236) will decrease coverage by 0.02%.
The diff coverage is 33.33%.

❗ Current head 784a8c4 differs from pull request most recent head e5d9751. Consider uploading reports for the commit e5d9751 to get more accurate results

@@             Coverage Diff              @@
##                dev   #14463      +/-   ##
============================================
- Coverage     38.54%   38.53%   -0.02%     
+ Complexity     4557     4555       -2     
============================================
  Files          1237     1237              
  Lines         43462    43420      -42     
  Branches       4814     4784      -30     
============================================
- Hits          16754    16731      -23     
+ Misses        24853    24836      -17     
+ Partials       1855     1853       -2     
Impacted Files Coverage Δ
...scheduler/data/quality/DataQualityApplication.java 0.00% <0.00%> (ø)
...scheduler/plugin/task/dq/utils/SparkArgsUtils.java 0.00% <0.00%> (ø)
...ata/quality/execution/SparkRuntimeEnvironment.java 73.68% <80.00%> (-2.79%) ⬇️

... and 36 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sonarcloud
Copy link

sonarcloud bot commented Jul 10, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

30.0% 30.0% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@zhongjiajie zhongjiajie added the 3.2.0 for 3.2.0 version label Jul 10, 2023
@zhongjiajie zhongjiajie added this to the 3.2.0 milestone Jul 10, 2023
Copy link
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks

@zhongjiajie zhongjiajie merged commit 75d29f6 into apache:dev Jul 10, 2023
IT-Kwj pushed a commit to IT-Kwj/dolphinscheduler that referenced this pull request Jul 14, 2023
zhongjiajie pushed a commit that referenced this pull request Jul 20, 2023
biaoma-ty pushed a commit to Kasma-Inc/dolphinscheduler that referenced this pull request Aug 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.2.0 for 3.2.0 version backend improvement make more easy to user or prompt friendly
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants