Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement][Test] Remove dependency of powermock #11405

Closed
10 tasks done
EricGao888 opened this issue Aug 10, 2022 · 5 comments · Fixed by #12311
Closed
10 tasks done

[Improvement][Test] Remove dependency of powermock #11405

EricGao888 opened this issue Aug 10, 2022 · 5 comments · Fixed by #12311
Assignees
@EricGao888
Labels
help wanted Extra attention is needed improvement make more easy to user or prompt friendly priority:high test
Milestone
3.2.0

Comments

@EricGao888
Copy link
Member

EricGao888 commented Aug 10, 2022 •
edited
Loading

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

Sub-Tasks

I have split the task into six sub-tasks by workload:

Examples

Currently we have several typical scenarios where we heavily rely on Powermock

  1. Mock static class
  2. Suppress Static Initialization Error
  3. Access private method in UT
    ... (to be continued)

I will give some examples and corresponding solutions for these scenarios respectively as follows:

  1. Add mockito-inline and use Mockito.mockStatic as an alternative.
  2. Simply mocking some special static classes with Mockito.mockStatic will cause Static Initialization Error. For these special cases, we need to refactor the origin class with some tricks to decouple the usage of static method.
  3. Change private method to protected

For examples, see: #11588

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@EricGao888 EricGao888 added improvement make more easy to user or prompt friendly Waiting for reply Waiting for reply labels Aug 10, 2022
@github-actions
Copy link

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

  • In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
  • If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

@EricGao888 EricGao888 self-assigned this Aug 10, 2022
@EricGao888 EricGao888 added test help wanted Extra attention is needed and removed Waiting for reply Waiting for reply labels Aug 10, 2022
@EricGao888 EricGao888 mentioned this issue Aug 10, 2022
Merged
@EricGao888
Copy link
Member Author

FYI here is a list of UT files which uses Powermock:

Found Occurrences in Directory dolphinscheduler with mask '*.java'  (192 usages found)
    Unclassified  (192 usages found)
        dolphinscheduler-alert-server  (1 usage found)
            org.apache.dolphinscheduler.alert  (1 usage found)
                AlertServerTest.java  (1 usage found)
        dolphinscheduler-api  (27 usages found)
            org.apache.dolphinscheduler.api.controller  (5 usages found)
                DataAnalysisControllerTest.java  (1 usage found)
                ProcessInstanceControllerTest.java  (1 usage found)
                ProcessTaskRelationControllerTest.java  (1 usage found)
                ResourcesControllerTest.java  (1 usage found)
                WorkerGroupControllerTest.java  (1 usage found)
            org.apache.dolphinscheduler.api.interceptor  (2 usages found)
                RateLimitInterceptorTest.java  (2 usages found)
            org.apache.dolphinscheduler.api.service  (20 usages found)
                BaseServiceTest.java  (3 usages found)
                DataAnalysisServiceTest.java  (2 usages found)
                DataSourceServiceTest.java  (4 usages found)
                LoggerServiceTest.java  (1 usage found)
                ResourcesServiceTest.java  (4 usages found)
                SchedulerServiceTest.java  (2 usages found)
                TenantServiceTest.java  (1 usage found)
                UdfFuncServiceTest.java  (3 usages found)
        dolphinscheduler-common  (10 usages found)
            org.apache.dolphinscheduler.common.utils  (10 usages found)
                CommonUtilsTest.java  (2 usages found)
                FileUtilsTest.java  (3 usages found)
                HadoopUtilsTest.java  (4 usages found)
                NetUtilsTest.java  (1 usage found)
        dolphinscheduler-datasource-api  (9 usages found)
            org.apache.dolphinscheduler.plugin.datasource.api.client  (4 usages found)
                CommonDataSourceClientTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.api.utils  (5 usages found)
                CommonUtilsTest.java  (5 usages found)
        dolphinscheduler-datasource-clickhouse  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.clickhouse  (4 usages found)
                ClickHouseDataSourceChannelTest.java  (4 usages found)
        dolphinscheduler-datasource-db2  (7 usages found)
            org.apache.dolphinscheduler.plugin.datasource.db2  (4 usages found)
                DB2DataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.db2.param  (3 usages found)
                Db2DataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-datasource-hive  (7 usages found)
            org.apache.dolphinscheduler.plugin.datasource.hive  (4 usages found)
                HiveDataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.hive.param  (3 usages found)
                HiveDataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-datasource-mysql  (14 usages found)
            org.apache.dolphinscheduler.plugin.datasource.mysql  (4 usages found)
                MySQLDataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.mysql.param  (3 usages found)
                MySQLDataSourceProcessorTest.java  (3 usages found)
            org.apache.dolphinscheduler.plugin.datasource.mysql.provider  (3 usages found)
                JDBCDataSourceProviderTest.java  (3 usages found)
            org.apache.dolphinscheduler.plugin.datasource.mysql.utils  (4 usages found)
                DataSourceUtilsTest.java  (4 usages found)
        dolphinscheduler-datasource-oracle  (7 usages found)
            org.apache.dolphinscheduler.plugin.datasource.oracle  (4 usages found)
                OracleDataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.oracle.param  (3 usages found)
                OracleDataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-datasource-postgresql  (7 usages found)
            org.apache.dolphinscheduler.plugin.datasource.postgresql  (4 usages found)
                PostgreSQLDataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.postgresql.param  (3 usages found)
                PostgreSQLDataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-datasource-presto  (7 usages found)
            org.apache.dolphinscheduler.plugin.datasource.presto  (4 usages found)
                PrestoDataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.presto.param  (3 usages found)
                PrestoDataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-datasource-redshift  (3 usages found)
            org.apache.dolphinscheduler.plugin.datasource.redshift.param  (3 usages found)
                RedshiftDataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-datasource-spark  (7 usages found)
            org.apache.dolphinscheduler.plugin.datasource.spark  (4 usages found)
                SparkDataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.spark.param  (3 usages found)
                SparkDataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-datasource-sqlserver  (7 usages found)
            org.apache.dolphinscheduler.plugin.datasource.sqlserver  (4 usages found)
                SQLServerDataSourceChannelTest.java  (4 usages found)
            org.apache.dolphinscheduler.plugin.datasource.sqlserver.param  (3 usages found)
                SQLServerDataSourceProcessorTest.java  (3 usages found)
        dolphinscheduler-log-server  (3 usages found)
            org.apache.dolphinscheduler.server.log  (3 usages found)
                LoggerRequestProcessorTest.java  (3 usages found)
        dolphinscheduler-master  (18 usages found)
            org.apache.dolphinscheduler.server.master  (3 usages found)
                SubProcessTaskTest.java  (3 usages found)
            org.apache.dolphinscheduler.server.master.processor  (5 usages found)
                CacheProcessorTest.java  (1 usage found)
                TaskAckProcessorTest.java  (3 usages found)
                TaskKillResponseProcessorTest.java  (1 usage found)
            org.apache.dolphinscheduler.server.master.registry  (3 usages found)
                MasterRegistryClientTest.java  (3 usages found)
            org.apache.dolphinscheduler.server.master.runner  (4 usages found)
                MasterTaskExecThreadTest.java  (1 usage found)
                WorkflowExecuteRunnableTest.java  (3 usages found)
            org.apache.dolphinscheduler.server.master.service  (3 usages found)
                FailoverServiceTest.java  (3 usages found)
        dolphinscheduler-server  (4 usages found)
            org.apache.dolphinscheduler.server.utils  (4 usages found)
                ProcessUtilsTest.java  (4 usages found)
        dolphinscheduler-service  (8 usages found)
            org.apache.dolphinscheduler.service.alert  (4 usages found)
                AlertClientServiceTest.java  (3 usages found)
                ProcessAlertManagerTest.java  (1 usage found)
            org.apache.dolphinscheduler.service.log  (3 usages found)
                LogClientServiceTest.java  (3 usages found)
            org.apache.dolphinscheduler.service.process  (1 usage found)
                ProcessServiceTest.java  (1 usage found)
        dolphinscheduler-task-dvc  (5 usages found)
            org.apache.dolphinscheduler.plugin.task.dvc  (5 usages found)
                DvcTaskTest.java  (5 usages found)
        dolphinscheduler-task-emr  (8 usages found)
            org.apache.dolphinscheduler.plugin.task.emr  (8 usages found)
                EmrAddStepsTaskTest.java  (4 usages found)
                EmrJobFlowTaskTest.java  (4 usages found)
        dolphinscheduler-task-jupyter  (5 usages found)
            org.apache.dolphinscheduler.plugin.task.jupyter  (5 usages found)
                JupyterTaskTest.java  (5 usages found)
        dolphinscheduler-task-mlflow  (5 usages found)
            org.apache.dolphinler.plugin.task.mlflow  (5 usages found)
                MlflowTaskTest.java  (5 usages found)
        dolphinscheduler-task-openmldb  (1 usage found)
            org.apache.dolphinscheduler.plugin.task.openmldb  (1 usage found)
                OpenmldbTaskTest.java  (1 usage found)
        dolphinscheduler-task-python  (1 usage found)
            org.apache.dolphinscheduler.plugin.task.python  (1 usage found)
                PythonTaskTest.java  (1 usage found)
        dolphinscheduler-task-sagemaker  (5 usages found)
            org.apache.dolphinscheduler.plugin.task.sagemaker  (5 usages found)
                SagemakerTaskTest.java  (5 usages found)
        dolphinscheduler-task-spark  (4 usages found)
            org.apache.dolphinscheduler.plugin.task.spark  (4 usages found)
                SparkTaskTest.java  (4 usages found)
        dolphinscheduler-task-zeppelin  (4 usages found)
            org.apache.dolphinscheduler.plugin.task.zeppelin  (4 usages found)
                ZeppelinTaskTest.java  (4 usages found)
        dolphinscheduler-worker  (4 usages found)
            org.apache.dolphinscheduler.server.worker.processor  (3 usages found)
                TaskDispatchProcessorTest.java  (3 usages found)
            org.apache.dolphinscheduler.server.worker.runner  (1 usage found)
                TaskExecuteThreadTest.java  (1 usage found)

@EricGao888 EricGao888 mentioned this issue Aug 12, 2022
Open
12 tasks
@EricGao888 EricGao888 mentioned this issue Aug 22, 2022
Merged
@EricGao888 EricGao888 added this to the 3.1.0 milestone Aug 23, 2022
@EricGao888 EricGao888 mentioned this issue Aug 25, 2022
Closed
3 tasks
@EricGao888 EricGao888 mentioned this issue Sep 8, 2022
Merged
@EricGao888 EricGao888 modified the milestones: 3.1.0, 3.2.0 Sep 20, 2022
@EricGao888 EricGao888 mentioned this issue Sep 21, 2022
Merged
4 tasks
@jieguangzhou jieguangzhou mentioned this issue Sep 21, 2022
Closed
3 tasks
@EricGao888 EricGao888 mentioned this issue Sep 23, 2022
Closed
3 tasks
@EricGao888
Copy link
Member Author

EricGao888 commented Sep 23, 2022 •
edited
Loading

I will try to finish all the sub-tasks in Sept. and Oct. However, anyone interested and willing to help are very welcome to pick those sub-tasks. To help with this issue, just create a new issue for the sub-task you want to pick and comment with the issue link below.

This was referenced Sep 26, 2022
Closed
Closed
@fuchanghai
Copy link
Member

Are you fixing the dolphinscheduler-common module @EricGao888

@EricGao888
Copy link
Member Author

Are you fixing the dolphinscheduler-common module @EricGao888

Yes, I will take care of the rest ones.

This was referenced Sep 27, 2022
Closed
Closed
Closed
This was referenced Oct 9, 2022
Merged
Closed
@rickchengx rickchengx mentioned this issue Oct 10, 2022
Merged
This was referenced Oct 10, 2022
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EricGao888 EricGao888

Labels
help wanted Extra attention is needed improvement make more easy to user or prompt friendly priority:high test
Projects
None yet
Milestone
3.2.0
2 participants
@fuchanghai @EricGao888