Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GitHub action to run SonarQube for METdataio pull requests and feature branches #289

Closed
12 of 28 tasks
JohnHalleyGotway opened this issue Apr 3, 2024 · 2 comments · Fixed by #290, #291, #293 or #298
Closed
12 of 28 tasks
Labels
component: code optimization Code optimization issue priority: high High Priority requestor: USAF United States Air Force type: new feature Make it do something new

Comments

@JohnHalleyGotway
Copy link
Contributor

JohnHalleyGotway commented Apr 3, 2024

Describe the New Feature

This issue is to add a new SonarQube workflow to GitHub actions to automate the static code analysis for all pull requests. In addition, add a manual trigger workflow dispatch option where the reference branch can be manually specified.

Recommend adding this workflow to both the develop branch and the current main_v*so that the workflow dispatch option can be made available.

Recommend pushing results to a new SonarQube project named METdataio GHA at needham.rap.ucar.edu.

Recommend having the workflow report bad status if the number of SonarQube findings are increased relative to the SonarQube reference.

See issue dtcenter/MET#2379 and its two linked PR's as an example, but the implementation for a python-only repo should be more straight-forward.

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

  • Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Needed for the Air Force - 2771024

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

The following SonarQube issues are closely related:

New Feature Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added priority: high High Priority type: new feature Make it do something new component: code optimization Code optimization issue requestor: USAF United States Air Force labels Apr 3, 2024
@JohnHalleyGotway JohnHalleyGotway added this to the METdataio-3.0.0 milestone Apr 3, 2024
JohnHalleyGotway added a commit that referenced this issue Apr 3, 2024
@JohnHalleyGotway JohnHalleyGotway linked a pull request Apr 4, 2024 that will close this issue
14 tasks
@JohnHalleyGotway
Copy link
Contributor Author

Followed these instructions to better integrate SonarQube with GitHub:
https://docs.sonarsource.com/sonarqube/10.2/devops-platform-integration/github-integration/

To create a new GitHub app for the DTCenter organization (https://github.com/apps/dtcenter-sonarqube-integration) to support SonarQube integration.

Installed that app in the DTCenter org settings (https://github.com/organizations/dtcenter/settings/apps) for 6 repositories: METplus, MET, METviewer, METplotpy, METcalcpy, METdataio

Followed these instructions to setup GitHub user authentication for our SonarQube server:
https://docs.sonarsource.com/sonarqube/10.2/instance-administration/authentication/github/

To create a new GitHub app for the DTCenter organization (https://github.com/apps/dtcenter-sonarqube-user-auth) to support GitHub user authentication at our SonarQube server.

Members of the DTCenter GitHub organization are now able to authenticate at needham.rap.ucar.edu using their GitHub credentials.

I was able to re-use two existing GitHub actions provided by SonarSource:

sonarsource/sonarqube-scan-action@master
sonarsource/sonarqube-quality-gate-action@master

I did add steps before calling these actions to check out the code and configure the SonarQube properties file.
Note that I did parse the version number from docs/version and include that in the properties file. So the version of the code will now be attached to the scan.

@JohnHalleyGotway
Copy link
Contributor Author

JohnHalleyGotway commented Apr 4, 2024

I notice the following exception in the METdataio SonarQube scan. Need to ask @hsoh-u about this exception.

It also appears in the Nightly Build output in seneca:/d1/projects/MET/MET_regression/sonarqube_METdataio/NB20240403/run_sonarqube_20240403.log.

ERROR: Error when running: 'node -v'. Is Node.js available during analysis?
org.sonar.plugins.javascript.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis?
	at org.sonar.plugins.javascript.nodejs.NodeCommand.start(NodeCommand.java:85)
	at org.sonar.plugins.javascript.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:217)
	at org.sonar.plugins.javascript.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:173)
	at org.sonar.plugins.javascript.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:146)
	at org.sonar.plugins.javascript.eslint.BridgeServerImpl.initNodeCommand(BridgeServerImpl.java:253)
	at org.sonar.plugins.javascript.eslint.BridgeServerImpl.startServer(BridgeServerImpl.java:181)
	at org.sonar.plugins.javascript.eslint.BridgeServerImpl.startServerLazily(BridgeServerImpl.java:293)
	at org.sonar.plugins.javascript.eslint.AbstractEslintSensor.execute(AbstractEslintSensor.java:73)
	at org.sonar.plugins.javascript.eslint.CssRuleSensor.execute(CssRuleSensor.java:96)
	at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
	at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:88)
	at org.sonar.scanner.sensor.ModuleSensorsExecutor.lambda$execute$1(ModuleSensorsExecutor.java:61)
	at org.sonar.scanner.sensor.ModuleSensorsExecutor.withModuleStrategy(ModuleSensorsExecutor.java:79)
	at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:61)
	at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:82)
	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
	at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
	at org.sonar.scanner.scan.SpringProjectScanContainer.scan(SpringProjectScanContainer.java:398)
	at org.sonar.scanner.scan.SpringProjectScanContainer.scanRecursively(SpringProjectScanContainer.java:394)
	at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:363)
	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
	at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
	at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:139)
	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
	at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
	at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:71)
	at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:65)
	at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
	at com.sun.proxy.$Proxy0.execute(Unknown Source)
	at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
	at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
	at org.sonarsource.scanner.cli.Main.execute(Main.java:112)
	at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
	at org.sonarsource.scanner.cli.Main.main(Main.java:61)
Caused by: java.io.IOException: Cannot run program "node": error=2, No such file or directory
	at java.base/java.lang.ProcessBuilder.start(Unknown Source)
	at java.base/java.lang.ProcessBuilder.start(Unknown Source)
	at org.sonar.plugins.javascript.nodejs.ProcessWrapperImpl.startProcess(ProcessWrapperImpl.java:43)
	at org.sonar.plugins.javascript.nodejs.NodeCommand.start(NodeCommand.java:82)
	... 38 common frames omitted
Caused by: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
	at java.base/java.lang.ProcessImpl.start(Unknown Source)
	... 42 common frames omitted

JohnHalleyGotway added a commit that referenced this issue Apr 4, 2024
…es for SonarQube. These have been relocated to internal/scripts/sonarqube for consistency with other METplus repos. I checked the crontab entry on seneca for the met_test user that the latter version is used, and not this one I'm deleting.
@JohnHalleyGotway JohnHalleyGotway moved this from 🔖 Ready to 👀 In review in METplus-Analysis-6.0.0 Development Apr 4, 2024
@JohnHalleyGotway JohnHalleyGotway linked a pull request Apr 4, 2024 that will close this issue
bikegeek added a commit that referenced this issue Apr 4, 2024
* Per #289, add draft sonarqube.xml workflow.

* Per #289, add logic for handling the sonar-project.properties file.

* Per #289, remove scanning of my feature branch and also tweak run script.

* Per #289, update the run_sonarqube.sh script to check that SONAR_TOKEN and SONAR_HOST_URL are both set.

* Per #289, add logic to define SONAR_PROJECT_VERSION

* Per #289 fix typo in ProjectName where METdatio should really be METdataio

* Per #273, fix typo in properties file

* Per #279, update run_sonarqube.sh script to push to a SonarQube branch that matches the source code branch

* Per #279, update the PR template.

* Per #289, remove the internal_tests directory which only contains files for SonarQube. These have been relocated to internal/scripts/sonarqube for consistency with other METplus repos. I checked the crontab entry on seneca for the met_test user that the latter version is used, and not this one I'm deleting.

* Issue #289 Added METreformat to the list of sonar.sources

---------

Co-authored-by: bikegeek <[email protected]>
bikegeek added a commit that referenced this issue Apr 4, 2024
* Per #289, same set of changes from PR #290, but for the main_v2.1 branch instead.

* Update sonar-project.properties

include checks for METreformat code

---------

Co-authored-by: bikegeek <[email protected]>
@bikegeek bikegeek moved this from 👀 In review to ✅ Done in METplus-Analysis-6.0.0 Development Apr 4, 2024
@JohnHalleyGotway JohnHalleyGotway linked a pull request Apr 5, 2024 that will close this issue
bikegeek pushed a commit that referenced this issue Apr 8, 2024
* Rename METdbLoad/tests to METdbLoad/test for internal consistency, and try adding the sonar.coverage.exclusions setting to prevent new test code from counting against code coverage.

* Try defining the exclusion directories using **

* Update the sonar.coverage.exclusions SonarQube setting to match filenames rather than just directories.
@JohnHalleyGotway JohnHalleyGotway linked a pull request Apr 19, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment