Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: [dataproc] updated docs and code sturcture for Dataproc metric configuration #8462

Merged
merged 2 commits into from
Sep 28, 2022
Merged

chore: [dataproc] updated docs and code sturcture for Dataproc metric configuration #8462

merged 2 commits into from
Sep 28, 2022

Conversation

gcf-owl-bot[bot]
Copy link
Contributor

@gcf-owl-bot gcf-owl-bot bot commented Sep 27, 2022 •
edited
Loading

  • Regenerate this pull request now.

Committer: @akshatbhargava123
PiperOrigin-RevId: 477101789

Source-Link: googleapis/googleapis@2e19de1

Source-Link: https://github.com/googleapis/googleapis-gen/commit/c12032a59778fdd173d4cfe82c472c0596043865
Copy-Tag: eyJwIjoiamF2YS1kYXRhcHJvYy8uT3dsQm90LnlhbWwiLCJoIjoiYzEyMDMyYTU5Nzc4ZmRkMTczZDRjZmU4MmM0NzJjMDU5NjA0Mzg2NSJ9

@trusted-contributions-gcf trusted-contributions-gcf bot added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Sep 27, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@snippet-bot
Copy link

snippet-bot bot commented Sep 27, 2022 •
edited
Loading

Here is the summary of changes.

You are about to add 153 region tags.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@trusted-contributions-gcf trusted-contributions-gcf bot added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@diegomarquezp diegomarquezp added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@gcf-owl-bot
Copy link
Contributor Author

gcf-owl-bot bot commented Sep 27, 2022

Owl bot is regenerating pull request 8462...
Build name: ZWM0NWFkMzUtNGNlYy00MWJjLWI0ODctOWM0MTlmZjZkZGVm

@lqiu96 lqiu96 mentioned this pull request Sep 27, 2022
Closed
@trusted-contributions-gcf trusted-contributions-gcf bot added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@trusted-contributions-gcf trusted-contributions-gcf bot added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Sep 27, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@gcf-owl-bot
Copy link
Contributor Author

gcf-owl-bot bot commented Sep 27, 2022

Owl bot is regenerating pull request 8462...
Build name: MDNiYWUwMzItYWJmNC00ZmI4LThmNzQtYjU2ZDcyYWZkNjJm

@trusted-contributions-gcf trusted-contributions-gcf bot added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Sep 27, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@trusted-contributions-gcf trusted-contributions-gcf bot added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Sep 27, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@lqiu96
Copy link
Contributor

lqiu96 commented Sep 27, 2022 •
edited
Loading

Owl-bot is adding in the staging-directory for some reason.

Was reprocessed and removed

@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@trusted-contributions-gcf trusted-contributions-gcf bot added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 27, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@lqiu96
Copy link
Contributor

lqiu96 commented Sep 28, 2022

@suztomo

Original googleapis-gen commit for java-dataproc adds in Metric.java and MetricOrBuilder.java 7 days ago: https://github.com/googleapis/googleapis-gen/commit/a41cc2ab3df78ee0b84f67499e7562de02ab34c0

Another googleapis-gen commit for java-datproc deletes Metric.java and MetricOrBuilder.java yesterday:
https://github.com/googleapis/googleapis-gen/commit/c12032a59778fdd173d4cfe82c472c0596043865

Java-Dataproc

Since java-dataproc owlbot PR was not merged, it looks like owlbot combined the two PRs: googleapis/java-dataproc#922

  • Metric.java and MetricOrBuilder.java are not added to the PR (which looks like the correct state)

Monorepo

In google-cloud-java, the java-dataproc PR was merged 7 days ago: #8433

  • Added in Metric.java and MetricOrBuilder.java

New Owlbot PR for java-dataproc came in yesterday: #8462

  • Does NOT remove Metric.java or MetricOrbuilder.java even though the second googleapis-gen commit removes the two java files.

Is this the correct behavior? I thought this may be a bug.

  1. If you look at the regenerated PR (it shows owlbot staging dir): 9c84ff9 -- It doesn't have the two java files in there.
  2. The post processor doesn't delete the files since it doesn't exist in the staging dir (if I'm understanding this correctly)

If this is the intended behavior, this may be an issue if protos and the corresponding java files are deleted. Can you confirm if this is the behavior or if I'm missing something?

Fix

  1. Manually delete the two files?

@suztomo
Copy link
Member

suztomo commented Sep 28, 2022 •
edited
Loading

Failure in "ci / units (8)":

Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project proto-google-cloud-dataproc-v1: Compilation failure: Compilation failure: 
Error:  /home/runner/work/google-cloud-java/google-cloud-java/java-dataproc/proto-google-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/Metric.java:[118,9] cannot find symbol
Error:    symbol:   variable internal_static_google_cloud_dataproc_v1_Metric_descriptor
Error:    location: class com.google.cloud.dataproc.v1.ClustersProto
Error:  /home/runner/work/google-cloud-java/google-cloud-java/java-dataproc/proto-google-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/Metric.java:[125,9] cannot find symbol
Error:    symbol:   variable internal_static_google_cloud_dataproc_v1_Metric_fieldAccessorTable
Error:    location: class com.google.cloud.dataproc.v1.ClustersProto
Error:  /home/runner/work/google-cloud-java/google-cloud-java/java-dataproc/proto-google-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/Metric.java:[669,11] cannot find symbol
Error:    symbol:   variable internal_static_google_cloud_dataproc_v1_Metric_descriptor
Error:    location: class com.google.cloud.dataproc.v1.ClustersProto
Error:  /home/runner/work/google-cloud-java/google-cloud-java/java-dataproc/proto-google-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/Metric.java:[676,11] cannot find symbol
Error:    symbol:   variable internal_static_google_cloud_dataproc_v1_Metric_fieldAccessorTable
Error:    location: class com.google.cloud.dataproc.v1.ClustersProto
Error:  /home/runner/work/google-cloud-java/google-cloud-java/java-dataproc/proto-google-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/Metric.java:[709,11] cannot find symbol
Error:    symbol:   variable internal_static_google_cloud_dataproc_v1_Metric_descriptor
Error:    location: class com.google.cloud.dataproc.v1.ClustersProto
Error:  -> [Help 1]

owlbot-java log: https://pantheon.corp.google.com/cloud-build/builds;region=global/adc924e4-6e44-4efb-983b-fbf65dea6ae8;step=5?project=repo-automation-bots , says "In monorepo but no owl-bot-staging. Formatting changes in the last commit". Searching for owlbot-java invocation that actually extracted owl-bot-staging directory.

Selected "Regenerate this pull request now."

How the files are supposed to be removed

java-dataproc/.OwlBot.yaml defines "deep-remove-regex". deep-remove-regex: tells Owl Bot which files to remove before copying in files from googleapis/googleapis-gen (from the document).

https://github.com/googleapis/google-cloud-java/blob/main/java-dataproc/.OwlBot.yaml#L16

deep-remove-regex:
- "/samples/snippets/generated"
- "/grpc-google-.*/src"
- "/proto-google-.*/src"
- "/google-.*/src"

google-cloud-java/java-dataproc/proto-google-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/Metric.java should match the 3rd item "/proto-google-.*/src. The files in /proto-google-.*/src should have been removed when owlbot-java ran with working directory of google-cloud-java/java-dataproc.

Before owlbot-java postprocessor finishes, the pull request has commit 5573023
. https://gist.github.com/suztomo/6c48106fda143b4d399529dae85e2daa . This will soon be rebased by OwlBot Java postprocessor. Metric.java is not in the list.

OwlBot Java PostProcessor log

https://pantheon.corp.google.com/cloud-build/builds;region=global/713b7ce7-dc86-4921-8f05-b513b1f30077;step=5?project=repo-automation-bots gpaste/5082812242722816

It's indeed reading from owl-bot-staging.

Step #5: Processing monorepo
Step #5: Extracting contents from owl-bot-staging
Step #5: /workspace/google-cloud-java/java-dataproc /workspace/google-cloud-java

How is it executing deep-remove-regex? To figure out by running the owlbot command by myself.

@gcf-owl-bot
Copy link
Contributor Author

gcf-owl-bot bot commented Sep 28, 2022

Owl bot is regenerating pull request 8462...
Build name: YWU0OGM2MDAtZTNmYy00M2MyLTkxYzctOTc3NDc5MGQzMWU3

@trusted-contributions-gcf trusted-contributions-gcf bot added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Sep 28, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 28, 2022
@trusted-contributions-gcf trusted-contributions-gcf bot added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 28, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 28, 2022
@suztomo suztomo mentioned this pull request Sep 28, 2022
Merged
gcf-merge-on-green bot pushed a commit that referenced this pull request Sep 28, 2022
@suztomo
build: adding /java-dataproc to OwlBot config (#8476)
8c5c39e 
An experiment to add /java-dataproc to OwlBot configuration file

This is part of investigation of #8462
@gcf-owl-bot
Copy link
Contributor Author

gcf-owl-bot bot commented Sep 28, 2022

Owl bot is regenerating pull request 8462...
Build name: OWRiOTVhOWEtYmQzZC00NjkzLTljNmEtMjk3NzQzNTA0Njlk

@gcf-owl-bot
chore: updated docs and code sturcture for Dataproc metric configuration
44ba2f2 
Committer: @akshatbhargava123
PiperOrigin-RevId: 477101789

Source-Link: googleapis/googleapis@2e19de1

Source-Link: https://github.com/googleapis/googleapis-gen/commit/c12032a59778fdd173d4cfe82c472c0596043865
Copy-Tag: eyJwIjoiamF2YS1kYXRhcHJvYy8uT3dsQm90LnlhbWwiLCJoIjoiYzEyMDMyYTU5Nzc4ZmRkMTczZDRjZmU4MmM0NzJjMDU5NjA0Mzg2NSJ9
@trusted-contributions-gcf trusted-contributions-gcf bot added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 28, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 28, 2022
@trusted-contributions-gcf trusted-contributions-gcf bot added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 28, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 28, 2022
@suztomo
Copy link
Member

suztomo commented Sep 28, 2022 •
edited
Loading

The experiment #8476 succeeded:

image

The log file of OwlBot service invocation now has the entry below. First it removes the file and then generates Java code from protos.

    "textPayload": "Step #5: rm  /tmp/tmp-1-0gwwndV7nKFT/dest/java-dataproc/proto-google-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/Metric.java",

@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 28, 2022
@suztomo suztomo merged commit b0e22b7 into main Sep 28, 2022
@suztomo suztomo deleted the owl-bot-copy-java-dataproc branch September 28, 2022 19:38
@suztomo suztomo mentioned this pull request Sep 28, 2022
Closed
suztomo added a commit to suztomo/google-cloud-java that referenced this pull request Oct 4, 2022
@suztomo
build: adding /java-dataproc to OwlBot config (googleapis#8476)
bf079f7 
An experiment to add /java-dataproc to OwlBot configuration file

This is part of investigation of googleapis#8462
suztomo pushed a commit to suztomo/google-cloud-java that referenced this pull request Oct 4, 2022
@gcf-owl-bot @suztomo
chore: [dataproc] updated docs and code sturcture for Dataproc metric…
2146600 
… configuration (googleapis#8462)

* chore: updated docs and code sturcture for Dataproc metric configuration

Committer: @akshatbhargava123
PiperOrigin-RevId: 477101789

Source-Link: googleapis/googleapis@2e19de1

Source-Link: googleapis/googleapis-gen@c12032a
Copy-Tag: eyJwIjoiamF2YS1kYXRhcHJvYy8uT3dsQm90LnlhbWwiLCJoIjoiYzEyMDMyYTU5Nzc4ZmRkMTczZDRjZmU4MmM0NzJjMDU5NjA0Mzg2NSJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@suztomo suztomo suztomo approved these changes

Assignees
No one assigned
Labels
owl-bot-copy
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lqiu96 @suztomo @diegomarquezp @yoshi-kokoro