Skip to content

Commit

Permalink
Make scrub_preprocessed_java_class local only
Browse files Browse the repository at this point in the history
Summary:
Noticed several continuous release builds failing with errors similar to:
```
[2023-08-11T14:29:12.035-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (preprocess_java_classes) [re_worker_upload], and 0 other actions
[2023-08-11T14:29:19.036-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 29_statics_internal.jar) [re_action_cache], and 405 other actions
[2023-08-11T14:29:26.136-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 6_preconditions.jar) [re_upload], and 32214 other actions
[2023-08-11T14:29:33.235-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 6_preconditions.jar) [re_upload], and 23739 other actions
[2023-08-11T14:29:40.335-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 344_canaries.jar) [re_upload], and 15291 other actions
[2023-08-11T14:29:47.336-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 34295_multiverse.jar) [re_upload], and 7010 other actions
[2023-08-11T14:29:48.875-07:00] Action failed: fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) (scrub_preprocessed_java_class 29938_FBReactNativeFXAccessLibrarySpecAndroid.jar)
[2023-08-11T14:29:48.877-07:00] Stderr: dotslash error: problem with `/re_cwd/third-party/toolchains/jdk/dotslash_java`
[2023-08-11T14:29:48.877-07:00] caused by: failed to download artifact
[2023-08-11T14:29:48.877-07:00] caused by: backend fetch failed
[2023-08-11T14:29:48.877-07:00] caused by: CASDaemonService::materializeDotslashAndCache failed with ApplicationException
[2023-08-11T14:29:48.877-07:00] caused by: Loadshedding: load shedding due to qps limit
[2023-08-11T14:29:48.987-07:00] Action failed: fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) (scrub_preprocessed_java_class 28507_bkactioncaaretrieveblockstore.jar)
...
followed by several instances of the same error on different 'scrub_preprocessed_java_class' actions)
```
https://www.internalfb.com/skycastle/run/3012908150711616126

It seems likely that as soon as `preprocess_java_classes` finishes more than 30k `scrub_preprocessed_java_class` actions are scheduled at the same time, all of which run on RE, which all try to fetch the same dotslash artifact at the same time and hit qps limits.

We should be able to work around this by just running it locally.

Reviewed By: IanChilds

Differential Revision: D48283036

fbshipit-source-id: 19b2602b60c535bdc1337d86b3744c1bbf53baf9
  • Loading branch information
christolliday authored and facebook-github-bot committed Aug 15, 2023
1 parent 035cfaa commit a6c60ff
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion prelude/android/preprocess_java_classes.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ def get_preprocessed_java_classes(ctx: AnalysisContext, input_jars = {"artifact"
jar_name = unscrubbed_output_jar.basename
output_jar = ctx.actions.declare_output("preprocessed_java_classes/output_dir/{}".format(jar_name))
scrub_cmd = cmd_args(zip_scrubber, unscrubbed_output_jar, output_jar.as_output())
ctx.actions.run(scrub_cmd, category = "scrub_preprocessed_java_class", identifier = jar_name)
ctx.actions.run(scrub_cmd, category = "scrub_preprocessed_java_class", identifier = jar_name, local_only = True)
output_jars_to_owners[output_jar] = target_label

return output_jars_to_owners

0 comments on commit a6c60ff

Please sign in to comment.