From a6c60ffaed1d78e5e740df2bd0c2bd6f99050a9c Mon Sep 17 00:00:00 2001 From: Chris Tolliday Date: Tue, 15 Aug 2023 11:23:23 -0700 Subject: [PATCH] Make scrub_preprocessed_java_class local only Summary: Noticed several continuous release builds failing with errors similar to: ``` [2023-08-11T14:29:12.035-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (preprocess_java_classes) [re_worker_upload], and 0 other actions [2023-08-11T14:29:19.036-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 29_statics_internal.jar) [re_action_cache], and 405 other actions [2023-08-11T14:29:26.136-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 6_preconditions.jar) [re_upload], and 32214 other actions [2023-08-11T14:29:33.235-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 6_preconditions.jar) [re_upload], and 23739 other actions [2023-08-11T14:29:40.335-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 344_canaries.jar) [re_upload], and 15291 other actions [2023-08-11T14:29:47.336-07:00] Waiting on fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) -- action (scrub_preprocessed_java_class 34295_multiverse.jar) [re_upload], and 7010 other actions [2023-08-11T14:29:48.875-07:00] Action failed: fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) (scrub_preprocessed_java_class 29938_FBReactNativeFXAccessLibrarySpecAndroid.jar) [2023-08-11T14:29:48.877-07:00] Stderr: dotslash error: problem with `/re_cwd/third-party/toolchains/jdk/dotslash_java` [2023-08-11T14:29:48.877-07:00] caused by: failed to download artifact [2023-08-11T14:29:48.877-07:00] caused by: backend fetch failed [2023-08-11T14:29:48.877-07:00] caused by: CASDaemonService::materializeDotslashAndCache failed with ApplicationException [2023-08-11T14:29:48.877-07:00] caused by: Loadshedding: load shedding due to qps limit [2023-08-11T14:29:48.987-07:00] Action failed: fbsource//fbandroid/apps/fb4a:fbandroid_no_face_e2e_staticdi_dextr_splitarsc_dlstr_xzs_hdpi_x86_release (fbsource//tools/build_defs/config/platform/android:fb4a#e9f2c405f2ad03ef) (scrub_preprocessed_java_class 28507_bkactioncaaretrieveblockstore.jar) ... followed by several instances of the same error on different 'scrub_preprocessed_java_class' actions) ``` https://www.internalfb.com/skycastle/run/3012908150711616126 It seems likely that as soon as `preprocess_java_classes` finishes more than 30k `scrub_preprocessed_java_class` actions are scheduled at the same time, all of which run on RE, which all try to fetch the same dotslash artifact at the same time and hit qps limits. We should be able to work around this by just running it locally. Reviewed By: IanChilds Differential Revision: D48283036 fbshipit-source-id: 19b2602b60c535bdc1337d86b3744c1bbf53baf9 --- prelude/android/preprocess_java_classes.bzl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prelude/android/preprocess_java_classes.bzl b/prelude/android/preprocess_java_classes.bzl index e8b9fcb2..acd8fda1 100644 --- a/prelude/android/preprocess_java_classes.bzl +++ b/prelude/android/preprocess_java_classes.bzl @@ -57,7 +57,7 @@ def get_preprocessed_java_classes(ctx: AnalysisContext, input_jars = {"artifact" jar_name = unscrubbed_output_jar.basename output_jar = ctx.actions.declare_output("preprocessed_java_classes/output_dir/{}".format(jar_name)) scrub_cmd = cmd_args(zip_scrubber, unscrubbed_output_jar, output_jar.as_output()) - ctx.actions.run(scrub_cmd, category = "scrub_preprocessed_java_class", identifier = jar_name) + ctx.actions.run(scrub_cmd, category = "scrub_preprocessed_java_class", identifier = jar_name, local_only = True) output_jars_to_owners[output_jar] = target_label return output_jars_to_owners