Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve compilation time for specific files (ClangBuildAnalyzer report included) #57569

Open
Calinou opened this issue Feb 3, 2022 · 2 comments
Open

Improve compilation time for specific files (ClangBuildAnalyzer report included) #57569

Calinou opened this issue Feb 3, 2022 · 2 comments
Labels
enhancement performance topic:buildsystem

Comments

@Calinou
Copy link
Member

Calinou commented Feb 3, 2022 •
edited
Loading

Related to #41271 and #42048.

Godot version

4.0.dev (fc09d78)

System information

Fedora 34, Clang 12.0.1, ClangBuildAnalyzer 1.3.0

Issue description

We should figure out how to decrease the compilation time of specific C++ files that take the longest to compile. This was measured on an Intel Core i7-6700K:

**** Time summary:
Compilation (3442 times):
  Parsing (frontend):         4841.4 s
  Codegen & opts (backend):    785.5 s

**** Files that took longest to parse (compiler frontend):
 38652 ms: artifacts/editor_fonts.linuxbsd.tools.64.llvm.json
 31744 ms: artifacts/editor_translation.linuxbsd.tools.64.llvm.json
 24071 ms: artifacts/text_server_adv.linuxbsd.tools.64.llvm.json
 19170 ms: artifacts/rtlightmap_hdr.gen.linuxbsd.tools.64.llvm.json
 13625 ms: artifacts/editor_node.linuxbsd.tools.64.llvm.json
 11168 ms: artifacts/hb-subset.linuxbsd.tools.64.llvm.json
 10993 ms: artifacts/hb-subset-plan.linuxbsd.tools.64.llvm.json
 10725 ms: artifacts/visual_script_editor.linuxbsd.tools.64.llvm.json
 10558 ms: artifacts/editor_help.linuxbsd.tools.64.llvm.json
 10350 ms: artifacts/hb-ot-face.linuxbsd.tools.64.llvm.json

**** Files that took longest to codegen (compiler backend):
 15259 ms: artifacts/bvh_builder_twolevel.linuxbsd.tools.64.llvm.json
 11608 ms: artifacts/bvh_builder_sah_spatial.linuxbsd.tools.64.llvm.json
 10898 ms: artifacts/bvh_intersector_hybrid4_bvh4.linuxbsd.tools.64.llvm.json
 10591 ms: artifacts/rtcore_builder.linuxbsd.tools.64.llvm.json
  9919 ms: artifacts/state.linuxbsd.tools.64.llvm.json
  9747 ms: artifacts/bvh_builder_sah_mb.linuxbsd.tools.64.llvm.json
  9200 ms: artifacts/hb-subset.linuxbsd.tools.64.llvm.json
  8507 ms: artifacts/bvh_intersector_stream_bvh4.linuxbsd.tools.64.llvm.json
  6798 ms: artifacts/variant_call.linuxbsd.tools.64.llvm.json
  6532 ms: artifacts/bvh_intersector1_bvh4.linuxbsd.tools.64.llvm.json

**** Templates that took longest to instantiate:
 14476 ms: std::__detail::__hyperg<long double> (1137 times, avg 12 ms)
 14041 ms: std::__detail::__hyperg<float> (1137 times, avg 12 ms)
 13764 ms: std::unique_ptr<std::thread::_State> (648 times, avg 21 ms)
 11214 ms: std::__uniq_ptr_data<std::thread::_State, std::default_delete<std::t... (648 times, avg 17 ms)
 11136 ms: std::basic_string<char16_t>::_M_construct<const char16_t *> (2324 times, avg 4 ms)
 11116 ms: std::__uniq_ptr_impl<std::thread::_State, std::default_delete<std::t... (648 times, avg 17 ms)
 10737 ms: std::basic_string<char32_t>::_M_construct<const char32_t *> (2324 times, avg 4 ms)
  9818 ms: std::__detail::__hyperg_reflect<long double> (1137 times, avg 8 ms)
  9743 ms: std::__detail::__hyperg_reflect<float> (1137 times, avg 8 ms)
  9300 ms: std::make_shared<std::mutex> (697 times, avg 13 ms)
  9135 ms: __gnu_cxx::__to_xstring<std::basic_string<wchar_t>, wchar_t> (1162 times, avg 7 ms)
  8771 ms: std::allocate_shared<std::mutex, std::allocator<std::mutex>> (697 times, avg 12 ms)
  8496 ms: std::shared_ptr<std::mutex>::shared_ptr<std::allocator<std::mutex>> (697 times, avg 12 ms)
  8301 ms: std::__shared_ptr<std::mutex, __gnu_cxx::_S_atomic>::__shared_ptr<st... (697 times, avg 11 ms)
  8270 ms: std::basic_string<wchar_t>::_M_construct<wchar_t *> (2324 times, avg 3 ms)
  7676 ms: std::__shared_count<__gnu_cxx::_S_atomic>::__shared_count<std::mutex... (697 times, avg 11 ms)
  7557 ms: std::__detail::__cyl_bessel_j<float> (1137 times, avg 6 ms)
  7403 ms: std::__detail::__cyl_bessel_j<long double> (1137 times, avg 6 ms)
  7309 ms: __gnu_cxx::__to_xstring<std::basic_string<char>, char> (1161 times, avg 6 ms)
  7051 ms: std::basic_string<char32_t> (1162 times, avg 6 ms)
  7002 ms: std::basic_string<char16_t>::basic_string (1162 times, avg 6 ms)
  6859 ms: std::basic_string<char32_t>::basic_string (1162 times, avg 5 ms)
  6846 ms: std::basic_string<char> (1162 times, avg 5 ms)
  6788 ms: std::basic_string<char16_t> (1162 times, avg 5 ms)
  6376 ms: std::unordered_map<int, int> (302 times, avg 21 ms)
  6334 ms: std::basic_string<wchar_t> (1162 times, avg 5 ms)
  6050 ms: std::__detail::__cyl_bessel_i<float> (1137 times, avg 5 ms)
  5888 ms: std::__detail::__cyl_bessel_i<long double> (1137 times, avg 5 ms)
  5723 ms: std::__detail::__bessel_jn<float> (1137 times, avg 5 ms)
  5623 ms: std::__detail::__bessel_jn<long double> (1137 times, avg 4 ms)

**** Template sets that took longest to instantiate:
 67236 ms: ClassDB::bind_method<$> (6466 times, avg 10 ms)
 65760 ms: create_method_bind<$> (6422 times, avg 10 ms)
 50085 ms: std::__and_<$> (27356 times, avg 1 ms)
 44208 ms: CowData<$>::resize (13303 times, avg 3 ms)
 38888 ms: Vector<$>::resize (10577 times, avg 3 ms)
 30175 ms: MethodBindTRC<$>::MethodBindTRC (2623 times, avg 11 ms)
 28518 ms: std::__detail::__hyperg<$> (2274 times, avg 12 ms)
 27288 ms: std::basic_string<$> (4681 times, avg 5 ms)
 25752 ms: std::unique_ptr<$> (1169 times, avg 22 ms)
 25352 ms: MethodBindTRC<$>::call (2623 times, avg 9 ms)
 22272 ms: std::__or_<$> (18465 times, avg 1 ms)
 21998 ms: std::basic_string<$>::_M_construct<$> (7039 times, avg 3 ms)
 21042 ms: std::__uniq_ptr_data<$> (1169 times, avg 18 ms)
 20815 ms: std::__uniq_ptr_impl<$> (1169 times, avg 17 ms)
 20510 ms: MethodBindT<$>::MethodBindT (2935 times, avg 6 ms)
 20437 ms: std::basic_string<$>::_M_construct_aux<$> (6999 times, avg 2 ms)
 20434 ms: std::is_convertible<$> (17183 times, avg 1 ms)
 19601 ms: std::basic_string<$>::basic_string (5897 times, avg 3 ms)
 19561 ms: std::__detail::__hyperg_reflect<$> (2274 times, avg 8 ms)
 18687 ms: __gnu_cxx::__stoa<$> (18551 times, avg 1 ms)
 18013 ms: std::chrono::duration<$> (11325 times, avg 1 ms)
 17329 ms: Vector<$>::Vector (17750 times, avg 0 ms)
 16835 ms: CowData<$>::_copy_on_write (13770 times, avg 1 ms)
 16445 ms: __gnu_cxx::__to_xstring<$> (2323 times, avg 7 ms)
 16196 ms: Vector<$>::push_back (3997 times, avg 4 ms)
 14961 ms: std::__detail::__cyl_bessel_j<$> (2274 times, avg 6 ms)
 12100 ms: Vector<$> (4132 times, avg 2 ms)
 11938 ms: std::__detail::__cyl_bessel_i<$> (2274 times, avg 5 ms)
 11865 ms: std::_Hashtable<$> (1041 times, avg 11 ms)
 11414 ms: MethodBindT<$>::call (2935 times, avg 3 ms)

**** Functions that took longest to compile:
  7257 ms: embree::State::parse(embree::Ref<embree::TokenStream>) (thirdparty/embree/kernels/common/state.cpp)
  1642 ms: VisualShaderEditor::VisualShaderEditor() (editor/plugins/visual_shader_editor_plugin.cpp)
  1264 ms: RenderingServer::_bind_methods() (servers/rendering_server.cpp)
   953 ms: embree::sse2::BVHBuilderMSMBlur::BuilderT<embree::NodeRefPtr<4>, emb... (thirdparty/embree/kernels/bvh/bvh_builder_sah_mb.cpp)
   844 ms: RenderingDevice::_bind_methods() (servers/rendering/rendering_device.cpp)
   825 ms: EditorNode::EditorNode() (editor/editor_node.cpp)
   733 ms: embree::sse2::BVHBuilderMSMBlur::BuilderT<embree::NodeRefPtr<4>, emb... (thirdparty/embree/kernels/bvh/bvh_builder_sah_mb.cpp)
   726 ms: embree::sse2::BVHBuilderMSMBlur::BuilderT<embree::NodeRefPtr<4>, emb... (thirdparty/embree/kernels/bvh/bvh_builder_sah_mb.cpp)
   716 ms: create_editor_theme(Ref<Theme>) (editor/editor_themes.cpp)
   684 ms: fill_default_theme(Ref<Theme>&, Ref<Font> const&, Ref<Texture2D>&, R... (scene/resources/default_theme/default_theme.cpp)
   618 ms: _register_variant_builtin_methods() (core/variant/variant_call.cpp)
   580 ms: GDScriptFunction::disassemble(Vector<String> const&) const (modules/gdscript/gdscript_disassembler.cpp)
   569 ms: BaseMaterial3D::_bind_methods() (scene/resources/material.cpp)
   568 ms: register_global_constants() (core/core_constants.cpp)
   553 ms: embree::sse2::BVHBuilderMSMBlur::BuilderT<embree::NodeRefPtr<4>, emb... (thirdparty/embree/kernels/bvh/bvh_builder_sah_mb.cpp)
   520 ms: embree::sse2::GeneralBVHBuilder::BuilderT<embree::sse2::GeneralBVHBu... (thirdparty/embree/kernels/bvh/bvh_builder.cpp)
   498 ms: GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Call... (modules/gdscript/gdscript_vm.cpp)
   468 ms: match (thirdparty/pcre2/src/pcre2_match.c)
   438 ms: embree::sse2::BVHNIntersectorStream<4, 1, true, embree::sse2::Triang... (thirdparty/embree/kernels/bvh/bvh_intersector_stream_bvh4.cpp)
   419 ms: embree::sse2::GeneralBVHBuilder::BuilderT<embree::sse2::GeneralBVHBu... (thirdparty/embree/kernels/bvh/bvh_builder.cpp)
   418 ms: embree::sse2::BVHNIntersectorStream<4, 1, false, embree::sse2::Trian... (thirdparty/embree/kernels/bvh/bvh_intersector_stream_bvh4.cpp)
   391 ms: embree::sse2::BVHNIntersectorStream<4, 1, true, embree::sse2::Triang... (thirdparty/embree/kernels/bvh/bvh_intersector_stream_bvh4.cpp)
   384 ms: embree::sse2::BVHNIntersectorStream<4, 1, false, embree::sse2::Trian... (thirdparty/embree/kernels/bvh/bvh_intersector_stream_bvh4.cpp)
   374 ms: embree::Stat::print(std::ostream&) (thirdparty/embree/kernels/common/stat.cpp)
   372 ms: ShaderTypes::ShaderTypes() (servers/rendering/shader_types.cpp)
   367 ms: yyparse(glslang::TParseContext*) (thirdparty/glslang/glslang/MachineIndependent/glslang_tab.cpp)
   362 ms: embree::sse2::BVHNIntersectorStream<4, 1, false, embree::sse2::Trian... (thirdparty/embree/kernels/bvh/bvh_intersector_stream_bvh4.cpp)
   357 ms: void embree::sse2::BVHNIntersectorStream<4, 1, true, embree::sse2::T... (thirdparty/embree/kernels/bvh/bvh_intersector_stream_bvh4.cpp)
   346 ms: void embree::sse2::BVHNIntersectorStream<4, 1, false, embree::sse2::... (thirdparty/embree/kernels/bvh/bvh_intersector_stream_bvh4.cpp)
   332 ms: ParticlesMaterial::_bind_methods() (scene/resources/particles_material.cpp)

**** Function sets that took longest to compile / optimize:
 11070 ms: CowData<$>::_ref(CowData<$> const&) (2132 times, avg 5 ms)
  8094 ms: CowData<$>::resize(int) (1599 times, avg 5 ms)
  7655 ms: CowData<$>::_unref(void*) (4398 times, avg 1 ms)
  7323 ms: embree::State::parse(embree::Ref<$>) (2 times, avg 3661 ms)
  6256 ms: void ClassDB::_add_class<$>() (4581 times, avg 1 ms)
  6034 ms: void embree::TaskScheduler::spawn_root<$>(unsigned long const&, unsi... (116 times, avg 52 ms)
  5725 ms: GetTypeInfo<$>::get_class_info() (4088 times, avg 1 ms)
  5036 ms: CowData<$>::_copy_on_write() (1656 times, avg 3 ms)
  4900 ms: VariantCasterAndValidate<$>::cast(Variant const**, unsigned int, Cal... (3269 times, avg 1 ms)
  4840 ms: CallableCustomMethodPointer<$>::call(Variant const**, int, Variant&,... (994 times, avg 4 ms)
  3773 ms: MethodBindTRC<$>::call(Object*, Variant const**, int, Callable::Call... (2443 times, avg 1 ms)
  3238 ms: CallableCustomMethodPointer<$>::get_object() const (994 times, avg 3 ms)
  2969 ms: embree::sse2::GeneralBVHBuilder::BuilderT<$>::recurse(embree::sse2::... (22 times, avg 134 ms)
  2554 ms: MethodBindT<$>::_gen_argument_type_info(int) const (2635 times, avg 0 ms)
  2517 ms: MethodBindT<$>::call(Object*, Variant const**, int, Callable::CallEr... (2736 times, avg 0 ms)
  2468 ms: embree::TaskScheduler::ClosureTaskFunction<$>::execute() (154 times, avg 16 ms)
  2290 ms: void call_get_argument_type_info_helper<$>(int, int&, PropertyInfo&) (2693 times, avg 0 ms)
  1995 ms: embree::sse2::BVHNIntersectorStream<$>::intersect(embree::Accel::Int... (5 times, avg 399 ms)
  1928 ms: embree::sse2::BVHNIntersectorKHybrid<$>::occluded(embree::vint_impl<... (18 times, avg 107 ms)
  1828 ms: void register_builtin_method<$>(Vector<$> const&, Vector<$> const&) (595 times, avg 3 ms)
  1758 ms: embree::sse2::BVHNIntersector1<$>::intersect(embree::Accel::Intersec... (9 times, avg 195 ms)
  1753 ms: embree::sse2::GeneralBVHBuilder::BuilderT<$>::createLargeLeaf(embree... (20 times, avg 87 ms)
  1584 ms: void embree::sse2::BVHNIntersectorStream<$>::occludedCoherent<$>(emb... (10 times, avg 158 ms)
  1509 ms: embree::sse2::BVHBuilderMSMBlur::BuilderT<$>::recurse(embree::sse2::... (4 times, avg 377 ms)
  1462 ms: embree::sse2::BVHBuilderMSMBlur::BuilderT<$>::createLargeLeaf(embree... (4 times, avg 365 ms)
  1367 ms: MethodBindTRC<$>::_gen_argument_type_info(int) const (1251 times, avg 1 ms)
  1337 ms: embree::sse2::BVHNIntersectorKHybrid<$>::intersectCoherent(embree::v... (10 times, avg 133 ms)
  1190 ms: embree::sse2::BVHNIntersector1<$>::occluded(embree::Accel::Intersect... (10 times, avg 119 ms)
  1167 ms: void ClassDB::register_class<$>() (580 times, avg 2 ms)
  1164 ms: MethodBindTR<$>::call(Object*, Variant const**, int, Callable::CallE... (794 times, avg 1 ms)

*** Expensive headers:
1088267 ms: core/object/class_db.h (included 812 times, avg 1340 ms), included via:
  navigation_server_2d.linuxbsd.tools.64.llvm.json navigation_server_2d.h  (3578 ms)
  godot_navigation_server.linuxbsd.tools.64.llvm.json godot_navigation_server.h navigation_server_3d.h  (3242 ms)
  navigation_server_3d.linuxbsd.tools.64.llvm.json navigation_server_3d.h  (3071 ms)
  editor_vcs_interface.linuxbsd.tools.64.llvm.json editor_vcs_interface.h  (3040 ms)
  rendering_device.linuxbsd.tools.64.llvm.json rendering_device.h  (2517 ms)
  noise_texture.linuxbsd.tools.64.llvm.json noise_texture.h open_simplex_noise.h image.h resource.h resource_uid.h ref_counted.h  (2374 ms)
  ...

1058277 ms: core/object/method_bind.h (included 813 times, avg 1301 ms), included via:
  noise_texture.linuxbsd.tools.64.llvm.json noise_texture.h open_simplex_noise.h image.h resource.h resource_uid.h ref_counted.h class_db.h  (2358 ms)
  rendering_device.linuxbsd.tools.64.llvm.json rendering_device.h class_db.h  (2331 ms)
  scroll_bar.linuxbsd.tools.64.llvm.json scroll_bar.h range.h control.h shortcut.h input_event.h resource.h resource_uid.h ref_counted.h class_db.h  (2313 ms)
  upnp_device.linuxbsd.tools.64.llvm.json upnp_device.h ref_counted.h class_db.h  (2244 ms)
  editor_fonts.linuxbsd.tools.64.llvm.json editor_fonts.h theme.h resource.h resource_uid.h ref_counted.h class_db.h  (2223 ms)
  gdscript_function.linuxbsd.tools.64.llvm.json gdscript_function.h ref_counted.h class_db.h  (2195 ms)
  ...

894631 ms: scene/main/node.h (included 415 times, avg 2155 ms), included via:
  cpu_particles_3d.linuxbsd.tools.64.llvm.json cpu_particles_3d.h visual_instance_3d.h node_3d.h  (4525 ms)
  grid_map.linuxbsd.tools.64.llvm.json grid_map.h node_3d.h  (4514 ms)
  audio_listener_3d.linuxbsd.tools.64.llvm.json audio_listener_3d.h node_3d.h  (4189 ms)
  decal.linuxbsd.tools.64.llvm.json decal.h visual_instance_3d.h node_3d.h  (4103 ms)
  skeleton_ik_3d.linuxbsd.tools.64.llvm.json skeleton_ik_3d.h skeleton_3d.h node_3d.h  (4083 ms)
  xr_nodes.linuxbsd.tools.64.llvm.json xr_nodes.h camera_3d.h node_3d.h  (4032 ms)
  ...

834187 ms: core/object/object.h (included 818 times, avg 1019 ms), included via:
  scroll_bar.linuxbsd.tools.64.llvm.json scroll_bar.h range.h control.h shortcut.h input_event.h resource.h resource_uid.h ref_counted.h class_db.h method_bind.h binder_common.h  (1926 ms)
  editor_fonts.linuxbsd.tools.64.llvm.json editor_fonts.h theme.h resource.h resource_uid.h ref_counted.h class_db.h method_bind.h binder_common.h  (1894 ms)
  noise_texture.linuxbsd.tools.64.llvm.json noise_texture.h open_simplex_noise.h image.h resource.h resource_uid.h ref_counted.h class_db.h method_bind.h binder_common.h  (1852 ms)
  editor_scale.linuxbsd.tools.64.llvm.json os.h engine.h main_loop.h input_event.h resource.h resource_uid.h ref_counted.h class_db.h method_bind.h binder_common.h  (1838 ms)
  editor_run_native.linuxbsd.tools.64.llvm.json editor_run_native.h box_container.h container.h control.h shortcut.h input_event.h resource.h resource_uid.h ref_counted.h class_db.h method_bind.h binder_common.h  (1837 ms)
  video_stream_theora.linuxbsd.tools.64.llvm.json video_stream_theora.h resource_loader.h resource.h resource_uid.h ref_counted.h class_db.h method_bind.h binder_common.h  (1809 ms)
  ...

715677 ms: core/object/ref_counted.h (included 715 times, avg 1000 ms), included via:
  noise_texture.linuxbsd.tools.64.llvm.json noise_texture.h open_simplex_noise.h image.h resource.h resource_uid.h  (2387 ms)
  upnp_device.linuxbsd.tools.64.llvm.json upnp_device.h  (2352 ms)
  scroll_bar.linuxbsd.tools.64.llvm.json scroll_bar.h range.h control.h shortcut.h input_event.h resource.h resource_uid.h  (2334 ms)
  editor_fonts.linuxbsd.tools.64.llvm.json editor_fonts.h theme.h resource.h resource_uid.h  (2248 ms)
  gdscript_function.linuxbsd.tools.64.llvm.json gdscript_function.h  (2243 ms)
  video_stream_theora.linuxbsd.tools.64.llvm.json video_stream_theora.h resource_loader.h resource.h resource_uid.h  (2230 ms)
  ...

650965 ms: scene/main/scene_tree.h (included 418 times, avg 1557 ms), included via:
  scene_tree.linuxbsd.tools.64.llvm.json  (3030 ms)
  grid_map.linuxbsd.tools.64.llvm.json grid_map.h node_3d.h node.h  (2597 ms)
  godot_navigation_server.linuxbsd.tools.64.llvm.json godot_navigation_server.h navigation_server_3d.h class_db.h navigation_region_3d.h node_3d.h  (2522 ms)
  editor_scene_importer_gltf.linuxbsd.tools.64.llvm.json editor_scene_importer_gltf.h gltf_state.h gltf_accessor.h gltf_document.h gltf_document_extension_convert_importer_mesh.h importer_mesh_instance_3d.h node_3d.h node.h  (2516 ms)
  gltf_document_extension.linuxbsd.tools.64.llvm.json gltf_document.h gltf_document_extension_convert_importer_mesh.h importer_mesh_instance_3d.h node_3d.h node.h  (2485 ms)
  gltf_buffer_view.linuxbsd.tools.64.llvm.json gltf_buffer_view.h gltf_document.h gltf_document_extension_convert_importer_mesh.h importer_mesh_instance_3d.h node_3d.h node.h  (2334 ms)
  ...

581604 ms: scene/gui/control.h (included 262 times, avg 2219 ms), included via:
  editor_resource_picker.linuxbsd.tools.64.llvm.json editor_resource_picker.h editor_file_dialog.h box_container.h container.h  (4847 ms)
  progress_bar.linuxbsd.tools.64.llvm.json progress_bar.h range.h  (4798 ms)
  plugin_config_dialog.linuxbsd.tools.64.llvm.json plugin_config_dialog.h check_box.h button.h base_button.h  (4713 ms)
  editor_run_native.linuxbsd.tools.64.llvm.json editor_run_native.h box_container.h container.h  (4643 ms)
  texture_button.linuxbsd.tools.64.llvm.json texture_button.h base_button.h  (4637 ms)
  texture_rect.linuxbsd.tools.64.llvm.json texture_rect.h  (4617 ms)
  ...

579168 ms: scene/main/canvas_item.h (included 330 times, avg 1755 ms), included via:
  touch_screen_button.linuxbsd.tools.64.llvm.json touch_screen_button.h node_2d.h  (4167 ms)
  light_2d.linuxbsd.tools.64.llvm.json light_2d.h node_2d.h  (4020 ms)
  cpu_particles_2d.linuxbsd.tools.64.llvm.json cpu_particles_2d.h node_2d.h  (3972 ms)
  canvas_modulate.linuxbsd.tools.64.llvm.json canvas_modulate.h node_2d.h  (3875 ms)
  polygon_2d.linuxbsd.tools.64.llvm.json polygon_2d.h node_2d.h  (3850 ms)
  skeleton_modification_2d_ccdik.linuxbsd.tools.64.llvm.json skeleton_modification_2d_ccdik.h skeleton_2d.h node_2d.h  (3837 ms)
  ...

555129 ms: core/io/resource.h (included 785 times, avg 707 ms), included via:
  noise_texture.linuxbsd.tools.64.llvm.json noise_texture.h open_simplex_noise.h image.h  (2404 ms)
  scroll_bar.linuxbsd.tools.64.llvm.json scroll_bar.h range.h control.h shortcut.h input_event.h  (2350 ms)
  editor_fonts.linuxbsd.tools.64.llvm.json editor_fonts.h theme.h  (2263 ms)
  video_stream_theora.linuxbsd.tools.64.llvm.json video_stream_theora.h resource_loader.h  (2256 ms)
  tab_container.linuxbsd.tools.64.llvm.json tab_container.h container.h control.h shortcut.h input_event.h  (2235 ms)
  editor_scale.linuxbsd.tools.64.llvm.json os.h engine.h main_loop.h input_event.h  (2226 ms)
  ...

548193 ms: core/io/resource_uid.h (included 785 times, avg 698 ms), included via:
  noise_texture.linuxbsd.tools.64.llvm.json noise_texture.h open_simplex_noise.h image.h resource.h  (2395 ms)
  scroll_bar.linuxbsd.tools.64.llvm.json scroll_bar.h range.h control.h shortcut.h input_event.h resource.h  (2342 ms)
  editor_fonts.linuxbsd.tools.64.llvm.json editor_fonts.h theme.h resource.h  (2256 ms)
  video_stream_theora.linuxbsd.tools.64.llvm.json video_stream_theora.h resource_loader.h resource.h  (2248 ms)
  editor_scale.linuxbsd.tools.64.llvm.json os.h engine.h main_loop.h input_event.h resource.h  (2218 ms)
  tab_container.linuxbsd.tools.64.llvm.json tab_container.h container.h control.h shortcut.h input_event.h resource.h  (2215 ms)
  ...

It seems files generated for embedding binary data take a lot of time to be even read by the compiler. (The time displayed is the sum of all C++ files that included the files in question.)

Is there a way to speed up the embedding of large binary files?

Steps to reproduce

Note: The instructions below are only tested on Linux. Regardless of the operating system, you must use Clang to compile Godot – using GCC or MSVC won't work.

  • Download a binary release for https://github.com/aras-p/ClangBuildAnalyzer or compile it from source.
  • cd to the Godot source repository.
  • Run the following commands in order (assuming you made the ClangBuildAnalyzer binary executable and moved it to your PATH):
ClangBuildAnalyzer --start artifacts

scons use_llvm=yes CCFLAGS="-ftime-trace" -j$(nproc)

# Copy all Clang time trace JSON files to the artifacts directory, so they can be read by ClangBuildAnalyzer.
# There will be some naming conflicts, so the report won't be fully accurate.
# TODO: Figure out a better way to handle this.
cp ../**/*.llvm.json artifacts -n

ClangBuildAnalyzer --stop artifacts capture_file

ClangBuildAnalyzer --analyze capture_file
@Calinou Calinou added this to the 4.0 milestone Feb 3, 2022
@Calinou Calinou mentioned this issue Feb 4, 2022
Merged
@TechnoPorg
Copy link
Contributor

I'm not sure how effective this would be, but perhaps embedding the binary files using the linker would be faster? It would skip the entire stage of parsing the C++ source, which seems to be a place where things slow down. I found a blog post giving a good overview of how to do that, here: https://csl.name/post/embedding-binary-data/

This would be more complicated to do across multiple platforms, but since this would just be doing one very specific function, it shouldn't be too hard to create a script for Godot that takes a binary file, and outputs an object file in the target format, sort of like the Python shader builders. Alternatively, an easier solution would be to include a stripped-down version of https://github.com/netwide-assembler/nasm in Godot, since I believe it supports all the platforms that Godot does.

@Calinou
Copy link
Member Author

Calinou commented Jul 1, 2022 •
edited
Loading

This would be more complicated to do across multiple platforms, but since this would just be doing one very specific function, it shouldn't be too hard to create a script for Godot that takes a binary file, and outputs an object file in the target format, sort of like the Python shader builders. Alternatively, an easier solution would be to include a stripped-down version of netwide-assembler/nasm in Godot, since I believe it supports all the platforms that Godot does.

This is an interesting idea, but relying on NASM at build-time would require an additional dependency to compile Godot. Also, a different solution is required on ARM. It also wouldn't work for WebAssembly.

Embedding files at link-time is worth pursuing as well, but it's scoped to the linker you're currently using. This means it wouldn't work when using MSVC or for WebAssembly. It may also not work with linkers such as mold (I haven't checked).

@akien-mga akien-mga removed this from the 4.0 milestone Feb 22, 2023
@Calinou Calinou mentioned this issue Mar 1, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement performance topic:buildsystem
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Calinou @akien-mga @TechnoPorg