Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speedup Ninja backend with many extract_objects or targets #13879

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Speedup Ninja backend with many extract_objects or targets #13879

wants to merge 10 commits into from

Conversation

bonzini
Copy link
Contributor

@bonzini bonzini commented Nov 6, 2024 •
edited
Loading

This speeds up the target generation of QEMU, which is very slow.

  • Before: 41 seconds (total time 174 seconds)
  • After: 29 21 seconds (total time 150 134 seconds)

@bonzini bonzini force-pushed the speedups branch 4 times, most recently from c7bd527 to f501fa9 Compare November 6, 2024 16:08
eli-schwartz
eli-schwartz requested changes Nov 6, 2024
View reviewed changes
mesonbuild/backend/backends.py Outdated
Comment on lines 479 to 486
proj_dir_to_build_root: str) -> T.Tuple[T.List[str], T.List[build.BuildTargetTypes]]:
obj_list: T.List[str] = []
deps: T.List[build.BuildTargetTypes] = []
proj_dir_to_build_root: str,
obj_list: T.List[str], deps: T.List[build.BuildTargetTypes]) -> None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dcbaker I suspect you may have some opinions about this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--verbose? :)

mesonbuild/compilers/compilers.py Outdated Show resolved Hide resolved
@bonzini bonzini force-pushed the speedups branch 4 times, most recently from 47d8542 to 8c0faca Compare November 7, 2024 09:35
@bonzini bonzini changed the title Speedup Ninja backend with many extract_objects Speedup Ninja backend with many extract_objects or targets Nov 7, 2024
@bonzini bonzini force-pushed the speedups branch 2 times, most recently from 234adf6 to bc660cb Compare November 7, 2024 12:13
@eli-schwartz
Copy link
Member

Pushed 1b15bd0 and 58d1efb for now. The backend changes look very interesting, though also take quite a bit more effort to understand what the code is doing both before and after. :D

@bonzini
compilers: cache the results of is_source()
f1fbed6 
is_source() is called almost 900000 times in a QEMU setup.  Together with
the previously added caching, this basically removes _determine_ext_objs()
from the profile when building QEMU.

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
utils: cache build directory files
c1f4f6b 
get_target_generated_sources often calls File.from_built_relative on
the same file, if it is used by many sources.  This is a somewhat
expensive call both CPU- and memory-wise, so cache the creation
of build-directory files as well.

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
ninjabackend: use File.from_built_relative()
589510f 
Do not reinvent it in NinjaBackend.determine_ext_objs(), so as to use
the recently added caching of the results of File.from_built_relative().

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini bonzini force-pushed the speedups branch 3 times, most recently from 14150bd to 84b08ff Compare November 20, 2024 07:16
@bonzini
Copy link
Contributor Author

bonzini commented Nov 20, 2024

The backend changes look very interesting, though also take quite a bit more effort to understand what the code is doing both before and after. :D

Fair enough! I've split the changes more finely, which should be easier to both review and apply individually. If you prefer to have multiple PRs let me know.

@bonzini
ninjabackend: prefer "in" to regex search
f3dfac4 
Regexes can be surprisingly slow.  This small change brings
ninja_quote() from 12 to 3 seconds when building QEMU.
Before:

   ncalls  tottime  percall  cumtime  percall
  3734443    4.872    0.000   11.944    0.000

After:

   ncalls  tottime  percall  cumtime  percall
  3595590    3.193    0.000    3.196    0.000

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
arglist: optimize __init__()
281b960 
"Inline" CompilerArgs.__iter__() into CompilerArgs.__init__(), so that
replace list(Iterable) is replaced by the much faster list(List).

Before:

   ncalls  tottime  cumtime
    19268    0.163    3.586 arglist.py:97(__init__)

After:

   ncalls  tottime  cumtime
    18674    0.211    3.442 arglist.py:97(__init__)

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
arglist: optimize flush_pre_post(), and __iadd__() with it
d054f7f 
Unless an argument is marked as Dedup.OVERRIDDEN, pre_flush_set and
post_flush_set will always be empty and the loops in flush_pre_post()
will not be doing anything interesting:

        for a in self.pre:
            dedup = self._can_dedup(a)
            if a not in pre_flush_set:
                # This just makes new a copy of self.pre
                new.append(a)
                if dedup is Dedup.OVERRIDDEN:
                    # this never happens
                    pre_flush_set.add(a)

        for a in reversed(self.post):
            dedup = self._can_dedup(a)
            if a not in post_flush_set:
                # Here self.post is reversed twice
                post_flush.appendleft(a)
                if dedup is Dedup.OVERRIDDEN:
                    # this never happens
                    post_flush_set.add(a)
        new.extend(post_flush)

In this case it's possible to avoid expensive calls and loops, instead
relying as much on Python builtins as possible.  Track whether any options
have that flag and if not just concatenate pre, _container and post.

Before:

   ncalls  tottime  cumtime
    45127    0.251    4.530 arglist.py:142(__iter__)
    81866    3.623    5.013 arglist.py:108(flush_pre_post)
    76618    3.793    5.338 arglist.py:273(__iadd__)

After:

    35647    0.156    0.627 arglist.py:160(__iter__)
    78998    2.627    3.603 arglist.py:116(flush_pre_post)
    73774    3.605    5.049 arglist.py:292(__iadd__)

The time in __iadd__ is reduced because it calls __iter__, which flushes
pre and post.

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
arglist: post is only appended to, make it a list
db0cadc 
self.post is only ever appended to on the right hand.  However,
it is then reversed twice in flush_pre_post(), by using "for a in
reversed.post()" and appendleft() within the loop.  It would be tempting
to use appendleft() in __iadd__ to avoid the call to reversed(), but that
is not a good idea because the loop of flush_pre_post() is part of a slow
path.  It's rather more important to use a fast extend-with-list-argument
in the fast path where needs_override_check if False.

For clarity, and to remove the temptation, make "post" a list instead
of a deque.

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
backends: avoid extend() in _flatten_object_list
2130bbb 
Accumulate into lists that are passed by the caller, thus avoiding
allocations and calls to extend() on recursive extract_objects().

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
backends: remove unused argument
3593e90 
The proj_dir_to_build_root argument of determine_ext_objs() is always empty,
remove it.

Signed-off-by: Paolo Bonzini <[email protected]>
@bonzini
backends: avoid os.path.join in common case of _flatten_object_list
58a072d 
proj_dir_to_build_root is empty by default, in fact always except on
some cases of the VS2010 backend.

Add it after the fact in flatten_object_list(), which reduces the
numbers of os.path.join().

Signed-off-by: Paolo Bonzini <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dcbaker dcbaker Awaiting requested review from dcbaker dcbaker is a code owner

@jpakkane jpakkane Awaiting requested review from jpakkane jpakkane is a code owner

@eli-schwartz eli-schwartz Awaiting requested review from eli-schwartz

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bonzini @eli-schwartz