Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse thread pool for threaded tests #37

Closed
Swatinem opened this issue Dec 4, 2023 · 1 comment
Closed

Reuse thread pool for threaded tests #37

Swatinem opened this issue Dec 4, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@Swatinem
Copy link

Swatinem commented Dec 4, 2023

Being able to run tests multi-threaded simply by adding DIVAN_THREADS=XXX to get a feeling for contention is super nice.
However it seems like every benchmark run is using scoped threads under the hood.

Running a threaded benchmark through samply record, I end up with well beyond 6k "tracks", each of which is extremely short lived and it is pretty much impossible to select any of the background threads to do proper profiling.

It also appears that a large portion of the main thread time is actually spent creating / destroying threads themselves, at least on macOs where I tested this:

Bildschirmfoto 2023-12-04 um 11 09 35

@nvzqz nvzqz added the enhancement New feature or request label Dec 6, 2023
@nvzqz nvzqz closed this as completed in 4c94258 Nov 25, 2024
@nvzqz
Copy link
Owner

nvzqz commented Nov 25, 2024

Hey @Swatinem sorry for taking forever on this. As part of Divan's JSON work, I am refactoring Divan's internals and part of that refactor was adding a thread pool. I decided to extract the thread pool out to address your issue, and this fix is now available in v0.1.16.

In case you're curious about how it works:

  1. The main thread creates a Task, which is a pointer to a TaskShared pinned on the stack. TaskShared stores the function to run, along with other fields for coordinating threads.

  2. New threads are spawned if the requested amount is not available. Each receives tasks over an associated channel.

  3. The main thread sends the Task over the channels to the requested amount of threads. Upon receiving the task, each auxiliary thread will execute it and then decrement the task's reference count.

  4. The main thread executes the Task like auxiliary threads. It then waits until the reference count is 0 before returning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants