Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multi-threaded compression and dictionary #4189

Open
sreendra opened this issue Nov 5, 2024 · 6 comments
Open

multi-threaded compression and dictionary #4189

sreendra opened this issue Nov 5, 2024 · 6 comments
Assignees
Labels

Comments

@sreendra
Copy link

sreendra commented Nov 5, 2024

Is your feature request related to a problem? Please describe.
We have an application which is multi threaded and are planning to compress using a digested dictionary. When i used the same ctx for compression, intermittently i am getting Access violation reading location. Then after changing it to non-static, the issue subsided as mentioned in the manual ?

Describe the solution you'd like
Can we make it 1 time init and use it across threads ? This saves init time/thread ?

Describe alternatives you've considered
NA

Additional context
NA

@Cyan4973 Cyan4973 changed the title Hi @Cyan, multi-threaded compression and dictionary Nov 5, 2024
@Cyan4973
Copy link
Contributor

Cyan4973 commented Nov 5, 2024

Dictionaries can be shared between threads.

@sreendra
Copy link
Author

sreendra commented Nov 6, 2024

Thanks for the info @Cyan4973 .

I have not benchmarked how long it is taking for compression context and decompression context initialization. But assuming it's an overhead on top of every request ?

Is it possible to make it as a static variable to be leveraged across multiple threads ?

@Cyan4973 Cyan4973 self-assigned this Nov 6, 2024
@Cyan4973
Copy link
Contributor

Cyan4973 commented Nov 6, 2024

A compression context can be re-employed,
but it cannot be shared concurrently across multiple threads.

Re-using a context is generally good for speed and reduced memory pressure.
The initialization cost also gets reduced considerably.

@sreendra
Copy link
Author

sreendra commented Nov 6, 2024

Sure @Cyan4973 ...So i have to live with it is it ? Because my app is a multi threaded scenario.

Thank you for your prompt responses

@Cyan4973
Copy link
Contributor

Ensure that each thread has its own compression context.

Then they can work in parallel, sharing the same dictionary.

@sreendra
Copy link
Author

Sure @Cyan4973 ...Thank you...

Are there any plans in future to even make this context shareable across threads because it saves lot of time along with storage ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants