Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream::Encoder::recommended_input_size requires specifying the writer generic #290

Open
Firestar99 opened this issue Jul 9, 2024 · 1 comment

Comments

@Firestar99
Copy link

Firestar99 commented Jul 9, 2024

My use-case is to stream directly from an rkyv serializer through zstd into a file (10MiB to 5GiB compressed). I'm using stream::Encoder to do that, and noticed it has this nice method available:

/// Return a recommendation for the size of data to write at once.
pub fn recommended_input_size() -> usize { [...] }

I going to assume rkyv may only arrive with a few bytes each, so I thought it would be best to use a BufWriter before feeding it into zstd, like that:

BufWriter::with_capacity(
	zstd::stream::Encoder::recommended_input_size(),
	zstd::stream::Encoder::new(write, 0)?.auto_finish(),
)

EDIT: It lowered compression time from 10s to 2s for 300MiB.

However the example above does not compile, as recommended_input_size is within an impl block of Encoder, which requires specifying the writer type as a generic. Even though the method internally does not care for the generic at all. So instead you have to do something like this, which doesn't feel like how it's intended to be used, especially since the generic is the wrong type anyway.

BufWriter::with_capacity(
	zstd::stream::Encoder::<Vec<u8>>::recommended_input_size(),
	zstd::stream::Encoder::new(write, 0)?.auto_finish(),
)
@gyscos
Copy link
Owner

gyscos commented Jul 9, 2024

Hi, and thanks for the report!

Indeed, it's a bit annoying having to specify a type here while it has no impact on what the function actually runs.
An alternative could be to make it a free function instead. It's currently available as zstd_safe::CCtx::in_size(), but we could re-export/wrap it under a better name (not sure what this name should be yet).

Alternatively we could implement this method just for a single instance of the generic type, for example Encoder<()>. It wouldn't be usable as a regular encoder, but rustc would be able to resolve Encoder::recommended_input_size like that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants