Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support writer owns the schema #56

Open
ZENOTME opened this issue Nov 20, 2024 · 3 comments
Open

Support writer owns the schema #56

ZENOTME opened this issue Nov 20, 2024 · 3 comments

Comments

@ZENOTME
Copy link
Contributor

ZENOTME commented Nov 20, 2024

Currently, the writer needs to take a lifetime param and this will make it hard to include in struct sometimes. E.g.

struct OutWriter<'a> {
  avro_schema: avro::Schema
  inner_writer: avro::Writer<'a,Vec<u8>>
}

Even though OutWriter owns the avro_schema, it still needs to define the lifetime param explicitly. And this lifetime will continue to propagate to other types. Self-reference seems a complicated problem https://users.rust-lang.org/t/how-can-we-teach-people-about-self-referential-types/65362/2. So could we just provide a writer who owns the schema and avoids the lifetime param?🤔

@martin-g
Copy link
Member

Do you need to keep the inner_writer as a field ?
You can create an instance when you need to write something and then discard it, e.g. in OutWriter::write() method.

@ZENOTME
Copy link
Contributor Author

ZENOTME commented Nov 26, 2024

Do you need to keep the inner_writer as a field ? You can create an instance when you need to write something and then discard it, e.g. in OutWriter::write() method.

For example, I want user to use the writer like following:

outside_writer.write()
outside_writer.write() // user can call write multiple times.
outside_writer.close()

In this case, I have to cache the content of each write. When the user calls close, create an Avro writer and write all cached content at once.
But if I can keep the inner_writer as a field, I don't need to cache the content. I think this will cause more memory consumption, at most twice I think.

@martin-g
Copy link
Member

You can experiment and propose a PR but I think it will lead to too many API breaks and probably many clone()s of the Schema.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants