Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support writing character-compressed data in SAS files #75

Closed
evanmiller opened this issue Aug 1, 2016 · 2 comments
Closed

Support writing character-compressed data in SAS files #75

evanmiller opened this issue Aug 1, 2016 · 2 comments

Comments

@evanmiller
Copy link
Contributor

SAS files often use RLE compression to reduce file size (this is distinct from binary compression described in #21). ReadStat can read but not write RLE-compressed files.

@evanmiller
Copy link
Contributor Author

One potential issue is that with compressed rows, it may be impossible to predict the number of data pages required by the file, since compressed rows have variable length. This is a problem because ReadStat uses a streaming design for writing out a file, and the total page count is required at the beginning of the file.

@evanmiller
Copy link
Contributor Author

Implemented in c6f1a0a

Enable with

readstat_writer_set_compression(writer, READSTAT_COMPRESS_ROWS);

Note that with the RLE compression algorithm, the entire file will be built in memory before being written out. This is to address the page count issue mentioned above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant