Page size error when parsing sas7bdat file

[curtisalexander]

Issue

I receive the following error when parsing a rather large sas7bdat file.

Format: SAS data file (SAS7BDAT)
ReadStat: Error parsing page 32767, bytes 0-131071
Error processing .\_rand_ds.sas7bdat: Invalid file, or file has unsupported features

Dataset

The dataset I am using for testing has 3,800,000 rows and 110 columns. Of greater import is that it has 33,195 (i.e. > 32,767) pages if I run a proc contents on the file from within SAS. I can take the same dataset and cut it down so that it has < 32,767 pages and I can parse without issue.

OS

I get the above error only when I run on 64-bit Windows (x86 processor). I built the executable — ReadStat_App.exe — using Visual Studio 19 and the newly added Visual Studio solution for 1.1.5.

If I build on 64-bit Linux (x86 processor), I can parse the file without error. To me this suggests a challenge with macOS / Linux C integer sizes vs. Windows C integer sizes.

Troubleshooting

Note that I'm glad to provide the raw dataset I'm using for testing (it is ~ 4.6GB in size). Alternatively I can provide the SAS program I utilized to generate the dataset if you have access to SAS (the program just creates random data to produce a large file).

Or if I can assist by simply rebuilding and testing from a different commit, I'm glad to do so.

[evanmiller]

Hi, thanks for the report. Do you see the errors described in #225 when compiling on Windows? Given the magic 4 GB file size I am guessing there is an overflowing 32-bit variable somewhere.

[curtisalexander]

Yes, I see the warnings in #225.

I see a slew of other warnings as well. That could be because I am utilizing VS19 or it could be my error as I don't ever work with VS.

[evanmiller]

If you look at the Appveyor log linked in that issue, you will see many signed/unsigned warnings, among others.

See if this branch fixes things:

https://github.com/WizardMac/ReadStat/tree/windows-largefile

[curtisalexander]

That branch fixes things! I can read the large file now. I tested both displaying the sas7bdat metadata as well as reading thesas7bdat and subsequently writing the file out as a csv.

And that was a super quick turnaround!

[curtisalexander]

Do you know roughly when you would cut a new release with these updates? Or would you move these updates into another branch (say a dev branch)? Just curious as I'm ultimately binding to the C library and am bringing in your (and other ReadStat contributor's) work in as a git submodule. Just trying to figure out which branch I should be looking to bring in.

Thanks again for the assistance!

[evanmiller]

@curtisalexander Excellent news!

Generally I do releases about every 3 months with the accumulated changes, with a 1 month beta period prior. master always reflects the most recent release.

I'll be moving these changes into dev once I get some other Windows build stuff resolved (hopefully tonight). So I'd point to dev if you want the latest and greatest and don't mind if things break now and then.

[curtisalexander]

Sounds good - thank you for the clarification!

[curtisalexander]

Closing - fixed by dc76fb1.