Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osmpbf: use cgo/czlib for decompressions, with build tag #19

Merged
merged 1 commit into from
Jan 9, 2021
Merged

Conversation

paulmach
Copy link
Owner

The pbf data comes in blocks, each block is zlib compressed. Decompressing this data takes about 33% of the total read time on my benchmarks. I added support for a cgo version, https://github.com/DataDog/czlib with a build tag.

> CGO_ENABLED=0 go test -bench . 
BenchmarkLondon-12             3         341655178 ns/op        1245167664 B/op  4544349 allocs/op

> CGO_ENABLED=1 go test -bench . 
BenchmarkLondon-12             4         266967016 ns/op        1236313254 B/op  4492042 allocs/op

> benchcmp
benchmark              old ns/op     new ns/op     delta
BenchmarkLondon-12     341655178     266967016     -21.86%

benchmark              old allocs     new allocs     delta
BenchmarkLondon-12     4544349        4492042        -1.15%

benchmark              old bytes      new bytes      delta
BenchmarkLondon-12     1245167664     1236313254     -0.71%

The performance benefit is pretty large but I'm not sure what the community thinking around cgo is, so leaving this hear for now. I'm hoping the build tag support will make it usable.

@invisiblefunnel
Copy link

invisiblefunnel commented Dec 19, 2020

I think this is a good change. Another option is to let the caller override the zlib reader function. No strong preference.

@paulmach paulmach merged commit 585eb15 into master Jan 9, 2021
@paulmach paulmach deleted the cgo branch March 23, 2021 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants