Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breaking change in Thrift library #292

Closed
darabos opened this issue Jun 22, 2020 · 7 comments · Fixed by lynxkite/lynxkite#36
Closed

Breaking change in Thrift library #292

darabos opened this issue Jun 22, 2020 · 7 comments · Fixed by lynxkite/lynxkite#36

Comments

@darabos
Copy link

darabos commented Jun 22, 2020

apache/thrift#2181 broke this library. Probably only the generated code is broken. I'll try to regenerate it and see if that fixes things.

go/src/github.com/xitongsys/parquet-go/parquet/parquet.go:631:37: not enough arguments in call to iprot.ReadStructBegin
	have ()
	want (context.Context)
go/src/github.com/xitongsys/parquet-go/parquet/parquet.go:637:57: not enough arguments in call to iprot.ReadFieldBegin
	have ()
	want (context.Context)
[...]
@darabos
Copy link
Author

darabos commented Jun 22, 2020

I can't regenerate it because there hasn't been a Thrift compiler release since the breaking change. All Go packages depending on the HEAD of their dependencies is sure an interesting setup... I guess I can manually fix the generated code?

@xitongsys
Copy link
Owner

hi, @darabos
You should use the specific version of thrift library

github.com/apache/thrift v0.0.0-20181112125854-24918abba929

@ToutPetitAdrien
Copy link

Hi @darabos
I've got the same error in my project's build and fix it by running:

go mod init

This initialize go.mod tool for managing Go modules. I didn't figure out why is this working yet.

@GordonPman
Copy link

hi, @darabos
You should use the specific version of thrift library

github.com/apache/thrift v0.0.0-20181112125854-24918abba929

Sorry I am new here and this might be a stupid question but how do I pull this specific verison? I tried go get github.com/apache/[email protected] but this throws me this error here: "go: cannot use path@version syntax in GOPATH mode". Would this be correct and is the issue caused by another problem?

@xitongsys
Copy link
Owner

hi, @GordonPman
This thrift version is specified in go.mod.
As I know, go mod can control the version automatically. I will give you a step-by-step description, hope it's helpful:

  1. create a new directory
zxt@as01:~/workspace/tmp$ mkdir a
zxt@as01:~/workspace/tmp$ ls
a
  1. copy the example file to this directory
zxt@as01:~/workspace/tmp/a$ cp ~/go/src/github.com/xitongsys/parquet-go/example/local_flat.go ./
zxt@as01:~/workspace/tmp/a$ ls
local_flat.go
  1. run go mod init in the directory.
zxt@as01:~/workspace/tmp/a$ ls
local_flat.go
zxt@as01:~/workspace/tmp/a$ go mod init a
go: creating new go.mod: module a
zxt@as01:~/workspace/tmp/a$ ls
go.mod  local_flat.go
zxt@as01:~/workspace/tmp/a$
  1. go run your code. go mod will get the right version of all your dependences
zxt@as01:~/workspace/tmp/a$ go run local_flat.go
go: finding github.com/xitongsys/parquet-go-source latest
go: finding github.com/xitongsys/parquet-go v1.5.2
go: downloading github.com/xitongsys/parquet-go v1.5.2
go: downloading github.com/xitongsys/parquet-go-source v0.0.0-20200611103345-88e6b030070d
go: extracting github.com/xitongsys/parquet-go-source v0.0.0-20200611103345-88e6b030070d
go: extracting github.com/xitongsys/parquet-go v1.5.2
go: downloading github.com/apache/thrift v0.0.0-20181112125854-24918abba929
go: downloading github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: downloading github.com/klauspost/compress v1.9.7
go: extracting github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: extracting github.com/apache/thrift v0.0.0-20181112125854-24918abba929
go: extracting github.com/klauspost/compress v1.9.7
go: finding github.com/apache/thrift v0.0.0-20181112125854-24918abba929
go: finding github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: finding github.com/klauspost/compress v1.9.7
2020/06/26 02:06:35 Write Finished
2020/06/26 02:06:35 [{StudentName 20 10 51 true 18439 0} {StudentName 21 11 51.1 false 18439 0} {StudentName 22 12 51.2 true 18439 0} {StudentName 23 13 51.3 false 18439 0} {StudentName 24 14 51.4 true 18439 0} {StudentName 20 15 51.5 false 18439 0} {StudentName 21 16 51.6 true 18439 0} {StudentName 22 17 51.7 false 18439 0} {StudentName 23 18 51.8 true 18439 0} {StudentName 24 19 51.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 30 53 true 18439 0} {StudentName 21 31 53.1 false 18439 0} {StudentName 22 32 53.2 true 18439 0} {StudentName 23 33 53.3 false 18439 0} {StudentName 24 34 53.4 true 18439 0} {StudentName 20 35 53.5 false 18439 0} {StudentName 21 36 53.6 true 18439 0} {StudentName 22 37 53.7 false 18439 0} {StudentName 23 38 53.8 true 18439 0} {StudentName 24 39 53.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 50 55 true 18439 0} {StudentName 21 51 55.1 false 18439 0} {StudentName 22 52 55.2 true 18439 0} {StudentName 23 53 55.3 false 18439 0} {StudentName 24 54 55.4 true 18439 0} {StudentName 20 55 55.5 false 18439 0} {StudentName 21 56 55.6 true 18439 0} {StudentName 22 57 55.7 false 18439 0} {StudentName 23 58 55.8 true 18439 0} {StudentName 24 59 55.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 70 57 true 18439 0} {StudentName 21 71 57.1 false 18439 0} {StudentName 22 72 57.2 true 18439 0} {StudentName 23 73 57.3 false 18439 0} {StudentName 24 74 57.4 true 18439 0} {StudentName 20 75 57.5 false 18439 0} {StudentName 21 76 57.6 true 18439 0} {StudentName 22 77 57.7 false 18439 0} {StudentName 23 78 57.8 true 18439 0} {StudentName 24 79 57.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 90 59 true 18439 0} {StudentName 21 91 59.1 false 18439 0} {StudentName 22 92 59.2 true 18439 0} {StudentName 23 93 59.3 false 18439 0} {StudentName 24 94 59.4 true 18439 0} {StudentName 20 95 59.5 false 18439 0} {StudentName 21 96 59.6 true 18439 0} {StudentName 22 97 59.7 false 18439 0} {StudentName 23 98 59.8 true 18439 0} {StudentName 24 99 59.9 false 18439 0}]

@GordonPman
Copy link

hi, @GordonPman
This thrift version is specified in go.mod.
As I know, go mod can control the version automatically. I will give you a step-by-step description, hope it's helpful:

  1. create a new directory
zxt@as01:~/workspace/tmp$ mkdir a
zxt@as01:~/workspace/tmp$ ls
a
  1. copy the example file to this directory
zxt@as01:~/workspace/tmp/a$ cp ~/go/src/github.com/xitongsys/parquet-go/example/local_flat.go ./
zxt@as01:~/workspace/tmp/a$ ls
local_flat.go
  1. run go mod init in the directory.
zxt@as01:~/workspace/tmp/a$ ls
local_flat.go
zxt@as01:~/workspace/tmp/a$ go mod init a
go: creating new go.mod: module a
zxt@as01:~/workspace/tmp/a$ ls
go.mod  local_flat.go
zxt@as01:~/workspace/tmp/a$
  1. go run your code. go mod will get the right version of all your dependences
zxt@as01:~/workspace/tmp/a$ go run local_flat.go
go: finding github.com/xitongsys/parquet-go-source latest
go: finding github.com/xitongsys/parquet-go v1.5.2
go: downloading github.com/xitongsys/parquet-go v1.5.2
go: downloading github.com/xitongsys/parquet-go-source v0.0.0-20200611103345-88e6b030070d
go: extracting github.com/xitongsys/parquet-go-source v0.0.0-20200611103345-88e6b030070d
go: extracting github.com/xitongsys/parquet-go v1.5.2
go: downloading github.com/apache/thrift v0.0.0-20181112125854-24918abba929
go: downloading github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: downloading github.com/klauspost/compress v1.9.7
go: extracting github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: extracting github.com/apache/thrift v0.0.0-20181112125854-24918abba929
go: extracting github.com/klauspost/compress v1.9.7
go: finding github.com/apache/thrift v0.0.0-20181112125854-24918abba929
go: finding github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: finding github.com/klauspost/compress v1.9.7
2020/06/26 02:06:35 Write Finished
2020/06/26 02:06:35 [{StudentName 20 10 51 true 18439 0} {StudentName 21 11 51.1 false 18439 0} {StudentName 22 12 51.2 true 18439 0} {StudentName 23 13 51.3 false 18439 0} {StudentName 24 14 51.4 true 18439 0} {StudentName 20 15 51.5 false 18439 0} {StudentName 21 16 51.6 true 18439 0} {StudentName 22 17 51.7 false 18439 0} {StudentName 23 18 51.8 true 18439 0} {StudentName 24 19 51.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 30 53 true 18439 0} {StudentName 21 31 53.1 false 18439 0} {StudentName 22 32 53.2 true 18439 0} {StudentName 23 33 53.3 false 18439 0} {StudentName 24 34 53.4 true 18439 0} {StudentName 20 35 53.5 false 18439 0} {StudentName 21 36 53.6 true 18439 0} {StudentName 22 37 53.7 false 18439 0} {StudentName 23 38 53.8 true 18439 0} {StudentName 24 39 53.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 50 55 true 18439 0} {StudentName 21 51 55.1 false 18439 0} {StudentName 22 52 55.2 true 18439 0} {StudentName 23 53 55.3 false 18439 0} {StudentName 24 54 55.4 true 18439 0} {StudentName 20 55 55.5 false 18439 0} {StudentName 21 56 55.6 true 18439 0} {StudentName 22 57 55.7 false 18439 0} {StudentName 23 58 55.8 true 18439 0} {StudentName 24 59 55.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 70 57 true 18439 0} {StudentName 21 71 57.1 false 18439 0} {StudentName 22 72 57.2 true 18439 0} {StudentName 23 73 57.3 false 18439 0} {StudentName 24 74 57.4 true 18439 0} {StudentName 20 75 57.5 false 18439 0} {StudentName 21 76 57.6 true 18439 0} {StudentName 22 77 57.7 false 18439 0} {StudentName 23 78 57.8 true 18439 0} {StudentName 24 79 57.9 false 18439 0}]
2020/06/26 02:06:35 [{StudentName 20 90 59 true 18439 0} {StudentName 21 91 59.1 false 18439 0} {StudentName 22 92 59.2 true 18439 0} {StudentName 23 93 59.3 false 18439 0} {StudentName 24 94 59.4 true 18439 0} {StudentName 20 95 59.5 false 18439 0} {StudentName 21 96 59.6 true 18439 0} {StudentName 22 97 59.7 false 18439 0} {StudentName 23 98 59.8 true 18439 0} {StudentName 24 99 59.9 false 18439 0}]

This worked for me. Thanks a lot. Really appreciate that you took the time!

@ottensjors
Copy link

Hi guys,

I’m experiencing the same issue as described above.

I tried @xitongsys fix that helped out @GordonPman. But unfortunately this didn’t work and I’m left with same issue.

Set up I’m using:

Ubuntu 20.04 LTS
Go 1.15

Could there be an incompatibility issue between parquet-go and Go 1.15?

The go.mod of parquet-go states “go 1.13”.

Please let me know if I need to. Provide any other info.

zolstein pushed a commit to zolstein/parquet-go that referenced this issue Jun 23, 2023
…itongsys#289)

* refactor packages to use encoding.Values container

* refactor page and dictionary creation to use encoding.Values

* go vet fix

* reduce memory footprint of encoding.Values

* refactor encoding.Encoding to use simple Go types

* port parquet-go package to use pair of values+offsets to represent byte arrays

* add fuzz tests back

* optimize DELTA_LENGTH_BYTE_ARRAY decoding (xitongsys#291)

* optimize DELTA_LENGTH_BYTE_ARRAY decoding

* add link to online documentation

* fix

* add a unit test for decodeByteArrayLengths

* Update encoding/delta/length_byte_array_amd64.s

Co-authored-by: Kevin Burke <[email protected]>

* optimize DELTA_LENGTH_BYTE_ARRAY encoding (xitongsys#292)

Co-authored-by: Kevin Burke <[email protected]>

* account for size of offsets buffer when benchmarking throughput

* optimize DELTA_BYTE_ARRAY decoding (xitongsys#294)

* PR feedback

Co-authored-by: Kevin Burke <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants