Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Offer content type JSON Lines and format gzip #91

Closed
acka47 opened this issue Apr 23, 2018 · 8 comments · Fixed by #126
Closed

Offer content type JSON Lines and format gzip #91

acka47 opened this issue Apr 23, 2018 · 8 comments · Fixed by #126

Comments

@acka47
Copy link
Contributor

acka47 commented Apr 23, 2018

As in lobid-resources, see http://lobid.org/resources/api#content_types
@acka47
Copy link
Contributor Author

acka47 commented Apr 24, 2018

And also support format gzip via content header.

@acka47 acka47 changed the title Offer content type JSON Lines Offer content type JSON Lines and format gzip Apr 24, 2018
@fsteeg fsteeg added the ready label May 14, 2018
@fsteeg fsteeg self-assigned this May 14, 2018
@fsteeg fsteeg added working and removed ready labels Jun 13, 2018
fsteeg added a commit that referenced this issue Jun 19, 2018
@fsteeg
Generalize RDF response case in controller
db4b465 
See #91
fsteeg added a commit that referenced this issue Jun 19, 2018
@fsteeg
Support bulk requests via format parameter and content negotiation
8bec747 
See #91
fsteeg added a commit that referenced this issue Jun 19, 2018
@fsteeg
Support bulk requests via format parameter and content negotiation
93e1ec4 
See #91
fsteeg added a commit that referenced this issue Jun 19, 2018
@fsteeg
Support bulk requests via format parameter and content negotiation
b52479f 
See #91
fsteeg added a commit that referenced this issue Jun 19, 2018
@fsteeg
Create source using streams and Source.from, not Source.actorRef
f640e08 
Details:

https://www.playframework.com/documentation/2.6.x/JavaStream
https://doc.akka.io/docs/akka/current/stream/stream-flows-and-basics.html
https://doc.akka.io/japi/akka/current/akka/stream/javadsl/Source.html

See: #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Create source with custom Iterator to avoid memory issues
01a4faf 
See #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Add Content-Disposition header for bulk request download in browser
ad14314 
See #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Add bulk requests to API documentation
eb1b4f6 
See #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Generalize RDF response case in controller
25b4124 
See #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Support bulk requests via format parameter and content negotiation
355626f 
See #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Create source using streams and Source.from, not Source.actorRef
0ae8c51 
Details:

https://www.playframework.com/documentation/2.6.x/JavaStream
https://doc.akka.io/docs/akka/current/stream/stream-flows-and-basics.html
https://doc.akka.io/japi/akka/current/akka/stream/javadsl/Source.html

See: #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Create source with custom Iterator to avoid memory issues
fff8268 
See #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Add Content-Disposition header for bulk request download in browser
9251d5b 
See #91
fsteeg added a commit that referenced this issue Jun 20, 2018
@fsteeg
Add bulk requests to API documentation
6f3a158 
See #91
@fsteeg fsteeg mentioned this issue Jun 20, 2018
Merged
@fsteeg fsteeg added review and removed working labels Jun 20, 2018
@fsteeg
Copy link
Member

fsteeg commented Jun 20, 2018

Deployed to stage, see:

http://stage.lobid.org/gnd/search?q=ehrenfeld&format=bulk

Tested uncompressed request for all corporate bodies:

curl "http://stage.lobid.org/gnd/search?q=type:CorporateBody&format=bulk" > bulk.jsonl

This yields a 1.7 GB file. Took about:

  • 1:30 minutes on the same machine
  • 2:30 minutes on our local network
  • 6:45 minutes on our Eduroam WLAN

Tested same request, but compressed (handled by the Apache proxy):

curl --header "Accept-Encoding: gzip" "http://stage.lobid.org/gnd/search?q=type:CorporateBody&format=bulk" > bulk.gz

This yields a 174 MB file. Took about:

  • 1:30 minutes on the same machine
  • 1:30 minutes on our local network
  • 1:30 minutes on our Eduroam WLAN

See also documentation: http://stage.lobid.org/gnd/api#content_types

@fsteeg fsteeg assigned acka47 and unassigned fsteeg Jun 20, 2018
@acka47
Copy link
Contributor Author

acka47 commented Jun 21, 2018

You should also be able to get jsonlines and gzip for a filter query. I tried curl --header "Accept: application/x-jsonlines" "http://stage.lobid.org/gnd/search?filter=%2B%28type%3APlaceOrGeographicName%29" > geographika.jsonl and am currently pulling the whole GND.

Furthermore, as we discussed online we will use jsonl instead of bulk (and adjust this in lobid-resources at a later point as well).

@acka47
Copy link
Contributor Author

acka47 commented Jun 21, 2018

Downloading the whole GND as gzip (1,5 GB, unzipped 14 GB) took just 13 minutes. So, this definitely works like a charm.

@acka47 acka47 assigned fsteeg and unassigned acka47 Jun 21, 2018
fsteeg added a commit that referenced this issue Jun 22, 2018
@fsteeg
Use jsonl as format value for bulk requests
11dd61d 
For consistency with `html`, `json`, etc.

See #91
fsteeg added a commit that referenced this issue Jun 22, 2018
@fsteeg
Support filter param in bulk requests
81b7f48 
See #91
@fsteeg
Copy link
Member

fsteeg commented Jun 22, 2018

Deployed to stage:

http://stage.lobid.org/gnd/search?filter=ehrenfeld&format=jsonl
http://stage.lobid.org/gnd/api#content_types

@fsteeg fsteeg assigned acka47 and unassigned fsteeg Jun 22, 2018
@acka47
Copy link
Contributor Author

acka47 commented Jun 22, 2018

Looks good. One minor problem is left, though, which actually was there before this ticket. http://lobid.org/gnd/4074335-4.jsonl also gives back JSON and not JSON lines as doest http://lobid.org/gnd/4074335-4.jsonfoo. We should only allow a colon : with more to follow after .json.

fsteeg added a commit that referenced this issue Jun 22, 2018
@fsteeg
Serve HTTP 415 responses for unsupported media types
0e2dd40 
Don't fall back to JSON if unsupported format was requested

See #91
@fsteeg
Copy link
Member

fsteeg commented Jun 22, 2018

Deployed to stage.

Unsupported:

http://stage.lobid.org/gnd/4074335-4.jsonl
http://stage.lobid.org/gnd/4074335-4.jsonfoo
http://stage.lobid.org/gnd/4074335-4.json:suggest
http://stage.lobid.org/gnd/search?format=jsonfoo

Supported:

http://stage.lobid.org/gnd/search?format=json:suggest
http://stage.lobid.org/gnd/search?format=jsonl

@acka47
Copy link
Contributor Author

acka47 commented Jun 22, 2018

+1

@acka47 acka47 removed their assignment Jun 22, 2018
@dr0i dr0i added deploy and removed review labels Jun 26, 2018
fsteeg added a commit that referenced this issue Jun 27, 2018
@fsteeg
Merge branch '91-bulk' of https://github.com/hbz/lobid-gnd
1ee4a6d 
Resolves #91

See http://lobid.org/gnd/api#content_types
@dr0i dr0i removed the deploy label Jun 27, 2018
@fsteeg fsteeg mentioned this issue Jun 27, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support bulk requests hbz/lobid-gnd
3 participants
@fsteeg @acka47 @dr0i