-
Notifications
You must be signed in to change notification settings - Fork 490
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Search API: document how the dateSort field support date range queries (i.e. datasets published in the last week) #2291
Comments
@pdurbin would it be possible for you to add this information into the Search API guide? I have a feeling more and more people will want to be able to search on this to only retrieve the freshest published data for their integrations with us. |
…n -- e.g. why is it not going through DataFileServiceBean.save
We seem to be surviving just fine without this documentation. Closing! 😄 |
I'm working on this exact task ("show me datasets published in the last week using the Search API") with @erikbuunk as part of IQSS/dataverse-metrics#33 so I'm re-opening this because I'd might as well document this in the API Guide while I'm at it. |
I just made pull request #6264 |
document how to search date ranges via API #2291
@erinspace from https://osf.io/share/ asked "Is there a way we can query for all results within a certain date range? Ideally, at least at day granularity? For example, we’d love to be able to query for all results between 2015-05-01 and 2015-05-10."
A similar question came up from the Dataverse Community Meeting 2015. I'm pretty sure in came up in the context of the upcoming Archivematica integration as well.
Anyway, most of an email I sent to people at @CenterForOpenScience as an answer. It seems to be working for their use case so I should probably document this as part of the Search API.
Related is @markwilkinson asking about which fields can be searched. This "dateSort" field is not driven by dynamic metadata. Rather, it's hard coded in https://github.com/IQSS/dataverse/blob/4.0/src/main/java/edu/harvard/iq/dataverse/search/SearchFields.java#L155 but I don't have any plans to change it. Data driven fields (from metadata blocks) are currently only exposed via an API that's blocked from the outside. Making that API public should probably be a separate ticket and is related to #1510.
The email to @CenterForOpenScience
I don't document the "dateSort" field at http://guides.dataverse.org/en/latest/api/search.html but it's what we us internally to mean "create date or published date" (we use published date when it's available) but it's searchable, a real* date type (so we can do range queries on it), and what we use for sorting on the home page (we default to "newest first").
There's granularity down to seconds at least, from what I can tell. See also https://cwiki.apache.org/confluence/display/solr/Working+with+Dates
Others have asked for something similar so we should probably make a ticket about this... make range queries over dates a supported and documented feature of the Search API.
I kind of assume only datasets are of interest (not files or dataverses) but you can include all types. Note that depending on the number of results, you might need to iterate with the "start" cursor per the API Guide:
curl -s --globoff "https://dataverse.harvard.edu/api/search?key=$API_TOKEN&type=dataset&sort=date&order=asc&q=*&fq=dateSort:[2015-05-01T00\:00\:00Z+TO+2015-05-10T00\:00\:00Z]" | jq '.data.items[] | {name,published_at}' | head -12
Related: #70 and #370
The text was updated successfully, but these errors were encountered: