Digitization Folk Fest Ingest #2011
Comments
|
– There is a spreadsheet somewhere that maps the folkfest stuff to Swift IDs. I'll try to chase that down, or Sarah knows the details. – We talked a bit about how we'll want to deal with Preservation/AIPs, and previous archival efforts, and the cheap interim solution I landed on was that we'll just take the tarred (or zipped or whatever the package is) copy of whatever we pull out of Swift, extend the book (and future) models w/ – We're just looking to attach one big high quality PDF of the work for things like this and newspapers, not per-page images, and I'm unsure if that's what we'll get out of Swift or if this will require some preprocessing. – The fulltext is in a metadata file whose name I forget right now. We'll have to find a parser for it if one exists, and figure out how to store the data – I'm thinking as an attachment if the parsing isn't incredibly complex, or in a text field on a not-normally-loaded join table if we need to heavily pre-process the data upfront and keep the indexable copy somewhere we can get at it easily? The reason not to store the full text on the table itself is that that will make reading the row back from the DB for search results and general metadata viewing way to costly – we'll only ever need the fulltext data for (re-)indexing (I guess this may argue for the pre-process into join table approach bc we'll need to reindex that data on save? Thinking ahead, we don't want to have to pull the text file from the Cloud just to re-save a file, so likely we want that in the DB somewhere, in a cheap-to-reindex form) |
|
|
Thanks! |
|
FYI: Here is a Google Drive folder with all the Folk Fest items from SWIFT - it's big at 11 GB but we pulled everything so you could see what's there. Have fun with the files! |
…fest_ingest ## Context The first set of digitization content will be the Folk Fest Programs. This set was chosen because it is small (there are 36) and well described. This adds attributes to store the artifacts that we'll need - in how we'll want to deal with Preservation/AIPs, and previous archival efforts. The historical archive will never be loaded or made available for UI purposes and we'll have no reason to ever want to load it, so the attachment's only purpose is, in the future, somebody can find the old copy of the data. - fulltext in an easy to process format. We want this in a join table so that it doesn't penalize us when we frequently request the object but close enough that we can re-index as required. Related to #2011 ## What's New - Add `has_one_attached :historical_archive` to Book Model which takes the full archive for all directories relating to that object and archive it in case it's needed. Will not appear in the UI - `fulltext` will be extracted from the ALTO xml. Should be easy to access in a join table but not the main object table so that results stay efficient. - Attach one high quality PDF
@sarahseverson I made an assumption that this was exactly how the material was in Swift: In directories with the noid label. But there's some indication in Matt's note that the content might have come compressed
What format was it in? I've assumed that we should store it as |
|
Peel preservation examples was prepared by @kgood to show some of the differences in structures in Swift. |
|
@kgood @sarahseverson @henryzhang87 and I met today. We discussed some questions I had:
Yes, though there might be some differences; newspapers might be the biggest exception.
We came up with three ideas
ACTION: Henry will set Tricia up on geoffy with access to the OpenStack Swift API My hope is that we will be able to do # 3 when this gets rolling
Swift downloads directories without compression exactly as Sarah/Tianyu provided above noid/type [tiff/jpeg/etc]/1.tar Some swift directories won't have pdfs: like the Images will be tiff or jpeg
ACTION: Kenton will follow up with Tricia about this. |
|
re: 1 - here is an example of a spreadsheet for a Newspaper upload to IA where we have a NOID, code (three letters) and some individual item-level metadata (year, month, day, pages) |
|
We had further discussion about the
Next steps: |
|
Based on discussion with @kgood @pgwillia @sarahseverson Recording the metadata aspect of the discussion here so that it is captured and can then be incorporated into our AIP specifications for PMPY. The new AIP generated on ingest will include in the metadata a link to the original (i.e., legacy) AIP in the form of a PREMIS relationship type property. The metadata should be: relSubType:sup [PID of legacy AIP], where the PID is likely to be UUID (as this is how we are naming AIPs) and, for reference, the full URI for the property is https://id.loc.gov/vocabulary/preservation/relationshipSubType/sup (which indicates that the new AIP supersedes the original or legacy AIP). We also discussed the desire for recording the container in which the original or legacy AIP is located, which can also be captured through a metadata property: relators:rps [name of storage container], where the name of the storage container could be something like OpenStack/Swift, and the full URI for the property is http://id.loc.gov/vocabulary/relators/rps (which indicates that it is a repository for the object) |
Is your feature request related to a problem? Please describe.
The first set of digitization content will be the Folk Fest Programs. This set was chosen because it is small (there are 36) and well described.
AddRemovehas_one_attached :historical_archiveto Book Model which takes the full archive for all directories relating to that object and archive it in case it's needed. Will not appear in the UIhas_one_attached :historical_archivefrom Book Model and replace with metadata attribute relators:rps (http://id.loc.gov/vocabulary/relators/rps) to link to the complete AIP stored elsewhere.Initial ingest assumes that we'll push the content (triples csv, pdf, alto) to the server for ingest from a rake script.
Follows #2010
The text was updated successfully, but these errors were encountered: