Implement prototype search for guidance policy

[jason-upchurch]

Summary

This ticket is a prototype implementation ticket to continue work done under issue #3489, #3488, #3527. The main focus of this ticket is to test searchability of a handful of pdfs using a limited scope and functionality as a proof-of-concept/prototype. Follow-on work/tickets is/are identified in the high-level completion criteria below.

High-level completion criteria

• identify some (e.g., 5-10) native pdfs from content-s3 that should be made searchable (content work, cc @dorothyyeager @kathycarothers )
  • technical considerations: should these be ones already uploaded through wagtail? Is it easiest just to identify and not necessarily deal with organizational questions for prototype, e.g., file system layout, name mapping/convention, etc?
• provide sitemap to search.gov (back-end cms: cc @jason-upchurch )
  • technical considerations: what tags, metadata
• use search.gov platform as initial platform for returning search results (unsure, possibly coordination between design, back-end cms, front-end cms: cc @JonellaCulmer @patphongs @johnnyporkchops @jason-upchurch )
  • technical considerations: does the platform sufficiently allow user feedback on functionality or search expectations?
• open additional tickets as needed. Some examples (not necessarily in sequential order):
  • creating organization scheme and process flow for uploading documents intended to be searched to wagtail/s3 directly (possibly combined effort by content and cms back end cc @dorothyyeager @kathycarothers @johnnyporkchops @patphongs @jason-upchurch )
  • building automated sitemap generation based on existing/added/deleted content and create update policy/schedule, e.g., once per day to be consistent with search.gov (back-end cms: cc @jason-upchurch )
  • front-end cms ticket to build search box/functionality to policy/guidance page (front-end cms work: cc @patphongs @johnnyporkchops @jason-upchurch )
  • front-end design ticket to generate requirements for search box and overall search functionality (design work with coordination with cms front-end cc @JonellaCulmer @patphongs @johnnyporkchops @jason-upchurch )
  • possible ticket to transition or plan transition of i14y search architecture to the sitemap.xml architecture (larger scope and resource question: @PaulClark2 @AmyKort @patphongs @rfultz @johnnyporkchops @jason-upchurch )
  • others

[dorothyyeager]

Here's the list of some URLs already in Wagtail that will be needed for the guidance policy page and could be used for testing:

https://www.fec.gov/resources/cms-content/documents/fedreg_notice_2019-10_07292019.pdf
https://www.fec.gov/resources/cms-content/documents/candgui.pdf
https://www.fec.gov/resources/cms-content/documents/guideline-for-presentation-good-order.pdf
https://www.fec.gov/resources/cms-content/documents/fecfrm1m.pdf
https://www.fec.gov/resources/cms-content/documents/fecfrm1mi.pdf
https://www.fec.gov/resources/cms-content/documents/fecfrm5.pdf
https://www.fec.gov/resources/cms-content/documents/fedreg_notice2003-9.pdf
https://www.fec.gov/resources/cms-content/documents/enforcementprocedures_hearingtranscript-6-11-2003.pdf
https://www.fec.gov/resources/cms-content/documents/comment_democracy21_05222019.pdf
https://www.fec.gov/resources/cms-content/documents/comment_campaign_legal_center_05232019.pdf

Not PDFs, but if we want to test the press releases:
https://www.fec.gov/updates/fec-statement-on-carey-fec/
https://www.fec.gov/updates/fec-provides-guidance-following-us-district-court-decision-crew-v-fec-316-f-supp-3d-349-ddc-2018/

[dorothyyeager]

With the list above and a related spreadsheet shared with @jason-upchurch, I'm passing this issue on to him to work on the rest! I'm available for any content-type questions that might come up. cc: @JonellaCulmer

[jason-upchurch]

@dorothyyeager @JonellaCulmer thank you--I have issue now. Following are the locations where the pdfs @dorothyyeager identified live:

• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fecfrm1m.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fedreg_notice_2019-10_07292019.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/candgui.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/guideline-for-presentation-good-order.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fecfrm1mi.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fecfrm5.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fedreg_notice2003-9.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/enforcementprocedures_hearingtranscript-6-11-2003.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/comment_democracy21_05222019.pdf
• https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/comment_campaign_legal_center_05232019.pdf

For the time being, I can limit scope to pdfs. I'm not quite sure the press release directories given above map to the s3 bucket at all (for example, there is no statement-on-carey-fec "directory" in the bucket, so I'm not sure the underlying document type/location and may be out of scope for the prototype (that is, I don't think that the preceding research ticket investigated anything beyond making pdfs in s3 searchable).

[jason-upchurch]

tentative xml sitemap. Sent to search.gov and awaiting next steps to begin search testing.

<?xml version="1.0" encoding="UTF-8"?>
<urlset>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fecfrm1m.pdf</loc>
<lastmod>2017-08-17T09:25:48+00:00</lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fedreg_notice_2019-10_07292019.pdf</loc>
<lastmod>2019-09-17T08:00:33+00:00</lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/candgui.pdf</loc>
<lastmod>2018-10-03T11:07:29+00:00</lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/guideline-for-presentation-good-order.pdf</loc>
<lastmod>2020-02-05T13:12:21+00:00</lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fecfrm1mi.pdf</loc>
<lastmod>2018-09-21T14:50:43+00:00</lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fecfrm5.pdf</loc>
<lastmod>2018-03-16T10:58:46+00:00</lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/fedreg_notice2003-9.pdf</loc>
<lastmod></lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/enforcementprocedures_hearingtranscript-6-11-2003.pdf</loc>
<lastmod></lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/comment_democracy21_05222019.pdf</loc>
<lastmod></lastmod>
</url>
<url>
<loc>https://cg-47928592-406c-4536-8234-99b896e8d57d.s3-us-gov-west-1.amazonaws.com/cms-content/documents/comment_campaign_legal_center_05232019.pdf</loc>
<lastmod></lastmod>
</url>
</urlset>

[jason-upchurch]

For posterity: search.gov attempted to index based on above but hit 403 error. We asked cloud.gov to whitelist search.gov's IP address. This was done and we are awaiting the indexing to test success/failure.

coordinating with @dorothyyeager @AmyKort to plan for next steps per @PaulClark2 cc @pkfec

[dorothyyeager]

sitemap ticket that @rfultz started: #3280

[dorothyyeager]

robots ticket from @rfultz that could play into this work: #2849

[jason-upchurch]

Prototype complete and functional. Continuing feature work under new issue(s), e.g., #3580 and #3581