Create automated sitemap.xml generation for policy/guidance pdfs

[jason-upchurch]

@jason-upchurch commented on Fri Feb 28 2020

Summary

From #3554, we need to build automated sitemap generation based on existing/added/deleted content and create update policy/schedule, e.g., once per day to be consistent with search.gov

No longer blocked: This ticket is presently blocked by fecgov/openFEC#3580, which creates the strategy for dealing with policy/guidance documents in an unambiguous way to enable automatic sitemap generation (which will enable automatic index updating).

When working on this ticket, we need to consider: #3280

Completion criteria

• sitemap.xml is generated correctly based on established convention/strategy
  • current strategy is to provide a hard-coded list of resources that need to be searchable
• search.gov uses domains we specify (e.g., change proxy url to "correct one")
• provide search.gov url from github so that they update sitemap from repo
• research robots.txt and indexing multiple sitemaps
• research domain partitioning such that global search does not pollute guidance search
• research options for a "highly automated process" for generating sitemaps such that people will not need to maintain them by hand

Technical considerations

• ATO
• AWS/cloud.gov v server
• We appear to house all of our scheduled tasks in openFEC repo, thus transferred this issue to openFEC and will use existing framework

---

@jason-upchurch commented on Tue Mar 03 2020

Related issue opened with cloud.gov to consider AWS Lambda or Scheduled Task: cloud-gov/s3-broker#38

---

@jason-upchurch commented on Tue Mar 03 2020

@lbeaufort advised in sprint planning to use caution with respect to this work as it pertains to ATO.

cc @wjiangFEC @PaulClark2

---

@jason-upchurch commented on Tue Mar 03 2020

requesting ATO for review. Issue open with cloud.gov to check feasibility of deploying cloud.gov only solution. Investigating contingency plans, i.e., server based, in the event cloud.gov is unable to support application.

[JonellaCulmer]

Duplicate ticket, so closing in favor of #3804