[WIP] Test sitemap on dev

[johnnyporkchops]

Summary (required)

Test branch for use with test sitemaps sites on search.gov: fec_content_s3_dev (fec_content_s3_dev) and DEV.FEC.GOV(copy of WWW) (dev-fec.gov)

• Resolves #issue_number Develop www.fec.gov sitemap #5522

(Include a summary of proposed changes and connect issue below)

Required reviewers

(Include who is required to review prior to merge. For example: One designer and two front end developer reviews are required prior to merge)

Impacted areas of the application

General components of the application that this PR will affect:

Screenshots

(Include a screenshot of the new/updated features in context (“in the wild”). If it is an interface change, include both before and after screenshots)

Related PRs

Related PRs against other branches:

branch	PR
fix/other_pr	link
feature/other_pr	link

How to test

(Include any information that may be helpful to the reviewer(s). This might include links to sample pages to test or any local environmental setup that is unusual such as environment variable (never credentials), API version to point to, etc)

System architecture updates (if applicable)

(If this pull request changes our current system diagram, include a description of those changes here and create a new ticket to update the system diagram)

[codecov]

Codecov Report

Merging #5735 (15aa918) into develop (381911f) will decrease coverage by 0.02%.
Report is 56 commits behind head on develop.
The diff coverage is 76.92%.

@@             Coverage Diff             @@
##           develop    #5735      +/-   ##
===========================================
- Coverage    78.22%   78.20%   -0.02%     
===========================================
  Files          285      285              
  Lines         9307     9302       -5     
  Branches       650      650              
===========================================
- Hits          7280     7275       -5     
  Misses        2027     2027              

Files Changed	Coverage Δ
fec/home/wagtail_hooks.py	100.00% <ø> (+4.16%)	⬆️
fec/fec/settings/base.py	88.57% <33.33%> (-1.63%)	⬇️
fec/search/views.py	71.26% <50.00%> (ø)
fec/fec/tests/test_robots.py	95.65% <100.00%> (+0.19%)	⬆️
fec/fec/urls.py	80.76% <100.00%> (+0.76%)	⬆️
fec/search/tests/test_views.py	100.00% <100.00%> (ø)

[johnnyporkchops]

When testing the wagtail-generated sitemap on dev, there are non-existent children pages listed in the since-deleted section,
registration-and-reporting . This section was replaced by help-for-candidates-and-committees and is redirected there. These are pages created by 18f in the very early stages of the site. With ids like 6, 7, and 16, they were among the first pages created.
Wagtail was in version 1 then, so it have been caused by a since-resolved bug in Wagtail where deleting a parent did not properly remove its children from the database. Just a theory.

Content team confirmed in slack thread that these can safely be removed. These pages have been removed from dev and production databases

To remove these pages from sitemap, we will need to remove them from the home_custompage(or whatever type it is) table and then the wagtailccore_pagetables. To avoid migration error that come up later when foreign key constraints hang around in the revisions table after their parental keys have been removed everywhere else., we'll remove their references from wagtailcore_revisions tables too. Make sure to remove based on pk (page id).

Used this command to get the id of the page and the type of page from the URL (30 is CustomPage)

SELECT id, content_type_id
FROM wagtailcore_page
where url_path = '/home/registration-and-reporting/ssf-filing-requirements/';

id | content_type_id 
----+-----------------
44 |              30

Below is a list of the pages to be removed . ID, url and lastmod date of each::
Remove, in this order, from home_custompage --then -- wagtailcore_page --then -- wagtailcore_revisions based on its page ID. Can do bulk removal statements like examples below:

DELETE
FROM home_custompage
WHERE page_ptr_id in (44,66,69,80,88);


DELETE
FROM wagtailcore_page
WHERE id in (44,66,69,80,88);

DELETE
FROM wagtailcore_revision
WHERE content ->> 'pk' in ('44','66','69','80','88');

• ID: 6
  https://dev.fec.gov/registration-and-reporting/essentials-house-and-senate-candidates-and-committees/
  2015-10-27

• ID: 7
  https://dev.fec.gov/registration-and-reporting/get-e-filing-password/
  2016-05-27

• ID: 12
  https://dev.fec.gov/registration-and-reporting/administrative-fines/
  2017-05-17

• ID: 16
  https://dev.fec.gov/registration-and-reporting/know-your-filing-requirements/
  2016-08-29

• ID: 21
  https://dev.fec.gov/registration-and-reporting/quarterly-reports/
  2017-05-18

• ID: 22
  https://dev.fec.gov/registration-and-reporting/pre-election-reports/
  2017-05-18

• ID: 23
  https://dev.fec.gov/registration-and-reporting/48-hour-notices/
  2017-05-18

• ID: 25
  https://dev.fec.gov/registration-and-reporting/request-additional-information/
  2017-05-18

• ID: 26
  https://dev.fec.gov/registration-and-reporting/missed-deadline/
  2017-04-07

• ID: 28
  https://dev.fec.gov/registration-and-reporting/post-general-election-reports/
  2017-05-18

• ID: 29
  https://dev.fec.gov/registration-and-reporting/register-now/
  2016-02-02

• ID: 42
  https://dev.fec.gov/registration-and-reporting/ssf-registration/
  2017-04-03

• ID: 44
  https://dev.fec.gov/registration-and-reporting/ssf-filing-requirements/
  2016-11-04

• ID: 66, Page type : 1
  https://dev.fec.gov/registration-and-reporting/essentials-corporations-and-labor-organizations/
  2016-03-31

• ID:69
  https://dev.fec.gov/registration-and-reporting/party-committee-filing-requirements/
  2016-08-29

• ID: 80
  https://dev.fec.gov/registration-and-reporting/nonconnected-committee-filing-requirements/
  2016-11-21

• ID: 88
  https://dev.fec.gov/registration-and-reporting/terminating-a-committee/
  2017-04-03

[johnnyporkchops]

Closing, testing is complete