Make sure retired and redirected URLs don't show up in global search results

[dorothyyeager]

Summary

What we are after:
Users need the global search to direct them to current pages. Some retired pages that we've unpublished and redirected are turning up in global search results. We need to make sure only active pages show up in those results, and unpublished pages that we haven't deleted don't.

Related issues

Other search improvement tickets

• Research better ways of not forgetting major sections to index #3287 - Figure out how not to exclude website sections from search indexing.
• Make plan for scheduling regular updating of global search #3283 - Make plan for scheduling future search indexing

Most recent search.gov indexing PR

• Updates for global site search #3278: Updates for global site search

Background

Some H4CC and other pages are active for a time, but then unpublished and redirected to more current content.

For example, the now-unpublished page with a URL at https://www.fec.gov/help-candidates-and-committees/making-disbursements-pac/coordinated-communications-nonconnected-pac/ (8740 )redirects to our main page for coordinated communications at https://www.fec.gov/help-candidates-and-committees/candidate-taking-receipts/coordinated-communications/. (4371)

The older page shows up in search results though (see highlighted):

Similar thing for https://www.fec.gov/help-candidates-and-committees/making-disbursements-political-party/contributions-made-by-party-committees/ (8152) which was unpublished and redirected to https://www.fec.gov/help-candidates-and-committees/making-disbursements-political-party/contributions-made-party-committees/ (10479), but older page is still turning up in results.

Completion criteria

• make list of URLS that need to be removed (all of them have been unpublished and now redirect to a more current page):

• https://www.fec.gov/help-candidates-and-committees/candidate-taking-receipts/contribution-limits-candidates/ (6077)

• https://www.fec.gov/help-candidates-and-committees/candidate-taking-receipts/who-can-and-cannot-contribute/ (6075)

• https://www.fec.gov/help-candidates-and-committees/making-disbursements-political-party/contributions-made-by-party-committees/ (8152)

• https://www.fec.gov/help-candidates-and-committees/taking-receipts-pac/contributions-nonconnected-pacs-limited-liability-companies-llcs/ (8694)

• https://www.fec.gov/help-candidates-and-committees/taking-receipts-pac/contributions-nonconnected-pacs-partnerships/ (8688)

• https://www.fec.gov/help-candidates-and-committees/making-disbursements-pac/coordinated-communications-nonconnected-pac/ (8740)

• https://www.fec.gov/help-candidates-and-committees/making-disbursements-pac/independent-expenditures-nonconnected-pac/ (8743)

• https://www.fec.gov/help-candidates-and-committees/making-disbursements-pac/notices-required-nonconnected-pac-solicitations-nonconnected-pac/ (THIS PAGE WAS DELETED) (20)

• https://www.fec.gov/help-candidates-and-committees/filing-pac-reports/nonconnected-committee-independent-expenditures/ (84)

• https://www.fec.gov/help-candidates-and-committees/taking-receipts-ssf/contributions-by-partnerships-and-llcs-to-ssfs (9420)

• https://www.fec.gov/help-candidates-and-committees/making-disbursements-ssf-or-connected-organization/making-electioneering-communications (9509)

• https://www.fec.gov/help-candidates-and-committees/making-disbursements-ssf-or-connected-organization/making-independent-expenditures-ssf-corporation-labor-organization (9491)

• https://www.fec.gov/help-candidates-and-committees/keeping-pac-records-nonconnected/misappropriated-nonconnected-pac-funds/ (THIS PAGE WAS DELETED ) (89)

• https://www.fec.gov/help-candidates-and-committees/filing-ssf-reports/independent-expenditures/ (48)

• https://www.fec.gov/press/resources-journalists/presidential-public-funding/ (113)

• manually remove the URLs from global search

• confirm that these URLs do not show up on our site's global search.

Future work

-Consider an archival policy for pages that have been retired. Would changing the URL (by adding "retired" or "old" or "archived-older-page" after we retire a page avoid this issue?
-Make sure we put "removing pages from global search" into future tickets for similar work when we switch a page to a different template.

[patphongs]

Content team can take a look at transition pages that are being indexed here: https://github.com/fecgov/fec-cms/blob/master/fec/search/management/data/transition_pages.json

Please look at what pages we can delete, keep, and add.

[dorothyyeager]

I've updated the ticket with the list of URLs that are showing up in search even though retired and redirected. cc @bmathesonFEC @patphongs

[johnnyporkchops]

@dorothyyeager , I filled in page IDs for each page with the exception of a few. @dorothyyeager Can you help me locate the two pages unchecked above?

[dorothyyeager]

Hi @johnnyporkchops - I looked in Wagtail and will confirm that it appears neither of those two pages exists anymore. They were both redirected ages ago and I suspect they were deleted. This is maybe a good learning lesson though. Seems like it's better just to unpublish a page and not take the extra step of deleting it?

[johnnyporkchops]

@thanks @dorothyyeager!

[johnnyporkchops]

@dorothyyeager These have all been removed from search.gov index and do not seem to be showing up in search. Can you please test as well.

[dorothyyeager]

Will do. When I tested a couple, the contribution limit one still comes up for me as a suggestion, so want to make sure I give these a good thorough test.

[johnnyporkchops]

@patphongs , I noticed that we have advanced in the web_app_json file

{
    "document_id": "app-2",
    "title": "Advanced data",
    "path": "https://www.fec.gov/data/advanced/",
    "created": "2017-04-01",
    "language": "en",
    "promote": true,
    "description": "Search, filter and download data",
    "tags": "raising, spending, candidates, committees, contributions, advanced data, campaign finance data, disclosure"
  },

We should change this to browse-data .

[dorothyyeager]

Seeing this in results when I search "contribution limits" (it's one of the pages in the list above). Is it possible that this won't take until the next time global search is manually indexed? @johnnyporkchops @patphongs

[dorothyyeager]

I also am still seeing the two pages that were completely deleted. Redirects are working in all of these cases. For the screencap above, I wonder if that "suggested" thing is what is triggering that page to still show up?

For the rest of them, I don't see them in the search anymore. Thanks for all your work on this @johnnyporkchops. If you need me to put in a ticket on the weird things (the two deleted pages and the suggested page), let me know.

[johnnyporkchops]

The IDs of the two deleted pages mentioned above are 20 and 89. Attempts to delete them from the index from the command line return "Resource not found", so I am not sure how we can remove them from search.

[dorothyyeager]

@johnnyporkchops Is there any way to take the "suggested" off the contribution limits page? That will at least reduce the visibility of it in the search results. The other page is not really one that a lot of people look for. In both case, redirects work so clicking on the wrong result still ends up getting you to the right page.

In the meantime, we'll make sure the content team knows that deleting retired pages is not a good thing to do.

[dorothyyeager]

I've put in tickets to resolve the issues identified with three pages.

#3524 - Figuring out how to remove deleted pages out of global search

#3523 - Changing the suggested result for "contribution limits" to the current page rather than a deleted one.

The results have been fixed for the other pages listed, so we can close this ticket and move on to the next one in 11.6.