Make it possible to search filings by the things they cite

[mlissner]

@ss108 just added authority extraction for all of the filings in the RECAP Archive. This feature is really cool, but now we gotta build on it.

The next really cool thing we can and should do is allow filings to be searched by the things they cite. The way this will work, I think, is that we'll enhance the "Cited by" sidebar to have two sub-sections, probably separated by tabs. Here's how it looks now:

I think the trick will be to give it a section for opinions citing to the current one, and a section for filings citing the current one.

At the bottom of the current sidebar, we have a button that says "View Citing Opinions". When you click it, you are taken to the opinion search, where a basic search has been started for you, which you can refine however you choose.

We will need a similar button at the bottom of the filings section, except clicking it will take you to the RECAP Archive.

That'll take care of the UI part of this. Then, on the search side, we'll want to index the all the cited IDs in every filing, and make them searchable, like we do for opinions. This will require a new index, I believe, and a re-index from the DB of every item that cites another item, or maybe we can use the re-index API to use the old index, with a custom job to add the citation data to Elastic (it's only 22M rows, it'd be nice not to have to re-index all 400M for it!).

For now we need to know if a re-index is needed and maybe get these fields in place if so. Later, once Elastic is launched, we can do the rest of this, I think.

---

Open Questions

1. Do we load the filings sidebar section when the page loads even though they aren't visible until you switch to their tab or do we do it when somebody switches to their tab? The latter is better for performance of the page and of our server, the former is better for the user that clicks the tab (less waiting).

2. Do we have a button for citation alerts, so people can get a notification when a new filing cites something? It's a cool (and under-used) feature. But we'll need Add "Create an Alert" for RECAP searches #612 first.

3. Is this complicated enough that it should get something bigger and better than the sidebar? Maybe it's time to do a custom page for all this?

[mlissner]

This just needs analysis and possibly some schema work before we launch, but we don't need to build it until some time later.

[albertisfu]

@mlissner I've analyzed the requirements of this feature and these are my findings and questions.

According to how the Cited by section on Opinions sidebar works.

We're going to add a new section:

Cited by these documents: (RECAPDocuments)

• RECAPDocument example 1 (Link to the RECAPDocument detail page)
• RECAPDocument example 2 (Link to the RECAPDocument detail page)
  ...
  View citing documents (Button pointing to ?type=r&q=cites%3A(current_opinion_pk))

When clicking the View citing documents we're going to take the user to the RECAP Search results page.

Here we will display all the RECAP documents that cite the opinion mentioned in the query: cites:(opinion_pk)

If this is correct, the only change needed in the ES RECAPDocument mapping (for child documents) is the inclusion of a new field cites, which will hold all the Opinion IDs cited by the RECAPDocument based on the OpinionsCitedByRECAPDocument instances.

• One question: do we need to add this new field at the parent document level (cites in DocketDocument)? I think it would only be useful if we want to allow users to search by parties (which are only found in DocketDocument) in combination with the cites field. Otherwise, adding it to the RECAPDocument mapping should be enough, since each RECAPDocument also contains all the Docket fields. This would make it possible to query docket fields (except for parties) along with cites.

• Another question about some additional elements that are present in the current cites search for opinions:
  

• The # References to this case label displays the depth (The number of times the cited opinion was cited in the citing document), this field is also present in the OpinionsCitedByRECAPDocument model.
  Should we also display this label on the RECAP search results that matched the cited opinion? If so we'd also need to add the citation_depth field to the ES RECAPDocument mapping.
  Also, the text label should be # References from this case, since in this case, it means the number of times the RECAPDocument cited the Opinion the user is searching for?

• In the Opinion search results we have the Cited by # opinions link which points to a new search based on the current Opinion ID. I assume this is not required for RECAP search results since RECAPDocuments are not cited by opinions or other RECAPDocuments, correct?

• In brief, it appears we would only need one or two new fields (cites and citation_depth) to be added to the RECAPDocument mapping, assuming we agree on the requirements mentioned above.

But you mentioned:

This will require a new index, I believe
Were you referring to the new fields described above in the current RECAPDocument mapping, or were you suggesting a completely new index to also hold the citations contained in RECAPDocuments, similar to how we handle Parentheticals for Opinions?

If we add just one or two new fields to the RECAPDocument mapping, this will necessitate a mapping update and a re-index. I'll do some tests to see if it's feasible to use the Alias and Re-index APIs to avoid a complete re-indexing process.

[mlissner]

do we need to add this new field at the parent document level (cites in DocketDocument)?

No. Document-level is fine.

Should we also display this label on the RECAP search results that matched the cited opinion?

Nah, seems complicated, we can skip it.

I assume this is not required for RECAP search results since RECAPDocuments are not cited by opinions or other RECAPDocuments, correct?

Correct.

In brief, it appears we would only need one or two new fields (cites and citation_depth) to be added to the RECAPDocument mapping

Great!

Were you referring to the new fields described above in the current RECAPDocument mapping, or were you suggesting a completely new index to also hold the citations contained in RECAPDocuments, similar to how we handle Parentheticals for Opinions?

No, I just meant that we'd need a re-index.

Thank you.

[mlissner]

This is done and I even made a youtube video about it (on the FLP channel).