CompareReferences tool #7930

orlicohen · 2022-07-07T20:17:08Z

New tool to compare references.
(Linked to #6837)

codecov · 2022-07-07T21:44:15Z

Codecov Report

Merging #7930 (a736923) into master (804d1e2) will decrease coverage by 0.017%.
The diff coverage is 83.402%.

@@               Coverage Diff               @@
##              master     #7930       +/-   ##
===============================================
- Coverage     87.078%   87.062%   -0.017%     
- Complexity     36997     37007       +10     
===============================================
  Files           2217      2218        +1     
  Lines         173849    173758       -91     
  Branches       18791     18769       -22     
===============================================
- Hits          151385    151277      -108     
- Misses         15832     15896       +64     
+ Partials        6632      6585       -47

Impacted Files	Coverage Δ
.../hellbender/tools/reference/CompareReferences.java	`66.667% <66.667%> (ø)`
...ute/hellbender/utils/reference/ReferenceUtils.java	`73.529% <66.667%> (-5.418%)`	⬇️
...ls/reference/CompareReferencesIntegrationTest.java	`77.465% <77.465%> (ø)`
...bender/tools/reference/ReferenceSequenceTable.java	`85.311% <85.311%> (ø)`
...tute/hellbender/tools/reference/ReferencePair.java	`86.486% <86.486%> (ø)`
...ools/reference/ReferenceSequenceTableUnitTest.java	`97.872% <97.872%> (ø)`
...bender/utils/reference/ReferenceUtilsUnitTest.java	`75.556% <100.000%> (+6.984%)`	⬆️
...er/tools/spark/sv/evidence/BreakpointEvidence.java	`66.822% <0.000%> (-11.916%)`	⬇️
...on/FindBreakpointEvidenceSparkIntegrationTest.java	`96.296% <0.000%> (-3.704%)`	⬇️
...rg/broadinstitute/hellbender/utils/SVInterval.java	`90.000% <0.000%> (-2.857%)`	⬇️
... and 31 more

droazen · 2022-07-18T18:18:03Z

@orlicohen It looks like you accidentally rebased a bunch of unrelated commits -- we'll have to repair the branch before we can review

… featute

…sequence present

jamesemery

Looks good overall. A few bigger comments and mostly a few minor comments and requests for some more documentation.

jamesemery · 2022-07-18T19:16:23Z

src/main/java/org/broadinstitute/hellbender/engine/GATKTool.java

@@ -697,6 +697,10 @@ public final <T extends Feature> Object getHeaderForFeatures( final FeatureInput
        return hasFeatures() ? features.getHeader(featureDescriptor) : null;
    }

+    public final GATKPath getReferencePath(){


Public methods in GATKTool definitely need javadocs explaining themselves.

jamesemery · 2022-07-18T19:16:37Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+ *
+ */
+@CommandLineProgramProperties(
+        summary = "",


Fill these summaries in.

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

jamesemery · 2022-07-18T19:18:22Z

src/main/java/org/broadinstitute/hellbender/utils/SequenceDictionaryUtils.java

@@ -469,6 +469,7 @@ public static Set<String> getContigNames(SAMSequenceDictionary dict) {
        for (SAMSequenceRecord dictionaryEntry : dict.getSequences())
            contigNames.add(dictionaryEntry.getSequenceName());
        return contigNames;
+


accidental, remove

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferencePair.java

jamesemery · 2022-07-18T19:52:44Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferencePair.java

+     * @return the status set as a formatted String
+     */
+    public String statusAsString(){
+        String output = "";


Hmm... this currently is missing any indication of WHAT lines the status' are being generated by. I don't see where those are getting printed and it looks like we forgot to do that. I think we need to point the user at the specific lines that are problemeatic somehow for example "chr3 mismatches with reference A and B" for example.

@jamesemery We said that we would print just the statuses per reference pair in a compact tabular form -- linking the statuses to individual TableRows is not a feature that has been implemented yet, and is probably out of scope for this PR.

jamesemery · 2022-07-18T19:57:51Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+     */
+    private void writeTableToStdOutput(ReferenceSequenceTable table){
+        // print header
+        List<String> columnNames = table.getColumnNames();


I think now is the time to go back and merge these two writers. They are both printing the same data (and are both also writing the header for example). I would advocate simply providing the table writer an output stream that prints to stdout here. I think that will be much cleaner rather than having these two slightly incongruent methods.

jamesemery · 2022-07-18T20:09:24Z

src/test/java/org/broadinstitute/hellbender/testutils/ArgumentsBuilderTest.java

@@ -59,7 +59,8 @@ public void testAddFileArgument() throws Exception {
        Assert.assertEquals(args.getArgsArray(), new String[]{"--foo", file.getAbsolutePath()});
    }

-    private enum AnEnum { OPTION1, OPTION2}
+    private enum
+    AnEnum { OPTION1, OPTION2}


accidental undo

...est/java/org/broadinstitute/hellbender/tools/reference/CompareReferencesIntegrationTest.java

jamesemery · 2022-07-18T20:26:00Z

src/test/java/org/broadinstitute/hellbender/utils/reference/ReferenceUtilsUnitTest.java

+
+    @Test
+    public void testCalculateMD5(){
+        final File reference = new File( "src/test/resources/org/broadinstitute/hellbender/tools/reference/CompareReferences/hg19mini.fasta");


This test looks good but I would like to get an idea for how long it takes on a full reference. If you try running it on Homo_sapiens_assembly38.20.21.fasta how long does it take?

droazen

@orlicohen Posted an initial round of comments -- still need to review the tests

droazen · 2022-07-19T13:42:48Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+     *
+     * Note: If no output file provided, table will print to standard output.
+     */
+    @Argument(fullName = StandardArgumentDefinitions.OUTPUT_LONG_NAME, shortName = StandardArgumentDefinitions.OUTPUT_SHORT_NAME, doc = "If provided, file to output table.", optional = true)


In the doc = " " attribute for this argument, mention that the output is in TSV format. Eg., "If specified, output reference sequence table in TSV format to this file. Otherwise print it to stdout."

@orlicohen looks like you missed this

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

droazen · 2022-07-19T13:44:16Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+    private GATKPath output;
+
+    @Argument(fullName = "md5-calculation-mode", shortName = "md5-calculation-mode", doc = "MD5CalculationMode indicating method of MD5 calculation.", optional = true)
+    private MD5CalculationMode md5CalculationMode = MD5CalculationMode.USE_DICT;


Think James is correct that we probably want to default to RECALCULATE_IF_MISSING

droazen · 2022-07-19T13:45:56Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+    @Argument(fullName = "display-sequences-by-name", doc = "If provided, the table by sequence name will be printed.", optional = true)
+    private boolean displaySequencesByName = false;
+
+    @Argument(fullName = "display-only-differing-sequences", doc = "If provided, only display sequence names ", optional = true)


"only display sequence names that differ in their actual sequence"

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

droazen · 2022-07-19T14:23:31Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+     * Given a sequence name, returns the set of its corresponding rows
+     *
+     * @param sequenceName The sequence name as a String
+     * @return the set of TableRows that contain the sequence name


"or an empty Set if there are no such rows"

droazen · 2022-07-19T14:24:01Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+     * @return the set of TableRows that contain the sequence name
+     */
+    public Set<TableRow> queryBySequenceName(String sequenceName){
+        return tableBySequenceName.get(sequenceName) == null ? Collections.emptySet() : tableBySequenceName.get(sequenceName);


Save the return value from get() in a local variable so that you don't have to call it twice.

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

droazen · 2022-07-19T14:30:39Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+        }
+
+        @Override
+        public boolean equals(Object o) {


Document the reason why equals() and hashCode() only check the md5

@orlicohen I would add a little more detail as to why you chose to do this specifically.

jamesemery · 2022-07-19T14:50:41Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+                }
+                break;
+            case ALWAYS_RECALCULATE:
+                md5 = ReferenceUtils.calculateMD5(source, referenceInterval);


There should be a check here if this md5 mismatches with the md5 from the dictionary (if it exists of course). If it ever does we should use the logger to throw a warning to user listing the two md5s and the contig name and the reference for which it is mismatching.

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

droazen

@orlicohen Here are the rest of my review comments

droazen · 2022-07-19T16:12:39Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+
+                if(sequenceNameFound == 2){
+                    pair.addStatus(ReferencePair.Status.DIFFER_IN_SEQUENCE);
+                } else if(sequenceNameFound == 1){


Add a comment explaining the logic here (eg., why 2 means DIFFER_IN_SEQUENCE and 1 means DIFFER_IN_SEQUENCES_PRESENT)

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

droazen · 2022-07-19T16:20:15Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+                int ref1Index = pair.getRef1ColumnIndex();
+                int ref2Index = pair.getRef2ColumnIndex();
+                Set<ReferenceSequenceTable.TableRow> rows = queryBySequenceName(sequenceName);
+                int sequenceNameFound = 0;


This is really counting the number of times a sequence name was found and the other reference was missing the sequence. Consider renaming to make this clearer. Eg., sequenceNameFoundInOneRef

droazen · 2022-07-19T16:35:35Z

src/test/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTableUnitTest.java

+
+import java.util.*;
+
+public class ReferenceSequenceTableUnitTest {


extends GATKBaseTest

droazen · 2022-07-19T16:43:03Z

src/test/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTableUnitTest.java

+                for (ReferenceSequenceTable.TableEntry entry : row.getEntries()) {
+                    if (entry.getColumnValue().equals(expectedSequenceName)) {
+                        contigPresent = true;
+                        continue;


Think you want break; here to break out of the inner loop

droazen · 2022-07-19T16:44:52Z

src/test/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTableUnitTest.java

+                        continue;
+                    }
+                }
+                Assert.assertTrue(contigPresent);


Add a String as a second arg to assertTrue() that explains what went wrong when the assert fails. Eg.,

"Contig " + expectedSequenceName + " not found in all rows returned by queryBySequenceName(expectedSequenceName)"

This will get output when the assert fails

src/test/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTableUnitTest.java

droazen · 2022-07-19T16:49:15Z

src/test/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTableUnitTest.java

+        }
+    }
+
+    public Map<ReferencePair, Set<ReferencePair.Status>> manuallySetReferencePairStatus(){


droazen · 2022-07-19T16:50:31Z

tempreferences/expected.testCompareReferencesRenamedSequence.table

@@ -0,0 +1,5 @@
+MD5	Length	hg19mini.fasta	hg19mini_1renamed.fasta


Remove the three files in the tempreferences directory from this PR using git rm

jamesemery · 2022-07-19T17:05:56Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+        return tableBySequenceName.get(sequenceName) == null ? Collections.emptySet() : tableBySequenceName.get(sequenceName);
+    }
+
+    private String calculateMD5(SAMSequenceRecord record, ReferenceDataSource source) {


This mehtod is somewhat confusing in name. calculateMD5 implies we are actually calculating the md5 but we aren't here and indeed there is a second method you added that does that. Change this to something like getMD5ForRecord() or some such.

src/main/java/org/broadinstitute/hellbender/utils/reference/ReferenceUtils.java

jamesemery · 2022-07-20T16:34:49Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+    @Override
+    public void traverse(){
+        ReferenceSequenceTable table = new ReferenceSequenceTable(referenceSources, md5CalculationMode);
+        logger.warn("Building reference sequence table.");


Make these two logger.wan() messages logger.info() messages

jamesemery

A couple things you missed and we few pretty minor comments but otherwise this is looking good. Let me know when you want me to look at that refacotred mode. When you do that refactor remember that you should probably have at least a unit test asserting that you can build the table from a sequence dictionary.

jamesemery · 2022-07-20T16:35:26Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+            writeTableBySequenceName(table);
+        }
+
+        logger.warn("Analyzing table.");


logger.info()

jamesemery · 2022-07-20T16:37:48Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferencePair.java

+        DIFFER_IN_SEQUENCE_NAMES,
+        // found sequences with same names but inconsistent MD5s between the 2 references
+        DIFFER_IN_SEQUENCE,
+        // a sequence is present in one reference but not the other


"at least one sequence"

src/main/java/org/broadinstitute/hellbender/utils/reference/ReferenceUtils.java

jamesemery · 2022-07-20T16:44:18Z

src/main/java/org/broadinstitute/hellbender/utils/reference/ReferenceUtils.java

+    /**
+     * Given a reference data source and a sequence interval, calculates the MD5 for the given sequence.
+     *
+     * Note: does not close the ReferenceDataSource it's passed.


Can you add a to this comment a line stating that this md5 code mimics the MD5 calculation code in {@link picard.sam.CreateSequenceDictionary} in that it allws IUPAC bases but uppercases all the bases.

jamesemery · 2022-07-20T16:44:43Z

src/test/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTableUnitTest.java

+    }
+
+    public boolean compareRows(ReferenceSequenceTable.TableRow row1, ReferenceSequenceTable.TableRow row2){
+        return row1.getMd5().equals(row2.getMd5()) && row1.getLength() == row2.getLength();


jamesemery · 2022-07-20T16:47:53Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+            case ALWAYS_RECALCULATE:
+                md5 = ReferenceUtils.calculateMD5(referencePath, referenceInterval);
+                if(md5FromDict != null && !md5FromDict.equals(md5)){
+                    logger.warn(String.format("MD5 Mismatch for sequence %s. Found '%s', but calculated '%s'", record.getSequenceName(), md5FromDict, md5));


add "This could indicate that the sequence dictionary is invalid"

jamesemery · 2022-07-20T16:50:11Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+/**
+ * Display reference comparison as a tab-delimited table and summarize reference differences.
+ *
+ * <p>This tool generates a MD5-keyed table comparing specified references and does an analysis to summarize the differences


This is a good javadoc for the class

jamesemery · 2022-07-20T16:51:07Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+    }
+
+    @Override
+    public Object onTraversalSuccess() {


If you aren't using it you can get rid of this override.

jamesemery · 2022-07-20T16:54:02Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+        }
+    }
+
+    public static class CompareReferencesOutputTableWriter extends TableWriter<ReferenceSequenceTable.TableRow> {


Add a quick javadoc here explaining this is necessary to write the table output

jamesemery · 2022-07-20T16:57:25Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+     * @return the corresponding TableRow from the tableByMD5
+     */
+    public TableRow queryByMD5(String md5){
+        if(!tableBuilt){


Since you repeat this line in a bunch of places. Can you make a helper method checkTableStatus() or something like that and pull out all these if(!tableBuilt) statements to call it for readability.

…truction

droazen

@orlicohen A few final comments for you, then this should be ready for merge. Tool is looking good!

droazen · 2022-07-21T13:34:58Z

src/main/java/org/broadinstitute/hellbender/engine/GATKTool.java

+     * @return path for the reference specified at the command line
+     */
+    public final GATKPath getReferencePath(){
+        return referenceArguments.getReferenceSpecifier();


Since referenceArguments is protected, this method is not needed and should be deleted. You can just access referenceArguments.getReferenceSpecifier() directly from your tool.

droazen · 2022-07-21T13:36:56Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+ * <pre>
+ * gatk CompareReferences \
+ *   -R reference1.fasta \
+ *   -ref-comp reference2.fasta


The argument is actually -refcomp (or --references-to-compare)

droazen · 2022-07-21T13:38:21Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+          USE_DICT,
+          RECALCULATE_IF_MISSING,
+          ALWAYS_RECALCULATE;
+    }


Add a one-line comment above each enum value explaining what each mode does.

droazen · 2022-07-21T13:41:42Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+    private void writeTable(ReferenceSequenceTable table){
+        TableColumnCollection columns = new TableColumnCollection(table.getColumnNames());
+        try(CompareReferences.CompareReferencesOutputTableWriter writer = output == null
+            ? new CompareReferences.CompareReferencesOutputTableWriter(new OutputStreamWriter(System.out), columns)


Since there is no test coverage for the stdout case, have you run the tool manually without -O and confirmed that the table gets printed to stdout correctly? It might be a good idea to also add an integration test that runs the tool without -O, and just makes sure it completes successfully/doesn't crash (don't need to check the actual output).

droazen · 2022-07-21T13:45:08Z

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java

+     * @param table
+     */
+    public void writeTableBySequenceName(ReferenceSequenceTable table){
+        List<String> output = new ArrayList<>();


Since it turns out we didn't need to save the output after all, you can remove this output array and just print directly in your main loop below. This cuts down on unnecessary memory usage.

droazen · 2022-07-21T14:12:33Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+            case ALWAYS_RECALCULATE:
+                md5 = ReferenceUtils.calculateMD5(referencePath, referenceInterval);
+                if(md5FromDict != null && !md5FromDict.equals(md5)){
+                    logger.warn(String.format("MD5 Mismatch for sequence %s. Found '%s', but calculated '%s'. Sequence dictionary may be invalid.", record.getSequenceName(), md5FromDict, md5));


This warning should mention the name of the reference involved. It should also say "Found %s in sequence dictionary" for clarity

droazen · 2022-07-21T14:17:46Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+                    pair.removeStatus(ReferencePair.Status.EXACT_MATCH);
+                }
+                if(!ref1Value.getColumnValue().equals(ref2Value.getColumnValue()) &&
+                        !(ref1Value.getColumnValue().equals(MISSING_ENTRY_DISPLAY_STRING) || ref2Value.getColumnValue().equals(MISSING_ENTRY_DISPLAY_STRING))){


Use the TableEntry.isEmpty() method whenever you need to check whether an entry is missing. Eg., ref1Value.isEmpty() instead of ref1Value.getColumnValue().equals(MISSING_ENTRY_DISPLAY_STRING). Comment also applies to the rest of the method below.

droazen · 2022-07-21T14:23:00Z

src/main/java/org/broadinstitute/hellbender/tools/reference/ReferenceSequenceTable.java

+        public String toString(){
+            String row = String.format("%s\t", this.md5);
+            for(int i = 0; i < entries.length; i++){
+                row += entries[i].columnValue;


Does the toString() method end up adding the MD5 twice, since it's in column 0?

droazen · 2022-07-21T14:23:37Z

src/main/java/org/broadinstitute/hellbender/utils/reference/ReferenceUtils.java

+    /**
+     * Given a reference data source and a sequence interval, calculates the MD5 for the given sequence.
+     *
+     * Note: does not close the ReferenceDataSource it's passed.


Comment is obsolete, since the method no longer takes a ReferenceDataSource

droazen · 2022-07-21T14:24:29Z

src/main/java/org/broadinstitute/hellbender/utils/reference/ReferenceUtils.java

+            throw new GATKException("Incorrect MessageDigest algorithm specified in calculateMD5()", exception);
+        }
+
+        try(final ReferenceDataSource source = ReferenceDataSource.of(referencePath.toPath(), true)) {


Add comment: we pass in "true" as the second arg to ReferenceDataSource.of() to tell it not to modify IUPAC bases or capitalization for us.

droazen

👍 Looks like all comments have been addressed! Will merge once tests pass

Addressed

orlicohen requested a review from droazen July 8, 2022 15:35

droazen self-assigned this Jul 8, 2022

orlicohen added 18 commits July 18, 2022 14:38

initial version of reference comparator tool

92bc20a

basic skeleton code

85bc87a

basic output table implementation

4335f4f

working on output table

aea17a5

updates to mode 1 comparison

7869a86

refactoring/cleaning up existing code and adding write to file output…

41ce2f6

… featute

table to file output and integration tests

fa26001

checking in references and expected outputs, minor tweaks

2671b43

added md5 calculation mode code and integration tests

8f74dd7

added table query functionality

006ada6

refactored TableEntry

b344c64

prelim unit tests

b2507a1

unit tests for query methods, initial table analysis

e3b1aac

unit tests updated

ee0db1a

initial table analysis, work in progress

6c00ada

table analysis for exact match and diffs in sequence, sequence name, …

233bc97

…sequence present

superset/subset table analysis

9a5cf45

finished table analysis and added documentation

437eb84

orlicohen force-pushed the oc_referencecomparator_6837 branch from a0e00bb to 437eb84 Compare July 18, 2022 18:40

jamesemery previously requested changes Jul 18, 2022

View reviewed changes

update to subset/superset logic, added case to analyze table unit test

5c887a1

droazen suggested changes Jul 19, 2022

View reviewed changes

jamesemery reviewed Jul 19, 2022

View reviewed changes

src/main/java/org/broadinstitute/hellbender/tools/reference/CompareReferences.java Show resolved Hide resolved

droazen suggested changes Jul 19, 2022

View reviewed changes

jamesemery reviewed Jul 19, 2022

View reviewed changes

src/main/java/org/broadinstitute/hellbender/utils/reference/ReferenceUtils.java Show resolved Hide resolved

droazen changed the title **DO NOT MERGE** CompareReferences tool CompareReferences tool Jul 20, 2022

responding to review comments

dc59744

jamesemery reviewed Jul 20, 2022

View reviewed changes

responding to review comments, refactored ReferenceSequenceTable cons…

70ec346

…truction

droazen suggested changes Jul 21, 2022

View reviewed changes

responding to final review comments

a736923

droazen approved these changes Jul 21, 2022

View reviewed changes

droazen merged commit 64f9953 into master Jul 21, 2022

droazen deleted the oc_referencecomparator_6837 branch July 21, 2022 17:34


		import java.util.*;

		public class ReferenceSequenceTableUnitTest {

		@@ -0,0 +1,5 @@
		MD5 Length hg19mini.fasta hg19mini_1renamed.fasta

CompareReferences tool #7930

CompareReferences tool #7930

Conversation

orlicohen commented Jul 7, 2022

codecov bot commented Jul 7, 2022 • edited Loading

Codecov Report

droazen commented Jul 18, 2022

jamesemery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droazen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droazen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamesemery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droazen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droazen left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 7, 2022 •

edited

Loading