Post-pilot revisions and open tasks for ACRE Guidelines and Surveys

[abogdanoski]

Timeline and responsibilities: https://docs.google.com/spreadsheets/d/1vr0DAO78W_oe-eAB7hOHzrIU83YgfdB8wlaRPKVyJcc/edit?usp=sharing.

Open items

Guidelines

Content

• In 1.1.1 outline in a bit more detail the workflow from candidate to declared paper @fhoces

• At the end of section 1.2, write a reminder to set up a revised reproduction package @fhoces

• Provide more guidance on how to add and scope claims in Scoping chapter @fhoces

• Template email for communications in unused data scenario @abogdanoski

• Draft chapter on "Concluding the Reproduction" w/ description of the desired outputs + guidance on how to structure reproduction package + info on how we will use the results of the reproduction. This would probably be an added chapter to be inserted between Ch. 6 and 7 in current draft. @fhoces

• Chapter 6: Draft introduction explaining diagram examples @fhoces

• @fhoces Add figures to Robustness chapter and notify @joelferg

• Ch. 7.2: Resources (@fhoces to draft a table with i) general resources for comp. reproducibility, ii) examplary projects, iii) exemplary authors)

• Fernando to review updated AEA template, then modify for ACRE purposes @fhoces @joelferg

• Ch. 8: Tips for reproducibility (draft complete, @khoeberling reviewed; comments in pull request)

• Ch. 9: Update licensing, including suggested citations format (@khoeberling)

• Review references and integrate bibtech @abogdanoski

• Ch. 10: Glossary of terms (draft complete, review pending @fhoces)

• Platform Code of Conduct @abogdanoski @khoeberling

Surveys

Survey 1
Content

• Add questions about pre-specifying robustness checks @fhoces
• Q6.8: add a choice to report p-values @abogdanoski
• Q6.8: make clear that they need to input numbers @fhoces
• Q6.8 force formats in fields @em-ng21 (?)

Survey 2

• Clarify order of reproducibility assessment questions: first about paper-level reproducibility assessment, then about each individual display item @fhoces
• Clarify connection between Survey 1 (claims) and Survey 2 (display items) @fhoces
• Develop grading rubrics, consider ways to automate student reports @fhoces @em-ng21
• Fix skip pattern with "I am not sure" option for scoring levels @fhoces [clarify what this means] @fhoces
• Add "data format" and "specific software" to improvement options @abogdanoski
• Add a question on whether differences in display item relate to pre-specified claims (or correspond to a different section of the DI that does not fundamentally connect to no relate the claim) @fhoces [reword or clarify what this means]
• Add questions where reproducers must score some parts of others reproducers (summary of claims, repro package, etc). @joelferg (?)
• Q7.2 Clarify that this is about improvements made by the reproducer + give examples of improvements possible (?)

Spreadsheet

• Make separate, view only example workbook and move all example sheets there @joelferg
• Populate example workbook with better/more concrete/more examples @joelferg @fhoces
• Confirm all links function correctly @joelferg
• Remove hyperlinks in .do files/make it so specific file types don't automatically become hyperlinks @joelferg
• Include something about apostrophes before +'s in improvement sheet (?)

---

Completed items:

Guidelines

• Add Stage 0, i.e. guidance on selecting a paper and communicating with authors to ask for missing reproduction package @fhoces
• Chapter 8: Additional resources @joelferg @em-ng21
• Template email for communications at Stage 0 @abogdanoski
• Chapter 4: Robustness @fhoces
• Build examples of "incomplete workflows" and feed them to the diagram builder @joelferg and @fhoces to workshop a solution
• Chapter 9: Contributions @em-ng21 @khoeberling
• In Ch. 2.1: explain how to distinguish between different data and code files (?)
• Develop examples for reproduction strategies @joelferg @fhoces Formatting and Aesthetics
• Convert tables and figures into HTML; check operability across browsers @em-ng21
• Create a cover @em-ng21
• Tips and resources for reproducibility chapter draft @em-ng21

Survey 1

Content

• Create an automated report @fhoces @em-ng21 Formatting
• set up navigation back and forth

Survey 2

Content

• In the Introduction, clearly state the desired outputs of the overall exercise and especially an updated reproduction package @fhoces

Diagram Builder @joelferg

• Automate diagram building
• Update the survey and guidelines with an example using "unused data sources"
• Fix bug with a comma separating files: option 1: coded up, option 2: add instructions to materials (ie survey and guidelines) to avoid this problem. (eg R_8wR08MAw9yCyQdH_ACRE_diagram.txt)
• Add a separation between unused data sources and unused data files.
• Figure out why some diagrams don't have a name for output (eg R_bvIfoWKuIvhj877_diagram.txt)
• Ask flexibility to question 10.1
• Ask about more annotations.
• Ask to add a vertical line (low priority)

General

Immediate

• Create a script to automate the list of contributors @em-ng21
• Allow unrestricted navigation back through both surveys (consider implementing breadcrumbs navigation system) @fhoces
• Create a template of reproduction package including recording sheets and diagram trees @fhoces @joelferg
• Language review/ copy editing @abogdanoski @khoeberling
• 1 slide for WEAI presentation with a timeline for key milestones @abogdanoski
• Match structure and numbering of questions in surveys with the structure of guidelines @abogdanoski
• Make explicit why we want them to report separate files for different code functions (e.g. based on coding conventions for reproducibility). @fhoces
• Check terminology and wording for consistency between guidelines and surveys @abogdanoski
  • scale with own data
  • scale with admin/confidential data
  • outputs -> display items