Skip to content

Metrics

Jump to bottom Edit New page
Sebastian Benthall edited this page Aug 13, 2020 · 61 revisions

This page lists the metrics that we're using to track the progress of the econ-ARK project as well as the current values for those metrics.

Summary

Priority Category Metric Name Value More Info In Use Last Updated
Crucial Outputs Papers that use HARK 8 link yes 2020-07-31
Crucial Outputs Papers that cite HARK 2 link yes 2020-07-31
Crucial Outputs Number of REMARKs 14 link yes 2020-06-18
Crucial Online Activity Major contributors 11 link yes 2020-06-18
Crucial Funding Number of grants 2 link yes 2020-06-25
Crucial Misc Features added to HARK 18 link yes 2020-08-13
Significant Outputs Courses that teach HARK 2 link yes 2020-06-25
Significant Outputs Conference talks about HARK 8 link yes 2020-06-25
Significant Outputs Number of DEMARKs 20 link yes 2020-06-25
Significant Events Events at central banks & government institutions 5 link yes 2020-06-25
Significant Events Professional events 7 link yes 2020-06-25
Significant Online Activity Pip installations 34,592 link yes 2020-06-25
Significant Online Activity Conda installations 4,592 link yes 2020-06-25
Significant Online Activity Forks on Github 137 link yes 2020-06-25
Significant Online Activity Total contributors to HARK repository 32 link yes 2020-06-25
Significant Online Activity Unique Cloners 69 link yes 2020-06-25
Significant Online Activity Issues submitted on Github 327 link yes 2020-06-25
Significant Online Activity % Issues Open 141 (43%) link yes 2020-06-25
Significant Online Activity % Issues With Replies in Last Month 52% link yes 2020-06-25
Significant Online Activity PRs submitted on Github 400 link yes 2020-06-25
Significant Online Activity % of Github PRs merged 388 (97%) link yes 2020-06-25
Significant Misc % test coverage of HARK 72% link yes 2020-06-25

Metric Details

Papers that use/cite HARK

Current total: 8 use, 2 cite (last checked 7/31/2020)

We find papers that cite HARK using a combination of Google Scholar, by searching for "econ-ark" and for the various DOIs for HARK listed on Zenodo. It's important to note that Google Scholar is only able to search abstracts and only for the past year, so we may end up wanting to supplement with Web of Science.

Searches:

List of papers that use HARK:

List of papers that cite but do not use HARK:

Papers found but not included in lists above:

Number of REMARKs

Current total: 14

The process for newly added REMARKs is outlined here, the authoritative list of remarks is the directories in this folder.

Remarks of particular note:

Number of major contributors

Current total: 11

Defining what one means by "major contribution" is never an easy task. Often we end up overvaluing the most legible and track-able contributions. The list below is only an initial attempt at definition, and the numerical values provided merely a ballpark figure.

Forms of "major contribution":

  • adding a single REMARK
  • adding at least three DEMARKs
  • adding a new model to HARK
  • adding at least three smaller tools or solvers
  • significantly increasing test coverage (> 5%)
  • significantly improving documentation (sustained documentation help over time)
  • contributing to the organization, structure, and/or strategic vision of the project

List of major contributors (in alphabetical order): (2020/03/06)

  • Sebastian Benthall
  • Chris Carroll
  • Edmund Crawley
  • Shauna Gordon-McKeon
  • Sumana Harihareswara
  • Jackie Kazil
  • David Low
  • Patrick Mogensen
  • Nathan Palmer
  • Mridul Seth
  • Matt White
  • Open Tech Strategies

Number of grants

Current total: 2

Grantors:

  • Sloan
  • Think Forward Initiative

Features added to HARK

Current total: 18

As part of the release process, the release manager (current @shaunagm) checks in the release notes for additional economic modeling features and adds them here. Submitters of pull requests will be asked to specify if their changes add any new features and/or families of features. This metric therefore gets updated with each release. To see features added over time, view the release notes for each version.

June 1 2020 to present (2020-06-25)

2020-08-08 - 0.10.7

  • Add a custom KrusellSmith Model (Feature 18)
  • Simulations now uses a dictionary history to store state history instead of _hist attributes (Feature 17)
  • Removed time flipping and time flow state, "forward/backward time" through data access (Feature 16)
  • Simulation draw methods are now individual distributions like Uniform, Lognormal, Weibull (Feature 15)

June 1 2019 to June 1 2020

2020-04-17 - 0.10.6

  • Add Bellman equations for cyclical model example (Feature 14)
  • read_shocks now reads mortality as well (Feature 13)
  • Discrete probability distributions are now classes (Feature 12)

2020-03-24 - 0.10.5

  • Default parameters dictionaries for ConsumptionSaving models have been moved from ConsumerParameters to nearby the classes that use them. (Feature 11)
  • Improvements and cleanup of ConsPortfolioModel, and adding the ability to specify an age-varying list of RiskyAvg and RiskyStd. (Feature 10)
  • Rewrite and simplification of ConsPortfolioModel solver. (Feature 9)

2020-03-05 - 0.10.4

  • Move non-reusable model code to examples directory, BayerLuetticke, FashionVictim now in examples instead of in HARK code #442 (Feature 8)
  • Load default parameters for ConsumptionSaving models #466 (Feature 7)
  • Improved implementation of parallelNelderMead #300 (Feature 6)

2019-12-12 - 0.10.3

Added constrained perfect foresight model solution. (Feature 5)

2019-10-03 - 0.10.2

No major changes.

From June 1 2018 to June 1 2019

  • Adding a framework for solving multidimensional models with non-convex value functions and occasionally binding constraints, based on the “G2EGM” method by Druedahl & Jorgensen (2017). (Feature 4)

  • Adding a framework for solving unidimensional models with a discrete-continuous choice structure, based on “The endogenous grid method for discrete-continuous dynamic choice models with (or without) taste shocks” by Iskhakov, Jorgensen, Rust, and Schjerning (Quantitative Economics 2017). (Feature 3)

  • Improved and expanded warnings for solution non-existence (and other undesirable properties) in foundational consumption-savings model. (Feature 2)

  • Generalized meta-parameters for aggregate productivity shocks models, enabling users to optimize the routines for their purposes. (Feature 1)

Courses that teach HARK

Current total: 2

Course Name Institution Instructor Term Link to More Info
Hands-On Heterogeneous Agent Macroeconomics Goethe University and SAFE (Sustainable Architecture for Finance in Europe) Chris Carroll, Matt White May 2018 (minicourse) syllabus
Open Source Macro Bootcamp The Becker Friedman Institute at the University of Chicago Chris Carroll July/August 2018 (two day training) program site
Hands-On Heterogeneous Agent Macroeconomics Summer School for Central Bankers Chris Carroll August 2019 (minicourse) syllabus
Hands-On Heterogeneous Agent Macroeconomics BI University, Oslo Chris Carroll Jan-Mar 2020 (minicourse) syllabus

Included in syllabus but not taught:

Conference talks about HARK

Current total: 8

Conference talks about HARK should be listed on the econ-ark Google calendar. Periodically, a team member will review the calendar and add listings here.

Date Presenter Venue Talk Title Link to More Info
September 13-14 , 2018 Chris Carroll Workshop on Housing, Credit and Heterogeneity: New Challenges for Stabilization Policies Heterogeneity, Housing, and Macroeconomic Policy: An Agenda link
Jun 19-21, 2018 Chris Carroll Computation in Economics and Finance Conference The Econ-ARK project link
June 8-9, 2018 Chris Carroll VESifo Venice Summer Institute Sticky Expectations and Consumption Dynamics link
May 24-25, 2018 Chris Carroll, Matt White Deutsche Bundesbank conference: International Conference on Household Finance. The Econ-ARK/HARK Toolkit for Heterogeneous Agent Modelling link
May 22-23, 2018 Chris Carroll European Central Bank conference Workshop on Household Heterogeneity and Monetary Economics and Presentation of the HARK Toolbox.
December 13–15, 2017 Chris Carroll Reserve Bank of Australia Quantitative Macroeconomics Workshop 2017 The Distribution of Wealth and the Marginal Propensity to Consume link
Dec 11-12, 2017 Chris Carroll Reserve Bank of New Zealand Conference on Heterogeneous Agents And Housing The normal science of heterogeneous agents macroeconomics link
Sep 21-22, 2017 Chris Carroll Heterogeneous Agents and Agent-based Modeling: The Intersection of Policy and Research Heterogeneous Macroeconomics and Reality link

Number of DEMARKs

Current total: 20

The number of DEMARKs is equivalent to the number of .ipynb files in the DEMARK/notebooks folder.

Note: 12 of the DEMARKS were added between June 1 2018 and June 1 2019 according to Github.

Events

By events, we mean in-person meetings hosted by, run by, or dedicated in large part to discussing or working on HARK. These can occur as part of a larger event, but are distinct from, say, giving a talk at a conference or meetup.

Events at central banks & government institutions

Current total: 5

Date Presenter Venue Title Link to More Info
May 2020 Edmund Crawley virtual presentation to Board of Governors of the Fed presentation of pandemic work
spring 2020 Chris Carroll virtual seminar "at" International Monetary Fund presentation of pandemic work
March 2019 Chris Carroll Norges Bank (the central bank of Norway) and Statistics Norway (the agency that compiles and controls the National Registry data) informal presentation
May 31 - Jun 6, 2018 Chris Carroll Bank of England untitled seminar link
Oct 5-6, 2017 Chris Carroll Danmarks Nationalbank, Deutsche Bundesbank and Norges Bank link
Professional events

Current total: 6

This is a catch-all category for outreach not captured by the category "courses", "conference talks", or "events at central banks & government institutions".

Date Presenter Venue Title Link to More Info
spring 2020 Chris Carroll Johns Hopkins University Presentation to the JHU university-wide consortium of researchers working on topics relating to Covid-19
May 6, 2019 Shauna Gordon-McKeon, Chris Carroll, Matt White PyCon development sprint
Jun 19-21, 2018 Chris Carroll, Matt White Computation in Economics and Finance Conference attended/co-organized "Developers Summit"
June 7, 2018 Chris Carroll Eurosystem Household Finance and Consumption Network, Bratislava
Feb 16, 2018 Chris Carroll University of Delaware seminar: Sticky Expectations and Consumption Dynamics link
Dec 18, 2017 Chris Carroll Australian National University seminar: The Distribution of Wealth and the Marginal Propensity to Consume link
October 4, 2017 Chris Carroll University of Copenhagen untitled seminar

Github issues & PRs

Current totals: 327 issues, 141 (43%) open; 400 pull requests, 12 (3%) open

Currently we're just getting all issues and PRs in the HARK repository, but ideally we would distinguish between issues/PRs from core contributors and from the community.

For now we use Shauna's old project to track % of issues responded to and % of PRs merged. We probably want a slightly different metric that's harder to measure.

% test coverage

Current total: 72%

More details here. To run, may need to start a new virtualenv, if so install coverage AND pytest. Then: coverage run --omit=/venv/ -m pytest. Run coverage html to get the html version.

Use https://tableconvert.com/ to get from coverage's output to markdown.

"Nice to have" metrics

These may be added to the list above when the crucial and significant metrics have been worked through.

  • Events
    • Overall number of events, number of attendees, number of volunteer teachers and mentors
  • Online activity
    • MyBinder usage
    • Website usage stats (separate from mybinder usage stats)
    • Total # of contributors on Github
    • Stars on Github
    • "Watchers" on Github
    • Mentions on social media
  • Misc
    • Number of collaborating/partner projects or organizations
Add a custom footer
Add a custom sidebar
Clone this wiki locally