Create computed nCommit & nAuthor per month by project

[noah-22]

@audrism

When analyzing a project's activity over time, commit counts and unique author counts per month (not counting one author multiple time if they have made multiple commits) are great metrics. Could we pre-compute this metric in one of our databases to allow for a query of Project.month.nCommits and Project.month.nAuthors?

[audrism]

OK, this may be a good candidate for CH as the number of records would exceed billion?

[audrism]

Also, what about number of blobs, types of files changed, APIs involved?

[noah-22]

Number and type of files changed, along with blobs and API's, would be very useful!

I propose a format here, in an existing clickhouse table conversation/issue, since this is now a clickhouse related issue.

[audrism]

/da1_data/basemaps/gz/P2mncFullS*.s
has project;month;nAuthors;nCommits

[noah-22]

This basemap is wonderful, thank you!

My group from the hackathon, Inflection Points, is interested in continuing research. We are interested in rootFork projects, though, which are often not the deforked (P) project. Blobs changed would also be a great metric to have. Could the map be expanded to include both of these?

[audrism]

Take a look at new P_metadata.S in mongodb: it might be good for sampling, though it does not have monthly blobs. You can always find P for rootFork via p2P map. Otherwise these projects should be nearly identical. Do you have instances where they are not?

[noah-22]

Is WoC.P_metadata.S an improved version of WoC.proj_metadata.S? They appears to have similar fields, with the addition of monthly commit count and core size. I have questions about these additions, though.

1. What is the purpose of the NumForks field and what advantage does it provide over CommunitySize? NumForks seems to often be 0, and is therefore only useful when P = RootFork.
2. Is Core defined by the authors that make up 80% of the contributions to the project? I've heard this metric mentioned before and am wondering if it the same used here.

I'll take it to be true that P is similar enough to the RootFork project that computing time series for both is not necessary.

[audrism]

1. NumForks is github-derived with all associated issues (often incorrect, many missing, hierarchical) while community size is more general, purely based on shared commits. RootFork is also derived from GH.
2. Correct
3. Generally, I would avoid using GH attributes like NumForks or RootFork since ghTorrent data is spotty and the
   actual fork relationships in GH are not always what you think they are. Yes, I would not bother getting a separate time series for root fork unless you have many examples to the contrary