Improved metrics: Measure and report the download count

[ShivamArora]

As a developer and owner of a package, I would like to see the download stats for my package.

The npm package repository on npmjs.com provides the download statistics for the libraries.

I would like to see the same happening on the pub.dev website.

This will help in getting to know how many people/projects are using my package.

[jonasfj]

See also #1229.

---

Defining the download count is a bit non-trivial though. Should we remove bots (how), if not travis-ci will dominate the result. Do we want to measure downloads, or number of times a client asked if a newer version was available?

I think we ultimately want a more transparent popularity metric along the lines of download count.

[ShivamArora]

I don't think asking for a newer version should count as a download.

By download count, I meant to measure the downloads made by different projects using the pub tool.

However, for a reference, can we check out how npm defines their download count?

[jonasfj]

By download count, I meant to measure the downloads made by different projects using the pub tool.

I only download a package to my machine once, then it's in my ~/.pub-cache/, no matter how many projects I used said package in.

However, if my travis setup isn't configured to cache it can download the package for every single test run.

[ShivamArora]

I only download a package to my machine once, then it's in my ~/.pub-cache/, no matter how many projects I used said package in.

The same happens for npm packages as well as they are stored in node_modules.

It would be worth if we can check out how the npm repository is measuring the download count of packages.

[isoos]

It would be worth if we can check out how the npm repository is measuring the download count of packages.

https://blog.npmjs.org/post/92574016600/numeric-precision-matters-how-npm-download-counts

The main gist:

npm’s download stats are naïve by design: they are simply a count of the number of HTTP 200 responses we served that were tarball files, i.e. packages. This means the number includes:

• automated build servers
• downloads by mirrors
• robots that download every package for analysis

[...]

So the count of “downloads” is much larger than the number of people who typed “npm install yourpackage” on any given day.

[...]

Bottom line: most packages get a trickle of downloads every day, and that’s not necessarily indicative that they’re being actively used. Only if your package is getting > 50 downloads/day can you be sure you’re seeing signal instead of noise. You will also get a burst of downloads whenever you publish a new package or a version of that package, because all the mirrors will download it.

[ShivamArora]

@isoos Do you have anything in mind regarding what could be the best approach to measure the download count?

[isoos]

Do you have anything in mind regarding what could be the best approach to measure the download count?

@ShivamArora: I don't think there is a good approach to it, by its nature, it cannot be solved correctly. Our current popularity metric is a proxy of the total download count. A lot of thought went into it, e.g. how to filter CI systems automatically, how to balance the long-tail distribution, and I believe it is much closer to the truth than the "feel-good" download counts.

Having said that, I think it can be a valid goal to expose a few more metrics that are also a proxy for popularity, because maybe a single metric won't ever solve it:

• the "feel-good" / "raw" download counts of the packages
• the number of visits on the package pages
• the number of API calls that requested data on a package (especially after these APIs are stable and public)
• search relevance / click-through-rate for a package (+ metrics on most searched-after expressions related to a package)

We are also planning to introduce some kind of community rating that also provides feedback about the quality and/or use of the package.

Note: we don't have any strict schedule for the above, and can't commit to it, but really open to ideas related to these.

[ShivamArora]

@isoos Cool.

I agree with the point that we should expose a few more metrics.

Having the feature of community rating would be pretty good. But don't limit it just to rating include some reviews as well.

Community Ratings and Reviews would be helpful for anyone to know about the quality of the package and what others feel about the package after using it.

Apart from that I would suggest another metric, if that's possible, which should provide information about the Documentation Quality.

Since good documentation is a lot more important than just using the package.

[bsutton]

@isoos

Our current popularity metric is a proxy of the total download count. A lot of thought went into it, e.g. how to filter CI systems automatically, how to balance the long-tail distribution, and I believe it is much closer to the truth than the "feel-good" download counts.

As a publisher I'm not certain what 'truth' you are talking about, as the current '0-100' popularity gives me no useful information.
Whilst I agree that it would be nice to filter out CI/bots etc its probably not a reality. As such a raw count actually is probably the best approach.

Raw counts will give a sense of scale.
The popularity score gives no sense of scale.
If I've only got 10 users then its probably time to dump the project. If there is 10K users then I'm been appreciated and its worth continuing the work :)

It is particularly unhelpful that the current metric is undocumented.
Is the range linear or exponential?
Is it somehow relative to other projects?

I have a feeling that the team tried to be too clever and we have ended up with something far worse than the raw numbers.

[mit-mit]

This is actively being considered. We're looking into options for how to get a representative count of downloads / pub gets, even for cases where the package is still in the local cache.

[bsutton]

@mit-mit 👍

[MaikuB]

I don't know how many authors are publishing prerelease versions but seeing download numbers for them could help authors get a sense of the number of users that helping to test out these versions. This is turn could assist with making a decision on if it's ready for a stable release

[AKushWarrior]

@mit-mit I think that when a user runs pub get, pub can check the differences from the previous pubspec.lock (what packages have been added/deleted) and then push those changes to the server, which applies them to the global count of usages for that package. That still doesn't solve the Travis CI issue, but it does solve the cache issue.

I know you guys have looked at NPM. https://crates.io (Rust's central package repository) also provides download stats, so it might be worth checking out how they do it.

[MaikuB]

To follow up on what I had mentioned, I had changes for a plugin that was in prerelease for a number of months that was then promoted to stable having figured it had been given enough time for the community to help test and give feedback. A couple of weeks later a bug has been found. IAs this is a plugin that's used by lot of the community, at least based on what I can infer from pub.dev and GitHub, I can only assume the prereleases hadn't gotten much usage to begin with. If there is some concern around displaying download counts publicly, perhaps consider having it visible only to the authors of the plugin. Though part of what would help in this particularly is to know the amount of downloads for a particular version. An example that does this is https://www.nuget.org/

[Jjagg]

If it's feasible, I have a use case where it would be very helpful to know why a package was installed, i.e. whether it was a direct dependency or transitive through package x (in percentages).

[jayoung-lee]

+1 to more straightforward metrics.

In the recent user study, users used the popularity score and Pub Points for decision making (e.g., skip packages with <80% popularity), but they didn't necessary understood the scores when using them.

[AKushWarrior]

+1 to more straightforward metrics.

In the recent user study, users used the popularity score and Pub Points for decision making (e.g., skip packages with <80% popularity), but they didn't necessary understood the scores when using them.

Well, nobody understands the scores, because they're not transparent. I don't see the issue with just exposing the raw metrics: that's both more useful and more informative.

[pichillilorenzo]

Is there any news or ETA about it? Thanks!

[sigurdm]

No - we don't have an ETA for this yet - but it is something we want to get to do.

[incrediblezayed]

+1

[therdm]

+1

[iapicca]

@mit-mit
this feature could help sniffing "likes" from bots

• [proposal] adopt measures against bots #6226

Why not? The downloads wrapping is a very popular cheating. The more downloads, the more often users choose your package, the more popular you become, and the more investments you get.

I wish good luck to the team to find a solution.

@xr0master
at the very least it will unveil if the issue of bots existed so far (I think so)

[PlaxXOnline]

+1

[IgorGZd]

+1

[edugemini]

+1

[ueman]

Please don't post +1. Use the 👍 emoji on the original post to vote it up. Comments without content don't add to the discussion and don't help prioritizing this issue. Adding a thumbs up does so.

[iapicca]

This is actively being worked on; we're adding various telemetry to enable this. But it will take a while before this is complete enough that we can start displaying it.

@mit-mit does your comment still reflect the status of the issue?

[mit-mit]

Yes, we're still working on it.

[renancaraujo]

Hello,

I wanted to chime in with the insight on how game changer this would be for us at @VeryGoodOpenSource

Two of the most popular packages that we have, very_good_cli and dart_frog_cli have thousands of reports usages (trough the now dead usage package) and still score relatively low in the current "popularity" metric.

Every metric has a caveat, and I think we can live with the ones under the possible "downloads". The real impact of these caveats relies more on how you communicate it to the people reading the chart rather than making it available or not.

As a follow-up on making "downloads" available as a metric, we can think on some capillaries that could increase the quality of life of most dart developers:

• Identifying download spikes after a release could indicate that people are updating the package to a new version. This can also be further extended to a "version" capillarity, just like we see on crates.io, see image below.
• Downloads per SDK local version could help package maintainers understand the impact of raising SDK version constraints.

Example of a download chart on crates.io, where the "version" capilarity is explored:

[bsutton]

I will add to @renancaraujo comments. The version details would be particularly interesting as we could then see when a particular version of Dart no longer needs to be supported.
We currently support from dart 2.12, but how many people are still using it?
This level of knowledge would help make our development more streamlined.

[Blacktaler]

It would be worth if we can check out how the npm repository is measuring the download count of packages.

https://blog.npmjs.org/post/92574016600/numeric-precision-matters-how-npm-download-counts

The main gist:

npm’s download stats are naïve by design: they are simply a count of the number of HTTP 200 responses we served that were tarball files, i.e. packages. This means the number includes:

• automated build servers
• downloads by mirrors
• robots that download every package for analysis

[...]

So the count of “downloads” is much larger than the number of people who typed “npm install yourpackage” on any given day.

[...]

Bottom line: most packages get a trickle of downloads every day, and that’s not necessarily indicative that they’re being actively used. Only if your package is getting > 50 downloads/day can you be sure you’re seeing signal instead of noise. You will also get a burst of downloads whenever you publish a new package or a version of that package, because all the mirrors will download it.

Ok, it seems a little bit harsh. But at least, are we able to see among how many users is our package popular? For example, how many users does it mean to be popular by 100%?

Sorry if it's the exact same thing as discussed above😃

[mit-mit]

Assigning to @sigurdm who started investigating this

[mit-mit]

Status update: we're actively working on this, and hope to have improved metrics in at least preview by end of year. Sorry this is taking a while, but there are a lot of considerations to work through.

[dadvir]

if i can jump in, and be a bit naive and (maybe) spiritual about this, I understand the Dart comity leaders concerns regarding a well-engineered and science-like metrics.
in a real-world scenario, you've your path chosen, and most of the times, clear and simple is much more better. the current scoring tells me nothing about should i use the package. this can be risky. also can be used to exploit by black hat coders.
consider an engineer, architect or sec-dev-ops looking into a Dart package, at the moment he's completely blind regarding the usage of it.

npm metrics proved to be a great tool for the community to decide whether or not to use a package. the Dart way, 0-100 percentage, simply doesn't cut it.

i suggest to imply provide a number, and note your science like comment about scoring, and let the wise women and men choose what they will :)

[iapicca]

the PR

• Experimental: Show download counts instead of popularity score #8126

seems to target this issue even if isn't mentioned directly

[Blacktaler]

if i can jump in, and be a bit naive and (maybe) spiritual about this, I understand the Dart comity leaders concerns regarding a well-engineered and science-like metrics. in a real-world scenario, you've your path chosen, and most of the times, clear and simple is much more better. the current scoring tells me nothing about should i use the package. this can be risky. also can be used to exploit by black hat coders. consider an engineer, architect or sec-dev-ops looking into a Dart package, at the moment he's completely blind regarding the usage of it.

npm metrics proved to be a great tool for the community to decide whether or not to use a package. the Dart way, 0-100 percentage, simply doesn't cut it.

i suggest to imply provide a number, and note your science like comment about scoring, and let the wise women and men choose what they will :)

I'm not quite sure that, what you said would be correct way too. Because there are rare packages/plugins, which are not used commonly, but necessary and not dangerous. So to work by your opinion they should add feedback section rather than likes and percentages.

[mit-mit]

seems to target this issue even if isn't mentioned directly

Yes, we're making good progress and should have some updates soon.

[bigbott]

5 years to show the known number of downloads. Ridiculous

Someone marked this my comment as abuse.
I think we have a different definition of the word abuse.

The fact that you did not fix issue opened 5 years ago, which can be fixed literally in 5 minutes is abuse.

[orestesgaolin]

The experimental flag seems to be available now under pub.dev/experimental

One thing that I'm not clear is for what time range the total download count is shown. Only after hovering over the number in the side bar you can see it's in the last 30 days.

Perhaps it should also be written openly on the Scores subpage

[iapicca]

The experimental flag seems to be available now under pub.dev/experimental

@orestesgaolin you made my day!
this could be a (or better, the final) nail in the coffin for getx popularity

[kyle-seongwoo-jun]

Hi @mit-mit,

Great work! Thank you to the Dart/Flutter team for the amazing job. I have a quick question:

I’m curious if the download count could be included in the pub.dev API response.

$ curl https://pub.dev/api/packages/riverpod/score | jq
{
  "grantedPoints": 160,
  "maxPoints": 160,
  "likeCount": 3434,
  "popularityScore": 0.9761956555517574,
  "tags": [
    ...
  ],
  "lastUpdated": "2024-11-20T23:32:33.305499"
}

Currently, the response only provides points, like count, and popularity. I believe adding the download count would be a valuable enhancement.

Thank you for considering this suggestion!

Relevant Code Reference:

pub-dev/app/lib/frontend/handlers/custom_api.dart

Lines 214 to 219 in dd4ecae

	/// Handles requests for
	// - /api/packages/<package>/score
	/// - /api/packages/<package>/versions/<version>/score
	Future<VersionScore> packageVersionScoreHandler(
	shelf.Request request, String package,
	{String? version}) async {

[sigurdm]

Thank you for considering this suggestion!

Yeah - we will make some kind of api for fetching the counts - the details are not worked out yet though.