How to measure adoption of Pip? Does anyone still use easy_install? PyPI census

[hickford]

The pythonwheels.com site measures the adoption of new tech by package maintainers.

From the other direction, it would be interesting to measure the adoption of packaging tech by users.

• Is everyone on the latest Pip version?
• How popular are wheel downloads compared to source distributions?
• Does anyone still use easy_install?
• Does anyone download eggs?
• How many people download packages from the web browser?

It might be possible to learn this from PyPI with the right logging. Pip and easy_install have distinct http user agents which include their precise version numbers.

---

• https://pypi.python.org/pypi/numpy
• https://pypi.python.org/pypi/requests
• https://pypi.python.org/pypi/pytz

[alex]

Using pypa/linehaul we're able to compute these statistics. @dstufft can share some of the numbers easily enough.

The tl;dr; is most folks are using pip, and it looks like most uses of easy_install these days come from setup_requires and setup.py test.

[pradyunsg]

Google BigQuery is useful. :)

The following query gives the result of how many downloads from PyPI happened from various clients (and their versions), in the past month:

SELECT
   details.installer.name, details.installer.version, COUNT(*) as download_count
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -1, "month"),
    CURRENT_TIMESTAMP()
  )
GROUP BY
  details.installer.name, details.installer.version
ORDER BY
  download_count DESC, details.installer.name ASC, details.installer.version DESC
LIMIT
  100

Running this at May 17, 2018, 12:05:13 PM IST, I got:

Click to expand

Row	details_installer_name	details_installer_version	download_count
1	pip	9.0.1	134582699
2	pip	10.0.1	118004637
3	pip	6.1.1	62803887
4	pip	9.0.3	47139548
5	pip	1.5.4	23256855
6	bandersnatch	2.0.0	22488790
7	pip	8.1.1	20827275
8	pip	6.0.8	11375879
9	bandersnatch	2.2.0	10311977
10	pip	10.0.0	10175888
11	bandersnatch	1.11	9716671
12	pip	8.1.2	9250362
13	pip	7.0.1	4767135
14	pip	1.5.6	4630838
15	bandersnatch	2.2.1	4495470
16	setuptools	39.1.0	3436419
17	setuptools	39.0.1	3084099
18	pip	7.1.2	2952558
19	Browser	null	2075401
20	pip	10.0.0b2	2000224
21	pip	9.0.2	1848474
22	pip	1.5.5	1488433
23	null	null	1249811
24	setuptools	28.8.0	1191750
25	setuptools	36.2.7	1075139
26	pip	1.4.1	999395
27	setuptools	20.7.0	980978
28	pip	7.1.0	970524
29	setuptools	3.3	863428
30	pip	9.0.0	699385
31	requests	2.18.4	691688
32	pip	8.0.3	641461
33	bandersnatch	2.1.3	611731
34	pex	1.2.6	602224
35	setuptools	0.9.8	586371
36	pip	8.0.2	514458
37	setuptools	25.2.0	451626
38	setuptools	18.5	403875
39	setuptools	38.5.1	401221
40	setuptools	33.1.1	400960
41	setuptools	38.4.0	350511
42	requests	2.6.0	341124
43	setuptools	38.2.4	296607
44	setuptools	12.2	272078
45	pex	1.3.2	251401
46	setuptools	12.0.5	218054
47	setuptools	36.5.0.post20170921	198229
48	pip	8.1.0	158730
49	OS	null	157984
50	pex	1.2.10	153304
51	setuptools	2.2	138811
52	setuptools	38.2.3	131705
53	setuptools	5.5.1	126601
54	pip	7.0.3	112030
55	setuptools	32.3.1	109628
56	setuptools	18.2	108019

---

Aggregated install-specific counts (I misplaced the query):

installer	download_count
pip	459495648
bandersnatch	47833402
setuptools	16771448
Browser	2075401
pex	1348972
null	1249811
requests	1092691
Homebrew	241166
OS	157984
distribute	143050
devpi	130353
Artifactory	111361
conda	12808
z3c.pypimirror	1

---

Is everyone on the latest Pip version?

Well, almost. :)

Does anyone still use easy_install?

Looks like setuptools is still a player in PyPI downloads.

How many people download packages from the web browser?

It's ~1% of just pip 10.0.1's usage and 0.45% of pip's usage.

---

I've not inspected package-specific/format-specific details though.

[pradyunsg]

Google BigQuery is useful. :)

The following query gives the result of how many downloads from PyPI happened from various clients (and their versions), in the past month:

SELECT
   details.installer.name, details.installer.version, COUNT(*) as download_count
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -1, "month"),
    CURRENT_TIMESTAMP()
  )
GROUP BY
  details.installer.name, details.installer.version
ORDER BY
  download_count DESC, details.installer.name ASC, details.installer.version DESC
LIMIT
  100

Running this at May 17, 2018, 12:05:13 PM IST, I got:

Click to expand

Row	details_installer_name	details_installer_version	download_count
1	pip	9.0.1	134582699
2	pip	10.0.1	118004637
3	pip	6.1.1	62803887
4	pip	9.0.3	47139548
5	pip	1.5.4	23256855
6	bandersnatch	2.0.0	22488790
7	pip	8.1.1	20827275
8	pip	6.0.8	11375879
9	bandersnatch	2.2.0	10311977
10	pip	10.0.0	10175888
11	bandersnatch	1.11	9716671
12	pip	8.1.2	9250362
13	pip	7.0.1	4767135
14	pip	1.5.6	4630838
15	bandersnatch	2.2.1	4495470
16	setuptools	39.1.0	3436419
17	setuptools	39.0.1	3084099
18	pip	7.1.2	2952558
19	Browser	null	2075401
20	pip	10.0.0b2	2000224
21	pip	9.0.2	1848474
22	pip	1.5.5	1488433
23	null	null	1249811
24	setuptools	28.8.0	1191750
25	setuptools	36.2.7	1075139
26	pip	1.4.1	999395
27	setuptools	20.7.0	980978
28	pip	7.1.0	970524
29	setuptools	3.3	863428
30	pip	9.0.0	699385
31	requests	2.18.4	691688
32	pip	8.0.3	641461
33	bandersnatch	2.1.3	611731
34	pex	1.2.6	602224
35	setuptools	0.9.8	586371
36	pip	8.0.2	514458
37	setuptools	25.2.0	451626
38	setuptools	18.5	403875
39	setuptools	38.5.1	401221
40	setuptools	33.1.1	400960
41	setuptools	38.4.0	350511
42	requests	2.6.0	341124
43	setuptools	38.2.4	296607
44	setuptools	12.2	272078
45	pex	1.3.2	251401
46	setuptools	12.0.5	218054
47	setuptools	36.5.0.post20170921	198229
48	pip	8.1.0	158730
49	OS	null	157984
50	pex	1.2.10	153304
51	setuptools	2.2	138811
52	setuptools	38.2.3	131705
53	setuptools	5.5.1	126601
54	pip	7.0.3	112030
55	setuptools	32.3.1	109628
56	setuptools	18.2	108019

---

Aggregated install-specific counts (I misplaced the query):

installer	download_count
pip	459495648
bandersnatch	47833402
setuptools	16771448
Browser	2075401
pex	1348972
null	1249811
requests	1092691
Homebrew	241166
OS	157984
distribute	143050
devpi	130353
Artifactory	111361
conda	12808
z3c.pypimirror	1

---

Is everyone on the latest Pip version?

Well, almost. :)

Does anyone still use easy_install?

Looks like setuptools is still a player in PyPI downloads.

How many people download packages from the web browser?

It's ~1% of just pip 10.0.1's usage and 0.45% of pip's usage.

---

I've not inspected package-specific/format-specific details though.

[pradyunsg]

^ this was fun.

[theacodes]

This is super cool. Does pipenv not identify itself separately from pip?

…

On Wed, May 16, 2018, 11:52 PM Pradyun Gedam ***@***.***> wrote: ^ this was fun. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#58 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc4JzAfhc9ktr_h_npmI1qEG0mbN3ks5tzR4_gaJpZM4DNOqT> .

[pradyunsg]

Does pipenv not identify itself separately from pip?

It uses pip to communicate with PyPI so, I don't see how we can separate the two. :/

[theacodes]

Hrm, there should be some way for us to do that. WDYT, @kennethreitz?

[kennethreitz]

Not that I know of. We just script pip.

[theacodes]

@pypa/pip-committers would it be possible to add an environment variable to override / augment pip's user-agent that pipenv could set? I think it's extremely valuable for us to be able to measure pipenv adoption.

[kennethreitz]

that's a @dstufft question :)

[theacodes]

Fairly trivial change to pip to make this happen pypa/pip#5424

If accepted, I'll also send a PR to pipenv to set this when invoking pip.