Added rustc_version to user agent data #10
Conversation
|
Failing CI appears unrelated to the contents of the PR. |
|
It's due to pypa/pip#9644, #11 will fix it for now. |
|
Ah, ok. Will rebase after that's merged. |
|
Looks like the tests need updated here. We'll also need to update the BigQuery table schema before we can merge this to add the new column. |
|
Is the BigQuery table schema in VCS somewhere I can send a PR, or is that something an admin does on the backend? |
|
@di lost track of this and now following up, is there anything I can do to help move the ball forward here? (Happy to rebase if that's the next step!) |
|
Sorry to leave you hanging here:
It is not, this is something that a admin would do on the backend.
I think next steps would be :
|
|
Will get this updated later today! |
|
Ok, rebased and test case added. |
|
Unfortunately CI seems to be busted, |
|
Before merging we need to add a new nested column to the |
|
Tried to do this, seems I don't have permissions on the public dataset :) ewdurbin@cloudshell:~$ bq show --schema --format=prettyjson bigquery-public-data:pypi.simple_requests > simple_requests-schema.json
ewdurbin@cloudshell:~$ bq show --schema --format=prettyjson bigquery-public-data:pypi.file_downloads > file_downloads-schema.json
ewdurbin@cloudshell:~$ cp simple_requests-schema.json simple_requests-schema-orig.json
ewdurbin@cloudshell:~$ cp file_downloads-schema.json file_downloads-schema-orig.json
ewdurbin@cloudshell:~$ vim simple_requests-schema.json
ewdurbin@cloudshell:~$ vim file_downloads-schema.json
ewdurbin@cloudshell:~$ diff -C 4 simple_requests-schema-orig.json simple_requests-schema.json
*** simple_requests-schema-orig.json 2022-03-17 09:54:08.037747262 +0000
--- simple_requests-schema.json 2022-03-17 09:54:50.337703919 +0000
***************
*** 108,115 ****
--- 108,119 ----
},
{
"name": "setuptools_version",
"type": "STRING"
+ },
+ {
+ "name": "rustc_version",
+ "type": "STRING"
}
],
"name": "details",
"type": "RECORD"
ewdurbin@cloudshell:~$ diff -C 4 file_downloads-schema-orig.json file_downloads-schema.json
*** file_downloads-schema-orig.json 2022-03-17 09:54:16.387738706 +0000
--- file_downloads-schema.json 2022-03-17 09:55:05.189688701 +0000
***************
*** 131,138 ****
--- 131,142 ----
},
{
"name": "setuptools_version",
"type": "STRING"
+ },
+ {
+ "name": "rustc_version",
+ "type": "STRING"
}
],
"name": "details",
"type": "RECORD"
ewdurbin@cloudshell:~$ bq update bigquery-public-data:pypi.simple_requests simple_requests-schema.json
BigQuery error in update operation: Access Denied: Table bigquery-public-data:pypi.simple_requests: Permission bigquery.tables.update denied on table bigquery-public-
data:pypi.simple_requests (or it may not exist).
ewdurbin@cloudshell:~$ bq update bigquery-public-data:pypi.file_downloads file_downloads-schema.json
BigQuery error in update operation: Access Denied: Table bigquery-public-data:pypi.file_downloads: Permission bigquery.tables.update denied on table bigquery-public-
data:pypi.file_downloads (or it may not exist). |
|
Schema files for reference: file_downloads-schema.json.txt |
|
Yeah, this needs to be done by the public datasets team. I've reached out to them and directed them to this issue as well. |
|
The I think we're good to merge this? |
|
Fantastic! Makes sense to me. |
|
Let's see what happens! |
|
@di how long do you think it'd take before data either starts flowing, or errors start flowing 😬 |
|
It's auto-deployed and pretty much immediately live in the dataset: SELECT
details.rustc_version,
COUNT(*) AS total_downloads
FROM
`bigquery-public-data.pypi.file_downloads`
WHERE
DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)
AND CURRENT_DATE()
GROUP BY
details.rustc_version
ORDER BY
total_downloads DESC
|
Added to pip in pypa/pip#9987