Determine new API URL structure for warehouse (starting with new JSON API)

[ctheune]

At the PyCon2014 sprint I have started to make bandersnatch easier to cache. This means moving away from XML-RPC in general.

I'm leveraging the existing /pypi//json API which already helps, but I'll need two more endpoints:

• get a list of all packages and their most recent serial
• get the changelog

I implemented the necessary code on a branch for PyPI:
https://bitbucket.org/ctheune/pypi/branch/ctheune-bandersnatch-json

However, I don't wanna force this through but have a decision how the URLs should look like.

Ideally we can implement this in both warehouse and PyPI in a way that bandersnatch can support both of them without breaking when you guys switch the public server (and I might be on vacation. ;) )

[dstufft]

So I have some ideas on both a new API for accessing data about PyPI and also some rough ideas for a new mirroring API in general. I'll take a look at what you have so far.

[dstufft]

So if I read your PR correctly, the new URLs would be https://pypi.python.org/json/changes and https://pypi.python.org/json/packages? If that's the case then I'm not really a big fan for adding those in Warehouse.

Ideally what i'd like to do is get a nice hypermedia based API setup probably rooted at /api/. Using something based on https://jsonapi.org/ is a possibility. There are a few options and need to dive into it to figure out what exactly needs done. Ideally the new API will also replace the existing JSON api and we can deprecate the old JSON api (but leave it in place until (or if!) it's no longer getting traffic.

[r1chardj0n3s]

I echo @dstufft in this. The question I have is whether we go all the way to /api/v0/ to future-proof us a little too. Unless we'd be happy with /api-v1/ or similar later on?

[dstufft]

So there are two ways to deal with that, one way is to version using the content type, so it's always /api/ but it'll select the version based on the content type, github uses this like: Accept: application/vnd.github.beta+json or Accept: application/vnd.github.v3+json. The other way is to do /api/v0/ etc. I lean towards using the content type but we'll need to figure out in general how we want to handle versioning going forward and how the code to handle that looks like.

[steveklabnik]

Just let me know if I can help out at all regarding JSON API stuff.

[brainwane]

As I understand it, this issue (designing and implementing a new Warehouse API) is a prerequisite for integrating twine into pip and thus dealing with pypa/packaging-problems#76 and pypa/packaging-problems#60 , per @dstufft's comment in pypa/twine#127. Is that correct? If so, I'd suggest we add this to one of our upcoming milestones.

[brainwane]

We talked about this issue in today's bug triage meeting and folks explained to me: Even though this may be necessary for some Twine improvements, this is not a ticket we will address before launch. This is a new feature and is best suited for post-launch; Warehouse needs to be done before we can improve twine.

[phildini]

Hello! Quick call-out that some of us would really enjoy the JSON API containing info like owners/maintainers before the XMLRPC API is shut down. See ticket linked right above. Cheers, thanks for all your work!

[brainwane]

I've marked #2914 as something we should address before shutting down legacy PyPI, but developing the structure for the new API can wait till after we shut down the legacy site.

[brainwane]

As we develop the new API we should consider #347 as well. And I've added this issue to the list of things we might work on at the PyCon sprints.

[theacodes]

I would also love an API for managing both my account and my projects. For some examples of where this is useful:

1. We have an account that owns all of the projects our organization publishes. I want to rotate its password every week.
2. Likewise, I want to audit all of my organization's project and verify that no more than n people have admin access to it.
3. I am actually in the process of migrating all of my projects in my personal account to a new account. It would be cool to do that programmatically.

I'm happy to help with the design and discussions around this (my day job is helping design APIs and implement clients for Google Cloud Platform).

[di]

I am actually in the process of migrating all of my projects in my personal account to a new account. It would be cool to do that programmatically.

We could probably just call this "the ability to add/remove collaborators via API" I think, since actual account migration is probably not something that happens very often.

[dstufft]

I'm hoping to carve out some ideas on this soon, maybe next week? Ideally the output of this ticket is the basic framework/skeleton of the API, and then further tasks can extend the functionality of it.

Defining APIs for PyPI is a tad bit trickier than the general case, because we generally have to design for a decade+ (for instance, XMLRPC got added, but it has not aged or scaled well! From my investigations so far, GraphQL would be a similar mistake). I'm almost certain that something Hypermedia based is the way forward here, but there's a lot of different ways to take that, we'll also need to be ensure to include all of the typical scaling things like pagination and the like.

[theacodes]

We could probably just call this "the ability to add/remove collaborators via API" I think, since actual account migration is probably not something that happens very often.

Yep just calling a a specific use case.

I'm hoping to carve out some ideas on this soon, maybe next week? Ideally the output of this ticket is the basic framework/skeleton of the API, and then further tasks can extend the functionality of it.

Sounds good, happy to review and be around to bounce ideas off of (I'm on IRC during PST working hours as thea).

Hypermedia based is the way forward here, but there's a lot of different ways to take that, we'll also need to be ensure to include all of the typical scaling things like pagination and the like.

Agreed - REST/JSON (and to some extent RPC/JSON) has more or less stood the test of time (in tech years, at least). Happy to provide feedback on that sort of stuff as well.

[dstufft]

@theacodes I guess I'm just not seeing what an IDL actually gets us here? The example of JSON Hyper Schema has JSON-Schema as part of it, it's just instead of your client hardcoding URLs and actions all over the place, it can discover them at runtime. You can also ship them as part of your client so that a network access isn't required in the common case (unless you introduce a new schema).

[dstufft]

@theacodes If it would be helpful, I'm happy to jump on a call or into IRC to go over the two things in a higher bandwidth setting instead of throwing github comments back and forth. I feel like there are probably some misconceptions on both sides about RPC and Hypermedia, and perhaps a higher bandwidth mechanism would help to work out what those are?

[theacodes]

I don't want to hold up progress. I would love to see a design doc or proof of concept if/when we have one.

[asmacdo]

POC is up #4078, I'm using this etherpad to document design proposal.
https://pad.sfconservancy.org/p/hypermedia_api_design

I did my best to incorporate the ideas discussed here, as well as in person discussions at pycon. I've set aside some time to keep working on this, so all feedback is welcome.

[brainwane]

Per discussion in IRC just now -- the author closed #4078 last year, and it's unclear whether this kind of feature would be welcome if someone were to try again at implementing it.

[brainwane]

Maintainers' opinions are welcome. Also, in my opinion, it would be easier to finalize design and implementation, test and review, and deploy this if we had funding for it.

[brainwane]

@asmacdo that Etherpad has now dissolved and reset -- did you keep a copy of your design proposal anywhere else?

Reminder to others that work on this could probably use funding.

[asmacdo]

@brainwane unfortunately I dont have a backup , but the PR could still be distilled down into a design proposal.

Key points:

• Create a resource based, hypermedia API
• Use Marshmallow to serialize
• use apispec to generate OpenAPI schema

Additional necessary features

• Pagination
• Filtering
• CDN caching (consider how pagination/filtering will affect)

[di]

Bit of a related update here: PEP 691 has been accepted.