build a proper API

[chadwhitacre]

Current: Ad-Hoc

https://github.com/gittip/www.gittip.com#api

Option 1: api.gittip.com

Like Twitter, GitHub, etc.

Option 2: Reddit-Style

This would be done w/ Aspen by converting all pages to negotiated simplates.

Surfaced with @clone1018 in IRC.

We do this for the stats page, using negotiated simplates:

https://www.gittip.com/about/stats.html
https://www.gittip.com/about/stats.json

Cf.:

http://www.reddit.com/
http://www.reddit.com/.json

http://www.reddit.com/user/whit537
http://www.reddit.com/user/whit537.json

[sigmavirus24]

I'll definitely work on this. Expect a pull request within a month or so (ideally)

[chadwhitacre]

@sigmavirus24 I would say start it as a repo under your account, and we can move it under the zetaweb organization when the time comes.

[chadwhitacre]

@sigmavirus24 Actually, I want you to assign this ticket to yourself. May I make you a zetaweb collaborator?

[sigmavirus24]

@whit537 I was thinking of doing the Reddit style API which is why I forked the repo. I would be honored if you added me and then I'll work on a separate feature branch.

[sigmavirus24]

So looking at #449, I'm wondering a couple things: are planning to use the API for Gittip.com as well or just maintain it for external purposes?

[chadwhitacre]

@sigmavirus24 It seems to me that api.gittip.com should be a service layer that is used by Gittip.com itself as well as third parties. No?

[sigmavirus24]

That's fine, I just didn't realize that was part of the plan. So in short you want

+---------------------------------------+
|           api.gittip.com              |
+---------------------------------------+
     ^                  ^
  ___|_______           |
| Gittip.com |          |
+-------------+      ___|__________
                    | Everyone else |
                    +---------------+

And the API should return:

• user information (authenticated and not)
• tips:
  • list
  • create
• account information (authenticated only)
• stats

And what else?

Also, how do we authenticate the user. Currently users can only sign in via Twitter or GitHub. OAuth? (OAuth2?)

[sigmavirus24]

So the reason I ask about authentication is because we authenticate users ourselves. How would a user otherwise authenticate with Gittip though (mainly via the API). Let's say we (or someone else) built a mobile (or desktop) app based on the API. Do they first get the user to authenticate via Twitter or GitHub first? This seems a bit unwieldy. I think there are other apps which already do this, but it just feels awkward to me. In that case, do we offer our own basic authentication (login/password) to make it simpler for a user to activate an OAuth token for the app? If we start doing our own OAuth for the API, do we incorporate #455 with this issue?

[chadwhitacre]

Right, that's a lot of work. Do you think we should tackle OAuth first?

[sigmavirus24]

I'm not entirely sure. I'll have to look more into aspen before giving you a solid response. Half of the time, I post questions to myself as well as to others. No one usually answers my questions anyway ;)

[lyndsysimon]

"No one usually answers my questions anyway ;)"

There's an easy solution to that - ask easier questions!

[sigmavirus24]

But that takes all the fun away from asking them!

[chadwhitacre]

:-)

!m @sigmavirus24
!m @lyndsysimon

[sigmavirus24]

@lyndsysimon since you seem interested in this as well, let me bounce an idea off of you (if you don't mind).

Even if we don't become an OAuth provider, I would think using OAuth secret/key pairs would be the best way of authenticating against the API. My concern is how the user/application developers obtain this. Obviously the user has to authenticate but how do they do so? Should we add the capability for "basic" (login/password) authentication? I'm trying to imagine all possible ways of using the API, one being a CLI. There's no way (assuming working on a remote machine) to open a page to "redirect" back to the application. Do we force users to procure their own secret/key pairs from Gittip.com?

The former seems reasonable, but then do we enable registrations that way? I would prefer not to. I like the way we encourage registrations now. Do we then only allow registered users to set up a username and password? (This is what I'm leaning towards, honestly.)

The latter seems unreasonable since keys are 20 or 24 characters (I can't remember which) and secrets are twice that length. That seems a bit of a labor for a normal user.

Obviously we'll need a separate endpoint for retrieving an authorization. The question then also becomes, since we're providing this alternate way of authenticating, do we allow it's user for the rest of the API or do we enforce the use of OAuth for the rest? I'd rather use the latter, especially since I don't think we should allow basic authentication on the web site either.

And this all almost seems like we're bulking up Gittip just to add the API. :-/

[chadwhitacre]

And what else?

Funds (#449). I think that's going to be an important part of the API, because it will enable an ecosystem of third parties implementing their own arbitrary algorithms for computing funds.

[chadwhitacre]

+1 from @hamstah in #sqlalchemy.

[sigmavirus24]

@whit537 +1 about?

---

Endpoints

A running list (I'll probably just update this comment for future reference):

• /authorizations (for OAuth purposes) [always requires authentication]
  • POST to it to create a new one, requires basic authentication
  • GET to receive a list of existing ones
  • /:id
    • GET to retrieve a specific one
    • PATCH to update
• /funds [always requires authentication]
  • GET retrieves a list of authenticated user's funds
  • POST creates a new fund
  • /:id
    • GET retrieves a specific fund
    • PATCH updates it
• /on {/github, /twitter, /google, /facebook} [public]
  • GET retrieves a list of all users (names only?) on each service
  • POST creates a new one (this is mostly for Gittip.com only) [again only internal to the API]
  • /on/:service/:id
    • GET retrieves just that user's information [public]
    • PATCH updates that user's information [requires authentication]
    • /account [requires authenticated]
      • GET would retrieve the account information for the authenticated user, credit card/bank account? (I'm not comfortable returning those necessarily)
      • PATCH would all the user to update the account information
• /tips [requires authentication]
  • GET a list of authenticated user's tips (or at least the tippees)
  • PUT a new tip
  • /tips/:id (is this even doable?)
    • GET would return the tip in specific
    • PATCH would update the tip
• /stats, GET only [public]

JSON representations

• A single authorization contains:
  • a unique id
  • a unique token/value
  • the date it was created
  • the date it was last modified
  • name of the application it is for
• A single fund should return:
  • a unique id
  • the name of the fund
  • the date it was created
  • a dictionary/hash of the users in the fund and their percentages, and the services they belong to
  • current balance of the fund
• A single user should return:
  • whether or not they have a valid credit card linked
  • whether or not they have a bank account linked
  • date they joined Gittip
  • current balance
  • their current balance
  • personal funding goal
  • personal statement
  • list of connected accounts
  • if the requesting user is authenticated, this should show the amount the authenticated user is tipping (if any)
• A single tip should return:
  • it's id
  • tippee
  • amount
  • ??
• Stats are already defined.

[sigmavirus24]

Also, we'll need some good validation for descriptive error messages (#479). The other thing is with /authorizations I'd like to suggest a page under a user's account where they can manage (revoke) existing authorizations.

[lyndsysimon]

(I wrote the below, and had to leave before fully proofreading it. My apologies if it's fractured. The short version is: I propose that we allow users to create API IDs and API keys in the web interface first. Those can be used for the API. Then once the API is complete and there is a need for it, we can implement a single endpoint to allow OAuth access to get the ID/key - without having to support OAuth throughout the API.)

Even if we don't become an OAuth provider, I would think using OAuth secret/key pairs would be the best way of authenticating against the API. My concern is how the user/application developers obtain this. Obviously the user has to authenticate but how do they do so? Should we add the capability for "basic" (login/password) authentication? I'm trying to imagine all possible ways of using the API, one being a CLI. There's no way (assuming working on a remote machine) to open a page to "redirect" back to the application. Do we force users to procure their own secret/key pairs from Gittip.com?

I think requiring OAuth for the API is unnecessarily weighty, from both sides. On the server, we're having to add code to support the authentication flow, and store/update OAuth tokens associated with an account. On the client, they're having to support redirection, which could be an issue with simple clients, like a nice wrapper around Curl. One might, for instance, want to hit the API from a cron job to find your account balance, which might then be displayed in a Conky widget. That would be hard to write if you had to deal with auth redirections.

The former seems reasonable, but then do we enable registrations that way? I would prefer not to. I like the way we encourage registrations now. Do we then only allow registered users to set up a username and password? (This is what I'm leaning towards, honestly.)

Registrations for Gittip? I say no - Gittip doesn't have user content at this point to speak of, and I think that's a good thing. It's not about what you write on Gittip, it's about your contributions on other sites.

The latter seems unreasonable since keys are 20 or 24 characters (I can't remember which) and secrets are twice that length. That seems a bit of a labor for a normal user.

I agree that it's cumbersome, but it's not uncommon. Thinking of things I've used recently that hit APIs, everything I can think of that ties into Eve Online's API uses this method. Example: Eve-Kill.net's Killboard

Obviously we'll need a separate endpoint for retrieving an authorization. The question then also becomes, since we're providing this alternate way of authenticating, do we allow it's user for the rest of the API or do we enforce the use of OAuth for the rest? I'd rather use the latter, especially since I don't think we should allow basic authentication on the web site either.

I disagree. If we're using a "secret key" method, then the user's username (or API ID) and API key would be sufficient. That could be sent with any request over SSL, and would relieve the server of the requirement to maintain state. If we use OAuth, then we can assign a temporary API key that the client must renew by re-authenticating.

Actually, now that I think about it, that seems to be the best way to me. Require an API ID and key for every transaction to a secured resource. Implement a single endpoint that will allow the client to use OAuth to obtain an API ID and key, along with a user-accessible web page to do the same. That way, normal requests are all stateless, developers can write hosted applications on the Gittip API without touching user credentials, and people wanting to develop on their own can just grab the API ID and key from the page.

In addition, this would provide flexibility for more complex authorization schemes in the future - users could have multiple API IDs, each with permissions to access different parts of the API.

And this all almost seems like we're bulking up Gittip just to add the API. :-/

Indeed - but that's because of the way we're growing. The web interface drives the growth of the application. If we implemented features in the application first, adding an API would be simple. On the other hand, if we implement something through a web interface, we get time to think about it and experience to build upon by the time we get around to moving it to the core of the application itself - the quality of the core reflects this.

[sigmavirus24]

I think requiring OAuth for the API is unnecessarily weighty, from both sides. On the server, we're having to add code to support the authentication flow, and store/update OAuth tokens associated with an account.

Well we're already talking about becoming an OAuth provider, so assuming we do become one, it wouldn't be as weighty as you suggest.

For simple curl requests, I'm thinking we can also use the basic tokens that GitHub uses. They could be sent as an Authorization header.

As for adding authorization flow to Gittip, this would ideally be handled by Aspen. Other frameworks (e.g., flask) handle this for the user.

Registrations for Gittip? I say no

We agree. :)

Require an API ID and key for every transaction to a secured resource.

This is how a few APIs (that I know of) operate and frankly how I intended this workflow to be used here.

users could have multiple API IDs, each with permissions to access different parts of the API.

Again, I should have been more explicit, each application seeking to authenticate its user should have it's own authorization. This would also allow us to track rogue applications that might be stealing from users.

---

To sum up (and be more explicit than before):

• I'm thinking only about a RESTful API
• In regards to authentication:
  • Non-Gittip.com applications should use OAuth for authentication
  • All authorizations created by the user for authentication purposes should be viewable via the web interface and revokable from there as well

My one question remains: is there a simple way to allow a user to authenticate with Gittip besides having to redirect from Twitter or GitHub (or future services)?

---

Finally, one nit-pick:

One might, for instance, want to hit the API from a cron job to find your account balance, which might then be displayed in a Conky widget.

Assuming you mean the amount Gittip lists you as current receiving, then this should be accessible through non-authenticated requests.

[chadwhitacre]

Convo: https://botbot.me/freenode/gittip/1796418/

Topics:

• whether api.gittip.com is a separate repo
• whether Gittip.com depends entirely on api.gittip.com

[sigmavirus24]

Thanks for linking this Chad.

[jeromegn]

Mostly this: http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api

[nairboon]

#402 and #977 ask for a history API, something like
/on/:service/:id/history for a list of received tips

[chadwhitacre]

I'm fine with our ad-hoc API as it's developing. We actually have clients being built now so a change at this point would be harder to pull off. Let's revisit in a couple years.

[eric-s-raymond]

Just a note that while the API is probably of little practical importance, being seen to have an API is crucial to gittip's positioning.

Truth: "where's the JSON RPC interface?" was one of the first questions I had about the site, even though I was well aware there simply wouldn't be that much it could do.

[chadwhitacre]

@eric-s-raymond Note that we do have a published JSON API, with a few clients already started:

https://github.com/gittip/www.gittip.com#api