Think through how to use Dredd against production deployments

[honzajavorek]

I'm filing this issue as a base for a debate and design decisions around runnning Dredd against production deployments of APIs. Dredd was developed as an API design tool and thus offers behavior which suits to the development phase. Using Dredd in production brings different requirements, possibly even contradictory to the ones in place.

We experimented with this use case in the past and there's this Post-Deployment tab in Apiary, which is able to run Dredd as a service, against any publicly accessible deployment:

This part of the product never made it to the general availability really, and is planned to be superseded by something more shiny, but something still featuring running Dredd against production.

At the same time there are feature requests like #1019 from @erajkovic and AFAIK our very own @apiaryio/adt uses Dredd as a post-deployment check.

I'd like to discuss challenges here and possibly to design a solution to this problem or to decide this isn't something Dredd should do at all.

Challenges

Some challenges I see from the top of my head:

• Dredd modifies real data. Should it only run safe HTTP methods, such as GET?
• When the API description had not been used for design, but it is created ex post, it is often incomplete. While normally Dredd follows the document as a spec to test, in this case it is expected to make assumptions and test those: Add an option to use field_name as default value when missing  #1019 While this is similar to "generating your unit tests from code" and theoretically doesn't make much sense for contract testing, there are still reasons why people want to do this:
  • Production deployment can influence how the API behaves (auth, gateways, ...) and people want to ensure it still behaves as described
  • Smoke tests
  • Even though the API description document was generated from code, they want to ensure it is up-to-date and there are still no discrepancies between the implementation

[kylef]

AFAIK our very own @apiaryio/adt uses Dredd as a post-deployment check.

In ADT our services are idempotent and stateless which makes this pretty simple for us. Given same input we always return same response. There is no database or state.

[pksunkara]

Also, if we want to run dredd post-deployment, we would need to make API Blueprint better at data representation with multiple samples and the option to choose which sample.

[honzajavorek]

@pksunkara Do you think this should be solved on a level of language design? Is there a similar feature in OpenAPI? IMHO it could be solved by having multiple API descriptions for every environment and DRYing them out somehow (Hercule, apiaryio/api-blueprint#20).

[pksunkara]

Different descriptions for different environments is an entirely separate problem which would be solved by the transclusion mentioned above in your comment. But the issue I am talking is a different one which can only be better represented by an example.

For example, I am running a Blog service and I want to test it's API in production. Since we are dealing with real data in production (even though it maybe a demo account), that would mean we can't mock the data. Let's model our data structures:

# Blogpost
+ post: A sample post in our blog (string)

I have 2 endpoints dealing with the above Blogpost model. One is viewing a blog post (GET /posts/1234), another is editing a Blogpost (PATCH /posts/1234)

Now, testing the first endpoint on real world data is easy. We just need to have a demo account with a demo post "A sample post in our blog". But how would you test the second endpoint? Let's say the request of the second endpoint that is sent by dredd is as follows:

{
  "post": "Modified post"
}

Then, the actual blog post will change and also the response returned will be different than the sample which you can assume from the Blogpost data structure in API Blueprint. This is what I would call the "Multi Sample Problem".

[honzajavorek]

In the original post I mention a possible solution:

Dredd modifies real data. Should it only run safe HTTP methods, such as GET?

Dredd is able to test what you describe, using hooks and asserting over a schema. But you have a good point that to make testing on top of real data useful, we need to be able to test workflows. Current Dredd testing is conceptually an isolated unit testing, which isn't possible nor really desired on production.

[w-vi]

In my previous job we used to have complex live testing and needed to test workflows as it was a web based game server for flash games. We had test scenarios where two bot players played againts each other in a predictable way. We used to have a simple python based framework which interpreted the scripts it was fairly specific but used to have variables (you could save a value from a response for later use) so that you could accomodate dynamic values like ids etc . Otherwise it was similar to Dredd as it validated the expected to real results of calls although it allowed for regex expressions instead of values. I think for Dredd most of the code is there the only bit missing is the ability to chain calls. Hooks can save values for later use and replace values before the calls as well as validate so I think we should focus on making the chain of calls easy to use. I don't think that API Blueprint or Swagger file suffice in this case as an input I think you need extra input describing the flow.

[honzajavorek]

With @netmilk we had an idea Dredd would have better separated the concerns of constructing the list of transactions and the transactions running.

Imagine Dredd could get a list of transactions representing a workflow instead of an API description, and whatever it gets, it replays it on top of the tested API, asserting the expected responses.

And there are even already some formats able to capture a plain collection of transactions (Postman).

So implementing workflows, in a primitive form, isn’t so impossible, it just needs some decoupling in the codebase.