Metadata insertion plugin

[JnMik]

Summary

We want to be able to add metadata on our customers.
These metadata could be forwarded to upstream APIs when request pass through Kong.

Our most basic use case : We proxy multiple 3-party APIs and want our customers to have one single API key. Their 3-party API key would be hidden in their metadata, and passed along with the request.

Apply metadata-insertion plugin on an API endpoint

The metadata-insertion plugin can be added on an API with a configuration like this (it looks like request-transformer plugin):

config = {
          add = {
            querystring = {
              "var1: %value1%",
              "var2: %value2%"
            },
            headers = {
              "var1: %value1%",
              "var2: %could_be_value2_from_transitory_store_if_provided%"
            }
          }
        }

Placeholder replacement order

The %placeholder% for metadata will be resolved based on three data source (cassandra, redis, transitory store)

What is the transitory store ?

Transitory store allows other plugins to prepare metadata on the fly when kong receive a request.
(View Tieske suggestion here #550)
One of the test cover the use case by feeding the transitory store with fixtures.

Plugin flow

To add metadata in database, POST request can be done on /consumers/:username_or_id/metadata/

• Finish Cassandra DAO #1 First the plugin verify if the request is done by an authenticated consumer. If not, the plugin force an HTTP_UNAUTHORIZED response to indicate that Metadata is necessary on that api endpoint. (We do not want the upstream API to receive incomplete request).

• How to distribute (release process) Kong? #2 Then the plugin loads the persistent metadata of that user in memory (Redis if already in memory, fallback on cassandra).

• /bin/kong global script #3 Then the transitory store is loaded as well in memory, replacing persistent metadata if same key already exist.

• Polishing the controllers + more integration tests #4 Finally %placeholer% are replaced, so the metadata are added to target Headers & Querystring.

Debug tool

I've added a debug.lua library for my POC, I understand that I should probably remove it if the pull request ever gets approved.

Headers & Querystring factory

Also I believe my request_headers_factory and request_querystring_factory could be used in the request-transformer plugin.

Full changelog

Implement metadata-insertion plugin
Comments are welcome, i've never done Lua / OpenResty before.

Issues resolved

Fix #550

Edit: Changed my summary to explain the big picture of how the plugin works.

[JnMik]

@thibaultcha Sorry for the lost of comment history, I was struggling with rebase so I created a new branch

Ref: #1961

[thibaultcha]

So from #1961:

On a side note, I am quite unsure how I feel about such a plugin. I do understand the promise, but am clearly questioning if the approach as a plugin is the best one.

The rest of the comment history does not apply anymore.

[JnMik]

So let's get back to business now that my PR git history looks ok

@thibaultcha Can you tell me more about what other approach is possible ?
I really want to be able to use metadatas with Kong, and easily keep my production environment updated as the new tags gets out (without the need to recompile my custom plugin etc).

Thx !

[JnMik]

I believe having it as a plugin offers flexibility.

And allowing other plugins to be able to also feed the ngx.ctx.transitory_store seems a great value for people who don't want persistent metadata but instead doing some parsing over the actual request or any other scenarios possible. @Tieske seemed to have some idea in mind regarding this.

[thibaultcha]

I am most of all reluctant to adding more plugins before implementing the plugins development API, which, among other things, could implement such helpers around the ngx.ctx table for example. It is an API we have been talking about for a long time, but the more plugins we add, the longer it will take us to implement it and the more plugins we will break when doing so...

[JnMik]

I see ! Seems like a legit meditation.
Is there any ETA for that plugins development API ?
Will it be simple to use and develop for it ?
How much time you think it could take for the refactor of one plugin ? (Like this one)

I'm just trying to figure out the future here, because I might be forced to find another metadata solution if it takes months to implement, we all have commitments to deliver hehe

Let me know !
thx

[thibaultcha]

Hi,

We shall not accept community plugins in the core anymore. Feel free to publish this plugin on your own following the distributing your plugin guide. You are also encouraged to advertise it on the mailing list, and we will soon provide a "community plugins" section on getkong.org/plugins/ on which you will be able to document it.

In the future, we plan to take out of this core repository all of the plugins we currently have as well.

Thank you!

[thibaultcha]

The main drawback here is the maintenance overhead such "alien" plugins add to the Kong team.

To answer your questions @JnMik:

Is there any ETA for that plugins development API ?

Probably some time this year.

Will it be simple to use and develop for it ?

Yes, very!

How much time you think it could take for the refactor of one plugin ? (Like this one)

For maintaining a given plugin with the upcoming changes, one might have to move a bit of logic here or there until we declare it stable (thinking when we will add new "contexts/hooks" like access(), etc...).

In terms of the Public Lua API (things like get_body(), set_uri() etc...), use of the raw ngx API should still be supported, so possibly no maintenance overhead. However the Kong API will try to be more efficient by providing a layer of caching between user calls and actual ngx calls on a per-request basis.

A bigger concern might be the DAO and the model. We will very likely change them quite a lot to improve and make Kong more flexible. This might mean that Consumers metadata will end up in the core model anyways 😉

[JnMik]

Not sure anymore if this is worth distributing my own custom plugin or wait for the consumers metadata to be in the core model. It tooks 4 months receiving a follow up here, I feel it could take years before the metadata shows up in the core model lol

I'll think about it.
thx

[thibaultcha]

We have a lot on our plate. (referring to why the model is yet to be rewritten)

We had uncertainties and decisions to make before coming back to the currently proposed plugins.

[thibaultcha]

I personally think it is worth it, since there is no support for such a feature in Kong itself yet, and as you highlighted, there might not be for a little while, still, and I know some users would be interested. Up to you!

[bskiefer]

@JnMik Did you abandon the project? Can't believe we still don't have it and it's been 2 years