feat(quay): add quay plugin

[fmenesesg]

#60
This is the first version of a Quay plugin that shows the tags, timestamp, last_update, and SHA of an image related to a backstage entity.

[tumido]

I haven't reviewed the code fully yet, however I have some architectural concerns here. Can you please explain me why do we need a backend plugin in here? It seems like it's just a API proxy which doesn't communicate with Backstage backend/database/catalog/anything at all. What is the justification for a backend plugin in here? Can't we either talk to Quay directly from the frontend or proxy it via the native Backstage proxy (similarly to what ArgoCD or Prometheus plugins do)?

[fmenesesg]

@tumido thanks for your feedback, and you are right, by now on this version, the backend plugin is just a proxy, but what I was thinking is that when we add the ability to get the vulnerabilities per tag, that logic should be on the backend plugin.

[tumido]

@fmenesesg hm, interesting thought, how would you imagine this being handled on the backend? As far as I know, Backstage has no notion of this kind in its catalog. Would we need to extend the database with a new model, perhaps? Is that what we want? Backstage's event processing is in its infancy as well currently, so any backend solution wouldn't be reactive, we would have to resort to a polling solution. So far it seems to me way like a way heavier and very complicated solution to implement something like that in backstage backend compared to fetching the data directly on frontend. If I compare this with for example Snyk integration, they also have a frontend plugin only. I think mainly because of the backstage backend not being capable/suitable for this type of processing.

However I may be missing something here. Do you have a solution in mind?

[fmenesesg]

@tumido what I have in mind was to mimic on the backend the same behavior that quay has on their code.

[tumido]

@fmenesesg well.. I'm afraid that adding that to Backstage backend serves not much of a purpose. The only benefit may be caching these results so we optimize traffic. However that would mean you'd either have to extend the backstage RBAC model or build a per-user cache. Either way I don't see a way how to create that easily in a safe and scalable fashion.

Quay itself is performing these queries on frontend... This is telling something about the UX of Quay API. I don't think we should solve that on Backstage backend.

[tumido]

I have 2 main concerns regarding the API calls aggregations into a backend plugin compared to many frontend requests.

Timeouts

The proposed solution via backend plugin results in following sequence, correct?

sequenceDiagram
    Backstage Front-end->>Backstage Back-end: /api/quayPlugin/getTags
    Backstage Back-end->>Quay API: /getTags?page=1
    Backstage Back-end->>Quay API: /getTags?page=2
    Backstage Back-end->>Quay API: /getTags?page=...
	Quay API-->>Backstage Back-end: Page 1 response
	Quay API-->>Backstage Back-end: Page 2 response
	Quay API-->>Backstage Back-end: Page x response
	Backstage Back-end-->>Backstage Front-end: Response

Loading

Imagine any of the queries from backend to Quay API erroring out. Backstage backend can either retry that query or return error. On error frontend would retry resulting in repeating all the queries again.

sequenceDiagram
    Backstage Front-end->>Backstage Back-end: /api/quayPlugin/getTags
    Backstage Back-end->>Quay API: /getTags?page=1
    Backstage Back-end->>Quay API: /getTags?page=2
    Backstage Back-end->>Quay API: /getTags?page=...
	Quay API-->>Backstage Back-end: Page 1 response
	Quay API-->>Backstage Back-end: Page 2 ERROR
	Backstage Back-end-->>Backstage Front-end: ERROR
    Backstage Front-end->>Backstage Back-end: retry /api/quayPlugin/getTags

Loading

If it is Backstage backend who retries a single query to Quay API, then there's a substantial risk of timing out the original request from backstage frontend.

sequenceDiagram
    Backstage Front-end->>Backstage Back-end: /api/quayPlugin/getTags
    Backstage Back-end->>Quay API: /getTags?page=1
    Backstage Back-end->>Quay API: /getTags?page=2
    Backstage Back-end->>Quay API: /getTags?page=...
	Quay API-->>Backstage Back-end: Page 1 response
	Quay API-->>Backstage Back-end: Page 2 ERROR
    Backstage Back-end->>Quay API: /getTags?page=2
	Quay API-->>Backstage Back-end: Page x response
    Note over Backstage Front-end: TIMEOUT

Loading

This is solvable by advanced caching or delayed processing/notifying user via webhooks/polling that the results are ready.

Request cancellations

Imagine user refreshing a page or leaves the page before receiving quay plugin response. The original backstage frontend to backstage backend request has to be cancelled. Than you'd have to create additional handler to cancel all pending async requests between Backstage backend and Quay API. Otherwise just a single user by hitting F5 multiple times can saturate the backstage backend bandwidth.

I'm not sure all of this work is a worthy investment just because of a single API endpoint. There are too many sideeffects that needs to be handled properly. I think the cost of maintenance is too big, compared to what you can already do via issuing multiple requests on the frontend side.

[fmenesesg]

Ok, I got what you said, I'm going to change the implementation to do something similar to what the Argo plugin does

[tumido]

@fmenesesg do you need any help to move this forward? I can contribute if you want. We'd love to have the plugin available. 🙂

[fmenesesg]

@tumido I'm out on vacation i will return on 2 weeks. If you want you can start with the removal of the backend plugin in order to use the built in proxy that backstage has. I will start working when I come back.

[tumido]

I gave it a rework (+rebase), removed the backend plugin and used the Proxy plugin instead.

That being said, it still needs a bit more work on CORS since all proxied requests now return:

API calls must be invoked with an X-Requested-With header if called from a browser

So.. Work in progress, just sharing my current results.

[tumido]

@fmenesesg I've updated the docs to include your fix and resolved merge conflict. PTAL 🙂

Thank you for letting me collaborate on this 👍

[fmenesesg]

@tumido LGTM, Thanks for your help with this 👍

@fmenesesg I've updated the docs to include your fix and resolved merge conflict. PTAL slightly_smiling_face

Thank you for letting me collaborate on this +1