Teams Incoming Webhook deprecation

[trenslow]

Hi Flux team,

I've been happily using the MS Teams incoming webhook functionality for quite some time now. Annoyingly, they are now deprecating the O365 connectors in favor of Power Automate Workflows.

Unfortunately, the workflow webhook request requires a different schema than the O365 connector, causing the workflow to fail.

Is it possible for Flux to implement this new schema? I guess a new Provider type would have to be created and a migration process devised. According to the first link, the O365 connector will stop working on 1 Oct. 2024, so there's a bit of time to game-plan.

Thanks for your consideration

[Zheer09]

Definitely we need this to have a new schema power automate can have adaptivecard that can be created from here https://adaptivecards.io/

[stefanprodan]

If the current API we use is going away in October, wouldn't be better to move the msteams provider to the new API?

[aspexdaniel]

If the current API we use is going away in October, wouldn't be better to move the msteams provider to the new API?

There's no new API from msteams side, the solution provided by MS is to create a Workflow in PowerAutomate, which by default only accepts Adaptive Card message format. The workflow will generate an URL as the calling endpoint.

[trenslow]

If the current API we use is going away in October, wouldn't be better to move the msteams provider to the new API?

It seems like if the provider starts sending an AdaptiveCard instead of the old MessageCard, the migration from incoming webhook to power automate workflows will be relatively straightforward. The current incoming webhook connector accepts both MessageCards and AdaptiveCards, so it wouldn't break the current provider/alert config.

[Zheer09]

If the current API we use is going away in October, wouldn't be better to move the msteams provider to the new API?

It seems like if the provider starts sending an AdaptiveCard instead of the old MessageCard, the migration from incoming webhook to power automate workflows will be relatively straightforward. The current incoming webhook connector accepts both MessageCards and AdaptiveCards, so it wouldn't break the current provider/alert config.

But is it possible to change the payload of the alert that flux will send to that webhook? If it is possible that can work and change the payload to adaptivecard then create a power automate workflow webhook for teams

[muandane]

Well i tried a simple webhook on workflow and it's not working it gets parsing issues Action 'Send_each_adaptive_card' failed: The execution of template action 'Send_each_adaptive_card' failed: the result of the evaluation of 'foreach' expression '@triggerOutputs()?['body']?['attachments']' is of type 'Null'. The result must be a valid array.

[bbenouarets]

If other system administrators are having difficulties locating the affected teams:
I have written a small tool in Golang that uses the Graph API to output the affected teams.

Teams Webhook Finder

This has helped us enormously, as Microsoft does not offer its own solution for reading the affected channels and teams. We have over 350 teams in our company, which we would otherwise have had to search through manually.

[giladk1221]

Hi,
Do you have update about how can I implement Teams webhook using Power Automate with FluxCD?

@yuval987

[matheuscscp]

Hello all 👋

I will be working on this issue.

I have started a draft pull request here: https://github.com/fluxcd/notification-controller/pull/920/files

I'm trying to register a personal Microsoft 365 Business account for myself in order to test this inside MS Teams but I'm having some issues in this registration and am following up with Microsoft's technical and sales support. In the meantime, the Flux project would appreciate help from Azure/MS Teams users to test this PR for us. (cc @trenslow @Zheer09 @aspexdaniel)

Here's a preview image for notification-controller: ghcr.io/fluxcd/notification-controller:preview-a4001bad@sha256:d508e821ee3ceebb4a3da7aeca82513dc8d49fae1401067babafb560f5fb893d

If you're using kustomize to deploy the Flux v2.3 components, a patch like the one below would do the trick (deploy the preview image for notification-controller and add the msadaptivecard provider to the CRD):

- target:
    kind: Deployment
    name: notification-controller
  patch: |
    - op: replace
      path: /spec/template/spec/containers/0/image
      value: ghcr.io/fluxcd/notification-controller:preview-a4001bad@sha256:d508e821ee3ceebb4a3da7aeca82513dc8d49fae1401067babafb560f5fb893d
- target:
    kind: CustomResourceDefinition
    name: providers.notification.toolkit.fluxcd.io
  patch: |
    - op: add
      path: /spec/versions/2/schema/openAPIV3Schema/properties/spec/properties/type/enum/-
      value: msadaptivecard

I have deployed an HTTP server just to echo the JSON payload sent by notification-controller and I'm getting the following:

{
    "attachments": [
        {
            "content": {
                "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
                "body": [
                    {
                        "items": [
                            {
                                "size": "medium",
                                "style": "heading",
                                "text": "gitrepository/flux-system.flux-system",
                                "type": "TextBlock",
                                "wrap": true
                            },
                            {
                                "text": "stored artifact for commit 'test notification-controller msadaptivecard provid...'",
                                "type": "TextBlock",
                                "wrap": true
                            },
                            {
                                "facts": [
                                    {
                                        "title": "summary",
                                        "value": "msadaptivecard test"
                                    },
                                    {
                                        "title": "cluster",
                                        "value": "crystal"
                                    },
                                    {
                                        "title": "revision",
                                        "value": "main@sha1:6603cebe98dd0546a5ea70ed6ace1ac29e39be85"
                                    }
                                ],
                                "type": "FactSet"
                            }
                        ],
                        "type": "Container"
                    }
                ],
                "type": "AdaptiveCard",
                "version": "1.5"
            },
            "contentType": "application/vnd.microsoft.card.adaptive",
            "contentUrl": null
        }
    ],
    "type": "message"
}

This is how the JSON above looks like in https://adaptivecards.io/designer:

Events with the error severity should look like this:

It's possible that the top-level payload format {"type": "message","attachments": [...]} is exclusive to MS Teams, in which case it would make more sense to just re-write the code for the msteams provider instead of adding a new one called msadaptivecard. But in the end we must implement something that works through the PowerAutomate Incoming Webhook feature that can dispatch the Adaptive Card to MS Teams, and not the current MS Teams Incoming Webhook feature (which is the one being deprecated). I ask help from the interested users to test all those cases while I follow up with Microsoft on registering a test account for myself 🙏

[matheuscscp]

I was finally able to register for a Microsoft 365 account and test this 🎉

[stefanprodan]

@matheuscscp awesome work! The only issue I see with this is that the message is trimmed and in case of an error you'll not see all the details. Can you click on card in MSTeam and see the whole message?

[matheuscscp]

The only issue I see with this is that the message is trimmed and in case of an error you'll not see all the details. Can you click on card in MSTeam and see the whole message?

That's actually what notification-controller is forwarding from the client to the provider, MS Teams has nothing to do with it. See the JSON payload I posted above one more time:

{
    "attachments": [
        {
            "content": {
                "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
                "body": [
                    {
                        "items": [
                            {
                                "size": "medium",
                                "style": "heading",
                                "text": "gitrepository/flux-system.flux-system",
                                "type": "TextBlock",
                                "wrap": true
                            },
                            {
                                "text": "stored artifact for commit 'test notification-controller msadaptivecard provid...'",
                                "type": "TextBlock",
                                "wrap": true
                            },
                            {
                                "facts": [
                                    {
                                        "title": "summary",
                                        "value": "msadaptivecard test"
                                    },
                                    {
                                        "title": "cluster",
                                        "value": "crystal"
                                    },
                                    {
                                        "title": "revision",
                                        "value": "main@sha1:6603cebe98dd0546a5ea70ed6ace1ac29e39be85"
                                    }
                                ],
                                "type": "FactSet"
                            }
                        ],
                        "type": "Container"
                    }
                ],
                "type": "AdaptiveCard",
                "version": "1.5"
            },
            "contentType": "application/vnd.microsoft.card.adaptive",
            "contentUrl": null
        }
    ],
    "type": "message"
}

The event.Message is stored artifact for commit 'test notification-controller msadaptivecard provid...', source-controller was the one who sent the payload to notification-controller with this ellipsis at the end of the message.

If I send the same payload with a longer message via curl, we can see everything (thanks to wrap: true in the Adaptive Card configuration):

(Note that I also removed the single quotes from the source-controller message in my curl command, I wasn't able to escape it so I just removed it, there's no bug there.)

[stefanprodan]

Ok great, errors are send in full by the controllers so no issues here! Thanks

[trenslow]

@matheuscscp thanks so much for your efforts! i wont have time until next week to test it, but from the screenshots it looks really good

im not a go developer, but in the new code it seems if there is extra event metadata configured in the alert, e.g. region: europe-west1 then that will also be displayed in the adaptivecard facts, correct?

other than that and what @stefanprodan asked about, theres no more open questions for me! and i look forward to being able to integrate it

[stefanprodan]

if there is extra event metadata configured in the alert, e.g. region: europe-west1 then that will also be displayed in the adaptivecard facts, correct?

Yes

[matheuscscp]

Hello @trenslow, @Zheer09, @aspexdaniel (and any other Flux users planning to use the MS Teams provider).

The solution we are implementing for addressing the deprecation of the Office 365 Connector is checking the host of the address. If the suffix .webhook.office.com is present, the controller will keep the old behavior and send the existing payload format. Otherwise, a new payload format for Microsoft Adaptive Card will be sent, which is compatible with the new solution offered by Microsoft Teams: Incoming Webhook with Workflows.

Can you please check if your current URLs, i.e. the ones provided by the Office 365 Connector, have the suffix .webhook.office.com? We are assuming that all the Office 365 Connector URLs have this format. If this assumption is wrong and there are Office 365 Connector URLs that do not have this suffix, they will stop working if we release this solution.

Thanks in advance for your help!

[Zheer09]

Hello @matheuscscp from myside all the host of the address we use the suffix is as you provided .webhook.office.com

[trenslow]

The URLs for our Office 365 connectors also end in webhook.office.com @matheuscscp