-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bot Connector service is not using latest Adaptive Cards #87
Comments
Thanks @v-kydela. We actually began investigations over Teams. Let me share what I was looking into. I still haven't found the cause of this discrepancy.
|
looping in @tonyanziano |
@corinagum I looked through the Emulator code for handling attachments and I was unable to find any special logic for the rendering of video / media cards. Also, like you said, we are relying on Web Chat's default host config and do not pass a custom one in when rendering Web Chat. |
@tonyanziano thanks for the help |
I'm having the same issue with videos in AdaptiveCards, but VideoCard attachments are working fine. const card = CardFactory.videoCard(
'2018 Imagine Cup World Championship Intro',
[{ url: 'https://adaptivecardsblob.blob.core.windows.net/assets/AdaptiveCardsOverviewVideo.mp4' }],
[{
type: 'openUrl',
title: 'Lean More',
value: 'https://channel9.msdn.com/Events/Imagine-Cup/World-Finals-2018/2018-Imagine-Cup-World-Championship-Intro'
}],
{
subtitle: 'by Microsoft',
text: 'Microsoft\'s Imagine Cup has empowered student developers around the world to create and innovate on the world stage for the past 16 years. These innovations will shape how we live, work and play.'
}
);
await turnContext.sendActivity({
attachments: [card]
}); Also seems related to this Adaptive Cards issue microsoft/AdaptiveCards#2272 |
Thank you @tdurnford I did see that issue, but it looks like the discussion ended before Adaptive Cards 1.1 were ever supported in Web Chat |
It looks like the media element is being stripped out of the Adaptive Card somewhere between the Bot and WebChat. AdaptiveCard{
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"version": "1.1",
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"text": "**AdaptiveCard with Video**"
},
{
"type": "Media",
"poster": "https://adaptivecards.io/content/poster-video.png",
"sources": [
{
"mimeType": "video/mp4",
"url": "https://adaptivecardsblob.blob.core.windows.net/assets/AdaptiveCardsOverviewVideo.mp4"
}
]
}
]
} DirectLineimport { DirectLine } from 'botframework-directlinejs';
global.XMLHttpRequest = require('xhr2');
global.WebSocket = require('ws');
var directLine = new DirectLine({
secret: '<WEBCHAT_SECRET>'
});
directLine.postActivity({
from: { id: 'myUserId', name: 'myUserName' },
type: 'message',
text: 'start'
}).subscribe();
directLine.activity$
.filter(activity => activity.from.name !== 'myUserName' && activity.attachments)
.subscribe(
({ attachments: [{content}] }) => console.log("Card ", content)
); OutputCard {
type: 'AdaptiveCard',
version: '1.1',
body: [ { type: 'TextBlock', text: '**AdaptiveCard with Video**' } ]
} The Adaptive Card sent by the bot has a media element, but the attachment received and processed by DirectLineJs does not. |
@tdurnford is correct. The WebChat and Direct Line connector service Adaptive Cards version is 1.0.6 WebChat is on version ~1.1.3 https://github.com/microsoft/BotFramework-WebChat/blob/master/packages/bundle/package.json#L36 and is able to render the card. Since the emulator is receiving messages directly, and not through the connector service: it can render these cards. This issue should be moved to https://github.com/microsoft/BotFramework-Services/ |
Good catch @EricDahlvang. But why would the connector service even have an Adaptive Cards version if it's not responsible for rendering anything? Also, stripping out elements is useless because if the Adaptive Cards version is higher than the channel client supports, the card won't render even with the special elements removed. |
The DirectLine connector service maps the incoming activities from JSON to C# Objects, and since the connector service uses AdaptiveCards v1.0.6 it is not able to map AdaptiveCards attributes from v1.1.0 or greater. As a result, it drops items/media in the card like videos. Right or wrong, the limiting factor for AdaptiveCards in Web Chat is the AdaptiveCards version in the connector service. Ideally, the connector service would leave attachments as JSON and let the client handle the attachment scheme. |
As a temporary workaround, you can set the contentType of the attachment to a custom value as exemplified in the Customization Card Components Web Chat Sample and change the type back to an AdaptiveCard before Web Chat renders it. By changing the contentType, the DirectLine Connector Service treats the AdaptiveCard as JSON and does not map it to a C# Object. As a result, the card does not lose any of the elements that are only available in newer versions of AdaptiveCards. Adding an attachment middleware to Web Chat allows you to monitor for the custom attachment and change the type back to an AdaptiveCard. For more details, take a look at the code snippets below. Bot - NodeJs SDK Set the attachment contentType property to 'application/vnd.microsoft.card.custom' when sending the AdaptiveCard. const card = CardFactory.adaptiveCard({
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"version": "1.1",
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"text": "AdaptiveCard with Video"
},
{
"type": "Media",
"poster": "https://adaptivecards.io/content/poster-video.png",
"sources": [
{
"mimeType": "video/mp4",
"url": "https://adaptivecardsblob.blob.core.windows.net/assets/AdaptiveCardsOverviewVideo.mp4"
}
]
}
]
});
card.contentType = 'application/vnd.microsoft.card.custom';
await context.sendActivity({
attachments: [card]
});
Web Chat Attachment Middleware In the attachment middleware, change the custom contentType back to 'application/vnd.microsoft.card.adaptive'. const attachmentMiddleware = () => next => card => {
if (card.attachment.contentType === 'application/vnd.microsoft.card.custom'){
card.attachment.contentType = 'application/vnd.microsoft.card.adaptive'
}
return next(card)
};
window.WebChat.renderWebChat({
directLine,
attachmentMiddleware
}, document.getElementById('webchat')); Screenshot This is not a long term solution to the issue, and still needs to be addressed in the connector service. |
@mathurvarun84 - It is not fixed yet. Use the workaround. |
@corinagum we also faced the same issue. when we are sending media based adaptive cards (version 1.1) then the webchat is failing to show the videos (we have the latest version of webchat). However, the above workaround solves our issue. But we can't use the above workaround as we have other channels for our bot too and they can't render custom content type adaptive cards. Any permanent fix for this issue? If you want I can send you our card json as well. |
@mathurvarun84 - What channels? I don't think there are any other channels that support Adaptive Cards 1.1, but in any case that's irrelevant because this issue is about Direct Line. |
@mathurvarun84 If you only want to apply the work around to AdaptiveCards being sent to Web Chat, you can check if the incoming activity's channel id is 'directline' before changing the content type. This should allow you to send AdaptiveCards to channels that support them while still using the work around for Web Chat. const card = CardFactory.adaptiveCard(Card);
if (turnContext.activity.channelId === 'directline') {
card.contentType = 'application/vnd.microsoft.card.custom';
}
await turnContext.sendActivity({ attachments: [ card ]}) Hope this helps! |
Hi tdurnford Also, may I know why it is failing anyway if we have adaptive cards 1.2.0 in the webchat so ideally, it should support. One more thing....does webchat support application/vnd.microsoft.card.videocard?? If we send the content as vidoecard (created by botbuilder) will that work in webchat. |
@mathurvarun84 - Direct Line and Web Chat are not two distinct channels. Direct Line is a channel and Web Chat is a Direct Line client. That means Web Chat is a client application with a user interface that can be used to interact with the bot through the Direct Line channel. Do you mean you have two clients? Are you using Web Chat in addition to your own Direct Line client? |
I apologize... .yes we have two clients... we are using salesforce which uses directline channel and we use normal app service in azure to use webchat as the channel (via the token) |
In that case, you can add channel data to each activity sent from Web Chat to distinguish messages sent specifically from Web Chat. Web Chat Store <script src="https://unpkg.com/simple-update-in/dist/simple-update-in.production.min.js"></script> const store = createStore(
{},
({ dispatch}) => next => async action => {
if (action.type === 'DIRECT_LINE/POST_ACTIVITY') {
action = window.simpleUpdateIn(action, ['payload', 'activity', 'channelData', 'channel'], () => 'webchat');
} else if ( ... ) {
...
} ...
return next(action)
}
); Bot const card = CardFactory.adaptiveCard(Card);
if (turnContext.activity.channelData && turnContext.activity.channelData.channel === 'webchat') {
card.contentType = 'application/vnd.microsoft.card.custom';
}
await turnContext.sendActivity({ test: 'hello', attachments: [ card ]}) For more details, take a look at the Backchannel Piggyback on Outgoing Activities sample. Hope this helps! |
@mathurvarun84 - Does the Salesforce client support Adaptive Cards 1.1? |
@corinagum - Has this been fixed? |
Web Chat has been bumped to 1.2, but the issue linked above will finish up some of the work I missed. |
@corinagum - Has the Direct Line connector service been updated to Adaptive Cards 1.2 (or become version-independent)? Can this issue be closed or has it not been addressed? |
I just tested again and Direct Line is still stripping out media elements from Adaptive Cards. That's interesting because the new wrap property works in choice sets. I guess Direct Line removes unrecognized elements but not unrecognized properties. |
Is there service team aware of AC 1.0.6 in their code base? If not, we should. CK and Daniel are probably good starting point. |
Opened a bug in DevOps for investigation & costing. |
To the earlier questions; we have scheduled the work to update Adaptive Cards to v1.2. It is a little trickier than expected, some of our clients are still using Adaptive Cards 0.5 and the 1.2 version breaks compatibility with them and we're looking for a solution that doesn't break existing bots while unblocks those wishing to use 1.2 features. |
@jameslew - From what you're saying, am I correct in understanding that when Direct Line deserializes an Adaptive Card using the 1.2 schema it removes tokens that are used by the 0.5 schema, which some Direct Line clients use? If that is true then it's an even more persuasive case in favor of the recommendation made by TJ and me. To explain, there seem to be two possible fixes for the Web Chat problem:
I had already thought option 2 would be best because it would be a permanent solution. That is to say, we wouldn't run into this problem again for future versions of Adaptive Cards. But based on what you're saying, it sounds like option 2 would also allow for more backward compatibility in addition to more forward compatibility. |
@ckkashyap has a work item to bump Adaptive Cards in Direct Line channel to use 1.2. I am unsure how current version (1.0.x) vs. 1.2. Looking at AC changelog but didn't see anything related to it. Agree to @v-kydela that the best option is don't touch JSON. But need to fully understand why we do it today and what is the advantage. If we passthru JSON, what features we missed. When bumping to 1.2, if the behavior changed, we will need to find a way to tell customers how our change will break stuff, and schedule, etc. In this case, we will need to loop in PM to coordinate. FYI, in Direct Line Streaming Extensions, it is passthru. In my believe, we should reduce the difference of DL REST/WS and DL SE as much as we could. |
@jameslew when it this scheduled to be addressed? We continue to get complaints from customers on the WebChat project. |
I have status on this from our last standup -- we are waiting on a bugfixed version of Adaptive Cards and are targeting ~11/22 for a fix in production. |
Potential dumb question warning ;) A client is suggesting that when they use Web Sockets they don't see the issue. I confess I don't know too much about this, is there any truth/reason for this? |
It's a good question but I can't think of a reason for that as the processing occurs inbound from the bot, not outbound to the client (over GET vs. Web Socket). Not all cards trigger the issue in the current library. |
@dandriscoll - Does the fix account for future versions of Adaptive Cards by allowing Adaptive Cards that Direct Line doesn't understand pass through unchanged? Or does the fix just incorporate the latest version of Adaptive Cards without changing the underlying design of Direct Line? |
I am also experiencing the same issue as brett-1hw in #2912, the adaptive card is rendering but cuts off when it should display ActionSets. I am also interacting with the bot using DirectLine. |
Any news on the target release of 22nd Nov? |
Hi @paulio , |
hello @ckkashyap , you can confirm which components and versions we have to update ?? |
Hi @MIRo2k , |
FYI it seems to be working :) |
Thanks @paulio - I am going to close this issue. |
Hello, **
** using a 1.2.4 Adaptive Cards, on westeurope. this is the card payload in the sample, it works in the Bot Emulator but not in Web Chat. If I remove the ActionSet element it renders without error:
|
[Edit by corinagum]:
Conclusion: Bot Connector service needs to be updated with latest Adaptive Cards
Screenshots
Version
Describe the bug
Adaptive Cards 1.1 introduced media elements. Web Chat supports Adaptive Cards 1.1 but doesn't render those elements. However, Emulator uses Web Chat and does render them. This is impacting at least one customer: https://stackoverflow.com/a/56177322/2122672
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Any channel that supports Adaptive Cards 1.1 should render media elements
Additional context
Example card:
[Bug]
The text was updated successfully, but these errors were encountered: