Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement ad conversion attribution for Brave Ads #6536

Closed
jsecretan opened this issue Oct 17, 2019 · 5 comments · Fixed by brave/brave-core#4253
Closed

Implement ad conversion attribution for Brave Ads #6536

jsecretan opened this issue Oct 17, 2019 · 5 comments · Fixed by brave/brave-core#4253
Assignees
@tmancey @moritzhaller
Labels
enhancement feature/ads priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass-Linux QA Pass-macOS QA Pass-Win64 QA/Yes release-notes/include
Milestone
1.5.x - Release

Comments

@jsecretan
Copy link

Description

Many advertisers would like to confirm their "return on ad spend", that is, confirm that when they ran an ad, it resulted in the actual outcome they wanted (e.g. buying their product from a website, signing up for a lead form, etc). In the broader ad-tech ecosystem, this is typically accomplished using "conversion pixels", third party trackers that are loaded on checkout pages, and transmit cookies associated with an ad view. Brave, of course, blocks out these trackers. We are implementing an improved version of this capability in Brave Ads, that uses the same authorized by anonymous confirmation system that supports the rest of Brave Ads. This ticket includes the client side work to make this possible.

An example conversion URL is as follows:
https://www.onlinestore.shopping/checkoutfinal/123456/complete.html

If the number between checkoutfinal and complete.html is some kind of purchase ID, the URL pattern would use “*” to ignore it when recognizing the URL.

Catalog

To perform this reporting, the client will need to pull a new version of the catalog, v3. The creative set object will include two new fields called "conversionURLPattern" and "conversionType".

"conversionType": either "postview" or "postclick"
"conversionURLPattern": A pattern for an http/https site that conforms to the following: https://developer.chrome.com/extensions/match_patterns
"conversionObservationWindow": 1,7 or 30 (days)

Browser Setting

We should have an additional setting under "Ads Settings" that allows ad users to specifically opt out of ads conversions:

"Ad Conversions: Provides a private way for Brave advertisers to understand the outcomes of their advertising. Turning this off may result in you earning less because you will be eligible for fewer campaigns."

Ad Library Behavior

If ad conversions has been turned off in the browser, the ad libraries should not serve any creative sets that have associated conversion settings.

If ad conversions are still enabled for the browser, do as follows.

If the ad from the creative set has a "postview"/"postclick" conversion, and the user sees/clicks that ad respectively, start a timer associated with that creative set id, for an amount of time specified in the conversionObservationWindow. While that timer is active, check on page load for URLs matching the conversionURLPattern value. If it is observed, and a conversion has not already been tracked for this creative set, schedule a "conversion" type confirmation event (https://github.com/brave/brave-browser/wiki/Security-and-privacy-model-for-ad-confirmations) to fired off. The conversion event should be tracked in history so it does not repeat. If another ad in the creative set has seen and the conversion has not happened, reset the timer to the original time.

The actual send of the conversion confirmation should be on an exponential random delay with a 24 hour mean. This confirmation only indicates that a matching URL associated with the campaign was visited, not what the URL was.

NB: There are also restrictions on this URL imposed at the campaign execution side to further enforce privacy (e.g. site is a typical shopping/commerce site, where the advertiser can demonstrate control).
@jsecretan jsecretan added the priority/P2 A bad problem. We might uplift this to the next planned release. label Oct 17, 2019
@jsecretan jsecretan added priority/P3 The next thing for us to work on. It'll ride the trains. feature/ads and removed priority/P2 A bad problem. We might uplift this to the next planned release. labels Oct 17, 2019
@jsecretan
Copy link
Author

Depends on #5231

@moritzhaller moritzhaller mentioned this issue Nov 12, 2019
Merged
32 tasks
@moritzhaller moritzhaller mentioned this issue Dec 10, 2019
Closed
32 tasks
@tmancey tmancey self-assigned this Dec 16, 2019
@tmancey tmancey mentioned this issue Dec 18, 2019
Merged
32 tasks
@tmancey tmancey added this to the 1.5.x - Nightly milestone Jan 9, 2020
@tmancey tmancey added the QA/Yes label Jan 21, 2020
@yrliou yrliou mentioned this issue Jan 21, 2020
Closed
@tmancey tmancey mentioned this issue Jan 23, 2020
Merged
32 tasks
@LaurenWags
Copy link
Member

QA Blocked on #8362 which will be resolved by the following PRs:

This was referenced Mar 11, 2020
Closed
Closed
Closed
This was referenced Mar 12, 2020
Closed
Closed
@kjozwiak kjozwiak mentioned this issue Mar 12, 2020
Closed
84 tasks
@btlechowski btlechowski mentioned this issue Mar 12, 2020
Closed
@btlechowski
Copy link

No longer blocked on #8362

@btlechowski
Copy link

btlechowski commented Mar 16, 2020 •
edited
Loading

Test plan

Test case 1: Check for conversions item in ads catalog

  1. Connect to US
  2. Open https://ads-serve.bravesoftware.com/v2/catalog
  3. Search for conversions. The item cannot be empty

Test case 2: conversion of an ad from live catalog on staging

  1. Connect to US
  2. Run Brave with command line: /usr/bin/brave-browser --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true --brave-ads-debug
  3. Enable Rewards
  4. Open brave.com
  5. Trigger an ad
  6. Open https://travala.com/booking?check_in=*
  7. Check logs for Started ad conversion timer
  8. Wait 20 minutes
  9. Check logs for Ad conversion for

Test case 3: Sequential conversions of two ads

  1. Overwrite the catalog with Charles
    Catalog: different sets different patterns.txt
  2. Run Brave with command line: /usr/bin/brave-browser --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true --brave-ads-debug
  3. Enable Rewards
  4. Open brave.com
  5. Trigger an ad
  6. Depending on Ad shown use proper link:
    Ad AAA, use https://travala.com/booking?check_in=AAA
    Ad BBB, use https://travala.com/booking?check_in=BBB
  7. Check logs for Started ad conversion timer for first ad
  8. Wait 20 minutes
  9. Check logs for Ad conversion for for first ad
  10. Trigger another Ad (is you got Ad AAA before, you should get Ad BBB)
  11. Depending on Ad shown use proper link:
    Ad AAA, use https://travala.com/booking?check_in=AAA
    Ad BBB, use https://travala.com/booking?check_in=BBB
  12. Check logs for Started ad conversion timer for second ad
  13. Wait 20 minutes
  14. Check logs for Ad conversion for for second ad

Test case 4: Parallel conversions of two ads

  1. Overwrite the catalog with Charles
    Catalog: different sets different patterns.txt
  2. Run Brave with command line: /usr/bin/brave-browser --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true --brave-ads-debug
  3. Enable Rewards
  4. Open brave.com
  5. Trigger the first ad
  6. Trigger the second ad
  7. Open https://travala.com/booking?check_in=AAA
  8. Open https://travala.com/booking?check_in=BBB
  9. Check logs for Started ad conversion timer for first ad
  10. Check logs for Ad conversion for for first ad
  11. Check logs for Started ad conversion timer for second ad
  12. Check logs for Ad conversion for for second ad

Note: you many encounter: #8644

@kjozwiak kjozwiak mentioned this issue Mar 17, 2020
Closed
@LaurenWags
Copy link
Member

LaurenWags commented Mar 17, 2020 •
edited by GeetaSarvadnya
Loading

Verified passed with

Brave 1.5.112 Chromium: 80.0.3987.132 (Official Build) (64-bit)
Revision fcea73228632975e052eb90fcf6cd1752d3b42b4-refs/branch-heads/3987@{#974}
OS macOS Version 10.14.6 (Build 18G3020)

PASS Test Case 1 - Check for conversions item in ads catalog

Example from staging catalog:

"conversions":[
                  {
                     "observationWindow":86400,
                     "urlPattern":"https://travala.com/booking?check_in=*",
                     "type":"postview"
                  }
               ],

PASS Test Case 2 - conversion of an ad from live catalog on staging

Confirmed the following in the logs:

[12642:775:0316/172352.025398:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid 45d30d1a-4237-43ff-a66b-76bb15679aad with creative set id c7f96341-726f-4e5d-9494-5686db89664b triggered on 2020-03-17 03:27:17.000 UTC
[12642:775:0316/172352.026271:INFO:ad_conversion_tracking.cc(219)] Saving ad conversions state
[12641:775:0316/172352.026642:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: c7f96341-726f-4e5d-9494-5686db89664b
  uuid: 45d30d1a-4237-43ff-a66b-76bb15679aad
  type: conversion
[12641:775:0316/172352.026709:INFO:redeem_token.cc(57)] Redeem

And then a bit farther down:

[12641:775:0316/172352.760299:INFO:redeem_token.cc(431)] Successfully redeemed 221a6964-780f-4784-aeef-f9a5e4cd1fd4 confirmation id with 45d30d1a-4237-43ff-a66b-76bb15679aad creative instance id for conversion

PASS Test case 3: Sequential conversions of two ads

First ad (BBB):

[13151:775:0317/093534.639545:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid 36c84fcb-6388-45d8-81ed-5a5d986cd7ef with creative set id de5a82e1-17d6-47e0-a368-17f8f56dfeb3 triggered on 2020-03-17 13:35:34.000 UTC
[13151:775:0317/093534.640497:INFO:ad_conversion_tracking.cc(220)] Saving ad conversions state
[13151:775:0317/093534.640790:INFO:ad_conversion_tracking.cc(64)] Ad conversion queue is empty
[13150:775:0317/093534.641100:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: de5a82e1-17d6-47e0-a368-17f8f56dfeb3
  uuid: 36c84fcb-6388-45d8-81ed-5a5d986cd7ef
  type: conversion

[13150:775:0317/093535.320602:INFO:redeem_token.cc(431)] Successfully redeemed 2c5ac8f6-0180-4729-84d8-eac9999d4135 confirmation id with 36c84fcb-6388-45d8-81ed-5a5d986cd7ef creative instance id for conversion

Second ad (AAA):

[13180:775:0317/095016.470826:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid bc47f825-bfbe-40a5-9c64-ec9b12660359 with creative set id 64aa94c2-29b0-4494-8edc-9e60c2e86a7c triggered on 2020-03-17 13:50:16.000 UTC
[13180:775:0317/095016.471329:INFO:ad_conversion_tracking.cc(220)] Saving ad conversions state
[13179:775:0317/095016.471563:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: 64aa94c2-29b0-4494-8edc-9e60c2e86a7c
  uuid: bc47f825-bfbe-40a5-9c64-ec9b12660359
  type: conversion

[13179:775:0317/095017.062078:INFO:redeem_token.cc(431)] Successfully redeemed bfa087df-fea6-478e-a41f-aeb879163691 confirmation id with bc47f825-bfbe-40a5-9c64-ec9b12660359 creative instance id for conversion

PASS - Test case 4: Parallel conversions of two ads
Note - not truly parallel, second ad does not begin conversion until 1st ad is complete. Per discussion with @btlechowski, he's going to log an issue.
Did encounter some issues with the logging info which is issue #8644

First Ad (BBB):

[13473:775:0317/105902.798165:INFO:ad_conversion_tracking.cc(178)] Started ad conversion timer for creative_instance_id 36c84fcb-6388-45d8-81ed-5a5d986cd7ef with creative set id de5a82e1-17d6-47e0-a368-17f8f56dfeb3 which will trigger on 2020-03-17 15:08:08.000 UTC

[13562:775:0317/111555.327743:INFO:redeem_token.cc(431)] Successfully redeemed a32486b4-eeca-4642-b868-32a35c5b9740 confirmation id with 36c84fcb-6388-45d8-81ed-5a5d986cd7ef creative instance id for conversion

Second Ad (AAA):

[13563:775:0317/111554.663859:INFO:ad_conversion_tracking.cc(178)] Started ad conversion timer for creative_instance_id bc47f825-bfbe-40a5-9c64-ec9b12660359 with creative set id 64aa94c2-29b0-4494-8edc-9e60c2e86a7c which will trigger on 2020-03-17 15:15:44.000 UTC

[13562:775:0317/112513.251860:INFO:redeem_token.cc(431)] Successfully redeemed 83ba9c27-916f-4eb8-91b8-17887acc7a8b confirmation id with bc47f825-bfbe-40a5-9c64-ec9b12660359 creative instance id for conversion

Verification passed on

Brave 1.5.111 Chromium: 80.0.3987.132 (Official Build) (64-bit)
Revision fcea73228632975e052eb90fcf6cd1752d3b42b4-refs/branch-heads/3987@{#974}
OS Ubuntu 18.04 LTS

Test case 1: Verified live catalog on staging.
Connected to US.
Command line used: /usr/bin/brave-browser --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true

"conversions": [{
		"observationWindow": 86400,
		"urlPattern": "https://travala.com/booking?check_in=*",
		"type": "postview"
	}
]

Test case 2

  • Verified that opening the urlPattern ads the conversion to the queue
    Used https://travala.com/booking?check_in=* to trigger the conversion
[4529:1:0315/233920.729703:INFO:ads_impl.cc(670)] Site visited https://travala.com/booking?check_in=*, domain matches the last shown ad notification for https://travala.com/
[4529:1:0315/233920.729940:INFO:ads_impl.cc(681)] Already sustaining ad notification interaction for https://travala.com/booking?check_in=*
[4529:1:0315/233920.732577:INFO:ad_conversions.cc(221)] Saving ad conversions state
[4529:1:0315/233920.753037:INFO:ad_conversions.cc(179)] Started ad conversion timer for creative_instance_id baf8f645-b98b-4224-a7e6-2ff2d78a4141 with creative set id c7f96341-726f-4e5d-9494-5686db89664b which will trigger on 2020-03-10 22:40:18.000 UTC
  • Verified conversion was successful
[4529:1:0315/234018.754223:INFO:ad_conversions.cc(143)] Ad conversion for uuid baf8f645-b98b-4224-a7e6-2ff2d78a4141 with creative set id c7f96341-726f-4e5d-9494-5686db89664b triggered on 2020-03-12 01:42:10.000 UTC
[4529:1:0315/234018.754530:INFO:ad_conversions.cc(221)] Saving ad conversions state
[4529:1:0315/234018.754694:INFO:ad_conversions.cc(64)] Ad conversion queue is empty
[4496:4496:0315/234018.754943:INFO:ads_service_impl.cc(2136)] AdsService Event Log: {"data":{"type":"confirmation","timestamp":"2020-03-10T23:40:18Z","creativeInstanceId":"baf8f645-b98b-4224-a7e6-2ff2d78a4141","confirmationType":"conversion"}}
[4528:4528:0315/234018.755736:INFO:confirmations_impl.cc(1162)] Confirm action:
  creativeInstanceId: baf8f645-b98b-4224-a7e6-2ff2d78a4141
  creativeSetId: c7f96341-726f-4e5d-9494-5686db89664b
  confirmationType: conversion

Test case 3
Ad1:

[4438:1:0316/164627.938858:INFO:ad_conversion_tracking.cc(178)] Started ad conversion timer for uuid 36c84fcb-6388-45d8-81ed-5a5d986cd7ef and creative set id de5a82e1-17d6-47e0-a368-17f8f56dfeb3 which will trigger on 2020-03-16 15:51:00.000 UTC
[4438:1:0316/165100.939761:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid 36c84fcb-6388-45d8-81ed-5a5d986cd7ef with creative set id de5a82e1-17d6-47e0-a368-17f8f56dfeb3 triggered on 2020-03-16 20:26:50.000 UTC
[4437:4437:0316/165100.940781:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: de5a82e1-17d6-47e0-a368-17f8f56dfeb3
  uuid: 36c84fcb-6388-45d8-81ed-5a5d986cd7ef
  type: conversion

Ad2:

[4438:1:0316/170739.109889:INFO:ad_conversion_tracking.cc(178)] Started ad conversion timer for uuid bc47f825-bfbe-40a5-9c64-ec9b12660359 and creative set id 64aa94c2-29b0-4494-8edc-9e60c2e86a7c which will trigger on 2020-03-16 16:08:18.000 UTC
[4438:1:0316/170818.112508:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid bc47f825-bfbe-40a5-9c64-ec9b12660359 with creative set id 64aa94c2-29b0-4494-8edc-9e60c2e86a7c triggered on 2020-03-18 00:46:30.000 UTC
[4437:4437:0316/170818.113709:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: 64aa94c2-29b0-4494-8edc-9e60c2e86a7c
  uuid: bc47f825-bfbe-40a5-9c64-ec9b12660359
  type: conversion

Test case 4

[27893:1:0317/161316.746754:INFO:ad_conversion_tracking.cc(178)] Started ad conversion timer for creative_instance_id bc47f825-bfbe-40a5-9c64-ec9b12660359 with creative set id 64aa94c2-29b0-4494-8edc-9e60c2e86a7c which will trigger on 2020-03-17 14:43:37.000 UTC
[27893:1:0317/161652.748698:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid bc47f825-bfbe-40a5-9c64-ec9b12660359 with creative set id 64aa94c2-29b0-4494-8edc-9e60c2e86a7c triggered on 2020-03-17 14:43:37.000 UTC
[29015:1:0317/163401.910464:INFO:ad_conversion_tracking.cc(178)] Started ad conversion timer for creative_instance_id 36c84fcb-6388-45d8-81ed-5a5d986cd7ef with creative set id de5a82e1-17d6-47e0-a368-17f8f56dfeb3 which will trigger on 2020-03-17 15:44:29.000 UTC
[29015:1:0317/164429.912010:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid 36c84fcb-6388-45d8-81ed-5a5d986cd7ef with creative set id de5a82e1-17d6-47e0-a368-17f8f56dfeb3 triggered on 2020-03-17 15:44:29.000 UTC

Verification passed on

Brave 1.5.112 Chromium: 80.0.3987.132 (Official Build) (64-bit)
Revision fcea73228632975e052eb90fcf6cd1752d3b42b4-refs/branch-heads/3987@{#974}
OS Windows 10 OS Version 1803 (Build 17134.1006)

Test1: PASS
Staging caltalog:

"conversions": [
            {
              "observationWindow": 86400,
              "urlPattern": "https://travala.com/booking?check_in=*",
              "type": "postview"
            }

Test2: PASS
Verified the logs for ads conversion:

[4964:2176:0317/233924.501:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid a4ef68c9-5d04-4d0b-b88e-f4f561138dc8 with creative set id c7f96341-726f-4e5d-9494-5686db89664b triggered on 2020-03-17 18:09:24.000 UTC
[4964:2176:0317/233924.501:INFO:ad_conversion_tracking.cc(220)] Saving ad conversions state
[4964:2176:0317/233924.501:INFO:ad_conversion_tracking.cc(64)] Ad conversion queue is empty
[4400:9708:0317/233924.501:INFO:ads_service_impl.cc(2142)] AdsService Event Log: {"data":{"type":"confirmation","stamp":"2020-03-17T23:39:24Z","notificationId":"a4ef68c9-5d04-4d0b-b88e-f4f561138dc8","notificationType":"conversion"}}
[6400:10388:0317/233924.501:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: c7f96341-726f-4e5d-9494-5686db89664b
  uuid: a4ef68c9-5d04-4d0b-b88e-f4f561138dc8
  type: conversion

And then a bit farther down:

[6400:10388:0317/233925.251:INFO:redeem_token.cc(431)] Successfully redeemed d9fad3d5-7cdc-415d-9bf8-df85b339298e confirmation id with a4ef68c9-5d04-4d0b-b88e-f4f561138dc8 creative instance id for conversion

Test3: PASS
Sequential conversions of two ads

1st ad (BBB)

[6896:8132:0318/012442.367:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid 36c84fcb-6388-45d8-81ed-5a5d986cd7ef with creative set id de5a82e1-17d6-47e0-a368-17f8f56dfeb3 triggered on 2020-03-17 18:49:30.000 UTC
[6896:8132:0318/012442.367:INFO:ad_conversion_tracking.cc(220)] Saving ad conversions state
[6896:8132:0318/012442.367:INFO:ad_conversion_tracking.cc(64)] Ad conversion queue is empty
[2608:5712:0318/012442.367:INFO:ads_service_impl.cc(2142)] AdsService Event Log: {"data":{"type":"confirmation","stamp":"2020-03-18T01:24:42Z","notificationId":"36c84fcb-6388-45d8-81ed-5a5d986cd7ef","notificationType":"conversion"}}
[11440:8136:0318/012442.367:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: de5a82e1-17d6-47e0-a368-17f8f56dfeb3
  uuid: 36c84fcb-6388-45d8-81ed-5a5d986cd7ef
  type: conversion

[11440:8136:0318/012443.130:INFO:redeem_token.cc(431)] Successfully redeemed 93422288-c242-43b0-a5b6-be8e5ae29f5e confirmation id with 36c84fcb-6388-45d8-81ed-5a5d986cd7ef creative instance id for conversion

2nd Ad (AAA)

[1056:2152:0318/020017.319:INFO:ad_conversion_tracking.cc(143)] Ad conversion for uuid bc47f825-bfbe-40a5-9c64-ec9b12660359 with creative set id 64aa94c2-29b0-4494-8edc-9e60c2e86a7c triggered on 2020-03-17 19:13:27.000 UTC
[1056:2152:0318/020017.319:INFO:ad_conversion_tracking.cc(220)] Saving ad conversions state
[9148:8764:0318/020017.319:INFO:ads_service_impl.cc(2142)] AdsService Event Log: {"data":{"type":"confirmation","stamp":"2020-03-18T02:00:17Z","notificationId":"bc47f825-bfbe-40a5-9c64-ec9b12660359","notificationType":"conversion"}}
[12692:17676:0318/020017.320:INFO:confirmations_impl.cc(1170)] Confirm action:
  creative_set_id: 64aa94c2-29b0-4494-8edc-9e60c2e86a7c
  uuid: bc47f825-bfbe-40a5-9c64-ec9b12660359
  type: conversion

[12692:17676:0318/020018.081:INFO:redeem_token.cc(431)] Successfully redeemed 209fdfd7-d861-4d1c-9777-f5a205a52347 confirmation id with bc47f825-bfbe-40a5-9c64-ec9b12660359 creative instance id for conversion

Test 4: PASS
Parallel conversions of two ads
Note - not truly parallel, second ad does not begin conversion until 1st ad is complete.
Did encounter some issues with the logging info which is issue #8644

@btlechowski btlechowski mentioned this issue Mar 17, 2020
Closed
@btlechowski btlechowski mentioned this issue Apr 21, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tmancey tmancey

@moritzhaller moritzhaller

Labels
enhancement feature/ads priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass-Linux QA Pass-macOS QA Pass-Win64 QA/Yes release-notes/include
Projects
None yet
Milestone
1.5.x - Release
7 participants
@jsecretan @tmancey @moritzhaller @kjozwiak @LaurenWags @btlechowski @GeetaSarvadnya