Round robin ads delivery

[jsecretan]

Description

Right now, the ad libraries will take a look at ads eligible to serve and will randomly select eligible ones. However, sometimes this results in seeing the same ad multiple times in a row. We should try to round robin deliver eligible ads instead so they are seen more evenly by the user.

So for all ads which have not yet reached frequency cap, we should first choose ads that have been seen the least over the observation window. The new algorithm would look as follows.

1. Find all ads eligible to serve given daily, lifetime, and hourly limits.
2. Among the ads in 1, count the number of times each has been seen over the observation window. Take that are tied for being seen the fewest number of times.
3. Choose randomly among the set in 2.

[btlechowski]

@tmancey Is there a way of telling whether duplicate ad is shown after round robin? I contrast to simply showing a duplicate ad?

[btlechowski]

@tmancey Thank you for the call, it was really helpful. Based on it I wrote a detailed test plan.

Setting up ads catalog

1. Connect to US VPN
2. Open https://ads-serve.brave.com/v1/catalog and copy the catalog
3. Format the catalog with json formatter
4. Search for Personal Finance set with at least 4 ads(or other creativeSet that has more than 4 ads)
5. Change "perDay" to 30 for that set
   Change "totalMax" to 30
   Change "dailyCap" to 30
6. Delete all the ads in the segment except the 4
7. Change the payload.body and payload.title of the first ad to Test 1, do the same for other ads in the set but increment the number
8. Save the catalog
   

Setting Charles proxy

1. Open Tools->Rewrite...
2. Enable Enable Rewrite
3. Click Add on the bottom left
4. Change name to ads catalog
5. Under Location click Add
Protocol: https
Host: ads-serve.brave.com
Path: v1/catalog
6. Under Action click Add
Type : Body
Where : Response
Replace : paste the whole catalog
7. Open https://ads-serve.brave.com/v1/catalog and check if your modified catalog is served
   

Test Case 1 - one set

1. Set up catalog according to above instructions
2. If there are more than one Personal Finance segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
3. Set Charles proxy according to above instructions
4. Clean install
5. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
6. Enable Rewards
7. Change ads per hour to 5
8. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
9. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
10. Watch an ad every 15 minutes (important as there is limit on every ad to be shown once per hour)

Note: There are 4 ads in the set for Personal Finance segment. 4 ads total for Personal Finance segment

Expected behavior
Duplicate Ad is not shown till all other ads are not shown for the same segment type (Personal Finance)

Example for 4 ads in the segment:
no duplicates within first 4 ads
Test 4
Test 1
Test 2
Test 3

no duplicates within next 4 ads
Test 2
Test 4
Test 1
Test 3

no duplicates within next 4 ads
Test 1
Test 3
Test 4
Test 2

Test Case 2 - two sets of the same segment type

1. Set up catalog according to above instructions. Use Personal Finance segment. The number of ads in the set must be 2.
2. Set up catalog according to above instructions. Use a different Personal Finance segment. Rename the ads differently. The number of ads in the set must be 2.
3. If there are more than one Personal Finance segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
4. Set Charles proxy according to above instructions
5. Clean install
6. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
7. Enable Rewards
8. Change ads per hour to 5
9. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
10. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
11. Watch an ad every 15 minutes (important as there is limit on every ad to be shown once per hour)

Note: There are 2 ads per set for Personal Finance segment. There are 2 sets, so 4 ads total for Personal Finance segment

Expected behavior
Duplicate Ad is not shown till all other ads are not shown for the same segment type (Personal Finance)

Example for 4 ads in the segment:
no duplicates within first 4 ads
Test 1
Test 1 Set 2
Test 2
Test 2 Set 2

no duplicates within next 4 ads
Test 2
Test 2 Set 2
Test 1
Test 1 Set 2

no duplicates within next 4 ads
Test 1
Test 1 Set 2
Test 2 Set 2
Test 2

Test Case 3 - one ad for the segment

1. Set up catalog according to above instructions. Use Personal Finance segment. The number of ads in the set must be 1.
2. If there are more than one Personal Finance segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
3. Set Charles proxy according to above instructions
4. Clean install
5. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
6. Enable Rewards
7. Change ads per hour to 5
8. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
9. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
10. Watch an ad every 60 minutes (important as there is limit on every ad to be shown once per hour)

Note: There is an ad in the set for Personal Finance segment. 1 ad total for Personal Finance segment

Expected behavior
The ad is shown every hour.

Test Case 4 - two sets of different segment types

Note: the point of this TC is to check if round robins from different segments interfere with each other

1. Set up catalog according to above instructions. Use Personal Finance segment. The number of ads in the set must be 2. Name them PF T1 and PF T2
2. Set up catalog according to above instructions. Use a different Arts & Entertainment segment. Rename the ads differently. The number of ads in the set must be 2. Name them A&E T1 and A&E T2
3. If there are more than one Personal Finance/Arts & Entertainment segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
4. Set Charles proxy according to above instructions
5. Clean install
6. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
7. Enable Rewards
8. Change ads per hour to 5
9. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
10. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
11. Watch an ad after 15 minutes -> Expected to get an ad from Personal Finance segment
12. Watch an ad after 15 minutes -> Expected to get the other ad from Personal Finance segment
13. Open https://www.imdb.com or other page that classifies as Arts & Entertainment, make sure that Arts & Entertainment wins over Personal Finance
14. Watch an ad after 15 minutes -> Expected to get an ad from Arts & Entertainment segment
15. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
16. Watch an ad after 15 minutes -> Expected to get an ad from Personal Finance segment
17. Open https://www.imdb.com or other page that classifies as Arts & Entertainment
18. Watch an ad after 15 minutes -> Expected to get the other ad from Arts & Entertainment segment
19. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
20. Watch an ad after 15 minutes -> Expected to get the other ad from Personal Finance segment
21. Watch an ad after 15 minutes -> Expected to get an ad from Personal Finance segment
22. Open https://www.imdb.com or other page that classifies as Arts & Entertainment
23. Watch an ad after 15 minutes -> Expected to get an ad from Arts & Entertainment segment
24. Watch an ad after 15 minutes -> Expected to get the other ad from Arts & Entertainment segment

Expected result:
round robins from different segments do not interfere with each other

Example:
PF T2
PF T1
A&E T2
PF T1
A&E T1
PF T2
PF T1
A&E T2
A&E T1

Now divide by segments and search for duplicates within pairs:
Personal Finance
PF T2
PF T1

PF T1
PF T2

PF T1

Arts & Entertainment
A&E T2
A&E T1

A&E T2
A&E T1

[btlechowski]

Verification passed on

Brave	1.1.11 Chromium: 78.0.3904.108 (Official Build) beta (64-bit)
Revision	4b26898a39ee037623a72fcfb77279fce0e7d648-refs/branch-heads/3904@{#889}
OS	Ubuntu 18.04 LTS

Verified test plan from #5945 (comment)

Verification passed on

Brave	1.1.19 Chromium: 79.0.3945.70 (Official Build) (64-bit)
Revision	65868db27916cebde51fc2465c9b34761e6383e2-refs/branch-heads/3945@{#869}
OS	Windows 7 Service Pack 1 (Build 7601.24530)

Verified test plan from #5945 (comment)

Verification PASSED on macOS 10.15.1 x64 using the following build:

Brave	1.1.19 Chromium: 79.0.3945.70 (Official Build) (64-bit)
Revision	65868db27916cebde51fc2465c9b34761e6383e2-refs/branch-heads/3945@{#869}
OS	macOS Version 10.15.1 (Build 19B88)

• Went through and verified the test plan outlined under Round robin ads delivery #5945 (comment)