Add Phishing Detection Integration [Main PR] #3206
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Task/Issue URL: https://app.asana.com/0/1207943168535188/1208149630394247/f Tech Design URL: https://app.asana.com/0/481882893211075/1207220724600204/f CC: **Description**: Implement macOS Phishing Protection integration via TabExtension. **Steps to test this PR**: 1. The tests are in a later PR, but from the first PR all you can test is that the dataActivities are started 2. it should write to disk in `/System/Volumes/Data/Users/thom/Library/Application Support/com.duckduckgo.macos.browser.debug/hashPrefixes.json` <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
Task/Issue URL: https://app.asana.com/0/0/1208206015949665/f Tech Design URL: https://app.asana.com/0/481882893211075/1207220724600204/f CC: **Description**: Implement URL checking + redirecting to duck://error page for phishing detection. **Steps to test this PR**: 1. Visit privacy-test-pages.site/security/badware/phishing.html 2. Ensure the browser navigates to a Duck error page, i.e. address bar says: `duck://error?reason=phishing&url=aHR0cDovL3ByaXZhY3ktdGVzdC1wYWdlcy5zaXRlL3NlY3VyaXR5L2JhZHdhcmUvcGhpc2hpbmcuaHRtbA&token=b2G1T-qOqTjZXNBAVYB6qSqOGEdTpXjQpvh4XjLKCDE:1725446933` <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
Task/Issue URL: https://app.asana.com/0/1199230911884351/1208149630394250/f Tech Design URL: https://app.asana.com/0/481882893211075/1207220724600204 CC: **Description**: Implement duck://error?kind=phishing handler so error pages correctly set tab.error, and load the relevant error page. **Steps to test this PR**: 1. Visit privacy-test-pages.site/security/badware/phishing.html 2. Ensure phishing error page is shown. <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
Task/Issue URL: https://app.asana.com/0/1199230911884351/1208149630394249/f Tech Design URL: https://app.asana.com/0/481882893211075/1207220724600204/f CC: **Description**: Implement AddressBarButtons to reflect designs from https://app.asana.com/0/0/1207896057014803/f **Steps to test this PR**: 1. Visit privacy-test-pages.site/security/badware/phishing.html 2. Ensure error page is shown 3. Click through the warning 4. Red alert circle should be shown for the privacy dashboard entrypoint button <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
Task/Issue URL: https://app.asana.com/0/1199230911884351/1208149630394246/f Tech Design URL: https://app.asana.com/0/481882893211075/1207220724600204/f CC: **Description**: Implement GeneralPreferencesView for Phishing Detection Error Page. Also add feature flag for enabling/disabling the preferences view. **Steps to test this PR**: 1. Go to Settings>General 2. Check there is a Malicious Site Protection section 3. Turn it off 4. Visit privacy-test-pages.site/security/badware/phishing.html 5. Check error page is not shown 6. Turn it back on 7. Retest, ensure the error page is shown <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
…ishing-detection-tests
Too many navigation events in short succession.
Task/Issue URL: https://app.asana.com/0/0/1208196336229421/f Tech Design URL: CC: **Description**: Implement test cases for phishing detection error page, tab extension, and privacy dashboard. **Steps to test this PR**: 1. Run the UnitTests + IntegrationTests 2. Visit https://privacy-test-pages.site/security/badware/phishing.html 3. Ensure warning is thrown 4. Click through warning <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
6 tasks
14 tasks
Task/Issue URL: https://app.asana.com/0/1204023833050360/1208253815417548/f Tech Design URL: CC: https://app.asana.com/0/1204023833050360/1207699541075655/f **Description**: Implement EventMapping and PhishingDetectionEvents firing for pixels defined in https://app.asana.com/0/1204023833050360/1207699541075655/f **Steps to test this PR**: 1. Start browser 2. Navigate to https://privacy-test-pages.site/security/badware/phishing.html 3. Ensure error page is thrown 4. Click through warning 5. Search app logs for "PixelKit", ensure you see something along the lines of: ``` 2024-09-09 14:26:43.734612+0100 DuckDuckGo[84803:28170724] [PixelKit] 👾[Standard-Fired] m_mac_phishing_detection_error-page-shown ["appVersion": "1.105.0", "client_side_hit": "false", "pixelSource": "browser-dmg"] 2024-09-09 14:26:46.267900+0100 DuckDuckGo[84803:28170291] [PixelKit] 👾[Standard-Fired] m_mac_phishing_detection_visit-site ["appVersion": "1.105.0", "pixelSource": "browser-dmg"] ``` <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
…-detection-integration
Task/Issue URL: https://app.asana.com/0/0/1208262609518788/f Tech Design URL: CC: **Description**: Fix navigation stack by doing .redirect in decidePolicy instead of webview.load. Also fix IntegrationTests bug that made .redirect impossible before. **Steps to test this PR**: 1. <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Definition of Done**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? --- ###### Internal references: [Pull Request Review Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f) [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) [Pull Request Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)
SabrinaTardio
approved these changes
Sep 9, 2024
not-a-rootkit
added a commit
to duckduckgo/BrowserServicesKit
that referenced
this pull request
Sep 9, 2024
<!-- Note: This checklist is a reminder of our shared engineering expectations. --> Please review the release process for BrowserServicesKit [here](https://app.asana.com/0/1200194497630846/1200837094583426). **Required**: Task/Issue URL: https://app.asana.com/0/1204023833050360/1207976613228509/f iOS PR: duckduckgo/iOS#3336 macOS PR: duckduckgo/macos-browser#3206 What kind of version bump will this require?: Minor **Optional**: Tech Design URL: https://app.asana.com/0/481882893211075/1207156899292810/f CC: https://app.asana.com/0/481882893211075/1207220724600204/f **Description**: Implement Phishing Detection library to facilitate end-to-end phishing detection feature. Including: 1. Background data updates 2. API client for updating data 3. Embedded datasets 4. Detection logic 5. Event firing <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Steps to test this PR**: 1. Build on macOS 2. Ensure signed in via use-login.duckduckgo.com (only available internally) 3. Quit the app 4. Visit https://privacy-test-pages.site/security/badware/phishing.html 5. Ensure error page is thrown 6. Click advanced 7. Click "Accept Risk" 8. Ensure page loads 9. Play around with navigations (back/forward, etc.) 10. Disable the feature in Settings>General 11. Try other test pages: - https://bad.third-party.site/security/badware/phishing-iframe-loader.html - https://bad.third-party.site/security/badware/phishing-meta-redirect.html - https://bad.third-party.site/security/badware/phishing-js-redirector-helper.html <!-- Before submitting a PR, please ensure you have tested the combinations you expect the reviewer to test, then delete configurations you *know* do not need explicit testing. Using a simulator where a physical device is unavailable is acceptable. --> **OS Testing**: * [ ] iOS 14 * [ ] iOS 15 * [ ] iOS 16 * [ ] macOS 10.15 * [ ] macOS 11 * [ ] macOS 12 --- ###### Internal references: [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) --------- Co-authored-by: Sabrina Tardio <[email protected]>
6 tasks
not-a-rootkit
added a commit
to duckduckgo/BrowserServicesKit
that referenced
this pull request
Sep 10, 2024
<!-- Note: This checklist is a reminder of our shared engineering expectations. --> Please review the release process for BrowserServicesKit [here](https://app.asana.com/0/1200194497630846/1200837094583426). **Required**: Task/Issue URL: https://app.asana.com/0/1204023833050360/1207976613228509/f iOS PR: duckduckgo/iOS#3336 macOS PR: duckduckgo/macos-browser#3206 What kind of version bump will this require?: Major **Optional**: Tech Design URL: https://app.asana.com/0/481882893211075/1207156899292810/f CC: https://app.asana.com/0/481882893211075/1207220724600204/f **Description**: Implement Phishing Detection library to facilitate end-to-end phishing detection feature. Including: 1. Background data updates 2. API client for updating data 3. Embedded datasets 4. Detection logic 5. Event firing <!-- Tagging instructions If this PR isn't ready to be merged for whatever reason it should be marked with the `DO NOT MERGE` label (particularly if it's a draft) If it's pending Product Review/PFR, please add the `Pending Product Review` label. If at any point it isn't actively being worked on/ready for review/otherwise moving forward (besides the above PR/PFR exception) strongly consider closing it (or not opening it in the first place). If you decide not to close it, make sure it's labelled to make it clear the PRs state and comment with more information. --> **Steps to test this PR**: 1. Build on macOS 2. Ensure signed in via use-login.duckduckgo.com (only available internally) 3. Quit the app 4. Visit https://privacy-test-pages.site/security/badware/phishing.html 5. Ensure error page is thrown 6. Click advanced 7. Click "Accept Risk" 8. Ensure page loads 9. Play around with navigations (back/forward, etc.) 10. Disable the feature in Settings>General 11. Try edge-case test pages: - https://bad.third-party.site/security/badware/phishing-iframe-loader.html - https://bad.third-party.site/security/badware/phishing-meta-redirect.html - currently not working - https://bad.third-party.site/security/badware/phishing-js-redirector-helper.html - currently not working The last two failing test cases are to be addressed in follow-up work. Currently this feature is hidden behind a feature-flag for internal-only builds. <!-- Before submitting a PR, please ensure you have tested the combinations you expect the reviewer to test, then delete configurations you *know* do not need explicit testing. Using a simulator where a physical device is unavailable is acceptable. --> **OS Testing**: * [ ] iOS 14 * [ ] iOS 15 * [ ] iOS 16 * [ ] macOS 10.15 * [ ] macOS 11 * [ ] macOS 12 --- ###### Internal references: [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943) --------- Co-authored-by: Sabrina Tardio <[email protected]>
privacy-test-pages is more reliable, since the latter caused a timeout.
not-a-rootkit
added a commit
to duckduckgo/iOS
that referenced
this pull request
Sep 10, 2024
<!-- Note: This checklist is a reminder of our shared engineering expectations. Feel free to change it, although assigning a GitHub reviewer and the items in bold are required.⚠️ If you're an external contributor, please file an issue first before working on a PR, as we can't guarantee that we will accept your changes if they haven't been discussed ahead of time. Thanks! --> Task/Issue URL: https://app.asana.com/0/0/1208253815417552/f Tech Design URL: CC: **Description**: iOS integration for BSK to implement phishing detection on macOS. macOS PR: duckduckgo/BrowserServicesKit#935 BSK PR: duckduckgo/macos-browser#3206 <!-- If at any point it isn't actively being worked on/ready for review/otherwise moving forward strongly consider closing it (or not opening it in the first place). If you decide not to close it, use Draft PR while work is still in progress or use `DO NOT MERGE` label to clarify the PRs state and comment with more information. --> **Steps to test this PR**: 1. <!-- Before submitting a PR, please ensure you have tested the combinations you expect the reviewer to test, then delete configurations you *know* do not need explicit testing. Using a simulator where a physical device is unavailable is acceptable. --> **Definition of Done (Internal Only)**: * [ ] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? **Copy Testing**: * [ ] Use of correct apostrophes in new copy, ie `’` rather than `'` **Orientation Testing**: * [ ] Portrait * [ ] Landscape **Device Testing**: * [ ] iPhone SE (1st Gen) * [ ] iPhone 8 * [ ] iPhone X * [ ] iPhone 14 Pro * [ ] iPad **OS Testing**: * [ ] iOS 15 * [ ] iOS 16 * [ ] iOS 17 **Theme Testing**: * [ ] Light theme * [ ] Dark theme --- ###### Internal references: [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Task/Issue URL: https://app.asana.com/0/0/1207334681074772/f
Tech Design URL: https://app.asana.com/0/481882893211075/1207156899292810/f
CC: https://app.asana.com/0/481882893211075/1207220724600204/f
Description:
Implement macOS phishing detection integration to facilitate end-to-end phishing error pages. This includes: embedded datasets, data updating, data storage, error page, tab extension, privacy dashboard, preferences page, and remote config.
Steps to test this PR:
The last two failing test cases are to be addressed in follow-up work. Currently this feature is hidden behind a feature-flag for internal-only builds so we've considered the risk low.
Definition of Done:
Internal references:
Pull Request Review Checklist
Software Engineering Expectations
Technical Design Template
Pull Request Documentation