NetworkService migration for Tor #2647

darkdh · 2019-06-10T23:42:53Z

brave-browser: a0b6c4d (75.0.3770.90)

When NetworkService is enabled, we only have access to profile-wise
proxy config. So we discard proxy config per url request path to have a
global proxy config for tor profile and insert username/password when it
is about to resolve proxy in net::ProxyResolutionService.

Note

new flow works no matter NetworkService enabled or not.
ProxyConfigServiceTor has to be NET exported to avoid //net circular
dependency.
ProxyConfigMonitor chromium overrides is for setting profile-wise
proxy config without setting proxy_config::prefs::kProxy which will be
interfered by other components(ex. DataReductionProxyService)

Submitter Checklist:

Test Plan:

Launch Brave with/without --enable-features=NetworkService and test following scenario separately

Go to https://check.torproject.org/ in non tor window, make sure it doesn't connect to tor network
Go to https://check.torproject.org/ in tor window, remember the ip address
Click New Tor Identity, it will trigger a auto refresh and ip should be different than step 2

Reviewer Checklist:

New files have MPL-2.0 license header.
Request a security/privacy review as needed.
Adequate test coverage exists to prevent regressions
Verify test plan is specified in PR before merging to source

After-merge Checklist:

The associated issue milestone is set to the smallest version that the
changes has landed on.
All relevant documentation has been updated.

patches/net-BUILD.gn.patch

iefremov · 2019-06-17T17:31:53Z

net/proxy_resolution/proxy_config_service_tor.cc

 const int kTorPasswordLength = 16;
 // Default tor circuit life time is 10 minutes
-constexpr base::TimeDelta kTenMins = base::TimeDelta::FromMinutes(10);
+constexpr base::TimeDelta kTenMins = base::TimeDelta::FromMinutes(1);


Doesn't look like ten minutes :)

ah, I was doing some testing and forgot to change back

iefremov · 2019-06-17T17:32:21Z

net/proxy_resolution/proxy_config_service_tor.cc

-#include "net/proxy_resolution/proxy_resolution_service.h"
-#include "net/url_request/url_request_context.h"
+#include "net/proxy_resolution/proxy_config_with_annotation.h"
+#include "net/base/registry_controlled_domains/registry_controlled_domain.h"


wrong sorting (just apply clang-format)

fixed in 4985fce

iefremov · 2019-06-17T17:33:48Z

net/proxy_resolution/proxy_config_service_tor.cc

-        proxy_url = std::string(scheme_ + "://" + host_ + ":" + port_);
-      }
+      std::string proxy_url =
+        std::string(scheme_ + "://" + host_ + ":" + port_);


whitespace could be fixed by clang-format

nit: use const std::string and generally const as much as possible

fixed in 4985fce

iefremov · 2019-06-17T17:38:53Z

net/proxy_resolution/proxy_config_service_tor.cc

-TorProxyConfigService::TorProxyConfigService(
-  const std::string& tor_proxy, const std::string& username,
-  TorProxyMap* tor_proxy_map) {
+ProxyConfigServiceTor::ProxyConfigServiceTor(const std::string& tor_proxy) {


why do we receive a string here instead of getting net::ProxyServer ? We could avoid all that parsing and use well-formed data structures

fixed in 51b3f3c

iefremov · 2019-06-17T17:43:49Z

browser/ui/browser_commands.cc

 }  // namespace

 namespace brave {

+void NewTorIdentityCallback(WebContents* current_tab, bool success) {


why success is unused?

fixed in 51b3f3c

iefremov · 2019-06-17T17:47:22Z

common/tor/tor_proxy_string_helper.cc

+    case version_info::Channel::DEV:
+      return std::string("socks5://127.0.0.1:9370");
+    case version_info::Channel::CANARY:
+      return std::string("socks5://127.0.0.1:9380");


isn't it possible to return GURL?

it is an uri not url, see ProxyServer::FromURI and ProxyServer::ToURI.

yea, socks5 is not a valid scheme for GURL afaik

iefremov · 2019-06-17T17:48:27Z

chromium_src/chrome/browser/net/proxy_config_monitor.cc

+  return std::make_unique<net::ProxyConfigServiceTor>(
+    tor::GetTorProxyString());
+}
+#endif


#endif // BUILDFLAG ...

fixed in 51b3f3c

iefremov · 2019-06-17T17:50:22Z

browser/tor/tor_profile_service_impl.cc

+  GURL url = request_url.ReplaceComponents(replacements);
+  tor_circuit_callback_ = std::move(callback);
+
+  DCHECK_CURRENTLY_ON(content::BrowserThread::UI);


typically such checks are placed in the very beginning of a method

fixed in 51b3f3c

iefremov · 2019-06-17T17:50:39Z

browser/tor/tor_profile_service_impl.cc

+                     base::Unretained(this), net::ERR_ABORTED, base::nullopt));
+  // Force lookup to erase the old circuit
+  storage_partition->GetNetworkContext()->
+    LookUpProxyForURL(url, std::move(proxy_lookup_client_ptr));


fixed in 4985fce

iefremov · 2019-06-17T17:52:29Z

generally looks good, but I will take more time to understand details.

iefremov · 2019-06-18T16:54:36Z

patches/net-proxy_resolution-proxy_resolution_service.cc.patch

+   if (current_state_ == STATE_NONE)
+     ApplyProxyConfigIfAvailable();
+
+  if (IsTorProxy(config_->value()) && raw_url.ref() == "NewTorCircuit")


IMO we should hide this under ENABLE_TOR, though not sure if it's possible

We try to make //net not to rely on any brave stuff to avoid circular dependency.

please update patches similar to discussions about sync patches

actually I'd really like to find some way to not have tor-specific stuff in ProxyResolutionService, in particular the tor_proxy_map_. Can we create a class a singleton class in Brave that keeps a map of ProxyResolutionService instance -> tor_proxy_map and use that inside a chromium_src injected method/macro?

iefremov · 2019-06-18T17:05:07Z

net/proxy_resolution/proxy_resolution_service_unittest.cc

+};
+
+TEST_F(ProxyResolutionServiceTest, TorProxy) {
+  MockAsyncProxyResolverFactory* factory =


nits: make_unique here and std::move below
const everywhere for local variables that do not change (proxy_uri, site_url, etc)

MockAsyncProxyResolverFactory will be used later

we don't actually care about factory->pending_requests().empty() so I will wrap it when constructing ProxyResolutionService

fixed in 58622cb

iefremov · 2019-06-18T17:06:04Z

net/proxy_resolution/proxy_resolution_service_unittest.cc

+  MockAsyncProxyResolverFactory* factory =
+      new MockAsyncProxyResolverFactory(false);
+  std::string proxy_uri("socks5://127.0.0.1:5566");
+  ProxyResolutionService service(


nit: we can avoid copypaste by initializing this stuff in the test class

fixed in 58622cb

iefremov · 2019-06-18T17:13:34Z

chromium_src/net/proxy_resolution/proxy_resolution_service.cc

+  ProxyConfigServiceTor tor_proxy_config_service(proxy_uri);
+  ProxyConfigWithAnnotation fetched_config;
+  tor_proxy_config_service.GetLatestProxyConfig(&fetched_config);
+  tor_proxy_config_service.SetUsername(


Some comments describing the key point would be highly appreciated

fixed in 58622cb

iefremov · 2019-06-18T17:22:38Z

browser/tor/tor_profile_service_impl.cc

+  bool success = proxy_info.has_value() && !proxy_info->is_direct();
+  if (tor_circuit_callback_) {
+    std::move(tor_circuit_callback_).Run(success);
+    binding_.Close();


shouldn't we close the binding in any case?

fixed in 58622cb

iefremov · 2019-06-18T17:34:00Z

browser/tor/tor_profile_service_impl.cc

-                                     &tor_proxy_map_,
-                                     true);
+void TorProfileServiceImpl::OnSetNewTorCircuitComplete(bool success) {
+  if (tor_circuit_callback_)


in what case this is empty? tests? Probably worth adding a comment

this function is not used, removed in 58622cb

iefremov · 2019-06-18T17:38:25Z

common/tor/tor_proxy_uri_helper.cc

+std::string GetTorProxyURI() {
+  switch (chrome::GetChannel()) {
+    case version_info::Channel::STABLE:
+      return std::string("socks5://127.0.0.1:9350");


why not introduce a named constant for 127.0.0.1 and maybe ports

fixed in 58622cb

iefremov · 2019-06-18T17:41:03Z

net/proxy_resolution/proxy_config_service_tor_unittest.cc

+  EXPECT_EQ(host_port.port(), 5566);
+  EXPECT_EQ(host_port.username(), isolation_key);
+
+  // persistent circuit isolation until timeout


small nit for all comments throughout the PR: normally the should look like a sentence, i.e. start with a Capital Letter and end with a dot

fixed in 58622cb

iefremov · 2019-06-18T17:45:18Z

LGTM with nits. My vote for merging this ASAP :) @bridiver @bsclifton @riastradh-brave WDYT?

bridiver · 2019-07-08T19:34:56Z

browser/profiles/brave_profile_manager_unittest.cc

@@ -12,13 +12,15 @@
 #include "base/files/scoped_temp_dir.h"
 #include "base/strings/utf_string_conversions.h"
 #include "brave/browser/profiles/tor_unittest_profile_manager.h"
+#include "brave/browser/tor/tor_launcher_factory.h"


we should probably keep all the tor stuff here inside ENABLE_TOR since we don't currently have an implementation for android

the same also applies for the tor-specific test files in gn

fixed in 03268dc482e62d0ff2c0573fb193528ca1508a28

bridiver · 2019-07-08T19:35:46Z

chromium_src/chrome/browser/net/proxy_config_monitor.cc

@@ -33,3 +35,6 @@ std::unique_ptr<net::ProxyConfigService> CreateProxyConfigServiceTor(
 #endif

 #include "../../../../../chrome/browser/net/proxy_config_monitor.cc"  // NOLINT
+
+// Required by lint
+#include "brave/chromium_src/chrome/browser/net/proxy_config_monitor.h"


this should go at the top, right?

can we actually disable this lint check for the chromium_src dir in chromium_src/CPPLINT.cfg?

fixed in 3bc4178

bridiver · 2019-07-08T19:38:34Z

browser/tor/tor_profile_service.cc

@@ -18,6 +16,11 @@
 #include "components/prefs/pref_registry_simple.h"
 #include "components/prefs/pref_service.h"

+#if BUILDFLAG(ENABLE_TOR)


shouldn't the entire TorProfileService be left out of gn if tor is not enabled?

fixed in 9bb8d68

bridiver · 2019-07-08T19:39:50Z

browser/tor/tor_profile_service_impl.cc

@@ -123,10 +124,12 @@ TorProfileServiceImpl::TorProfileServiceImpl(Profile* profile)
  tor_launcher_factory_ = TorLauncherFactory::GetInstance();
  tor_launcher_factory_->AddObserver(this);

+#if BUILDFLAG(ENABLE_TOR)


TorProfileServiceImpl should definitely not be built at all if tor is not enabled in gn

fixed in 9bb8d68

darkdh · 2019-07-10T00:53:54Z

CI passed based on result of
https://staging.ci.brave.com/job/brave-browser-build-pr/job/network_service_tor/38/
and
https://staging.ci.brave.com/job/brave-browser-build-pr/job/network_service_tor/39/

mkarolin · 2019-07-15T13:16:08Z

@darkdh this is marked as 0.68.x milestone, but I only see it in master (0.69.x). Are we planning to uplift to 0.68.x and 0.67.x? c76 bump is going to those branches and would need these changes to be there as well, I think.

darkdh · 2019-07-15T16:45:19Z

I must marked the milestone too early and train migration happened (it was meant to be nightly).
C76 bump depends on this so it should be wherever C76 bump uplifted to.

NetworkService migration for Tor

darkdh force-pushed the network_service_tor branch from cb2c3d2 to b3c9760 Compare June 11, 2019 01:26

darkdh marked this pull request as ready for review June 11, 2019 22:35

darkdh self-assigned this Jun 11, 2019

darkdh requested review from riastradh-brave, bridiver and bbondy June 11, 2019 22:36

darkdh force-pushed the network_service_tor branch from 071b483 to a473ce4 Compare June 11, 2019 22:51

bsclifton requested a review from iefremov June 12, 2019 16:11

darkdh commented Jun 13, 2019

View reviewed changes

patches/net-BUILD.gn.patch Show resolved Hide resolved

darkdh force-pushed the network_service_tor branch from 994a787 to 1fee089 Compare June 16, 2019 22:32

iefremov reviewed Jun 17, 2019

View reviewed changes

darkdh force-pushed the network_service_tor branch from 1fee089 to d62bea4 Compare June 17, 2019 18:02

iefremov reviewed Jun 18, 2019

View reviewed changes

Fix android build

5eeba9f

darkdh force-pushed the network_service_tor branch from 9256613 to 5eeba9f Compare July 8, 2019 17:31

bridiver reviewed Jul 8, 2019

View reviewed changes

darkdh added 2 commits July 8, 2019 12:47

Disable build/include in chromium_src

3bc4178

Fix enable_tor=false build

9bb8d68

darkdh force-pushed the network_service_tor branch 2 times, most recently from d6fb3aa to 48d0487 Compare July 9, 2019 16:49

Fix enable_tor=false tests

08e5837

darkdh force-pushed the network_service_tor branch from 48d0487 to 08e5837 Compare July 9, 2019 17:02

bridiver approved these changes Jul 9, 2019

View reviewed changes

darkdh merged commit 8b6960b into master Jul 10, 2019

darkdh deleted the network_service_tor branch July 10, 2019 00:54

darkdh added the feature/tor label Jul 15, 2019

darkdh mentioned this pull request Jul 15, 2019

Network service tor - 0.68.x #2920

Merged

darkdh modified the milestones: 0.68.x - Dev, 0.67.x - Beta Jul 15, 2019

darkdh mentioned this pull request Jul 15, 2019

Network service tor - 0.67.x #2924

Merged

darkdh added a commit that referenced this pull request Jul 15, 2019

Merge pull request #2647 from brave/network_service_tor

2598623

NetworkService migration for Tor

bsclifton pushed a commit that referenced this pull request Jul 16, 2019

Merge pull request #2647 from brave/network_service_tor

ea9edd1

NetworkService migration for Tor

iefremov mentioned this pull request Jul 16, 2019

Make Tor integration compatible with NetworkService brave/brave-browser#4312

Closed

riastradh-brave mentioned this pull request Jul 23, 2019

Limit URL requests to whitelisted schemes in windows with Tor brave/brave-browser#5357

Closed

NetworkService migration for Tor #2647

NetworkService migration for Tor #2647

Conversation

darkdh commented Jun 10, 2019 • edited Loading

Submitter Checklist:

Test Plan:

Reviewer Checklist:

After-merge Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkdh Jun 17, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iefremov Jun 17, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iefremov commented Jun 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkdh Jun 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iefremov commented Jun 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkdh commented Jul 10, 2019

mkarolin commented Jul 15, 2019

darkdh commented Jul 15, 2019

darkdh commented Jun 10, 2019 •

edited

Loading

darkdh Jun 17, 2019 •

edited

Loading

iefremov Jun 17, 2019 •

edited

Loading

darkdh Jun 18, 2019 •

edited

Loading