Include URI query string in proxied requests

[justinsteven]

Category

Bug

Feature/Issue Description

Q: Please give a brief summary of your feature/fix
A: Proxied requests with query string params (e.g. https://foo.example/bar?fizz=buzz) were being sent without the query string (i.e. https://foo.example/bar)

Q: Give a technical rundown of what you have changed (if applicable)
A: If the URI has a query then construct a path that includes the query string

Test Cases

Q: Describe your test cases, what you have covered and if there are any use cases that still need addressing.
A:

No test cases written. I've done simple testing in the lab only. I'd appreciate a careful look and further testing.

✔️ Proxied requests without a query string work, and are sent without a query string
✔️ Proxied requests with a query string (?foo) work, and are sent with the correct query string
✔️ Proxied requests with a query string (?foo=bar) work, and are sent with the correct query string

Wiki Page

If you are adding a new feature that is not easily understood without context, please draft a section to be added to the Wiki below.

N/A

[bcoles]

A wild justinsteven appears.

Strange. I'm pretty sure this used to work. I wonder when/how this stopped working... I'll take a look.

[bcoles]

I tried proxing a request on an old version of Firefox and the query string was sent ok.

But I encountered a different bug. The response body is never returned, regardless of whether the destination page was cross-origin. Tried on old Chrome and old Firefox, with and without this patch.

I don't see any immediate harm in including this patch, but the proxy needs further review and testing (by someone who isn't me).

[jackdwalker]

I'll try and spend some time this evening having a look.

[justinsteven]

@bcoles o/

You said:

I tried proxing a request on an old version of Firefox and the query string was sent ok

Interesting. It's definitely broken for me. I'll get to that soon.

I encountered a different bug. The response body is never returned, regardless of whether the destination page was cross-origin. Tried on old Chrome and old Firefox, with and without this patch.

I don't think I've seen that. I'll keep an eye out.

Oh, that is, unless beef is throwing ActiveRecord::ConnectionTimeoutError exceptions in your terminal. I've seen that, and it causes a lot of things in beef to break/hang/chug. See my dump below for some thoughts on that.

the proxy needs further review and testing (by someone who isn't me).

+1

I'm using proxy in the lab right now for a POC, but nothing I'd call real-world use. Someone who knows the guts of it should definitely take a look.

The brokennness

I've done varying tests with the hooked URL being http and https; the proxy destination URL being http and https; proxy clients being Firefox and curl. No luck on any account - the query string is never included by the hooked browser.

I've unwound my patch and the crazy reverse proxy setup I was using and it still seems buggy to me.

Ruby:

% ruby -v
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]

% bundle
Using rake 13.0.1
Using concurrent-ruby 1.1.6
Using i18n 1.8.4
Using minitest 5.14.1
Using thread_safe 0.3.6
Using tzinfo 1.2.7
Using zeitwerk 2.4.0
Using activesupport 6.0.3.2
Using activemodel 6.0.3.2
Using activerecord 6.0.3.2
Using public_suffix 4.0.5
Using addressable 2.7.0
Using ansi 1.5.0
Using io-like 0.3.1
Using archive-zip 0.12.0
Using console 1.8.2
Using nio4r 2.5.2
Using timers 4.3.0
Using async 1.26.2
Using async-io 1.30.0
Using async-dns 1.2.5
Using browserstack-local 1.3.0
Using buftok 0.2.0
Using bundler 2.1.4
Using byebug 11.1.3
Using mini_mime 1.0.2
Using mini_portile2 2.4.0
Using nokogiri 1.10.10
Using rack 2.2.3
Using rack-test 1.1.0
Using regexp_parser 1.7.1
Using xpath 3.2.0
Using capybara 3.33.0
Using childprocess 3.0.0
Using chunky_png 1.3.11
Using coderay 1.1.3
Using curb 0.9.10
Using daemons 1.3.1
Using diff-lcs 1.4.4
Using unf_ext 0.0.7.7
Using unf 0.1.4
Using domain_name 0.5.20190701
Using eventmachine 1.2.7
Using http_parser.rb 0.6.0
Using em-websocket 0.5.1
Using equalizer 0.0.11
Using erubis 2.7.0
Using espeak-ruby 1.0.4
Using event_emitter 0.2.6
Using execjs 2.7.0
Using ffi 1.13.1
Using ffi-compiler 1.0.1
Using geckodriver-helper 0.24.0
Using hashie 4.1.0
Using hashie-forbidden_attributes 0.1.1
Using http-cookie 1.0.3
Using http-form_data 2.3.0
Using http-parser 1.2.1
Using http 4.4.1
Using http-accept 1.7.0
Using io-console 0.5.6
Using reline 0.1.4
Using irb 1.2.4
Using json 2.3.1
Using maxmind-db 1.1.1
Using memoizable 0.4.2
Using method_source 1.0.0
Using mime-types-data 3.2020.0512
Using mime-types 3.3.1
Using mojo_magick 0.5.6
Using msgpack 1.3.3
Using msfrpc-client 1.1.2
Using multipart-post 2.1.1
Using ruby2_keywords 0.0.2
Using mustermann 1.1.1
Using naught 1.1.0
Using netrc 0.11.0
Using otr-activerecord 1.4.1
Using parseconfig 1.0.8
Using power_assert 1.2.0
Using pry 0.13.1
Using pry-byebug 3.9.0
Using rqrcode_core 0.1.2
Using rqrcode 1.1.2
Using qr4r 0.4.1
Using rack-protection 2.0.8.1
Using rdoc 6.2.1
Using rest-client 2.1.0
Using rr 1.2.1
Using rspec-support 3.9.3
Using rspec-core 3.9.2
Using rspec-expectations 3.9.2
Using rspec-mocks 3.9.1
Using rspec 3.9.0
Using rubyzip 2.3.0
Using rushover 0.3.0
Using selenium-webdriver 3.142.7
Using simple_oauth 0.3.1
Using tilt 2.0.10
Using sinatra 2.0.8.1
Using slack-notifier 2.3.2
Using sqlite3 1.4.2
Using sync 0.5.0
Using tins 1.25.0
Using term-ansicolor 1.7.1
Using test-unit 3.3.6
Using test-unit-context 0.5.1
Using test-unit-notify 1.0.4
Using test-unit-rr 1.0.5
Using test-unit-runner-tap 1.1.2
Using test-unit-full 0.0.5
Using thin 1.7.2
Using twitter 7.0.0
Using uglifier 4.2.0
Using websocket 1.2.8
Using websocket-client-simple 0.3.0
Using xmlrpc 0.3.0
Bundle complete! 41 Gemfile dependencies, 117 gems now installed.
Gems in the group development were not installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

My environment has the locale fix per #1290 (comment)

I'm at commit f3313fd of beef with the following changes:

% git diff
diff --git a/beef b/beef
index 1c85abab..569b995b 100755
--- a/beef
+++ b/beef
@@ -166,7 +166,8 @@ end
 # Connect to DB
 ActiveRecord::Base.logger = nil
 OTR::ActiveRecord.migrations_paths = [File.join('core', 'main', 'ar-migrations')]
-OTR::ActiveRecord.configure_from_hash!(adapter:'sqlite3', database:db_file)
+#OTR::ActiveRecord.configure_from_hash!(adapter:'sqlite3', database:db_file)
+OTR::ActiveRecord.configure_from_hash!(adapter:'sqlite3', database:db_file, pool:250)
 # Migrate (if required)
 context = ActiveRecord::Migration.new.migration_context
 if context.needs_migration?
diff --git a/config.yaml b/config.yaml
index d4d93e81..4ca906a2 100644
--- a/config.yaml
+++ b/config.yaml
@@ -8,7 +8,7 @@
 beef:
     version: '0.5.0.0-alpha-pre'
     # More verbose messages (server-side)
-    debug: false
+    debug: true
     # More verbose messages (client-side)
     client_debug: false
     # Used for generating secure tokens
@@ -18,15 +18,15 @@ beef:
     # Used by both the RESTful API and the Admin interface
     credentials:
         user:   "beef"
-        passwd: "beef"
+        passwd: "<SNIP>"
 
     # Interface / IP restrictions
     restrictions:
         # subnet of IP addresses that can hook to the framework
         permitted_hooking_subnet: ["0.0.0.0/0", "::/0"]
         # subnet of IP addresses that can connect to the admin UI
-        #permitted_ui_subnet: ["127.0.0.1/32", "::1/128"]
-        permitted_ui_subnet: ["0.0.0.0/0", "::/0"]
+        permitted_ui_subnet: ["127.0.0.1/32", "::1/128"]
+        #permitted_ui_subnet: ["0.0.0.0/0", "::/0"]
         # slow API calls to 1 every  api_attempt_delay  seconds
         api_attempt_delay: "0.05"

Regarding the ActiveRecord pool diff - that's a dirty hack I have to fix the following errors:

/home/justin/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:221:in `block in wait_poll': could not obtain a connection from t
he pool within 5.000 seconds (waited 137.317 seconds); all pooled connections were in use (ActiveRecord::ConnectionTimeoutError)

I don't know why I'm getting this error

• Does my reverse proxying setup cause weirdness or load in beef? 🤷
• Does proxying cause weirdness or load in beef? 🤷

But I got sick of seeing the error (and having beef go sideways until I bounced it) so setting the pool to a stupid high/unreasonable number has been a bit of a talisman for me. It probably warrants further investigation by someone who isn't me, or me when I have more time.

Back to the proxy.

With the hooked URL of http://127.0.0.2:3000/demos/basic.html, Proxy with a query string is definitely broken for same-site and cross-site requests

beef debug:

[11:59:14][>] Unsetting previously HB [127.0.0.1] used as Tunneling Proxy                                
[11:59:14][*] Using Hooked Browser with ip [127.0.0.1] as Tunneling Proxy                                

Use the proxy cross-site:

% curl -v -x http://127.0.0.1:6789 'http://1.1.1.1/path?query'    
* Expire in 0 ms for 6 (transfer 0x55e9b1ffaf50)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55e9b1ffaf50)
* Connected to 127.0.0.1 (127.0.0.1) port 6789 (#0)
> GET http://1.1.1.1/path?query HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 -1
* no chunk, no close, no size. Assume close to signal end                                                
<
ERROR: Cross Domain Request. The request was sent however it is impossible to view the response.         
* Closing connection 0

beef debug (Filtering for just the proxy stuff related to the above request):

[11:59:36][>] [Proxy] Proxy browser not set. Defaulting to first hooked browser [id: WCqc9Lr9hZJtoyuVXBdLQqHyDDVfcS2FXWU2zMzpT0LYz8FDuBO36NQD1rf63OwfrL8jcxd4mlyJtTYk]
[11:59:36][>] [PROXY] --> Forwarding request #62: domain[1.1.1.1:80], method[GET], path[/path], cross domain[true]
[11:59:37][>] http_db_object:
[11:59:37][>] {"id":62,"hooked_browser_id":"WCqc9Lr9hZJtoyuVXBdLQqHyDDVfcS2FXWU2zMzpT0LYz8FDuBO36NQD1rf63OwfrL8jcxd4mlyJtTYk","request":"GET /path HTTP/1.1\r\nHost: 1.1.1.1\r\nUser-Agent: curl/7.64.0\r\nAccept: */*\r\nProxy-Connection: Keep-Alive\r\n\r\n","allow_cross_domain":true,"response_data":null,"response_status_code":null,"response_status_text":null,"response_port_status":null,"response_headers":null,"method":"GET","content_length":"0","proto":"http","domain":"1.1.1.1","port":"80","has_ran":"waiting","path":"/path","response_date":null,"request_date":"2020-07-21T01:59:36.570Z"}
[11:59:37][>] result http_request_object
[11:59:37][>] {"id":62,"method":"GET","proto":"http","host":"1.1.1.1","port":"80","uri":"/path","headers":{"Host":"1.1.1.1","User-Agent":"curl/7.64.0","Accept":"*/*","Proxy-Connection":"Keep-Alive"},"allowCrossDomain":"true"}
[11:59:39][>] [PROXY] <-- Response for request #62 to [/path] on domain [1.1.1.1:80] correctly processed
[11:59:39][>] Event: 688.375s - [Console] log: [2020-07-21 11:59:38] [Requester] GET http://1.1.1.1:80/path - Data: undefined

(I don't know why beef is complaining about "Proxy browser not set", I've definitely set it)

If I watch the hooked browser's F12 "Network" tab, the request to http://1.1.1.1/path is sent without the querystring

---

Use the proxy same-site:

% curl -v -x http://127.0.0.1:6789 'http://127.0.0.2:3000/path?query'
* Expire in 0 ms for 6 (transfer 0x564ef2619f50)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x564ef2619f50)
* Connected to 127.0.0.1 (127.0.0.1) port 6789 (#0)
> GET http://127.0.0.2:3000/path?query HTTP/1.1
> Host: 127.0.0.2:3000
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 404
< content-length: 233
< content-type: text/html; charset=UTF-8
< server: Apache/2.2.3 (CentOS)
< x-cascade: pass
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
<
* Connection #0 to host 127.0.0.1 left intact
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL was not found on this server.</p><hr><address>Apache/2.2.3 (CentOS)</address></body></html>

beef debug (Filtering for just the proxy stuff related to the above request):

[12:17:31][>] [Proxy] Proxy browser not set. Defaulting to first hooked browser [id: WCqc9Lr9hZJtoyuVXBdLQqHyDDVfcS2FXWU2zMzpT0LYz8FDuBO36NQD1rf63OwfrL8jcxd4mlyJtTYk]
[12:17:31][>] [PROXY] --> Forwarding request #183: domain[127.0.0.2:3000], method[GET], path[/path], cross domain[true]
[12:17:31][>] http_db_object:
[12:17:31][>] {"id":183,"hooked_browser_id":"WCqc9Lr9hZJtoyuVXBdLQqHyDDVfcS2FXWU2zMzpT0LYz8FDuBO36NQD1rf63OwfrL8jcxd4mlyJtTYk","request":"GET /path HTTP/1.1\r\nHost: 127.0.0.2:3000\r\nUser-Agent: curl/7.64.0\r\nAccept: */*\r\nProxy-Connection: Keep-Alive\r\n\r\n","allow_cross_domain":true,"response_data":null,"response_status_code":null,"response_status_text":null,"response_port_status":null,"response_headers":null,"method":"GET","content_length":"0","proto":"http","domain":"127.0.0.2","port":"3000","has_ran":"waiting","path":"/path","response_date":null,"request_date":"2020-07-21T02:17:31.456Z"}
[12:17:31][>] result http_request_object
[12:17:31][>] {"id":183,"method":"GET","proto":"http","host":"127.0.0.2","port":"3000","uri":"/path","headers":{"Host":"127.0.0.2:3000","User-Agent":"curl/7.64.0","Accept":"*/*","Proxy-Connection":"Keep-Alive"},"allowCrossDomain":"true"}
[12:17:32][>] [PROXY] <-- Response for request #183 to [/path] on domain [127.0.0.2:3000] correctly processed
[12:17:33][>] Event: 1762.314s - [Console] log: [2020-07-21 12:17:32] [Requester] GET http://127.0.0.2:3000/path - Data: undefined

If I watch the hooked browser's F12 "Network" tab, the request to http://127.0.0.2:3000/path is sent without the querystring

[bcoles]

% curl -v -x http://127.0.0.1:6789 'http://127.0.0.2:3000/path?query'

Oh right... the proxy. Not the Rider/Requester. That would explain it.

Confirmed.

Without this patch, the URL query string is not sent. With the patch it is sent.

Edit: The issue related to the response body not showing the Proxy history tab exists before and after this patch.

Edit 2: The proxy and what used to be called the Rider/Requester are very similar in the UI, and in the server-side code, and in the client side hook API. The code should probably be merged into one extension (called "proxy"), with the option to switch the local proxy server (on port 6789) on or off in the config.

[justinsteven]

Oh right... the proxy. Not the Rider/Requester

😅 👍

That'll teach me to write a brief PR message

Glad we're on the same page. And yep, I know what you mean now, the Proxy history tab is broken for me too.