RCPP-89 Add 301/308 redirection support for HTTP transport requests #242

michael-wb · 2024-08-28T15:33:31Z

What, How & Why?

Since the 301/308 http redirection support is being removed from Core via realm/realm-core#7996, these changes add support to the CPP SDK network transport to support redirections.

Fixes #239

michael-wb · 2024-08-28T15:35:18Z

include/cpprealm/networking/http.hpp

+        virtual void send_request_to_server(::realm::networking::request &&request,
+                                            std::function<void(const response &)> &&completion) = 0;


git clang-format formatted these files differently than what is currently configured for realm-core. The .clang-format says it was based off the CLion configuration.

jbreams · 2024-08-29T18:47:55Z

include/cpprealm/networking/http.hpp

@@ -115,7 +115,7 @@ namespace realm::networking {
    [[maybe_unused]] void set_http_client_factory(std::function<std::shared_ptr<http_transport_client>()>&&);

    /// Built in HTTP transport client.
-    struct default_http_transport : public http_transport_client {
+    struct default_http_transport : public http_transport_client, public std::enable_shared_from_this<default_http_transport> {


why do we need to change the ownership/lifetime of this type from unique to shared?

I was using to ensure the object was still valid in the async callbacks. Fortunately (or unfortunately) like you said, the service object created and run for each call of this function is essentially updating this operation to be completely synchronous and blocking until the response is received from the server...

This "threading" (or lack thereof) for the transport implementation should be revisited in a separate task to ensure it is working in an appropriate asynchronous manner.

jbreams · 2024-08-29T18:55:17Z

src/cpprealm/internal/networking/network_transport.cpp

@@ -243,45 +246,45 @@ namespace realm::networking {
            socket.ssl_stream->set_logger(logger.get());
        }

-        realm::sync::HTTPHeaders headers;
+        realm::sync::HTTPClient<DefaultSocket> m_http_client = realm::sync::HTTPClient<DefaultSocket>(socket, logger);


you can construct this directly like realm::sync::HTTPClient<DefaultSocket> m_http_client(socket, logger) without any assignment.

Updated - I hadn't changed it from how the original code was doing it below.

jbreams · 2024-08-29T18:57:15Z

src/cpprealm/internal/networking/network_transport.cpp

+                    m_http_client.async_request(std::move(http_req),
+                                                [self = weak_from_this(), orig_request = std::move(request), cb = std::move(completion_block), redirect_count](const realm::sync::HTTPResponse &resp, const std::error_code &ec) mutable {
+                                                    constexpr std::string_view location_header = "location";
+                                                    auto transport = self.lock();


even though this "async_request" appears to be done asynchronously, the network::Service is actually unique to each request and is run synchronously at the end of this function. we shouldn't have to do anything to keep any of this alive for the duration of the request.

thinking about this a little more, does this mean that for each redirect we're creating a new network::Service and a new resolver thread?

jbreams · 2024-08-29T18:59:46Z

src/cpprealm/internal/networking/network_transport.cpp

-                    });
+                    // Pass along the original request so it can be resent to the redirected location URL if needed
+                    m_http_client.async_request(std::move(http_req),
+                                                [self = weak_from_this(), orig_request = std::move(request), cb = std::move(completion_block), redirect_count](const realm::sync::HTTPResponse &resp, const std::error_code &ec) mutable {


i think the signature for the handler for HTTPClient::async_request is void(HTTPResponse, std::error_code). so we dont need to take these by const ref.

Updated - this was from the original code, and it's actually better for resp not to be a const ref since the response object is being std::move'd when calling this function.

jbreams · 2024-08-29T19:03:15Z

src/cpprealm/internal/networking/network_transport.cpp

+                                                            // to perform redirections every time.
+                                                            std::string redirect_url;
+                                                            // Grab the new location from the 'Location' header and retry the request
+                                                            for (auto &[key, value]: resp.headers) {


the headers here is a std::map<std::string, std::string, HeterogeneousCaseInsensitiveCompare> so you should be able to just query for the location header by headers.find("location"). did that not work for some reason?

Updated - for some reason I was convinced this was a different type...

jbreams · 2024-08-29T19:29:40Z

src/cpprealm/internal/networking/network_transport.cpp

+                                                                // Otherwise, resend the original request to the new redirect URL path
+                                                                // Perform the entire operation again, since the remote host is likely changing and
+                                                                // a newe socket will need to be opened,
+                                                                orig_request.url = redirect_url;


i believe in addition to updating the url we need to remove the Authorization header from the request. I think this is the standard for how to follow redirects in various browsers like webkit/chromium. Technically you're supposed to remove the authorization header if the new request's URL is at a new origin, but webkit has historically just removed the authorization header unconditionally.

Yes - I wasn't sure of everything that was being updated as part of resending the packet after a redirect request, and the work to use redirection in the curl library showed that the Authorization header is removed.

Updated to remove this header before presenting the request for a redirect.

jbreams · 2024-08-29T19:31:34Z

src/cpprealm/internal/networking/network_transport.cpp

+                                                    }
+                                                    ::realm::networking::response res{static_cast<int>(resp.status), 0, {}, resp.body ? std::move(*resp.body) : "", std::nullopt};
+                                                    // Copy over all the headers
+                                                    for (auto &[k, v]: resp.headers) {


you could also do res.headers = http_headers(resp.headers.begin(), resp.headers.end())

michael-wb

Thanks for the review - I think updating the use of the service object by the default_http_transport class should be revisited as a separate task, since these changes will likely need additional testing than what is currently provided in the sync tests.

michael-wb · 2024-08-29T19:46:04Z

src/cpprealm/internal/networking/network_transport.cpp

@@ -243,45 +246,45 @@ namespace realm::networking {
            socket.ssl_stream->set_logger(logger.get());
        }

-        realm::sync::HTTPHeaders headers;
+        realm::sync::HTTPClient<DefaultSocket> m_http_client = realm::sync::HTTPClient<DefaultSocket>(socket, logger);


Updated - I hadn't changed it from how the original code was doing it below.

michael-wb · 2024-08-29T19:50:09Z

src/cpprealm/internal/networking/network_transport.cpp

-                    });
+                    // Pass along the original request so it can be resent to the redirected location URL if needed
+                    m_http_client.async_request(std::move(http_req),
+                                                [self = weak_from_this(), orig_request = std::move(request), cb = std::move(completion_block), redirect_count](const realm::sync::HTTPResponse &resp, const std::error_code &ec) mutable {


Updated - this was from the original code, and it's actually better for resp not to be a const ref since the response object is being std::move'd when calling this function.

michael-wb · 2024-08-29T20:00:03Z

src/cpprealm/internal/networking/network_transport.cpp

+                                                            // to perform redirections every time.
+                                                            std::string redirect_url;
+                                                            // Grab the new location from the 'Location' header and retry the request
+                                                            for (auto &[key, value]: resp.headers) {


Updated - for some reason I was convinced this was a different type...

michael-wb · 2024-08-29T20:44:57Z

include/cpprealm/networking/http.hpp

@@ -115,7 +115,7 @@ namespace realm::networking {
    [[maybe_unused]] void set_http_client_factory(std::function<std::shared_ptr<http_transport_client>()>&&);

    /// Built in HTTP transport client.
-    struct default_http_transport : public http_transport_client {
+    struct default_http_transport : public http_transport_client, public std::enable_shared_from_this<default_http_transport> {


I was using to ensure the object was still valid in the async callbacks. Fortunately (or unfortunately) like you said, the service object created and run for each call of this function is essentially updating this operation to be completely synchronous and blocking until the response is received from the server...

This "threading" (or lack thereof) for the transport implementation should be revisited in a separate task to ensure it is working in an appropriate asynchronous manner.

michael-wb · 2024-08-29T20:48:51Z

src/cpprealm/internal/networking/network_transport.cpp

+                                                                // Otherwise, resend the original request to the new redirect URL path
+                                                                // Perform the entire operation again, since the remote host is likely changing and
+                                                                // a newe socket will need to be opened,
+                                                                orig_request.url = redirect_url;


Yes - I wasn't sure of everything that was being updated as part of resending the packet after a redirect request, and the work to use redirection in the curl library showed that the Authorization header is removed.

Updated to remove this header before presenting the request for a redirect.

Added initial redirect support for CPP SDK

3db8f96

michael-wb self-assigned this Aug 28, 2024

michael-wb commented Aug 28, 2024

View reviewed changes

Michael Wilkerson-Barker added 3 commits August 28, 2024 12:11

Updated changelog

8918f13

Added max redirect support; reverted debug output that was left on

08490c0

Clang format

011cc91

jbreams reviewed Aug 29, 2024

View reviewed changes

michael-wb commented Aug 29, 2024

View reviewed changes

Updates from review

c85aaf4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RCPP-89 Add 301/308 redirection support for HTTP transport requests #242

RCPP-89 Add 301/308 redirection support for HTTP transport requests #242

michael-wb commented Aug 28, 2024

michael-wb Aug 28, 2024

jbreams Aug 29, 2024

michael-wb Aug 29, 2024

jbreams Aug 29, 2024

michael-wb Aug 29, 2024

jbreams Aug 29, 2024

jbreams Aug 29, 2024

jbreams Aug 29, 2024

michael-wb Aug 29, 2024

jbreams Aug 29, 2024

michael-wb Aug 29, 2024

jbreams Aug 29, 2024

michael-wb Aug 29, 2024

jbreams Aug 29, 2024

michael-wb left a comment

michael-wb Aug 29, 2024

michael-wb Aug 29, 2024

michael-wb Aug 29, 2024

michael-wb Aug 29, 2024

michael-wb Aug 29, 2024

		virtual void send_request_to_server(::realm::networking::request &&request,
		std::function<void(const response &)> &&completion) = 0;

RCPP-89 Add 301/308 redirection support for HTTP transport requests #242

Are you sure you want to change the base?

RCPP-89 Add 301/308 redirection support for HTTP transport requests #242

Conversation

michael-wb commented Aug 28, 2024

What, How & Why?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-wb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment