Skip to content

Commit

Permalink
add a warning if we think istio-proxy injection is causing problems
Browse files Browse the repository at this point in the history
We have encountered situations where the injection of istio-proxy in a
router pod (executing in kubernetes) causes strange networking errors
during uplink retrieval.

The root cause of the issue is that the router is executing and
attempting retrieve uplink schemas whilst the istio-proxy is modifying
network configuration at the same time.

This warning message will direct users to information which should help
them to configure their cluster or pod to avoid this problem.

fixes: #3533
  • Loading branch information
garypen committed Aug 7, 2023
1 parent 19ea4d9 commit 5d57ee9
Showing 1 changed file with 22 additions and 1 deletion.
23 changes: 22 additions & 1 deletion apollo-router/src/uplink/mod.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use std::error::Error as stdError;
use std::fmt::Debug;
use std::time::Duration;
use std::time::Instant;
Expand Down Expand Up @@ -359,7 +360,27 @@ where
Query: graphql_client::GraphQLQuery,
{
let client = reqwest::Client::builder().timeout(timeout).build()?;
let res = client.post(url).json(request_body).send().await?;
// It is possible that istio-proxy is re-configuring networking beneath us. If it is, we'll see an error something like this:
// level: "ERROR"
// message: "fetch failed from all endpoints"
// target: "apollo_router::router::event::schema"
// timestamp: "2023-08-01T10:40:28.831196Z"
// That's deeply confusing and very hard to debug. Let's try to help by printing out a helpful error message here
let res = client
.post(url)
.json(request_body)
.send()
.await
.map_err(|e| {
if let Some(hyper_err) = e.source() {
if let Some(os_err) = hyper_err.source() {
if os_err.to_string().contains("tcp connect error: Cannot assign requested address (os error 99)") {
tracing::warn!("If your router is executing within a kubernetes pod, this failure may be caused by istio-proxy injection. See https://github.com/apollographql/router/issues/3533 for more details about how to solve this");
}
}
}
e
})?;
tracing::debug!("uplink response {:?}", res);
let response_body: graphql_client::Response<Query::ResponseData> = res.json().await?;
Ok(response_body)
Expand Down

0 comments on commit 5d57ee9

Please sign in to comment.