feat(request): Implement circuit breaking

[2k-joker]

What

• Add circuit breaking capability to optimize for "dead end" requests
• Refactor request-response implementation to accommodate this feature but also maintain backwards compatibility

Why

The circuit breaker pattern is a great optimization technique especially for a microservices architecture

[2k-joker]

Testing Steps:

Configuration
This testing steps are based on the following configurations:

{:service_id=>"dummyjson.com",                                                        
 :max_failures_count=>10,                                                             
 :min_failures_count=>5,                                                              
 :failure_rate_threshold=>0.5,                                                        
 :sample_window=>300,                                                                 
 :open_circuit_sleep_window=>30,                                                      
 :error_codes_watchlist=>[],                                                          
 :maintenance_mode_header=>"X-Maintenance-Mode-Timeout"} 

1. Mount circuit breaker

HTTPigeon.configure { |c| c.mount_circuit_breaker = true }

2. [Optional] Make at least 5 successful requests (this is to offset the min_failures_count so we can test at the failure_rate_threshold)

> req = HTTPigeon::Request.new(base_url: 'https://dummyjson.com')
> req.run(path: '/http/200') # run 6 times
I, [2024-04-15T11:05:48.907323 #11684]  INFO -- : {"request":{"method":"get","url":"https://dummyjson.com/http/200","headers":{"User-Agent":"Faraday v2.7.6","Accept":"application/json","X-Request-Id":"c403a7b2-f41d-47af-b4c7-0d8b310be1e6"},"body":{},"host":"dummyjson.com","path":"/http/200"},"response":{"headers":{"access-control-allow-origin":"*","x-dns-prefetch-control":"off","x-frame-options":"SAMEORIGIN","strict-transport-security":"max-age=15552000; includeSubDomains","x-download-options":"noopen","x-content-type-options":"nosniff","x-xss-protection":"1; mode=block","x-ratelimit-limit":"100","x-ratelimit-remaining":"98","date":"Mon, 15 Apr 2024 15:05:48 GMT","x-ratelimit-reset":"1713193558","content-type":"application/json; charset=utf-8","content-length":"31","etag":"W/\"1f-avjefNgMMGfp/0DAyGrQhdM3dgk\"","vary":"Accept-Encoding","server":"railway"},"body":{"status":"200","message":"OK"},"status":200},"metadata":{"latency":0.283092,"identifier":"c403a7b2-f41d-47af-b4c7-0d8b310be1e6","protocol":"https"},"event_type":"http.outbound"}

> req.fuse.success_count
 => 6
> req.fuse.failure_count
 => 0 
> req.fuse.failure_rate
 => 0.0
> req.fuse.half_open?
 => false 
> req.fuse.open?
 => false

4. Make at least 5 failing requests

• The circuit should be half_open after this, and if your failure rate is at least 50%, the circuit should be :open too
• If not at >= 50% failure rate, make more failing requests, circuit should be open after this

> req.run(path: '/http/500') # run 5 times
I, [2024-04-15T11:06:38.119049 #11684]  INFO -- : {"request":{"method":"get","url":"https://dummyjson.com/http/500","headers":{"User-Agent":"Faraday v2.7.6","Accept":"application/json","X-Request-Id":"6c08ee46-b221-44b6-a004-bcfa742cad30"},"body":{},"host":"dummyjson.com","path":"/http/500"},"response":{"headers":{"access-control-allow-origin":"*","x-dns-prefetch-control":"off","x-frame-options":"SAMEORIGIN","strict-transport-security":"max-age=15552000; includeSubDomains","x-download-options":"noopen","x-content-type-options":"nosniff","x-xss-protection":"1; mode=block","x-ratelimit-limit":"100","x-ratelimit-remaining":"99","date":"Mon, 15 Apr 2024 15:06:38 GMT","x-ratelimit-reset":"1713193608","content-type":"application/problem+json; charset=utf-8","content-length":"136","etag":"W/\"88-Z2OQ5sBvrNHMIxaGoVQySXIz5ds\"","vary":"Accept-Encoding","server":"railway"},"body":{"status":"500","title":"Internal Server Error","type":"about:blank","detail":"Internal Server Error","message":"Internal Server Error"},"status":500},"metadata":{"latency":0.284771,"identifier":"6c08ee46-b221-44b6-a004-bcfa742cad30","protocol":"https"},"event_type":"http.outbound"}                          
/Users/khalilkum/Documents/apps/httpigeon/lib/httpigeon/middleware/circuit_breaker.rb:17:in `on_complete': the server responded with status 500 (HTTPigeon::Middleware::CircuitBreaker::FailedRequestError) 
I, [2024-04-15T11:06:44.507087 #11684]  INFO -- : {"event_type":"httpigeon.fuse.circuit_half_opened","service_id":"dummyjson.com","request_id":"ae235f3b-ec3f-4314-8e30-c6bb4f246b4a","circuit_state":"half_open","success_count":6,"failure_count":5,"failure_rate":0.45454545454545453,"recorded_at":1713193604}

> req.fuse.failure_rate
 => 0.45454545454545453 
3.1.4 :022 > req.fuse.failure_count
 => 5 
> req.fuse.half_open?
 => true 
> req.fuse.open?
 => false 
> req.run(path: '/http/500') # should open the fuse after this
I, [2024-04-15T11:07:11.784969 #11684]  INFO -- : {"event_type":"httpigeon.fuse.circuit_opened","service_id":"dummyjson.com","request_id":"84871081-17a5-4141-9b70-e6a139b732ab","circuit_state":"open","success_count":6,"failure_count":0,"failure_rate":0.0,"recorded_at":1713193631}
> req.fuse.half_open?
 => true 
> req.fuse.open?
 => true 
# resets the failure/count to give the circuit multiple opportunities to recover
> req.fuse.failure_rate
 => 0.0
> req.fuse.failure_count
 => 0

5. While the circuit is open, any request should be short-circuited

# Should be short-circuited (i.e handled by the `on_open_circuit` callback
> req.run(path: '/http/200')
I, [2024-04-15T11:27:57.539639 #11684]  INFO -- : {"event_type":"httpigeon.fuse.execution_skipped","service_id":"dummyjson.com","request_id":"061a71c7-7787-433a-82d6-4d6e6ebb7886","circuit_state":"open","success_count":1,"failure_count":0,"failure_rate":0.6666666666666666,"recorded_at":1713194877}
> req.run(path: '/http/500')
I, [2024-04-15T11:26:24.748938 #11684]  INFO -- : {"event_type":"httpigeon.fuse.execution_skipped","service_id":"dummyjson.com","request_id":"4d108854-4f36-4eb4-ae8f-6123d4262db4","circuit_state":"open","success_count":0,"failure_count":0,"failure_rate":1.0,"recorded_at":1713194784}

7. After 30 seconds (see configs above), the circuit should be back in half_open state only

• Any failing requests shouldn't re-open the circuit immediately,
• Any successful request should close the circuit and reset the failure_count

> sleep(30)
 => 30 
> req.fuse.open?
 => false 
> req.fuse.half_open?
 => true 
> req.fuse.failure_rate
 => 0.0
> req.run(path: '/http/500') # should go over the wire but not open circuit
> req.fuse.half_open?
 => true 
> req.fuse.open?
 => false
> req.run(path: '/http/200') # should close the circuit and reset failure count/rate
I, [2024-04-15T11:08:48.725243 #11684]  INFO -- : {"event_type":"httpigeon.fuse.circuit_closed","service_id":"dummyjson.com","request_id":"a13dbf34-dde5-4a12-8526-d0cd82205ee4","circuit_state":"closed","success_count":7,"failure_count":0,"failure_rate":0.0,"recorded_at":1713193728}
> req.fuse.half_open?
 => false 
> req.fuse.open?
 => false  

[harry-stebbins-dailypay]

🚀

[ShivamPatel17]

If I want to use different configs to call different endpoints in a service, would I just need to use a different service_id?

[ShivamPatel17]

nice!! didn't go through every case with the configs but tested with the steps you posted and looked over the code.

[2k-joker]

If I want to use different configs to call different endpoints in a service, would I just need to use a different service_id?

I think the idea of the circuitbreaker is really to check for service health in general and not just per endpoint. I contemplated supported passing a fuse per endpoint but imo that's not using the tool properly. I can be otherwise tho

[ShivamPatel17]

If I want to use different configs to call different endpoints in a service, would I just need to use a different service_id?

I think the idea of the circuitbreaker is really to check for service health in general and not just per endpoint. I contemplated supported passing a fuse per endpoint but imo that's not using the tool properly. I can be otherwise tho

Definitely makes sense for service health if that's something that gets checked often. But if service health is only checked on start up for example, I'd think adding a circuit breaker anywhere a request might retry is reasonable. What do you think?

[2k-joker]

If I want to use different configs to call different endpoints in a service, would I just need to use a different service_id?

I think the idea of the circuitbreaker is really to check for service health in general and not just per endpoint. I contemplated supported passing a fuse per endpoint but imo that's not using the tool properly. I can be otherwise tho

Definitely makes sense for service health if that's something that gets checked often. But if service health is only checked on start up for example, I'd think adding a circuit breaker anywhere a request might retry is reasonable. What do you think?

Oh service health as in not literally the healthcheck endpoint but more like tripping should be at the service level versus per endpoint level

[ShivamPatel17]

If I want to use different configs to call different endpoints in a service, would I just need to use a different service_id?

I think the idea of the circuitbreaker is really to check for service health in general and not just per endpoint. I contemplated supported passing a fuse per endpoint but imo that's not using the tool properly. I can be otherwise tho

Definitely makes sense for service health if that's something that gets checked often. But if service health is only checked on start up for example, I'd think adding a circuit breaker anywhere a request might retry is reasonable. What do you think?

Oh service health as in not literally the healthcheck endpoint but more like tripping should be at the service level versus per endpoint level

ahhhh understood 👍