Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mojaloop Connector - 504 gateway timeout response on high onload #3740

Closed
vijayg10 opened this issue Jan 29, 2024 · 2 comments
Closed

Mojaloop Connector - 504 gateway timeout response on high onload #3740

vijayg10 opened this issue Jan 29, 2024 · 2 comments
Assignees
Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service core-dev-squad oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it PI-23

Comments

@vijayg10
Copy link

vijayg10 commented Jan 29, 2024

Summary:
<Description regarding what the issue is (with important details) and where it is>

Severity:
(Low, Medium, High)

Priority:
(Critical, Medium, Low)

Expected Behavior

Steps to Reproduce

  1. Move into the harness - mojaloop/3670 branch
  2. Run the k6 tests for SDK inbound/outbound
  3. Based on config, the tests will show some 504 responses

Specifications

  • Component (if known): SDK
  • Version: v23.1.1
  • Platform: Docker-Compose/Linux
  • Subsystem: Ubuntu
  • Type of testing: k6 tests
  • Bug found/raised by: @aaronreynoza

Notes:

  • Configuring the amount of VUs for a given tests seem to affect the 504s responses

  • SDK contains lots of deepCopies with JSON.parse(JSON.stringify(object))

  • Severity when opened:

  • Priority when opened:

Tasks

@vijayg10 vijayg10 added bug Something isn't working or it has wrong behavior on a Mojaloop Core service oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it core-dev-squad labels Jan 29, 2024
@kleyow
Copy link

kleyow commented Feb 5, 2024

This is a non-issue atm after discussing with @aaronreynoza. He is not experiencing 504's with his latest changes.

@kleyow
Copy link

kleyow commented Mar 14, 2024

This story refers to testing the Outbound /transfers endpoint of the SDK.
I believe this issue is fixed in mojaloop/sdk-scheme-adapter#471

Details pull from the PR
Details below

// The unsubscribeTimeout is used to mitigate a believed issue happening in the
// parties lookup leg of transfers. When the same party is looked up multiple times in quick
// succession, the cache is subscribed to the same channel multiple times. We believe that
// requests that have just subscribed to the channel which have not received the message yet
// are getting unsubscribed when requests that have completed call unsubscribe.
// This leads the request state machine to timeout the request, fail and
// stall the service. This issue is only affects parties lookup since it is the only
// pub/sub that can use the same channel name, primarily in our ml-core-test-harness environment.

This prevents an unsubscribe happening if a party is being constantly looked up until the requests cools down which inturn stops from the statemachines hanging then returning a 504.
Repeated party lookups for a single payee are unlikely to happen in a real world scenario but is prevelant in performance testing.

I researched methods of maybe trying to lock channel pub/sub, disregarding a rewrite of the cache implementation this is the easiest solution for the specific edgecase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service core-dev-squad oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it PI-23
Projects
Status: Done
Development

No branches or pull requests

3 participants