503 from compose: Crash during parallel execution #7
Comments
|
The right solution here is to switch away from |
|
Can we contribute fixes back up to gocomposeapi itself? Because the whole point of these APIs is the code should do retries, etc. transparently. |
docwhat
changed the title
|
I see you've forked gocomposeapi and started using that fork. Do you have any issue open with the upstream folks? If they haven't been made aware of the problem then we can't expect them to fix it. |
|
I maintain a fork of |
|
I can reproduce this with a trivial example that bypasses pachelbel completely: package main
import (
"log"
"os"
"sync"
"github.com/benjdewan/gocomposeapi"
)
func main() {
client, err := composeapi.NewClient(os.Getenv("COMPOSE_API_TOKEN"))
if err != nil {
log.Fatal(err)
}
count := 2000
var wg sync.WaitGroup
wg.Add(count)
for i := 0; i < count; i++ {
go func() {
_, errs := client.GetAccount()
if len(errs) > 0 {
log.Println(errs)
}
wg.Done()
}()
}
wg.Wait()
}Which is really what I needed to file an issue against the parent project. |
|
After doing some additional digging, the error is not with 503 handling at all. We have no problem catching and handling 503 errors. The problem is that massively parallel invocations to if response.StatusCode != 200 {
errs = ProcessErrors(response.StatusCode, body)
}in |
|
I have a PR against the parent repo with a fix: compose/gocomposeapi#33 |
|
The next release of pachelbel, 6.0.1, will have the fix for this panic. It will not have request throttling to prevent a single pachelbel invocation from getting blocked by the Compose API (which wouldn't make the example using |
Pachelbel crashes during parallel execution.
The reason is that Compose.io sometimes returns 503 instead of valid json when running pachabel in parallel multiple times.
One can simulate it by GNU parallel:
The text was updated successfully, but these errors were encountered: