Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic when ingesting #2125

Closed
fingon opened this issue Jan 30, 2020 · 1 comment · Fixed by #2126
Closed

Panic when ingesting #2125

fingon opened this issue Jan 30, 2020 · 1 comment · Fixed by #2126

Comments

@fingon
Copy link
Contributor

fingon commented Jan 30, 2020 •
edited
Loading

m3coordinator log shows stacktrace (bit weird package names but it's built from my dev aiven branch; essentially few unrelated patches on top of commit 829d13e ) :

Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: panic: runtime error: index out of range
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: goroutine 24319 [running]:
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/query/models/strconv.QuoteSimple(...)
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/query/models/strconv/quote.go:218
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/query/models.Tags.quoteIDSimple(0x1d888e0, 0xc0004f9ec0, 0xc03766b180, 0xa, 0x12, 0x10c, 0x19, 0x0, 0xc0359e7620)
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/query/models/tags.go:171 +0x477
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/query/models.Tags.quotedID(0x1d888e0, 0xc0004f9ec0, 0xc03766b180, 0xa, 0x12, 0x0, 0xc040041000, 0xc0359e0000)
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/query/models/tags.go:132 +0x462
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/query/models.Tags.ID(0x1d888e0, 0xc0004f9ec0, 0xc03766b180, 0xa, 0x12, 0xed5c49ee9, 0x0, 0x10)
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/query/models/tags.go:69 +0x144
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/query/storage/m3.(*m3storage).Write(0xc00e1fdf50, 0x1d714c0, 0xc040041110, 0xc03fad0780, 0x1d4c840, 0xc03fad0780)
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/query/storage/m3/storage.go:578 +0xc3
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/query/storage/fanout.(*fanoutStorage).Write(0xc00e205cc0, 0x1d714c0, 0xc040041110, 0xc03fad0780, 0xc03a9d1680, 0xc03a9d1708)
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/query/storage/fanout/storage.go:414 +0x2f6
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/cmd/services/m3coordinator/ingest.(*downsamplerAndWriter).WriteBatch.func2()
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/cmd/services/m3coordinator/ingest/write.go:333 +0x18b
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: github.com/aiven/m3/src/x/sync.(*pooledWorkerPool).spawnWorker.func1(0xc00f3f6310, 0xc03a9d29a0, 0x15eea125293b29db, 0xc0089d15c0, 0x255f817900)
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/x/sync/pooled_worker_pool.go:147 +0x192
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]: created by github.com/aiven/m3/src/x/sync.(*pooledWorkerPool).spawnWorker
Jan 30 09:55:40 m3-coordinator-stats-eu-1 m3coordinator[47]:         /srv/jenkins/workspace/user-aiven-deps-f30/doc/dependencies/m3db/gopath/src/github.com/aiven/m3/src/x/sync/pooled_worker_pool.go:126 +0x66

this seems to occur also on low-volume test system so I could try isolating the offending data if stacktrace isn't enough.
@fingon
Copy link
Contributor Author

fingon commented Jan 30, 2020

I think I know what's causing it, (my) influxdb write endpoint assumes tags are short-lived. WriteBatch method it uses actually fires goroutine which uses the iter.Current() returned tags later.

I guess given bad enough luck, what happens is that the __name__ (only varying field in Tags in the iterator loop in src/query/api/v1/handler/influxdb/write.go) grows shorter at some point and hilarity ensues.

fingon added a commit to aiven/m3 that referenced this issue Jan 30, 2020
@fingon
[coordinator] Influxdb write endpoint tag copy fix
9dd3b16 
Reusing Tags objects directly over the iteration did not work.  Second
best performing option was to reuse the Tag objects within array, but
replace the Tag with the __name__ with new one.

Fixes issue m3db#2125.
@fingon fingon mentioned this issue Jan 30, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[coordinator] Influxdb write endpoint tag copy fix aiven/m3
1 participant
@fingon