[1.7] SIGSEGV in 1.7 branch #3585
Assignees
Labels
Comments
|
Unfortunately I don't have the core dumps since Fargate is serverless. I will set up another test to try to get that. I have a few more valgrind outputs though and some of them have more info. |
|
|
|
Output config: |
|
Go another stack trace: |
|
@PettitWesley do you have specific steps to reproduce the issue on #3585 (comment) ? which AWS VM Image/environment was used ? |
|
Same here: AWS ECS + loki + Fluent Bit v1.7.7 Log: Input: Inspect state: |
|
Looks like there might be multiple issues, here's a different one for the S3 buffer files/chunkio: |
|
@PettitWesley on that crash, how many chunks you had in fstore ? |
@edsiper I showed my output config in this comment: #3585 (comment) For the input, I think that doesn't matter, I set it up to receive logs of forward at a rate of 1000 per second. I just ran this on my dev machine which is amazon linux. There's nothing special about the set up.
|
|
@PettitWesley if is easy to reproduce in your environment, let's join a call to troubleshoot |
|
@edsiper I can only reproduce it in a long running test. I can't guarantee I can repro it quickly during a call. |
|
Is there a chance that this point the https://github.com/fluent/fluent-bit/blob/master/plugins/out_s3/s3_store.c#L178 |
|
similar Issue.. |
The following is the fluent-bit.conf file that I've specified: [SERVICE]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
[INPUT]
Name tail
Tag docker
Path /var/lib/docker/containers/*/*.log
Parser docker
DB /var/lib/fluent-bit/tail-docker.db
Skip_Long_Lines On
Buffer_Chunk_Size 32kb
Buffer_Max_Size 64kb
Mem_Buf_Limit 8mb
Refresh_Interval 30
Ignore_Older 1d
[FILTER]
Name nest
Match docker
Operation lift
Nested_under attrs
[FILTER]
Name modify
Match docker
Rename NOMAD_JOB_NAME nomad_job
Rename NOMAD_GROUP_NAME nomad_group
Rename NOMAD_TASK_NAME nomad_task
Rename NOMAD_ALLOC_ID nomad_alloc_id
[OUTPUT]
Name loki
Match docker
Host loki.service.consul
Port 3100
Label_Keys $stream,$nomad_job,$nomad_group,$nomad_task,$nomad_alloc_id
Drop_Single_Key true
Line_Format key_value
|
Same for me, if I use the "GO Loki plugin", 1.7.7 is quite stable, there seems to be an issue with Loki native plugin :( I cannot get the stack |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
This issue was closed because it has been stalled for 5 days with no activity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Describe the bug
I built the latest code in 1.7 last night: https://github.com/fluent/fluent-bit/tree/1.7
Deployed a bunch of containers on AWS Fargate. All but one of them died during the night.
Some had Valgrdind and gave me some output:
The text was updated successfully, but these errors were encountered: