Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The fluent-bit collects the mycat log. When the log is too long, the fluentbit collects the log from the beginning again. #736

Closed
liuxinbot opened this issue Sep 3, 2018 · 4 comments
Closed

The fluent-bit collects the mycat log. When the log is too long, the fluentbit collects the log from the beginning again. #736

liuxinbot opened this issue Sep 3, 2018 · 4 comments
Assignees
@edsiper
Labels
fixed question timeout no response or no updates as requested

Comments

@liuxinbot
Copy link

Bug Report

Describe the bug
The fluent-bit collects the mycat log. When the log is too long, the fluentbit collects the log from the beginning again.
To Reproduce

  • fluentbit conf
# ls
fluent-bit.conf  parsers.conf
# 
# cat fluent-bit.conf
[INPUT]
    Name tail
    tag mycat
    path /var/log/mycat.log
    Multiline On
    Parser_Firstline mycat_error_log_parser_head
    Parser_1 mycat_error_log_parser
    Path_Key file
[SERVICE]
    Flush           5
    Daemon          off
    Log_Level       debug
    Parsers_File    /fluent-bit/etc/parsers.conf

[FILTER]
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}
    Record AppName efk-mycat-test-ac234
    Record ConfigType Fluentbit
    Record CreatedBy orain.com
    Record DBType mysql
    Record ProxyType mycat
    Record Type Proxy

[OUTPUT]
    Name es
    Match *
    Include_Tag_Key True
    Host  elasticsearch-logging.kube-system.svc
    Port  9200
    Logstash_Format True
    Time_Key time
    Type  proxy
# 
# 
# cat parsers.conf

[PARSER]
  Name        mycat_error_log_parser_head
  Format      regex
  Regex       ^(?<time>\d{4}-\d{1,2}-\d{1,2}(\S)+\W(\S)+)(\s+)(?<logType>\S+)(\s+)(?<no>\S+)(\s+)(?<content>(.|\n)*)
  Time_Offset +0800
  Time_Format %Y-%m-%d %H:%M:%S.%L
  Time_Keep   Off

[PARSER]
  Name        mycat_error_log_parser
  Format      regex
  Regex       .*
  • The log of fluentbit
[2018/09/03 11:38:49] [debug] [task] destroy task=0x7f6f6f064000 (task_id=0)
[2018/09/03 11:38:49] [debug] [dyntag tail.0] 0x7f6f6f140440 destroy (tag=mycat)
[2018/09/03 11:39:44] [debug] [in_tail] append new file: /var/log/mycat.log
[2018/09/03 11:39:44] [debug] [in_tail] add to scan queue /var/log/mycat.log, offset=0
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 353
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 775
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 1204
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 1602
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 1919
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 2427
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 4065
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 4380
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 4692
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 5010
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 5322
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 5645
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 5967
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 6300
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 6627
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 6941
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 7276
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 7610
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 7945
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 8256
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 8645
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 9048
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 9726
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 10090
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 10429
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 10768
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 11103
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 11418
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 11807
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 12260
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 12713
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 13166
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 13619
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 14072
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 14525
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 14978
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 15431
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 15884
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 16337
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 16790
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 17243
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 17643
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 18043
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 18443
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 18843
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 19243
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 19643
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 20043
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 20443
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 20843
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 21243
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 21643
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 22144
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 22645
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 23012
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 23667
[2018/09/03 11:39:44] [debug] [input tail.0] [mem buf] size = 24321
[2018/09/03 11:39:44] [debug] [in_tail] file=/var/log/mycat.log read=32767 lines=59
[2018/09/03 11:39:44] [debug] [in_tail] file=/var/log/mycat.log read=14321 lines=0
[2018/09/03 11:39:44] [error] [in_tail] file=/var/log/mycat.log requires a larger buffer size, lines are too long. Skipping file.
[2018/09/03 11:39:49] [debug] [task->buffer] worker_id=0
[2018/09/03 11:39:49] [debug] [task] created task=0x7f6f6f064000 id=0 OK
[2018/09/03 11:39:49] [debug] [out_es] HTTP Status=200
[2018/09/03 11:39:49] [debug] [out_es Elasticsearch response
{"took":58,"errors":false,"items":[{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdg-XYKMlFHdwBd7QA","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdg-XYKMlFHdwBd7QB","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdg-XYKMlFHdwBd7QC","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdg-XYKMlFHdwBd7QD","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdg-XYKMlFHdwBd7QE","_version":1,"result":"created","_shards":{"tot
[2018/09/03 11:39:49] [debug] [task] destroy task=0x7f6f6f064000 (task_id=0)
[2018/09/03 11:39:49] [debug] [dyntag tail.0] 0x7f6f6f140440 destroy (tag=mycat)
[2018/09/03 11:40:43] [debug] [in_tail] append new file: /var/log/mycat.log
[2018/09/03 11:40:43] [debug] [in_tail] add to scan queue /var/log/mycat.log, offset=0
[2018/09/03 11:40:43] [debug] [input tail.0] [mem buf] size = 353
[2018/09/03 11:40:43] [debug] [input tail.0] [mem buf] size = 775
[2018/09/03 11:40:43] [debug] [input tail.0] [mem buf] size = 1204
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 1602
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 1919
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 2427
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 4065
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 4380
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 4692
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 5010
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 5322
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 5645
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 5967
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 6300
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 6627
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 6941
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 7276
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 7610
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 7945
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 8256
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 8645
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 9048
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 9726
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 10090
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 10429
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 10768
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 11103
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 11418
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 11807
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 12260
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 12713
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 13166
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 13619
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 14072
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 14525
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 14978
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 15431
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 15884
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 16337
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 16790
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 17243
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 17643
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 18043
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 18443
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 18843
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 19243
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 19643
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 20043
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 20443
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 20843
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 21243
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 21643
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 22144
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 22645
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 23012
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 23667
[2018/09/03 11:40:44] [debug] [input tail.0] [mem buf] size = 24321
[2018/09/03 11:40:44] [debug] [in_tail] file=/var/log/mycat.log read=32767 lines=59
[2018/09/03 11:40:44] [debug] [in_tail] file=/var/log/mycat.log read=14321 lines=0
[2018/09/03 11:40:44] [error] [in_tail] file=/var/log/mycat.log requires a larger buffer size, lines are too long. Skipping file.
[2018/09/03 11:40:49] [debug] [task->buffer] worker_id=0
[2018/09/03 11:40:49] [debug] [task] created task=0x7f6f6f064000 id=0 OK
[2018/09/03 11:40:49] [debug] [out_es] HTTP Status=200
[2018/09/03 11:40:49] [debug] [out_es Elasticsearch response
{"took":44,"errors":false,"items":[{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhNA4KMlFHdwBd7Q6","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhNA4KMlFHdwBd7Q7","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhNA4KMlFHdwBd7Q8","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhNA4KMlFHdwBd7Q9","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhNA4KMlFHdwBd7Q-","_version":1,"result":"created","_shards":{"tot
[2018/09/03 11:40:49] [debug] [task] destroy task=0x7f6f6f064000 (task_id=0)
[2018/09/03 11:40:49] [debug] [dyntag tail.0] 0x7f6f6f140440 destroy (tag=mycat)
[2018/09/03 11:41:44] [debug] [in_tail] append new file: /var/log/mycat.log
[2018/09/03 11:41:44] [debug] [in_tail] add to scan queue /var/log/mycat.log, offset=0
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 353
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 775
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 1204
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 1602
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 1919
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 2427
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 4065
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 4380
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 4692
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 5010
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 5322
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 5645
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 5967
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 6300
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 6627
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 6941
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 7276
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 7610
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 7945
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 8256
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 8645
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 9048
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 9726
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 10090
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 10429
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 10768
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 11103
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 11418
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 11807
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 12260
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 12713
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 13166
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 13619
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 14072
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 14525
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 14978
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 15431
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 15884
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 16337
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 16790
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 17243
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 17643
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 18043
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 18443
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 18843
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 19243
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 19643
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 20043
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 20443
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 20843
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 21243
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 21643
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 22144
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 22645
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 23012
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 23667
[2018/09/03 11:41:44] [debug] [input tail.0] [mem buf] size = 24321
[2018/09/03 11:41:44] [debug] [in_tail] file=/var/log/mycat.log read=32767 lines=59
[2018/09/03 11:41:44] [debug] [in_tail] file=/var/log/mycat.log read=14321 lines=0
[2018/09/03 11:41:44] [error] [in_tail] file=/var/log/mycat.log requires a larger buffer size, lines are too long. Skipping file.
[2018/09/03 11:41:49] [debug] [task->buffer] worker_id=0
[2018/09/03 11:41:49] [debug] [task] created task=0x7f6f6f064000 id=0 OK
[2018/09/03 11:41:49] [debug] [out_es] HTTP Status=200
[2018/09/03 11:41:49] [debug] [out_es Elasticsearch response
{"took":69,"errors":false,"items":[{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhbqaKMlFHdwBd7R0","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhbqaKMlFHdwBd7R1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhbqaKMlFHdwBd7R2","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhbqaKMlFHdwBd7R3","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true,"status":201}},{"index":{"_index":"logstash-2018.09.03","_type":"proxy","_id":"AWWdhbqaKMlFHdwBd7R4","_version":1,"result":"created","_shards":{"tot

...

  • Kibana show screenshot
    _15359468618581

  • The log of mycat
    mycat.log

Your Environment

  • Version used: Fluent Bit v0.12.9
  • Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.10.5
  • Operating System and version: Linux efk-mycat-test-ac234-proxy-0 3.10.0-693.el7.x86_64 Fix include paths #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 GNU/Linux

Any help would be appreciated.
@liuxinbot
Copy link
Author

The problem should be that the log line is too long beyond the fluentbit buffer, but I am curious why it read from the first line of the log again.

@edsiper edsiper self-assigned this Sep 3, 2018
@edsiper
Copy link
Member

edsiper commented Sep 3, 2018 •
edited
Loading

@liuxin95

thanks for reporting this issue.

I did some local tests with the information provided and these are my findings:

  • The file offset is never going back, meaning it never start reading from a previous position. Tested on normal mode and multiline mode. Behavior is expected.
  • When a long line is found, an error message is triggered and the file is not longer monitored. If you want to skip the long lines and continue processing other lines just append 'Skip_Long_Lines On' to the input tail section of your file.

@liuxinbot
Copy link
Author

@edsiper

Thank you very much for taking the time to reply.

As you said, my solution is also append 'Skip_Long_Lines On' to the input tail section to skip the long lines and continue processing other lines and it works fine.

But for the rigorous work attitude, I run a docker container to collect mycat log and print to stdout, when reading long logs, the fluentbit print info as
[2018/09/04 02:05:04] [error] [in_tail] file=/conf/mycat.log requires a larger buffer size, lines are too long. Skipping file..
Everything is as we expected but After half a minute or so，the fluentbit collect mycat log again. I found a clue from the output information
[2018/09/04 02:06:04] [debug] [in_tail] append new file: /conf/mycat.log.
Detailed information is as follows：

[root@10-10-88-198 fluentbit]# docker run -ti -v /root/fluentbit/conf:/conf fluent/fluent-bit:0.12.9 /fluent-bit/bin/fluent-bit -c /conf/fluent-bit.conf
Fluent-Bit v0.12.9
Copyright (C) Treasure Data

[2018/09/04 02:05:04] [ info] [engine] started
[2018/09/04 02:05:04] [debug] [in_tail] inotify watch fd=19
[2018/09/04 02:05:04] [debug] [in_tail] scanning path /conf/mycat.log
[2018/09/04 02:05:04] [debug] [in_tail] add to scan queue /conf/mycat.log, offset=0
[2018/09/04 02:05:04] [debug] [router] input=tail.0 'DYNAMIC TAG'
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 202
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 473
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 751
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 998
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 1164
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 1521
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 3008
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 3172
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 3333
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 3500
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 3661
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 3833
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 4004
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 4186
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 4362
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 4525
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 4709
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 4892
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 5076
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 5236
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 5474
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 5726
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 6253
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 6466
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 6654
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 6842
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 7026
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 7190
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 7428
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 7730
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 8032
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 8334
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 8636
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 8938
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 9240
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 9542
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 9844
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 10146
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 10448
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 10750
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 11052
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 11301
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 11550
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 11799
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 12048
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 12297
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 12546
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 12795
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 13044
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 13293
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 13542
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 13791
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 14141
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 14491
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 14707
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 15211
[2018/09/04 02:05:04] [debug] [input tail.0] [mem buf] size = 15714
[2018/09/04 02:05:04] [debug] [in_tail] file=/conf/mycat.log read=32767 lines=59
[2018/09/04 02:05:04] [debug] [in_tail] file=/conf/mycat.log read=14321 lines=0
[2018/09/04 02:05:04] [error] [in_tail] file=/conf/mycat.log requires a larger buffer size, lines are too long. Skipping file.
[0] mycat: [1535941306.170000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.<init>(PhysicalDBPool.java:101)) - total resouces of dataHost cap_host is :2"}]
[1] mycat: [1535941306.182000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.cache.CacheService.createLayeredPool(CacheService.java:125)) - create layer cache pool TableID2DataNodeCache of type encache ,default cache size 10000 ,default expire seconds18000"}]
[2] mycat: [1535941306.185000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.cache.DefaultLayedCachePool.createChildCache(DefaultLayedCachePool.java:80)) - create child Cache: TESTDB_ORDERS for layered cache TableID2DataNodeCache, size 50000, expire seconds 18000"}]
[3] mycat: [1535941306.324000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.config.classloader.DynaClassLoader.<init>(DynaClassLoader.java:34)) - dyna class load from ./catlet,and auto check for class file modified every 60 seconds"}]
[4] mycat: [1535941306.356000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:317)) - MyCat is ready to startup ..."}]
[5] mycat: [1535941306.357000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:329)) - Startup processors ...,total processors:8,aio thread pool size:16    
 each process allocated socket buffer pool  bytes ,a page size:2097152  a page's chunk number(PageSize/ChunkSize) is:512  buffer page's number is:40"}]
[6] mycat: [1535941306.357000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:330)) - sysconfig params:SystemConfig [processorBufferLocalPercent=100, frontSocketSoRcvbuf=1048576, frontSocketSoSndbuf=4194304, backSocketSoRcvbuf=4194304, backSocketSoSndbuf=1048576, frontSocketNoDelay=1, backSocketNoDelay=1, maxStringLiteralLength=65535, frontWriteQueueSize=2048, bindIp=0.0.0.0, serverPort=8066, managerPort=9066, charset=utf8, processors=8, processorExecutor=16, timerExecutor=2, managerExecutor=2, idleTimeout=1800000, catletClassCheckSeconds=60, sqlExecuteTimeout=300, processorCheckPeriod=1000, dataNodeIdleCheckPeriod=300000, dataNodeHeartbeatPeriod=10000, clusterHeartbeatUser=_HEARTBEAT_USER_, clusterHeartbeatPass=_HEARTBEAT_PASS_, clusterHeartbeatPeriod=5000, clusterHeartbeatTimeout=10000, clusterHeartbeatRetry=10, txIsolation=3, parserCommentVersion=50148, sqlRecordCount=10, bufferPoolPageSize=2097152, bufferPoolChunkSize=4096, bufferPoolPageNumber=40, maxResultSet=524288, bigResultSizeSqlCount=10, bufferUsagePercent=80, flowControlRejectStrategy=0, clearBigSqLResultSetMapMs=600000, defaultMaxLimit=100, sequnceHandlerType=2, sqlInterceptor=io.mycat.server.interceptor.impl.DefaultSqlInterceptor, sqlInterceptorType=select, sqlInterceptorFile=/usr/local/mycat/logs/sql.txt, mutiNodeLimitType=0, mutiNodePatchSize=100, defaultSqlParser=druidparser, usingAIO=0, packetHeaderSize=4, maxPacketSize=16777216, mycatNodeId=1]"}]
[7] mycat: [1535941306.417000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:53)) - useOffHeapForMerge = 1"}]
[8] mycat: [1535941306.417000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:54)) - memoryPageSize = 1m"}]
[9] mycat: [1535941306.418000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:55)) - spillsFileBufferSize = 1k"}]
[10] mycat: [1535941306.418000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:56)) - useStreamOutput = 0"}]
[11] mycat: [1535941306.418000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:57)) - systemReserveMemorySize = 384m"}]
[12] mycat: [1535941306.432000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:58)) - totalNetWorkBufferSize = 80MB"}]
[13] mycat: [1535941306.433000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:59)) - dataNodeSortedTempDir = /usr/local/mycat"}]
[14] mycat: [1535941306.436000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:122)) - mycat.memory.offHeap.size: 1005MB"}]
[15] mycat: [1535941306.445000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:437)) - using nio network handler "}]
[16] mycat: [1535941306.460000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:453)) - $_MyCatManager is started and listening on 9066"}]
[17] mycat: [1535941306.462000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:457)) - $_MyCatServer is started and listening on 8066"}]
[18] mycat: [1535941306.463000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:459)) - ==============================================="}]
[19] mycat: [1535941306.463000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:463)) - Initialize dataHost ..."}]
[20] mycat: [1535941306.463000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.initSource(PhysicalDBPool.java:303)) - init backend myqsl source ,create connections total 1 for hostM1 index :0"}]
[21] mycat: [1535941306.464000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[22] mycat: [1535941306.506000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-1-RW]", "content"=>"(io.mycat.backend.mysql.nio.handler.GetConnectionHandler.connectionAcquired(GetConnectionHandler.java:67)) - connected successfuly MySQLConnection [id=1, lastTime=1535941306506, user=root, schema=sbtest, old shema=sbtest, borrowed=true, fromSlaveDB=false, threadId=675, charset=utf8, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=efk-mycat-test-ac2340, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]"}]
[23] mycat: [1535941306.565000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.initSource(PhysicalDBPool.java:328)) - init result :finished 1 success 1 target count:1"}]
[24] mycat: [1535941306.566000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.init(PhysicalDBPool.java:274)) - cap_host index:0 init success"}]
[25] mycat: [1535941306.566000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.saveDataHostIndex(MycatServer.java:703)) - save DataHost index  cap_host cur index 0"}]
[26] mycat: [1535941306.575000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:501)) - ==============================================="}]
[27] mycat: [1535941306.575000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:502)) - Perform XA recovery log ..."}]
[28] mycat: [1535941306.579000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostS1 of schema sbtest"}]
[29] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-0-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=694, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[30] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-1-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=695, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[31] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=691, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[32] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-7-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=693, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[33] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=698, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[34] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=689, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[35] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-2-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=696, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[36] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=690, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[37] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=697, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[38] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=692, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[39] mycat: [1535942170.467000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=700, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[40] mycat: [1535942170.467000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=699, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[41] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[42] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-2-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[43] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-0-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[44] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[45] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[46] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-7-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[47] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-1-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[48] mycat: [1535942170.531000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[49] mycat: [1535942170.531000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[50] mycat: [1535942170.531000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[51] mycat: [1535942170.560000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[52] mycat: [1535942176.416000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:program err:java.io.IOException: Connection reset by peer ,ServerConnection [id=699, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]"}]
[53] mycat: [1535942176.797000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:program err:java.io.IOException: Connection reset by peer ,ServerConnection [id=700, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]"}]
[54] mycat: [1535942206.574000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.closeByIdleMany(PhysicalDatasource.java:336)) - too many ilde cons ,close some for datasouce  hostM1"}]
[55] mycat: [1535942206.575000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:too many idle con ,MySQLConnection [id=10, lastTime=1535942206452, user=root, schema=sbtest, old shema=sbtest, borrowed=false, fromSlaveDB=false, threadId=2517, charset=latin1, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=efk-mycat-test-ac2340, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]"}]
[56] mycat: [1535942206.576000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:too many idle con ,MySQLConnection [id=8, lastTime=1535942206552, user=root, schema=sbtest, old shema=sbtest, borrowed=false, fromSlaveDB=false, threadId=2518, charset=latin1, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=efk-mycat-test-ac2340, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]"}]
[2018/09/04 02:05:09] [debug] [task->buffer] worker_id=0
[2018/09/04 02:05:09] [debug] [task] created task=0x7feb70c49000 id=0 OK
[2018/09/04 02:05:09] [debug] [task] destroy task=0x7feb70c49000 (task_id=0)
[2018/09/04 02:05:09] [debug] [dyntag tail.0] 0x7feb70c890c0 destroy (tag=mycat)
[2018/09/04 02:06:04] [debug] [in_tail] append new file: /conf/mycat.log
[2018/09/04 02:06:04] [debug] [in_tail] add to scan queue /conf/mycat.log, offset=0
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 202
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 473
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 751
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 998
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 1164
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 1521
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 3008
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 3172
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 3333
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 3500
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 3661
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 3833
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 4004
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 4186
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 4362
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 4525
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 4709
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 4892
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 5076
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 5236
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 5474
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 5726
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 6253
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 6466
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 6654
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 6842
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 7026
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 7190
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 7428
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 7730
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 8032
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 8334
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 8636
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 8938
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 9240
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 9542
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 9844
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 10146
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 10448
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 10750
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 11052
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 11301
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 11550
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 11799
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 12048
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 12297
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 12546
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 12795
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 13044
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 13293
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 13542
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 13791
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 14141
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 14491
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 14707
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 15211
[2018/09/04 02:06:04] [debug] [input tail.0] [mem buf] size = 15714
[2018/09/04 02:06:04] [debug] [in_tail] file=/conf/mycat.log read=32767 lines=59
[2018/09/04 02:06:04] [debug] [in_tail] file=/conf/mycat.log read=14321 lines=0
[2018/09/04 02:06:04] [error] [in_tail] file=/conf/mycat.log requires a larger buffer size, lines are too long. Skipping file.
[0] mycat: [1535941306.170000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.<init>(PhysicalDBPool.java:101)) - total resouces of dataHost cap_host is :2"}]
[1] mycat: [1535941306.182000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.cache.CacheService.createLayeredPool(CacheService.java:125)) - create layer cache pool TableID2DataNodeCache of type encache ,default cache size 10000 ,default expire seconds18000"}]
[2] mycat: [1535941306.185000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.cache.DefaultLayedCachePool.createChildCache(DefaultLayedCachePool.java:80)) - create child Cache: TESTDB_ORDERS for layered cache TableID2DataNodeCache, size 50000, expire seconds 18000"}]
[3] mycat: [1535941306.324000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.config.classloader.DynaClassLoader.<init>(DynaClassLoader.java:34)) - dyna class load from ./catlet,and auto check for class file modified every 60 seconds"}]
[4] mycat: [1535941306.356000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:317)) - MyCat is ready to startup ..."}]
[5] mycat: [1535941306.357000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:329)) - Startup processors ...,total processors:8,aio thread pool size:16    
 each process allocated socket buffer pool  bytes ,a page size:2097152  a page's chunk number(PageSize/ChunkSize) is:512  buffer page's number is:40"}]
[6] mycat: [1535941306.357000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:330)) - sysconfig params:SystemConfig [processorBufferLocalPercent=100, frontSocketSoRcvbuf=1048576, frontSocketSoSndbuf=4194304, backSocketSoRcvbuf=4194304, backSocketSoSndbuf=1048576, frontSocketNoDelay=1, backSocketNoDelay=1, maxStringLiteralLength=65535, frontWriteQueueSize=2048, bindIp=0.0.0.0, serverPort=8066, managerPort=9066, charset=utf8, processors=8, processorExecutor=16, timerExecutor=2, managerExecutor=2, idleTimeout=1800000, catletClassCheckSeconds=60, sqlExecuteTimeout=300, processorCheckPeriod=1000, dataNodeIdleCheckPeriod=300000, dataNodeHeartbeatPeriod=10000, clusterHeartbeatUser=_HEARTBEAT_USER_, clusterHeartbeatPass=_HEARTBEAT_PASS_, clusterHeartbeatPeriod=5000, clusterHeartbeatTimeout=10000, clusterHeartbeatRetry=10, txIsolation=3, parserCommentVersion=50148, sqlRecordCount=10, bufferPoolPageSize=2097152, bufferPoolChunkSize=4096, bufferPoolPageNumber=40, maxResultSet=524288, bigResultSizeSqlCount=10, bufferUsagePercent=80, flowControlRejectStrategy=0, clearBigSqLResultSetMapMs=600000, defaultMaxLimit=100, sequnceHandlerType=2, sqlInterceptor=io.mycat.server.interceptor.impl.DefaultSqlInterceptor, sqlInterceptorType=select, sqlInterceptorFile=/usr/local/mycat/logs/sql.txt, mutiNodeLimitType=0, mutiNodePatchSize=100, defaultSqlParser=druidparser, usingAIO=0, packetHeaderSize=4, maxPacketSize=16777216, mycatNodeId=1]"}]
[7] mycat: [1535941306.417000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:53)) - useOffHeapForMerge = 1"}]
[8] mycat: [1535941306.417000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:54)) - memoryPageSize = 1m"}]
[9] mycat: [1535941306.418000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:55)) - spillsFileBufferSize = 1k"}]
[10] mycat: [1535941306.418000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:56)) - useStreamOutput = 0"}]
[11] mycat: [1535941306.418000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:57)) - systemReserveMemorySize = 384m"}]
[12] mycat: [1535941306.432000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:58)) - totalNetWorkBufferSize = 80MB"}]
[13] mycat: [1535941306.433000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:59)) - dataNodeSortedTempDir = /usr/local/mycat"}]
[14] mycat: [1535941306.436000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:122)) - mycat.memory.offHeap.size: 1005MB"}]
[15] mycat: [1535941306.445000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:437)) - using nio network handler "}]
[16] mycat: [1535941306.460000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:453)) - $_MyCatManager is started and listening on 9066"}]
[17] mycat: [1535941306.462000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:457)) - $_MyCatServer is started and listening on 8066"}]
[2018/09/04 02:06:09] [debug] [task->buffer] worker_id=0
[18] mycat: [1535941306.463000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:459)) - ==============================================="}]
[19] mycat: [1535941306.463000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:463)) - Initialize dataHost ..."}]
[2018/09/04 02:06:09] [debug] [task] created task=0x7feb70c49000 id=0 OK
[20] mycat: [1535941306.463000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.initSource(PhysicalDBPool.java:303)) - init backend myqsl source ,create connections total 1 for hostM1 index :0"}]
[21] mycat: [1535941306.464000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[22] mycat: [1535941306.506000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-1-RW]", "content"=>"(io.mycat.backend.mysql.nio.handler.GetConnectionHandler.connectionAcquired(GetConnectionHandler.java:67)) - connected successfuly MySQLConnection [id=1, lastTime=1535941306506, user=root, schema=sbtest, old shema=sbtest, borrowed=true, fromSlaveDB=false, threadId=675, charset=utf8, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=efk-mycat-test-ac2340, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]"}]
[23] mycat: [1535941306.565000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.initSource(PhysicalDBPool.java:328)) - init result :finished 1 success 1 target count:1"}]
[24] mycat: [1535941306.566000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.backend.datasource.PhysicalDBPool.init(PhysicalDBPool.java:274)) - cap_host index:0 init success"}]
[25] mycat: [1535941306.566000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.saveDataHostIndex(MycatServer.java:703)) - save DataHost index  cap_host cur index 0"}]
[26] mycat: [1535941306.575000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:501)) - ==============================================="}]
[27] mycat: [1535941306.575000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[WrapperSimpleAppMain]", "content"=>"(io.mycat.MycatServer.startup(MycatServer.java:502)) - Perform XA recovery log ..."}]
[28] mycat: [1535941306.579000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostS1 of schema sbtest"}]
[29] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-0-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=694, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[30] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-1-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=695, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[31] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=691, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[32] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-7-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=693, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[33] mycat: [1535942170.465000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=698, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[34] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=689, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[35] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-2-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=696, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[36] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=690, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[37] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=697, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[38] mycat: [1535942170.466000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=692, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[39] mycat: [1535942170.467000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=700, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[40] mycat: [1535942170.467000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.net.handler.FrontendAuthenticator.success(FrontendAuthenticator.java:226)) - ServerConnection [id=699, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]'test' login success"}]
[41] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[42] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-2-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[43] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-0-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[44] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[45] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[46] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-7-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[47] mycat: [1535942170.530000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-1-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[48] mycat: [1535942170.531000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[49] mycat: [1535942170.531000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-4-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[50] mycat: [1535942170.531000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[51] mycat: [1535942170.560000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-3-RW]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostM1 of schema sbtest"}]
[52] mycat: [1535942176.416000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-5-RW]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:program err:java.io.IOException: Connection reset by peer ,ServerConnection [id=699, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]"}]
[53] mycat: [1535942176.797000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[$_NIOREACTOR-6-RW]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:program err:java.io.IOException: Connection reset by peer ,ServerConnection [id=700, schema=sbtest, host=10.10.88.198, user=test,txIsolation=3, autocommit=true, schema=sbtest]"}]
[54] mycat: [1535942206.574000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.backend.datasource.PhysicalDatasource.closeByIdleMany(PhysicalDatasource.java:336)) - too many ilde cons ,close some for datasouce  hostM1"}]
[55] mycat: [1535942206.575000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:too many idle con ,MySQLConnection [id=10, lastTime=1535942206452, user=root, schema=sbtest, old shema=sbtest, borrowed=false, fromSlaveDB=false, threadId=2517, charset=latin1, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=efk-mycat-test-ac2340, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]"}]
[56] mycat: [1535942206.576000000, {"file"=>"/conf/mycat.log", "logType"=>"INFO", "no"=>"[Timer1]", "content"=>"(io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:too many idle con ,MySQLConnection [id=8, lastTime=1535942206552, user=root, schema=sbtest, old shema=sbtest, borrowed=false, fromSlaveDB=false, threadId=2518, charset=latin1, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=efk-mycat-test-ac2340, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]"}]
[2018/09/04 02:06:09] [debug] [task] destroy task=0x7feb70c49000 (task_id=0)
[2018/09/04 02:06:09] [debug] [dyntag tail.0] 0x7feb70c890c0 destroy (tag=mycat)

My fluentbit conf as follows:

[root@10-10-88-198 conf]# pwd
/root/fluentbit/conf
[root@10-10-88-198 conf]# ls
fluent-bit.conf  mycat.log  parsers.conf
[root@10-10-88-198 conf]# 
[root@10-10-88-198 conf]# cat fluent-bit.conf 
[INPUT]
    Name tail
    tag mycat
    path /conf/mycat.log
    Multiline On
    Parser_Firstline mycat_error_log_parser_head
    Parser_1 mycat_error_log_parser
    Path_Key file

[SERVICE]
    Flush           5
    Daemon          off
    Log_Level       debug
    Parsers_File    /conf/parsers.conf

[FILTER]
    Name record_modifier
    Match *

[OUTPUT]
    Name stdout
    Match *
[root@10-10-88-198 conf]# 
[root@10-10-88-198 conf]# 
[root@10-10-88-198 conf]# cat parsers.conf 
[PARSER]
  Name        mycat_error_log_parser_head
  Format      regex
  Regex       ^(?<time>\d{4}-\d{1,2}-\d{1,2}(\S)+\W(\S)+)(\s+)(?<logType>\S+)(\s+)(?<no>\S+)(\s+)(?<content>(.|\n)*)
  Time_Offset +0800
  Time_Format %Y-%m-%d %H:%M:%S.%L
  Time_Keep   Off

[PARSER]
  Name        mycat_error_log_parser
  Format      regex
  Regex       .*
[root@10-10-88-198 conf]#

@edsiper
Copy link
Member

edsiper commented Apr 3, 2019

I don't understand whats the latest issue reported.

@edsiper edsiper added timeout no response or no updates as requested and removed troubleshooting labels May 11, 2020
@edsiper edsiper closed this as completed May 11, 2020
@tmetodie tmetodie mentioned this issue Jan 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@edsiper edsiper

Labels
fixed question timeout no response or no updates as requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@edsiper @liuxinbot