how can I extend scrolling hits

[YTS85205107]

I have an index with at least 600,000 hits every minute. When I use elastalert2 to perform scrolling every minute, it takes about two seconds to scroll 10,000 hits each time. The scrolling index will take more than 1 minute to complete, causing the memory usage to increase. how can I optimize it?

The following is the template config of my elastalert2
config.yaml

rules_folder: rules

run_every:
  minutes: 1

buffer_time:
  minutes: 1 

es_host: 10.10.10.100
es_port: 9200
es_timeout: 60
use_ssl: True
verify_certs: False
ssl_show_warn: False

es_username: elastic
es_password: 

ssl_ca_certs: "/elastalert2/certs/elastic-certificates.p12"
ssl_cert: "/elastalert2/certs/http.p12"
ssl_key: "/elastalert2/certs/http.p12"

writeback_index: elastalert_status

alert_time_limit:
  days: 2

logging:
  version: 1
  incremental: false
  disable_existing_loggers: false
  formatters:
    logline:
      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'

  handlers:
    console:
      class: logging.StreamHandler
      formatter: logline
      level: DEBUG
      stream: ext://sys.stderr

    file:
      class : logging.FileHandler
      formatter: logline
      level: DEBUG
      filename: elastalert.log

  loggers:
    elastalert:
      level: WARN
      handlers: []
      propagate: true

    elasticsearch:
      level: WARN
      handlers: []
      propagate: true

    elasticsearch.trace:
      level: WARN
      handlers: []
      propagate: true

    '':  # root logger
      level: WARN
      handlers:
        - console
        - file
      propagate: false

rule.yaml

`name: "log increase"
type: spike
spike_height: 2.0
spike_type: up
index: nginx_all-*

#num_events: 1
timeframe:
  minutes: 1

filter:
- range:
    "@timestamp":
        gte: "now-1m"

scroll_size: 20000
realert:
  minutes: 0
query_timezone: "Asia/Taipei"

use_local_time: true

alert:
  - "slack"

slack:
slack_webhook_url: "https://hooks.slack.com/services/xxx/"
slack_proxy: "http://192.168.1.1:10001"
slack_username_override: ""
slack_channel_override: "#elk"

alert_text_type: alert_text_only
alert_text_args: ["@timestamp"]
alert_text: |
  Site: Nginx
  Alert: log increase at {0} UTC time`

Thanks.

[jertel]

For that much log/event volume you may need to use the use_count_query option, which will return counts, instead of all 10,000 records.

[YTS85205107]

thanks.