syncing files for period index_2635

[madskristiansen]

Describe the bug

This is a new installation - have 3 promtails running submitting logs, but I keep getting the following error message. Cant start Loki after this happens.

I have tried to clear out the chunks and indexes and started over a few times but same error with index_2635 keeps popping up.

2020-07-08 08:37:40.863: level=warn ts=2020-07-08T08:37:40.8630609Z caller=instance.go:174 message="could not create new stream for tenant" error="tenant 'fake' per-user streams limit exceeded, streams: 10000 exceeds calculated limit: 10000 (local limit: 10000, global limit: 0, global/ingesters: 0)"
2020-07-08 08:37:40.863: level=warn ts=2020-07-08T08:37:40.8630609Z caller=instance.go:174 message="could not create new stream for tenant" error="tenant 'fake' per-user streams limit exceeded, streams: 10000 exceeds calculated limit: 10000 (local limit: 10000, global limit: 0, global/ingesters: 0)"
2020-07-08 08:37:40.863: level=warn ts=2020-07-08T08:37:40.8630609Z caller=instance.go:174 message="could not create new stream for tenant" error="tenant 'fake' per-user streams limit exceeded, streams: 10000 exceeds calculated limit: 10000 (local limit: 10000, global limit: 0, global/ingesters: 0)"
2020-07-08 08:37:40.863: level=warn ts=2020-07-08T08:37:40.8630609Z caller=instance.go:174 message="could not create new stream for tenant" error="tenant 'fake' per-user streams limit exceeded, streams: 10000 exceeds calculated limit: 10000 (local limit: 10000, global limit: 0, global/ingesters: 0)"
2020-07-08 08:37:40.863: level=warn ts=2020-07-08T08:37:40.8630609Z caller=instance.go:174 message="could not create new stream for tenant" error="tenant 'fake' per-user streams limit exceeded, streams: 10000 exceeds calculated limit: 10000 (local limit: 10000, global limit: 0, global/ingesters: 0)"
2020-07-08 08:37:40.863: level=warn ts=2020-07-08T08:37:40.8630609Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=55.0064ms err="rpc error: code = Code(429) desc = Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased" msg="gRPC\n"
2020-07-08 08:37:40.863: level=debug ts=2020-07-08T08:37:40.8630609Z caller=logging.go:44 traceID=66ae8c9d85b038ff msg="POST /loki/api/v1/push (429) 97.0007ms"
2020-07-08 08:37:49.772: level=debug ts=2020-07-08T08:37:49.7721608Z caller=uploads.go:70 msg="uploading file for period index_2635"
2020-07-08 08:37:49.773: level=debug ts=2020-07-08T08:37:49.7731608Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=0s msg="gRPC (success)"
2020-07-08 08:37:49.775: level=debug ts=2020-07-08T08:37:49.7751602Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=0s msg="gRPC (success)"
2020-07-08 08:37:49.797: level=debug ts=2020-07-08T08:37:49.7973083Z caller=downloads.go:60 msg="syncing files for period index_2635"
2020-07-08 08:37:49.800: panic: runtime error: index out of range [1] with length 1
2020-07-08 08:37:49.800: 
2020-07-08 08:37:49.800: goroutine 372 [running]:
2020-07-08 08:37:49.800: github.com/grafana/loki/pkg/storage/stores/local.(*Shipper).checkStorageForUpdates(0xc0003085a0, 0x3d00d80, 0xc0000622e0, 0xc001608184, 0xa, 0xc0016226c0, 0x23, 0x0, 0x0, 0xc001f0ddd8, ...)
2020-07-08 08:37:49.800: 	/src/loki/pkg/storage/stores/local/downloads.go:35 +0x82a
2020-07-08 08:37:49.800: github.com/grafana/loki/pkg/storage/stores/local.(*Shipper).syncFilesForPeriod(0xc0003085a0, 0x3d00d80, 0xc0000622e0, 0xc001608184, 0xa, 0xc0016226c0, 0x4, 0x4)
2020-07-08 08:37:49.800: 	/src/loki/pkg/storage/stores/local/downloads.go:63 +0x258
2020-07-08 08:37:49.800: github.com/grafana/loki/pkg/storage/stores/local.(*Shipper).syncLocalWithStorage(0xc0003085a0, 0x3d00d80, 0xc0000622e0, 0x0, 0x0)
2020-07-08 08:37:49.800: 	/src/loki/pkg/storage/stores/local/shipper.go:253 +0x1eb
2020-07-08 08:37:49.800: github.com/grafana/loki/pkg/storage/stores/local.(*Shipper).loop(0xc0003085a0)
2020-07-08 08:37:49.800: 	/src/loki/pkg/storage/stores/local/shipper.go:175 +0x612
2020-07-08 08:37:49.800: created by github.com/grafana/loki/pkg/storage/stores/local.NewShipper
2020-07-08 08:37:49.800: 	/src/loki/pkg/storage/stores/local/shipper.go:125 +0x480

Configuration:

auth_enabled: false

server:
  http_listen_port: 3100
  log_level: debug

ingester:
  max_transfer_retries: 0 # Disable blocks transfers on ingesters shutdown or rollout.
  chunk_idle_period: 2h # Let chunks sit idle for at least 2h before flushing, this helps to reduce total chunks in store
  max_chunk_age: 2h  # Let chunks get at least 2h old before flushing due to age, this helps to reduce total chunks in store
  chunk_target_size: 1048576 # Target chunks of 1MB, this helps to reduce total chunks in store
  chunk_retain_period: 30s
  
  query_store_max_look_back_period: -1  # This will allow the ingesters to query the store for all data

  lifecycler:
    heartbeat_period: 5s
    interface_names:
      - vEthernet (HNSTransparent)
    join_after: 30s
    num_tokens: 512
    ring:
      heartbeat_timeout: 1m
      kvstore:
        consul:
          consistent_reads: true
          host: dk-nodetest1:8500
          http_client_timeout: 20s
        store: consul
      replication_factor: 1 # This can be increased and probably should if you are running multiple machines!

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb_shipper:
    shared_store: filesystem
    active_index_directory: E:/Loki_Data/index
    cache_location: /Loki_Data/cache
  filesystem:
    directory: /Loki_Data/chunks

limits_config:
  ingestion_burst_size_mb: 16
  ingestion_rate_mb: 10
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s  # No deletions, infinite retention

chunk_store_config:
  max_look_back_period: 0s  # No limit how far we can look back in the store

Expected behavior
Should be able to recover automatically.

Environment:
Bare metal

[cyriltovena]

/cc @sandeepsukhani

Which version are you using ? 1.5 or some master ? You also seems to suffer from too many labels issue, not sure what is you configuration for promtail but you might want to remove high cardinality labels. (I could be wrong and you just have a lot of logs in that case increase the stream limit).

[cyriltovena]

I've built an image that won't crash but throw an error if you want to give a go.

grafana/loki:improve-object-key-parsing-dd65479

[madskristiansen]

It is the latest 1.5.0 release.

Thanks, I will try with that image.

[cyriltovena]

It's latest.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.