Minio bucket is always emtpy

[suikast42]

Describe the bug

I setup a minio server with pre created bucket for mimir. With the mimir config

# Test observability.mimir[0]
# Do not use this configuration in production.
# It is for demonstration purposes only.
# Run Mimir in single process mode, with all components running in 1 process.
target: all,alertmanager,overrides-exporter
multitenancy_enabled: false
#bar: observability.mimir[0]
server:
  http_listen_port: 9009
  log_level: debug
  # Configure the server to allow messages up to 100MB.
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

common:
  storage:
    backend: s3
    s3:
      endpoint: "minio.service.consul:22375"
      access_key_id: ${MINIO_USER}
      secret_access_key: ${MINIO_PASSWORD}
      insecure: true
      bucket_name: mimir

# Blocks storage requires a prefix when using a common object storage bucket.
blocks_storage:
  backend: s3
  storage_prefix: blocks
  tsdb:
    retention_period: 168h
    dir: /data/ingester

alertmanager_storage:
  backend: s3
  storage_prefix: alarms

ruler_storage:
  backend: s3
  storage_prefix: ruler

compactor:
  sharding_ring:
    kvstore:
      store: memberlist

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
    replication_factor: 1


store_gateway:
  sharding_ring:
    replication_factor: 1

limits:
  max_label_names_per_series: 42
  # Allow ingestion of out-of-order samples up to 5 minutes since the latest received sample for the series.
  out_of_order_time_window: 5m

I excpect that mimir saves the TSDB files inside the bucket but the onliy that mirimir saves in the bucket is a filed named mimir_cluster_seed.json.

After restart the container all that is gone ( because I don't mapped this in a volume ).

Did I miss something ?

[pracucci]

I excpect that mimir saves the TSDB files inside the bucket but the onliy that mirimir saves in the bucket is a filed named mimir_cluster_seed.json.

Mimir upload blocks (of metrics) to the object storage at period interval. After you create a new Mimir cluster, the first blocks will be uploaded by ingesters about 3 hours after, and then every 2 hours at minute 00 of odd UTC hours (e.g. 01:00 UTC, 03:00 UTC, and so on).

[suikast42]

Mimir upload blocks (of metrics) to the object storage at period interval. After you create a new Mimir cluster, the first blocks will be uploaded by ingesters about 3 hours after, and then every 2 hours at minute 00 of odd UTC hours (e.g. 01:00 UTC, 03:00 UTC, and so on).

Thank you well.

I switch back to the local file storage. But the behaviour is the same.

The storage configuration:

blocks_storage:
  backend: filesystem
  bucket_store:
    sync_dir: /data/tsdb-sync
  filesystem:
    dir: /data/blocks
  tsdb:
    dir: /data/tsdb

I don't see any entires in /data/blocks. Bu the cluster config json like mino storage.

I think I don't realy understand the peristence semantic of mimir. My excpetion of using a object storage was that mimir does not need a local volume but at least a s3 bucket.

1. If I use a single instance with local file storage then it doesn't matter if the blocks stored in /data/tsdb or /data/blocks. Right?
2. If I use a cluster of mimir then I need for bucket_store like s3 and a local volume for the tsdb section right ?
3. If it so then I don't understand the benefit to use an object storage. Doen't this approach makes thinks to compilcate ? I assue the querier must do a lot of work for dedup the query results ?
4. I red in the documentation that I can set the data retention over

limits:
  compactor_blocks_retention_period: 10m

But mimir does not delete any data.

[pracucci]

If I use a single instance with local file storage then it doesn't matter if the blocks stored in /data/tsdb or /data/blocks. Right?

The TSDB dir (in your case /data/tsdb) is used by ingesters, while the /data/blocks is the "object storage" (in your case backed by a simple filesystem). Ingesters are expected to upload TSDB blocks to /data/blocks every 2h.

If I use a cluster of mimir then I need for bucket_store like s3 and a local volume for the tsdb section right ?

Correct. The TSDB directory should be persisted across restarts because it's where the ingesters store the WAL.

If it so then I don't understand the benefit to use an object storage. Doen't this approach makes thinks to compilcate ? I assue the querier must do a lot of work for dedup the query results ?

Data from ingesters is removed after few hours. Object storage provide long-term persistency.

compactor_blocks_retention_period: 10m

Retention is applied to the object storage, so /data/blocks in your case. Retention is enforced by compactor at period intervals (configurable, called "blocks clean period"). It's expected to work. If it doesn't, I would need more information about your compactor health (e.g. logs).

[suikast42]

I am more confused than before 😂.

Lets say I will configure mimir data retention for a week (168 h )

1. For local sotrage which parameters I have to set ?
2. For s3 bucket. Is it better to set a TTL or lifetime for the bucket or does it work as local storage ?

[suikast42]

I am testing now this setting for holding the metrics for 24h in local file storage

I expect that I only have to change the compactor_blocks_retention_period in the limit section to 72h for example to have a retetion for 3d. Is that right?

# Test observability.mimir[0]
# Do not use this configuration in production.
# It is for demonstration purposes only.

# Run Mimir in single process mode, with all components running in 1 process.
target: all,alertmanager,overrides-exporter
# Disable tendency support.
multitenancy_enabled: false

server:
  http_listen_port: 9009
  log_level: debug
  # Configure the server to allow messages up to 100MB.
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

blocks_storage:
  backend: filesystem
  bucket_store:
    sync_dir: /data/tsdb-sync
    ignore_blocks_within: 10h
  filesystem:
    dir: /data/blocks
  tsdb:
    dir: /data/tsdb
    # Note that changing this requires changes to some other parameters like
    # -querier.query-store-after,
    # -querier.query-ingesters-within and
    # -blocks-storage.bucket-store.ignore-blocks-within.
    retention_period: 24h
querier:
  query_ingesters_within: 12h
  query_store_after: 8h
ruler_storage:
  backend: filesystem
  filesystem:
    dir: /data/rules

alertmanager_storage:
  backend: filesystem
  filesystem:
    dir: /data/alarms

frontend:
  grpc_client_config:
    grpc_compression: snappy

frontend_worker:
  grpc_client_config:
    grpc_compression: snappy

ingester_client:
  grpc_client_config:
    grpc_compression: snappy

query_scheduler:
  grpc_client_config:
    grpc_compression: snappy

alertmanager:
  data_dir: /data/alertmanager
#  retention: 120h
  sharding_ring:
    replication_factor: 1
  alertmanager_client:
    grpc_compression: snappy

ruler:
  query_frontend:
    grpc_client_config:
      grpc_compression: snappy

compactor:
#  compaction_interval: 1m
#  deletion_delay: 2m
  max_closing_blocks_concurrency: 2
  max_opening_blocks_concurrency: 4
  symbols_flushers_concurrency: 4
  data_dir: /data/compactor
  sharding_ring:
    kvstore:
      store: memberlist


ingester:
  ring:
    replication_factor: 1

store_gateway:
  sharding_ring:
    replication_factor: 1

limits:
  # Limit queries to 5 years. You can override this on a per-tenant basis.
  max_total_query_length: 43680h
  max_label_names_per_series: 42
  # Allow ingestion of out-of-order samples up to 2 hours since the latest received sample for the series.
  out_of_order_time_window: 2h
  # delete old blocks from long-term storage.
  # Delete from storage metrics data older than 24 hours.
  compactor_blocks_retention_period: 24h
  ingestion_rate: 100000

[CMCDragonkai]

Is this:

Mimir upload blocks (of metrics) to the object storage at period interval. After you create a new Mimir cluster, the first blocks will be uploaded by ingesters about 3 hours after, and then every 2 hours at minute 00 of odd UTC hours (e.g. 01:00 UTC, 03:00 UTC, and so on).

Configurable?

[56quarters]

No, ingesters will only create 2h sized blocks.