Distributed loki, promtail with aws/s3 on k8s

[xpepermint]

I think I successfully installed loki-distrubuted and promtail on k8s using helm.

$ helm upgrade --install loki --namespace=loki grafana/loki-distributed -f values.yaml
$ helm upgrade --install promtail --namespace=loki grafana/promtail \
    --set config.lokiAddress=http://loki-loki-distributed-gateway.loki/loki/api/v1/push

I followed the documentation on aws/s3 and made the required changes in the values.yaml file:

loki:
  config: |
    auth_enabled: false

    server:
      http_listen_port: 3100

    distributor:
      ring:
        kvstore:
          store: memberlist

    memberlist:
      join_members:
        - loki-loki-distributed-memberlist

    ingester:
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      chunk_idle_period: 30m
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_retain_period: 1m
      max_transfer_retries: 0

    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m

    schema_config:
      configs:
        - from: 2021-04-10
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            prefix: loki_index_
            period: 24h

    storage_config:
      aws:
        s3: s3://XXX:YYY@REGION/BUCKET-NAME # secret :)
        # s3forcepathstyle: true
      boltdb_shipper:
        shared_store: aws
        active_index_directory: /var/loki/index
        cache_location: /var/loki/cache
        cache_ttl: 168h
      filesystem:
        directory: /var/loki/chunks

    chunk_store_config:
      max_look_back_period: 0s

    table_manager:
      retention_deletes_enabled: false
      retention_period: 0s

    query_range:
      align_queries_with_step: true
      max_retries: 5
      split_queries_by_interval: 15m
      cache_results: true
      results_cache:
        cache:
          enable_fifocache: true
          fifocache:
            max_size_items: 1024
            validity: 24h

    frontend_worker:
      frontend_address: loki-loki-distributed-query-frontend:9095

    frontend:
      log_queries_longer_than: 5s
      compress_responses: true

    compactor:
      # working_directory: /data/compactor
      shared_store: aws
      compaction_interval: 5m

    ruler:
      storage:
        type: local
        local:
          directory: /etc/loki/rules
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch
      alertmanager_url: https://alertmanager.xx
      external_url: https://alertmanager.xx

1. The first issue/question I have is, should I use the word s3 or aws when setting shared_store and object_store options. Documents and tutorials use sometimes s3 and sometimes aws.

2. The second thing which is not clear to me is how loki containers actually use S3 storage. When all containers are marked as Running, the S3 bucket remains empty. It remains empty even after that until I manually remove pods so that k8s can recreate them. This is the moment when data are flushed to S3. How can I verify that everything works correctly? Is there an amplitude by which the containers upload data to S3? How exactly data are later read from S3? Btw, I don't see any errors inside containers. It's not clear how this is supposed to work. I would ask a ninja to write a detailed explanation on this topic so I/we can follow the instructions and be safe that the system works as expected.

[xpepermint]

Update: After about 1h I now see fake and index directories in S3 bucket.

[MarjanJordanovski]

loki won't send logs to s3 if no new logs are created. So make sure new logs are created while loki is running (check loki-gateway pod logs to see if new POST requests are received). Also use chunk_idle_period: 1m if you want to see your s3 updated quicklier.
Also make sure that aws user with your XXX:YYY secrets has relevant policies attached so it can access s3.

And regarding storage_config, I'm using shared_store: s3 ,and that works for me.