error contacting frontend

[guleng]

Describe the bug
I deployed query frontend, but Loki keeps reporting this error

level=error ts=2020-10-29T12:31:39.866575302Z caller=worker_frontend_manager.go:96 msg="error contacting frontend" err="rpc error: code = Unavailable desc = connection closed"
level=error ts=2020-10-29T12:31:39.933130741Z caller=worker_frontend_manager.go:96 msg="error contacting frontend" err="rpc error: code = Unavailable desc = connection closed"
level=error ts=2020-10-29T12:31:39.988758706Z caller=worker_frontend_manager.go:96 msg="error contacting frontend" err="rpc error: code = Unavailable desc = connection closed"
...........

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (SHA or version)：2.0
2. Started Promtail：2.0

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

**Screenshots, loki config , query-frontend config **

 *loki config*
     config:
  auth_enabled: false
  distributor:
    ring:
      heartbeat_timeout: 10m
      kvstore:
        store: consul
        prefix: "collectors/"
        consul:
          host: "192.168.32.4:8500"

  ingester:
    chunk_idle_period: 30m    
    chunk_block_size: 262144    
    chunk_retain_period: 15m    
    max_transfer_retries: 60    
    chunk_target_size: 1536000  
    lifecycler:
      heartbeat_period: 5s   
      join_after: 30s   
      num_tokens: 512   
      tokens_file_path: /data/tokens
      
      ring:
        replication_factor: 1
        kvstore:
          store: consul
          prefix: "collectors/"
          consul:
            host: "192.168.32.4:8500"
            http_client_timeout: "20s"
            consistent_reads: true
      frontend_worker:
    frontend_address: query-frontend.loki.svc.cluster.local:3100   
    parallelism: 4             
  querier:
    query_ingesters_within: 168h   
  query_range:   
    cache_results: true    
    max_retries: 5
    align_queries_with_step: true
    split_queries_by_interval: 15m
    results_cache:
      cache:       
        enable_fifocache: true   
        default_validity: 1h    
        fifocache:
          max_size_bytes: 10G    
          max_size_items: 1000   
          validity: 1h   
  limits_config:   
    cardinality_limit: 1000000  
    ingestion_burst_size_mb: 128  
    ingestion_rate_strategy: global  
    ingestion_rate_mb: 64  
    max_entries_limit_per_query: 100000   
    enforce_metric_name: false   
    reject_old_samples: true   
    reject_old_samples_max_age: 168h   
  schema_config:
    configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
      chunks:
        prefix: chunk_
        period: 24h
    - from: 2020-10-29
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
      row_shards: 20 
  server:
    http_listen_port: 3100
  storage_config:
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      cache_ttl: 24h         
      shared_store: filesystem
    filesystem:
      directory: /data/loki/chunks
  chunk_store_config:
    cache_lookups_older_than: 36h  
    max_look_back_period: 720h  
  table_manager:
    poll_interval: 5m   
    creation_grace_period: 30m  
    retention_deletes_enabled: true
    retention_period: 720h
  compactor:
    working_directory: /data/loki/boltdb-shipper-compactor
    shared_store: filesystem
 ------------------------------------------------------------------
 **query-frontend config**
auth_enabled: false

# We don't want the usual /api/prom prefix.
http_prefix:

server:
  http_listen_port: 3100
schema_config:
  configs:
  - from: 2020-10-29
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

query_range:
  # make queries more cache-able by aligning them with their step intervals
  align_queries_with_step: true
  max_retries: 5
  # parallelize queries in 15min intervals
  split_queries_by_interval: 15m
  cache_results: true

  results_cache:
          #max_freshness: 10m
    cache:
      # We're going to use the in-process "FIFO" cache
      enable_fifocache: true
      fifocache:
        size: 1024
        validity: 24h

frontend:
  log_queries_longer_than: 5s
  downstream_url: loki.loki.svc.cluster.local:3100
  compress_responses: true

[n-saw]

@guleng how did you fix this.

I do see the missing http:// in downstream_url; was that the issue you had

See #3109

[cyriltovena]

I think his problem was that he was using both http and grpc worker. when using http you should only set downstream_url and when you use grpc you should set frontend_worker.frontend_address, not both. Le dim. 3 janv. 2021 à 23:17, nileshcs <[email protected]> a écrit :

…

@guleng <https://github.com/guleng> how did you fix this. I do see the missing http:// in downstream_url; was that the issue you had See #3109 <#3109> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2843 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIBF3KEEDWJ5OYZ427RJWDSYDUJJANCNFSM4TDV2U6A> .

[LinTechSo]

Hi. any Updates?
I added HTTP mode but get this error
level=error ts=2022-07-15T18:17:02.898913837Z caller=frontend_processor.go:69 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unimplemented desc = unknown service frontend.Frontend"

[naiame-neltheri]

@LinTechSo have you able to solve the issue?
I am having exact same error with HTTP mode

[dansl1982]

any updates here?