server returned HTTP status 401 Unauthorized (401): no org id #7081

mongoliaredmountain · 2022-09-07T04:30:08Z

I used helm install Loki and Promtail,But Promtail send 'caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"'

loki.yaml
loki的副本.md

promtail.yaml
promtail的副本.md

logs
level=error ts=2022-09-07T04:29:34.599308846Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:35.800382126Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:36.900329685Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:38.09956929Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:39.19990638Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:40.299893729Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:41.399964187Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:42.600158749Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:43.800863042Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:44.90022787Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:46.100090867Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:47.299770482Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:48.39999092Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"
level=error ts=2022-09-07T04:29:49.599192624Z caller=client.go:380 component=client host=loki-gateway msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id"

sbueringer · 2022-09-07T13:24:10Z

Not sure if it's a fix or just a workaround, but I had to do the following to make it work:

promtail values.yaml

config:
  # publish data to loki
  clients:
    - url: http://loki-gateway/loki/api/v1/push
      tenant_id: 1

and then in the values.yaml of the grafana chart

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Loki
      type: loki
      url: http://loki:3100
      jsonData:
        httpHeaderName1: 'X-Scope-OrgID'
      secureJsonData:
        httpHeaderValue1: '1'

fabriziopandini · 2022-09-07T13:29:52Z

+1 to get this fixed or documented, it took a while to get things working...

Grafana Agent Operator continously fails with: server returned HTTP status 401 Unauthorized (401): no org id ...and that is probably tracked upstream via: <grafana/loki#7081> (We are not interesting in having multi-tenant configuration and this is probably okay-ish.)

liguozhong · 2022-09-08T08:25:08Z

https://github.com/grafana/loki/tree/main/production/helm/loki
loki.auth_enabled : false .
helm's installation command Modify this configuration to false, you can disable the auth feature to fix this error
.

grafana/loki#7081 Also mentioned in: grafana/loki#7267 (comment)

mfordjody · 2022-12-17T09:34:10Z

good！

eweremono · 2023-03-15T13:34:25Z

This was very helpful, it enabled auth by default, you can change the value here to false and apply helm again

yahiya-ayoub · 2023-05-19T12:07:40Z

Hey guys, I got the same issue when I built the following structure:

enable loki gateway basic authentication and used the nginx ingress controller class to create aws network load balancer by setting the following options:

gateway:
  enabled: true
  replicas: 4
  verboseLogging: true
  autoscaling:
    enabled: false
    minReplicas: 3
    maxReplicas: 4
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: 50
  deploymentStrategy:
    type: RollingUpdate
  image:
    registry: docker.io
    repository: nginxinc/nginx-unprivileged
    tag: 1.19-alpine
    digest: null
    pullPolicy: IfNotPresent
  priorityClassName: null
  podAnnotations: {}
  podLabels: {}
  extraArgs: []
  extraEnv: []
  extraEnvFrom: []
  lifecycle: {}
  extraVolumes: []
  extraVolumeMounts: []
  podSecurityContext:
    fsGroup: 101
    runAsGroup: 101
    runAsNonRoot: true
    runAsUser: 101
  containerSecurityContext:
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
    allowPrivilegeEscalation: false
  resources: 
    requests:
        cpu: 1024m
        memory: 2048Mi
    limits:
      cpu: 1024m
      memory: 2048Mi
  terminationGracePeriodSeconds: 30
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "loki.gatewaySelectorLabels" . | nindent 10 }}
          topologyKey: kubernetes.io/hostname
  nodeSelector: {}
  tolerations: []
  service:
    port: 80
    type: ClusterIP
    clusterIP: null
    nodePort: null
    loadBalancerIP: null
    annotations: {}
    labels: {}
  ingress:
    enabled: enable
    ingressClassName: "ingress-nginx-private"
    annotations: {}
    hosts:
      - host: "loki-staging-dev.monitoring.floward.io"
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: loki-gateway-tls
        hosts:
          - "loki-staging-dev.monitoring.floward.io"
  basicAuth:
    enabled: true
    username: testuser
    password: "testpassword"
    htpasswd: >-
      {{ if .Values.loki.tenants }}


        {{- range $t := .Values.loki.tenants }}
      {{ htpasswd (required "All tenants must have a 'name' set" $t.name) (required "All tenants must have a 'password' set" $t.password) }}


        {{- end }}
      {{ else }} {{ htpasswd (required "'gateway.basicAuth.username' is required" .Values.gateway.basicAuth.username) (required "'gateway.basicAuth.password' is required" .Values.gateway.basicAuth.password) }} {{ end }}
    existingSecret: null
  readinessProbe:
    httpGet:
      path: /
      port: http
    initialDelaySeconds: 15
    timeoutSeconds: 1
  nginxConfig:
    logFormat: |-
      main '$remote_addr - $remote_user [$time_local]  $status '
              '"$request" $body_bytes_sent "$http_referer" '
              '"$http_user_agent" "$http_x_forwarded_for"';
    serverSnippet: ""
    httpSnippet: >-
      {{ if .Values.loki.tenants }}proxy_set_header X-Scope-OrgID $remote_user;{{ end }}
    customReadUrl: null
    customWriteUrl: null
    customBackendUrl: null
    file: |
      {{- include "loki.nginxFile" . | indent 2 -}}

I tried the solution of setting a tenant in promtail values file, but still getting the error
level=error ts=2023-05-19T12:07:02.507864685Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=401 tenant=1 error="server returned HTTP status 401 Unauthorized (401): <html>"

Note: I am using grafana/loki helm chart, do you have any other recommendations for helm chart?

dellnoantechnp · 2023-09-11T09:07:26Z

I fond this solution:

My Environment:

k8s: 1.24.x
Loki: loki-distributed-0.69.16 chart (appVersion 2.8.2)
Promtail: promtail-6.10.0 chart (appVersion 2.7.4)
Grafana: 10.0.2

1. If you want simplely enable multitenant support:

## loki-distributed-values.yaml
loki:
  config: |
    auth_enabled: true
  ingress:
    enabled: false
  gateway: 
    enabled: true
    basicAuth:
      enabled: false    <-- not support basic auth

## promtail-values.yaml
config:
  clients:
    - url: http://loki-distributed-gateway/loki/api/v1/push
      tenant_id: tenant-1

  snippets:
    pipelineStages:
      - cri: {}
      - match:
          selector: '{stream="stderr",container="promtail"}'
          action: drop
      - match: 
          selector: '{namespace="dev",stream="stdout"}'
          stages:
            - tenant:                                       <-- dynamic match tenantID
                source: namespace

In Grafana Web-UI:
config datasource.

Custom HTTP Headers

Header: X-Scope-OrgID
Value: tenant-1

2. If you want to use basic_auth `username` as `tenantID`

## loki-distributed-values.yaml
loki:
  config: |
    auth_enabled: true
  ingress:
    enabled: false
  gateway: 
    enabled: true
    basicAuth:
      enabled: true      <--  support basic auth
      username: admin
      password: admin-password
      existingSecret: null                                                       <--  only admin user      OR
      existingSecret: loki-multi-tenant-auth-secret              <-- multi tenant user as tenantID
    nginxConfig:
      file: |
          ............
                location = /loki/api/v1/push {
                  set $loki_api_v1_push_backend http://{{ include "loki.distributorFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }};
                  proxy_pass       $loki_api_v1_push_backend:3100$request_uri;
                  ## add X-Scope-OrgID header to backend endpoint.
                  proxy_set_header X-Scope-OrgID $remote_user;        <--- map remote_user to X-Scope-OrgID header
                  proxy_http_version 1.1;
                }
         ............

generate multi auth user using tenant id.

copy this bash script to generator .htpassword:

while read -r line; do
  USER=$(echo $line | awk '{print $1}');
  PASSWORD=$(echo $line | awk '{print $2}');
  printf "$USER:$(openssl passwd -apr1 $PASSWORD)\n" >> .htpasswd
  printf "user:\ %-20s updated" $USER
done < <(cat << EOF
user1  password1
user2  password2
tenant-3  password-3
EOF
)

Create loki-multi-tenant-auth-secret secret:

## kustomize
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generatorOptions:
  disableNameSuffixHash: true
secretGenerator:
  - name: loki-multi-tenant-auth-secret
    namespace: loki
    files:
      - .htpasswd

## promtail-values.yaml
config:
  clients:
    - url: http://loki-distributed-gateway/loki/api/v1/push
      basic_auth:
        username: tenant-3
        password: password-3

In Grafana Web-UI:
config datasource.

Auth:

Basic auth: enable

Basic Auth Details:

User: tenant-3
Password: password-3

Custom HTTP Headers

Header: X-Scope-OrgID
Value: tenant-3

mischavandenburg · 2023-12-30T11:02:12Z

I ran into this issue when deploying Loki to my homelab.

Then I used these values with the grafana/loki chart to get it to work:

    loki:
      auth_enabled: false
      commonConfig:
        replication_factor: 1
      storage:
        type: "filesystem"
    singleBinary:
      replicas: 1

Note that this is an installation for learning and hobby purposes.

vfzh · 2024-02-04T06:03:25Z

if you set the auth_basic: false, you need to set your promtail config.yaml like
clients:

url: http://$your_host:$your_port/loki/api/v1/push
tenant_id: fake

This change creates an initial configuration for Grafana Loki. This is surprisingly hard-won, and as a result, not yet fully hooked up to ArgoCD. This was applied manually with ``` helm upgrade --create-namespace --namespace logging logging . ``` There were a lot of reasons why this was hard-won. For starters, the Grafana Loki chart claimed there was a `global` section into which you could put `extraEnvFrom`, but that's not implemented (grafana/loki#11391). Next, every part of the installation guide (https://grafana.com/docs/loki/v2.9.x/setup/install/helm/install-scalable/) says to just copy-paste your S3 credentials. This is not a good idea for a variety of reasons, starting with them not being static, as they're being provisioned by Rook's CephObjectStore integration, and ending with it just not being good practice to put secrets into plaintext in a helm chart anyway. Fortunately, buried in the documentation and the helm chart are clues: it's possible to use `extraEnvFrom` to load the data from ConfigMap/Secret, but then you need to pass a special command-line argument `-config.expand-env=true`. See also, the aforementioned `global` section that does not actually work. But, after this adventure, we have centralized logging. There's still a few TODO issues, like figuring out authentication (see the sample on grafana/loki#7081 for guidance). There's also the question of how to automatically configure the Loki source in Grafana.

Zuckster · 2024-11-21T10:19:18Z

@dellnoantechnp
Im trying multi auth user as tenantID. Perhaps its not working, gateway give password mismatch, from alloy side 401 error. I hash my password, put in existingSecret, add proxy_set_header X-Scope-OrgID $remote_user; and still nothing. Maybe you can help me?

sbueringer mentioned this issue Sep 7, 2022

🌱 hack/observability: Bump Loki, Grafana and promtail charts kubernetes-sigs/cluster-api#7182

Merged

3 tasks

mongoliaredmountain closed this as completed Sep 26, 2022

deejgregor added a commit to opennms-forge/onms-k8s-poc that referenced this issue Oct 15, 2022

Workaround issue in loki helm chart with single binary

da7b85d

grafana/loki#7081 Also mentioned in: grafana/loki#7267 (comment)

deejgregor added a commit to opennms-forge/onms-k8s-poc that referenced this issue Oct 15, 2022

Workaround issue in loki helm chart with single binary

075a9c6

grafana/loki#7081 Also mentioned in: grafana/loki#7267 (comment)

deejgregor added a commit to opennms-forge/onms-k8s-poc that referenced this issue Oct 17, 2022

Workaround issue in loki helm chart with single binary

e16f884

grafana/loki#7081 Also mentioned in: grafana/loki#7267 (comment)

danivendetta mentioned this issue Nov 14, 2023

promtail cannot send log to loki-gateway server with error 401 Unauthorized message grafana/helm-charts#819

Open

vazkarvishal mentioned this issue Mar 5, 2024

fix: promtail failures connecting to local loki installation #12126

Merged

2 tasks

grafanabot mentioned this issue Mar 12, 2024

fix: promtail failures connecting to local loki installation [release-2.9.x] #12184

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server returned HTTP status 401 Unauthorized (401): no org id #7081

server returned HTTP status 401 Unauthorized (401): no org id #7081

mongoliaredmountain commented Sep 7, 2022

sbueringer commented Sep 7, 2022

fabriziopandini commented Sep 7, 2022

liguozhong commented Sep 8, 2022 •

edited

Loading

mfordjody commented Dec 17, 2022

eweremono commented Mar 15, 2023

yahiya-ayoub commented May 19, 2023 •

edited

Loading

dellnoantechnp commented Sep 11, 2023 •

edited

Loading

mischavandenburg commented Dec 30, 2023

vfzh commented Feb 4, 2024

Zuckster commented Nov 21, 2024

server returned HTTP status 401 Unauthorized (401): no org id #7081

server returned HTTP status 401 Unauthorized (401): no org id #7081

Comments

mongoliaredmountain commented Sep 7, 2022

sbueringer commented Sep 7, 2022

fabriziopandini commented Sep 7, 2022

liguozhong commented Sep 8, 2022 • edited Loading

mfordjody commented Dec 17, 2022

eweremono commented Mar 15, 2023

yahiya-ayoub commented May 19, 2023 • edited Loading

dellnoantechnp commented Sep 11, 2023 • edited Loading

My Environment:

1. If you want simplely enable multitenant support:

2. If you want to use basic_auth username as tenantID

generate multi auth user using tenant id.

copy this bash script to generator .htpassword:

mischavandenburg commented Dec 30, 2023

vfzh commented Feb 4, 2024

Zuckster commented Nov 21, 2024

liguozhong commented Sep 8, 2022 •

edited

Loading

yahiya-ayoub commented May 19, 2023 •

edited

Loading

dellnoantechnp commented Sep 11, 2023 •

edited

Loading

2. If you want to use basic_auth `username` as `tenantID`