Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proxy on grafana-server #6484

Closed
karasjoh000 opened this issue Apr 24, 2021 · 3 comments · Fixed by #7065
Closed

proxy on grafana-server #6484

karasjoh000 opened this issue Apr 24, 2021 · 3 comments · Fixed by #7065
Labels
wontfix

Comments

@karasjoh000
Copy link
Contributor

Bug Report

What happened:

TASK [ceph-grafana : wait for grafana to start] ********************************************************************************************************************************************************************
task path: /home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-grafana/tasks/configure_grafana.yml:112
Friday 23 April 2021  19:06:03 -0700 (0:00:04.962)       1:06:50.258 **********
Using module file /usr/lib/python3/dist-packages/ansible/modules/utilities/logic/wait_for.py
Pipelining is enabled.
<server> ESTABLISH SSH CONNECTION FOR USER: ceph
<server> SSH: EXEC sshpass -d12 ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'User="ceph"' -o ConnectTimeout=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o 'ProxyCommand=ssh -W %h:%p -q [email protected]' -o ControlPath=/home/john/.ansible/cp/%h-%r-%p server'/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=yfrafcacnidfanyemlweffqgbxxdcyuy] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-yfrafcacnidfanyemlweffqgbxxdcyuy ; /usr/bin/python3'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<server> (1, b'\n{"msg": "Timeout when waiting for 192.168.20.10:3000", "elapsed": 300, "failed": true, "invocation": {"module_args": {"host": "192.168.20.10", "port": 3000, "timeout": 300, "connect_timeout": 5, "delay": 0, "active_connection_states": ["ESTABLISHED", "FIN_WAIT1", "FIN_WAIT2", "SYN_RECV", "SYN_SENT", "TIME_WAIT"], "state": "started", "sleep": 1, "path": null, "search_regex": null, "exclude_hosts": null, "msg": null}}}\n', b"Warning: Permanently added 'server' (ECDSA) to the list of known hosts.\r\n")
<server> Failed to connect to the host via ssh: Warning: Permanently added 'server' (ECDSA) to the list of known hosts.
fatal: [ceph01]: FAILED! => changed=false
  elapsed: 300
  invocation:
    module_args:
      active_connection_states:
      - ESTABLISHED
      - FIN_WAIT1
      - FIN_WAIT2
      - SYN_RECV
      - SYN_SENT
      - TIME_WAIT
      connect_timeout: 5
      delay: 0
      exclude_hosts: null
      host: 192.168.20.10
      msg: null
      path: null
      port: 3000
      search_regex: null
      sleep: 1
      state: started
      timeout: 300
  msg: Timeout when waiting for 192.168.20.10:3000

RUNNING HANDLER [ceph-prometheus : service handler] ****************************************************************************************************************************************************************
task path: /home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-prometheus/handlers/main.yml:2
Friday 23 April 2021  19:11:08 -0700 (0:05:04.846)       1:11:55.104 **********

PLAY RECAP *********************************************************************************************************************************************************************************************************
ceph01                     : ok=594  changed=74   unreachable=0    failed=1    skipped=725  rescued=0    ignored=0
ceph02                     : ok=444  changed=38   unreachable=0    failed=0    skipped=643  rescued=0    ignored=0
ceph03                     : ok=454  changed=41   unreachable=0    failed=0    skipped=636  rescued=0    ignored=0


INSTALLER STATUS ***************************************************************************************************************************************************************************************************
Install Ceph Monitor           : Complete (0:08:20)
Install Ceph Manager           : Complete (0:10:58)
Install Ceph OSD               : Complete (0:11:51)
Install Ceph MDS               : Complete (0:05:21)
Install Ceph RGW               : Complete (0:05:17)
Install Ceph Client            : Complete (0:05:55)
Install Ceph Grafana           : In Progress (0:10:38)
        This phase can be restarted by running: roles/ceph-grafana/tasks/main.yml
Install Ceph Node Exporter     : Complete (0:05:55)

Friday 23 April 2021  19:11:08 -0700 (0:00:00.035)       1:11:55.140 **********
===============================================================================
ceph-grafana : wait for grafana to start ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 304.85s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-grafana/tasks/configure_grafana.yml:112 ----------------------------------------------------------------------------------------------------------------------
ceph-facts : get ceph current status ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 304.59s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-facts/tasks/facts.yml:278 ------------------------------------------------------------------------------------------------------------------------------------
ceph-osd : use ceph-volume to create bluestore osds ------------------------------------------------------------------------------------------------------------------------------------------------------- 162.25s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml:2 --------------------------------------------------------------------------------------------------------------------------------
ceph-container-engine : allow apt to use a repository over https (debian) --------------------------------------------------------------------------------------------------------------------------------- 133.88s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-container-engine/tasks/pre_requisites/debian_prerequisites.yml:8 ---------------------------------------------------------------------------------------------
ceph-osd : systemd start osd ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 112.22s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-osd/tasks/start_osds.yml:62 ----------------------------------------------------------------------------------------------------------------------------------
ceph-osd : ensure "/var/lib/ceph/osd/{{ cluster }}-{{ item }}" is present ---------------------------------------------------------------------------------------------------------------------------------- 86.76s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-osd/tasks/start_osds.yml:53 ----------------------------------------------------------------------------------------------------------------------------------
ceph-config : create ceph initial directories -------------------------------------------------------------------------------------------------------------------------------------------------------------- 78.61s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ------------------------------------------------------------------------------------------------------------------
ceph-config : create ceph initial directories -------------------------------------------------------------------------------------------------------------------------------------------------------------- 69.83s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ------------------------------------------------------------------------------------------------------------------
ceph-config : create ceph initial directories -------------------------------------------------------------------------------------------------------------------------------------------------------------- 69.65s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ------------------------------------------------------------------------------------------------------------------
ceph-config : create ceph initial directories -------------------------------------------------------------------------------------------------------------------------------------------------------------- 68.88s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ------------------------------------------------------------------------------------------------------------------
ceph-config : create ceph initial directories -------------------------------------------------------------------------------------------------------------------------------------------------------------- 68.05s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ------------------------------------------------------------------------------------------------------------------
ceph-config : create ceph initial directories -------------------------------------------------------------------------------------------------------------------------------------------------------------- 67.64s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ------------------------------------------------------------------------------------------------------------------
ceph-config : create ceph initial directories -------------------------------------------------------------------------------------------------------------------------------------------------------------- 67.21s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ------------------------------------------------------------------------------------------------------------------
ceph-container-engine : install container packages --------------------------------------------------------------------------------------------------------------------------------------------------------- 64.03s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-container-engine/tasks/pre_requisites/prerequisites.yml:26 ---------------------------------------------------------------------------------------------------
ceph-prometheus : ship systemd services -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 55.29s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-prometheus/tasks/systemd.yml:2 -------------------------------------------------------------------------------------------------------------------------------
ceph-rgw : create replicated pools for rgw ----------------------------------------------------------------------------------------------------------------------------------------------------------------- 36.85s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-rgw/tasks/rgw_create_pools.yml:52 ----------------------------------------------------------------------------------------------------------------------------
ceph-mon : generate systemd unit file for mon container ---------------------------------------------------------------------------------------------------------------------------------------------------- 34.62s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-mon/tasks/systemd.yml:2 --------------------------------------------------------------------------------------------------------------------------------------
ceph-node-exporter : ship systemd service ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 33.06s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-node-exporter/tasks/systemd.yml:2 ----------------------------------------------------------------------------------------------------------------------------
ceph-handler : copy mon restart script --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 32.64s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-handler/tasks/handler_mons.yml:9 -----------------------------------------------------------------------------------------------------------------------------
ceph-mgr : copy ceph key(s) if needed ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 31.18s
/home/john/Projects/ceph-dev/ceph-ansible/roles/ceph-mgr/tasks/common.yml:65 --------------------------------------------------------------------------------------------------------------------------------------

What you expected to happen:
not to fail
How to reproduce it (minimal and precise):

start the containerized deployment under a proxy.
Share your group_vars files, inventory and full ceph-ansibe log

---
dummy:
ceph_release_num:
  octopus: 15
ceph_origin: repository
ceph_repository: community
ceph_stable_release: octopus
monitor_interface: eno1
journal_size: 5120
public_network: 192.168.0.0/16
radosgw_interface: eno1
ceph_docker_http_proxy: http-proxy
ceph_docker_https_proxy: https-proxy
ceph_docker_no_proxy: no-proxy
containerized_deployment: true
dashboard_enabled: True
dashboard_protocol: http
dashboard_admin_user: admin
dashboard_admin_password: password
grafana_admin_user: admin
grafana_admin_password: password

Possible fix
Adding the proxy env in grafana template worked:

# This file is managed by ansible, don't make changes here - they will be
# overwritten.
[Unit]
Description=grafana-server
{% if container_binary == 'docker' %}
After=docker.service
Requires=docker.service
{% else %}
After=network.target
{% endif %}

[Service]
EnvironmentFile=-/etc/environment
{% if container_binary == 'podman' %}
ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
ExecStartPre=-/usr/bin/{{ container_binary }} rm --storage grafana-server
{% else %}
ExecStartPre=-/usr/bin/{{ container_binary }} stop grafana-server
{% endif %}
ExecStartPre=-/usr/bin/{{ container_binary }} rm grafana-server
ExecStart=/usr/bin/{{ container_binary }} run --rm --name=grafana-server \
{% if container_binary == 'podman' %}
  -d --log-driver journald --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
{% endif %}
  -v /etc/grafana:/etc/grafana:Z \
  -v /var/lib/grafana:/var/lib/grafana:Z \
  --net=host \
  --cpu-period={{ grafana_container_cpu_period }} \
  --cpu-quota={{ grafana_container_cpu_period * grafana_container_cpu_cores }} \
  --memory={{ grafana_container_memory }}GB \
  --memory-swap={{ grafana_container_memory * 2 }}GB \
  -e GF_INSTALL_PLUGINS={{ grafana_plugins|join(',') }} \
  -e http_proxy={{ ceph_docker_http_proxy }} \
  -e https_proxy={{ ceph_docker_https_proxy }} \
  -e no_proxy={{ ceph_docker_no_proxy }} \
  {{ grafana_container_image }}
{% if container_binary == 'podman' %}
ExecStop=-/usr/bin/sh -c "/usr/bin/{{ container_binary }} rm -f `cat /%t/%n-cid`"
{% else %}
ExecStop=-/usr/bin/{{ container_binary }} stop grafana-server
{% endif %}
KillMode=none
Restart=always
RestartSec=10s
TimeoutStartSec=120
TimeoutStopSec=15
{% if container_binary == 'podman' %}
Type=forking
PIDFile=/%t/%n-pid
{% endif %}

[Install]
WantedBy=multi-user.target

Environment:

  • OS (e.g. from /etc/os-release): ubuntu 20.04
  • Kernel (e.g. uname -a): Linux ceph01 5.4.0-72-generic Change the default disk numbering #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
@karasjoh000
Copy link
Contributor Author

just noticed this is a duplicate of #6481

karasjoh000 added a commit to karasjoh000/ceph-ansible that referenced this issue Apr 27, 2021
@karasjoh000
ceph-grafana: Add proxy env vars to grafana service template
ef83328 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: ceph#6484, ceph#6481

Signed-off-by: John Karasev <[email protected]>
karasjoh000 added a commit to karasjoh000/ceph-ansible that referenced this issue Apr 27, 2021
@karasjoh000
ceph-grafana: Add proxy env vars to grafana service template
1eecca9 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: ceph#6484, ceph#6481

Signed-off-by: John Karasev <[email protected]>
@github-actions
Copy link

github-actions bot commented May 9, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

@github-actions
Copy link

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

@karasjoh000 karasjoh000 mentioned this issue Jan 28, 2022
Merged
guits pushed a commit that referenced this issue Feb 7, 2022
@karasjoh000 @guits
ceph-grafana: Add proxy env vars to grafana service template
79ca442 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: #6484, #6481

Signed-off-by: John Karasev <[email protected]>
mergify bot pushed a commit that referenced this issue Feb 7, 2022
@karasjoh000
ceph-grafana: Add proxy env vars to grafana service template
fce9c51 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: #6484, #6481

Signed-off-by: John Karasev <[email protected]>
(cherry picked from commit 79ca442)
mergify bot pushed a commit that referenced this issue Feb 7, 2022
@karasjoh000
ceph-grafana: Add proxy env vars to grafana service template
c585bce 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: #6484, #6481

Signed-off-by: John Karasev <[email protected]>
(cherry picked from commit 79ca442)
guits pushed a commit that referenced this issue Feb 9, 2022
@karasjoh000 @guits
ceph-grafana: Add proxy env vars to grafana service template
9d7e3bf 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: #6484, #6481

Signed-off-by: John Karasev <[email protected]>
(cherry picked from commit 79ca442)
guits pushed a commit that referenced this issue Feb 9, 2022
@karasjoh000 @guits
ceph-grafana: Add proxy env vars to grafana service template
2e2d23c 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: #6484, #6481

Signed-off-by: John Karasev <[email protected]>
(cherry picked from commit 79ca442)
karasjoh000 added a commit to karasjoh000/ceph-ansible that referenced this issue Jul 13, 2022
@karasjoh000
ceph-grafana: Add proxy env vars to grafana service template
3bfa2c3 
When installing grafana plugins, the container will make http requests.
This requires http proxy otherwise installation cannot be performed. Passed
the proxy vars from all.yml as env args.
Fixes: ceph#6484, ceph#6481

Signed-off-by: John Karasev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
wontfix
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Grafana server proxy karasjoh000/ceph-ansible
1 participant
@karasjoh000