grafana cannot starting

[wuyifan18]

Bug Report

Which version of the demo you are using?
https://github.com/open-telemetry/opentelemetry-demo/releases/tag/1.7.0

Symptom

I use docker to deploy the demo. I run the commend '''docker compose up --force-recreate --remove-orphans --detach''', the demo can visit normally, but grafana exited and the URL http://localhost:8080/grafana/ can not visited.
Here is the log from grafana service:

grafana | logger=settings t=2023-12-20T11:59:00.727654279Z level=info msg="Starting Grafana" version=10.2.2 commit=161e3cac5075540918e3a39004f2364ad104d5bb branch=HEAD compiled=2023-11-20T12:28:00Z
grafana | logger=settings t=2023-12-20T11:59:00.727965332Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
grafana | logger=settings t=2023-12-20T11:59:00.727982464Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
grafana | logger=settings t=2023-12-20T11:59:00.727990941Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
grafana | logger=settings t=2023-12-20T11:59:00.727997676Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
grafana | logger=settings t=2023-12-20T11:59:00.728003397Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
grafana | logger=settings t=2023-12-20T11:59:00.728009914Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
grafana | logger=settings t=2023-12-20T11:59:00.728016432Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
grafana | logger=settings t=2023-12-20T11:59:00.728023596Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
grafana | logger=settings t=2023-12-20T11:59:00.728030571Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
grafana | logger=settings t=2023-12-20T11:59:00.728037768Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
grafana | logger=settings t=2023-12-20T11:59:00.728045365Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
grafana | logger=settings t=2023-12-20T11:59:00.728060819Z level=info msg=Target target=[all]
grafana | logger=settings t=2023-12-20T11:59:00.72807176Z level=info msg="Path Home" path=/usr/share/grafana
grafana | logger=settings t=2023-12-20T11:59:00.728078283Z level=info msg="Path Data" path=/var/lib/grafana
grafana | logger=settings t=2023-12-20T11:59:00.728086171Z level=info msg="Path Logs" path=/var/log/grafana
grafana | logger=settings t=2023-12-20T11:59:00.728092605Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
grafana | logger=settings t=2023-12-20T11:59:00.728098396Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
grafana | logger=settings t=2023-12-20T11:59:00.728104569Z level=info msg="App mode production"
grafana | logger=sqlstore t=2023-12-20T11:59:00.72850299Z level=info msg="Connecting to DB" dbtype=sqlite3
grafana | logger=migrator t=2023-12-20T11:59:00.729229551Z level=info msg="Starting DB migrations"
grafana | logger=migrator t=2023-12-20T11:59:00.778669727Z level=info msg="migrations completed" performed=0 skipped=517 duration=810.84µs
grafana | logger=secrets t=2023-12-20T11:59:00.779720803Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
grafana | logger=local.finder t=2023-12-20T11:59:00.871443309Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
grafana | logger=plugins.registration t=2023-12-20T11:59:01.023023657Z level=info msg="Plugin registered" pluginId=grafana-opensearch-datasource
grafana | logger=query_data t=2023-12-20T11:59:01.057022024Z level=info msg="Query Service initialization"
grafana | logger=live.push_http t=2023-12-20T11:59:01.062268371Z level=info msg="Live Push Gateway initialization"
grafana | logger=plugin.grafana-opensearch-datasource t=2023-12-20T11:59:01.742126548Z level=error msg="plugin process exited" path=/var/lib/grafana/plugins/grafana-opensearch-datasource/gpx_opensearch-datasource_linux_amd64 pid=23 error="signal: killed"
grafana | logger=infra.usagestats.collector t=2023-12-20T11:59:03.571806694Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
grafana | logger=provisioning.plugins t=2023-12-20T11:59:03.626287803Z level=error msg="Failed to read plugin provisioning files from directory" path=/etc/grafana/provisioning/plugins error="open /etc/grafana/provisioning/plugins: no such file or directory"
grafana | logger=provisioning.notifiers t=2023-12-20T11:59:03.626325647Z level=error msg="Can't read alert notification provisioning files from directory" path=/etc/grafana/provisioning/notifiers error="open /etc/grafana/provisioning/notifiers: no such file or directory"
grafana | logger=provisioning.alerting t=2023-12-20T11:59:03.626358044Z level=error msg="can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error="open /etc/grafana/provisioning/alerting: no such file or directory"
grafana | logger=provisioning.alerting t=2023-12-20T11:59:03.626367114Z level=info msg="starting to provision alerting"
grafana | logger=provisioning.alerting t=2023-12-20T11:59:03.626387208Z level=info msg="finished to provision alerting"
grafana | logger=grafanaStorageLogger t=2023-12-20T11:59:03.626770002Z level=info msg="Storage starting"
grafana | logger=http.server t=2023-12-20T11:59:03.629580643Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl=/grafana socket=
grafana | logger=ngalert.state.manager t=2023-12-20T11:59:03.636800951Z level=info msg="Warming state cache for startup"
grafana | logger=ngalert.state.manager t=2023-12-20T11:59:03.637495755Z level=info msg="State cache has been initialized" states=0 duration=693.067µs
grafana | logger=ngalert.scheduler t=2023-12-20T11:59:03.637525058Z level=info msg="Starting scheduler" tickInterval=10s
grafana | logger=ticker t=2023-12-20T11:59:03.637576086Z level=info msg=starting first_tick=2023-12-20T11:59:10Z
grafana | logger=ngalert.multiorg.alertmanager t=2023-12-20T11:59:03.645820881Z level=info msg="Starting MultiOrg Alertmanager"
grafana | logger=ngalert.migration t=2023-12-20T11:59:03.722720951Z level=info msg=Starting
grafana | logger=ngalert.migration t=2023-12-20T11:59:03.723141205Z level=info msg="No migrations to run"
grafana | logger=plugins.update.checker t=2023-12-20T11:59:04.454265581Z level=info msg="Update check succeeded" duration=809.87019ms
grafana | logger=grafana.update.checker t=2023-12-20T11:59:06.663074141Z level=info msg="Update check succeeded" duration=3.019257191s
grafana | logger=infra.usagestats t=2023-12-20T12:00:14.661350292Z level=info msg="Usage stats are ready to report"
grafana | logger=plugin.grafana-opensearch-datasource t=2023-12-20T12:00:53.738812021Z level=error msg="plugin process exited" path=/var/lib/grafana/plugins/grafana-opensearch-datasource/gpx_opensearch-datasource_linux_amd64 pid=30 error="signal: killed"
grafana | logger=plugin.grafana-opensearch-datasource t=2023-12-20T12:00:54.691272928Z level=error msg="plugin process exited" path=/var/lib/grafana/plugins/grafana-opensearch-datasource/gpx_opensearch-datasource_linux_amd64 pid=41 error="signal: killed"
grafana | logger=plugin.grafana-opensearch-datasource t=2023-12-20T12:00:54.691519419Z level=error msg="Failed to restart plugin" error="Unrecognized remote plugin message: \nThis usually means\n the plugin was not compiled for this architecture,\n the plugin is missing dynamic-link libraries necessary to run,\n the plugin is not executable by this process due to file permissions, or\n the plugin failed to negotiate the initial go-plugin protocol handshake\n\nAdditional notes about plugin:\n Path: /var/lib/grafana/plugins/grafana-opensearch-datasource/gpx_opensearch-datasource_linux_amd64\n Mode: -rwxr-xr-x\n Owner: 472 [grafana] (current: 472 [grafana])\n Group: 0 [root] (current: 0 [root])\n ELF architecture: EM_X86_64 (current architecture: amd64)\n"
grafana | logger=plugin.grafana-opensearch-datasource t=2023-12-20T12:00:55.732166508Z level=error msg="Failed to restart plugin" error="Unrecognized remote plugin message: \nThis usually means\n the plugin was not compiled for this architecture,\n the plugin is missing dynamic-link libraries necessary to run,\n the plugin is not executable by this process due to file permissions, or\n the plugin failed to negotiate the initial go-plugin protocol handshake\n\nAdditional notes about plugin:\n Path: /var/lib/grafana/plugins/grafana-opensearch-datasource/gpx_opensearch-datasource_linux_amd64\n Mode: -rwxr-xr-x\n Owner: 472 [grafana] (current: 472 [grafana])\n Group: 0 [root] (current: 0 [root])\n ELF architecture: EM_X86_64 (current architecture: amd64)\n"

[puckpuck]

Looks like #1301 broke this. I'm going to create a PR to get it fixed. I think we will need to do a 1.7.1 release

/cc @austinlparker