Nginx 1.22.0 doesn't start with OpenTelemetry enabled #199

jmadureira · 2022-07-28T09:23:02Z

Describe your environment
The issue was detected when using:

nginx:1.22.0-alpine
opentelemetry-cpp v.1.2.0
latest version of the nginx instrumentation

Steps to reproduce
It is possible to reproduce using the following Dockerfile and nginx.conf sample:

FROM nginx:1.22.0-alpine as builder

ENV OPENTELEMETRY_VERSION v1.2.0

RUN apk update \
  && apk add --update \
      alpine-sdk build-base cmake linux-headers libressl-dev pcre-dev zlib-dev \
      grpc grpc-dev curl-dev protobuf-dev c-ares-dev \
      re2-dev abseil-cpp

RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${OPENTELEMETRY_VERSION} \
  https://github.com/open-telemetry/opentelemetry-cpp.git \
  && cd opentelemetry-cpp \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_PREFIX_PATH=/install \
    -DWITH_OTLP=ON \
    -DWITH_OTLP_GRPC=ON \
    -DWITH_OTLP_HTTP=OFF \
    -DBUILD_TESTING=OFF \
    -DWITH_EXAMPLES=OFF \
    -DWITH_ABSEIL=ON \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    .. \
  && make -j2 \
  && make install

RUN git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
  && cd opentelemetry-cpp-contrib/instrumentation/nginx \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DNGINX_BIN=/usr/sbin/nginx \
    -DCMAKE_PREFIX_PATH=/install \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/nginx/modules \
    -DCURL_LIBRARY=/usr/lib/libcurl.so.4 \
    .. \
  && make -j2 \
  && make install

FROM nginx:1.22.0-alpine

RUN apk update \
  && apk add --update \
      grpc protobuf c-ares
      # gdb # uncomment this if you need to debug nginx

COPY --from=builder /usr/lib/nginx/modules/otel_ngx_module.so /usr/lib/nginx/modules/

COPY nginx/conf/nginx.conf /etc/nginx/nginx.conf

# CMD ["/usr/sbin/nginx-debug", "-g", "daemon off;"]

worker_processes  auto;
error_log  /dev/stderr;
pid        /tmp/nginx.pid;
worker_rlimit_nofile 8192;
worker_rlimit_core  500M;
working_directory /tmp;
debug_points abort;
load_module /usr/lib/nginx/modules/otel_ngx_module.so;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  server {
    listen 8080;

    location /status {
      # opentelemetry off;
      stub_status on;
      access_log  off;
      allow 127.0.0.1;
      deny all;
    }

    location / {
      root   html;
      index  index.html index.htm;
    }
  }
}

What is the expected behavior?
Nginx should start normally

What is the actual behavior?
Nginx logs the following messages and fails:

/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/07/28 09:16:02 [emerg] 1#1: dlopen() "/usr/lib/nginx/modules/otel_ngx_module.so" failed (Error relocating /usr/lib/nginx/modules/otel_ngx_module.so: _ZN4absl12lts_2021110216variant_internal21ThrowBadVariantAccessEv: symbol not found) in /etc/nginx/nginx.conf:8
nginx: [emerg] dlopen() "/usr/lib/nginx/modules/otel_ngx_module.so" failed (Error relocating /usr/lib/nginx/modules/otel_ngx_module.so: _ZN4absl12lts_2021110216variant_internal21ThrowBadVariantAccessEv: symbol not found) in /etc/nginx/nginx.conf:8

Additional context

The text was updated successfully, but these errors were encountered:

jmadureira · 2022-09-02T14:08:41Z

Seems to be addressed in PR #205.

DebajitDas · 2022-09-08T09:07:53Z

@jmadureira #205 takes care of "instrumentation/otel-webserver-module" directory which also has opentelemetry module for nginx.
This issue is referred to "intrumentation/nginx" directory, #205 does not take care of that.

rcjsuen · 2022-09-30T10:08:42Z

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.3.0
  Build:         2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

We have the same problem with k8s.gcr.io/ingress-nginx/controller:v1.3.0 which looks to be on 1.19.10 so I don't think this problem is unique to Nginx 1.22.0.

esigo · 2022-09-30T10:15:08Z

@rcjsuen, @jmadureira you may check my PR on nginx repo kubernetes/ingress-nginx#9062.

rcjsuen · 2022-09-30T10:47:39Z

@esigo Yeah, I also came across kubernetes/ingress-nginx#8585 and kubernetes/ingress-nginx#9017 while trying to figure out this symbol not found error.

At this rate it might be better to just wait for the official release of OpenTelemetry support in the Nginx controller in my case.

rcjsuen · 2022-09-30T17:56:37Z

Compiling gRPC manually seems like a workaround...? 🤔

FROM nginx:1.22.0-alpine as builder

RUN apk update \
  && apk add --update \
      alpine-sdk build-base cmake linux-headers libressl-dev pcre-dev zlib-dev \
      curl-dev protobuf-dev c-ares-dev \
      re2-dev abseil-cpp

ENV GRPC_VERSION v1.43.2
RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${GRPC_VERSION} \
  https://github.com/grpc/grpc \
  && cd grpc \
  && mkdir -p cmake/build \
  && cd cmake/build \
  && cmake \
    -DgRPC_INSTALL=ON \
    -DgRPC_BUILD_TESTS=OFF \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_BUILD_TYPE=Release \
    -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF \
    ../.. \
  && make -j2 \
    && make install

ENV OPENTELEMETRY_VERSION v1.3.0
RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${OPENTELEMETRY_VERSION} \
  https://github.com/open-telemetry/opentelemetry-cpp.git \
  && cd opentelemetry-cpp \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_PREFIX_PATH=/install \
    -DWITH_ZIPKIN=OFF \
    -DWITH_JAEGER=OFF \
    -DWITH_OTLP=ON \
    -DWITH_OTLP_GRPC=ON \
    -DWITH_OTLP_HTTP=OFF \
    -DBUILD_TESTING=OFF \
    -DWITH_EXAMPLES=OFF \
    -DWITH_ABSEIL=ON \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    .. \
  && make -j2 \
  && make install

RUN git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
  && cd opentelemetry-cpp-contrib/instrumentation/nginx \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DNGINX_BIN=/usr/sbin/nginx \
    -DCMAKE_PREFIX_PATH=/install \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/nginx/modules \
    -DCURL_LIBRARY=/usr/lib/libcurl.so.4 \
    .. \
  && make -j2 \
  && make install

FROM nginx:1.22.0-alpine

RUN apk update \
  && apk add --update \
      grpc protobuf c-ares
      # gdb # uncomment this if you need to debug nginx

COPY --from=builder /usr/lib/nginx/modules/otel_ngx_module.so /usr/lib/nginx/modules/

COPY nginx.conf /etc/nginx/nginx.conf

# CMD ["/usr/sbin/nginx-debug", "-g", "daemon off;"]

esigo · 2022-09-30T18:04:18Z

The symbols were not found for k8s-nginx mainly because the shared libraries were not copied over to the controller container from the init container:
https://github.com/kubernetes/ingress-nginx/blob/50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea/images/opentelemetry/rootfs/Dockerfile#L38-L45

rcjsuen · 2022-09-30T18:14:11Z

Yeah, I went down the path of manual compiling gRPC because I saw that in the ingress controller project.

jmadureira · 2022-10-03T08:37:48Z

Also tested with manual gRPC compilation and including the shared libraries and it works as expected:

FROM nginx:1.22.0-alpine as builder

ENV OPENTELEMETRY_VERSION v1.2.0

RUN apk update \
  && apk add --update \
      alpine-sdk build-base cmake linux-headers libressl-dev pcre-dev zlib-dev \
      curl-dev protobuf-dev c-ares-dev \
      re2-dev abseil-cpp

ENV GRPC_VERSION v1.43.2
RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${GRPC_VERSION} \
  https://github.com/grpc/grpc \
  && cd grpc \
  && mkdir -p cmake/build \
  && cd cmake/build \
  && cmake \
    -DgRPC_INSTALL=ON \
    -DgRPC_BUILD_TESTS=OFF \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_BUILD_TYPE=Release \
    -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF \
    ../.. \
  && make -j2 \
    && make install

RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${OPENTELEMETRY_VERSION} \
  https://github.com/open-telemetry/opentelemetry-cpp.git \
  && cd opentelemetry-cpp \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_PREFIX_PATH=/install \
    -DWITH_OTLP=ON \
    -DWITH_OTLP_GRPC=ON \
    -DWITH_OTLP_HTTP=OFF \
    -DBUILD_TESTING=OFF \
    -DWITH_EXAMPLES=OFF \
    -DWITH_ABSEIL=ON \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    .. \
  && make -j2 \
  && make install

RUN git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
  && cd opentelemetry-cpp-contrib/instrumentation/nginx \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DNGINX_BIN=/usr/sbin/nginx \
    -DCMAKE_PREFIX_PATH=/install \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/nginx/modules \
    -DCURL_LIBRARY=/usr/lib/libcurl.so.4 \
    .. \
  && make -j2 \
  && make install

FROM nginx:1.22.0-alpine

RUN apk update \
  && apk add --update \
      grpc protobuf c-ares
      # gdb # uncomment this if you need to debug nginx

COPY --from=builder /usr/lib/nginx/modules/otel_ngx_module.so /usr/lib/nginx/modules/
COPY --from=builder install/lib/lib* /etc/nginx/modules/

COPY nginx/conf/nginx.conf /etc/nginx/nginx.conf

# CMD ["/usr/sbin/nginx-debug", "-g", "daemon off;"]

It would be nice if the > 15 minutes of gRPC compilation could be avoided.

rcjsuen · 2022-10-03T10:55:52Z

Also tested with manual gRPC compilation and including the shared libraries and it works as expected

Thanks for verifying, @jmadureira!

It would be nice if the > 15 minutes of gRPC compilation could be avoided.

Agreed. This step burns through a lot of compute resources in the CI pipeline. Not sure if this is an issue with Alpine itself or what. There may also be additional cmake flags that could be configured to reduce the compilation time some more... 🤔

esigo · 2022-11-10T19:19:27Z

Hi,
here is the solution with static link:
kubernetes/ingress-nginx#9286

jmadureira added the bug Something isn't working label Jul 28, 2022

esigo mentioned this issue Oct 18, 2022

nginx instrumentation build cannot be loaded with undefined symbol errors #223

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nginx 1.22.0 doesn't start with OpenTelemetry enabled #199

Nginx 1.22.0 doesn't start with OpenTelemetry enabled #199

jmadureira commented Jul 28, 2022

jmadureira commented Sep 2, 2022

DebajitDas commented Sep 8, 2022

rcjsuen commented Sep 30, 2022

esigo commented Sep 30, 2022 •

edited

Loading

rcjsuen commented Sep 30, 2022

rcjsuen commented Sep 30, 2022

esigo commented Sep 30, 2022

rcjsuen commented Sep 30, 2022

jmadureira commented Oct 3, 2022 •

edited

Loading

rcjsuen commented Oct 3, 2022

esigo commented Nov 10, 2022

Nginx 1.22.0 doesn't start with OpenTelemetry enabled #199

Nginx 1.22.0 doesn't start with OpenTelemetry enabled #199

Comments

jmadureira commented Jul 28, 2022

jmadureira commented Sep 2, 2022

DebajitDas commented Sep 8, 2022

rcjsuen commented Sep 30, 2022

esigo commented Sep 30, 2022 • edited Loading

rcjsuen commented Sep 30, 2022

rcjsuen commented Sep 30, 2022

esigo commented Sep 30, 2022

rcjsuen commented Sep 30, 2022

jmadureira commented Oct 3, 2022 • edited Loading

rcjsuen commented Oct 3, 2022

esigo commented Nov 10, 2022

esigo commented Sep 30, 2022 •

edited

Loading

jmadureira commented Oct 3, 2022 •

edited

Loading