Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Envoy Lua filter causes a crash when code in either envoy_on_request or envoy_on_response calls requestInfo():dynamicMetadata() #4305

Closed
yskopets opened this issue Aug 30, 2018 · 7 comments
Closed

Envoy Lua filter causes a crash when code in either envoy_on_request or envoy_on_response calls requestInfo():dynamicMetadata() #4305

yskopets opened this issue Aug 30, 2018 · 7 comments
Assignees
@dio

Comments

@yskopets
Copy link
Member

Issue

Envoy Lua filter causes a crash when code inside function envoy_on_response(response_handle) calls response_handle:requestInfo():dynamicMetadata()

Description:

I'm trying to use Lua filter to implement redirect to a Login Page whenever HTTP Status 401 happens.

Here is my Envoy config (relevant part):

  http_filters:

  #
  # first, I want to save ':authority' in dynamic metadata for later use
  #
  - name: envoy.filters.http.header_to_metadata
    config:
      request_rules:
        - header: ':authority'
          on_header_present:
            metadata_namespace: my.gateway
            key: ':authority'
            type: STRING
          on_header_missing:
            metadata_namespace: my.gateway
            key: ':authority'
            value: 'unknown'
            type: STRING
          remove: false

  #
  # then, I want to replace HTTP Status 401 with HTTP Status 301
  #
  - name: envoy.lua
    config:
      inline_code: |
        function envoy_on_response(response_handle)
          if response_handle:headers():get(":status") == "401" then
            response_handle:headers():replace(":status", "301")
            response_handle:headers():replace("location", "https://login.example.org/login?redirect=https://" .. response_handle:requestInfo():dynamicMetadata():get("my.gateway")[":authority"])
          end
        end

and here are Envoy logs:

[2018-08-30 13:29:45.066][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x0
[2018-08-30 13:29:45.067][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<36> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-08-30 13:29:45.069][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #0 0x60d8af lua_rawgeti
[2018-08-30 13:29:45.072][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #1 0x61b9f2 luaL_unref
[2018-08-30 13:29:45.074][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #2 0x5ffadf Envoy::Extensions::HttpFilters::Lua::RequestInfoWrapper::~RequestInfoWrapper()
[2018-08-30 13:29:45.076][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #3 0x5ffdb6 Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<>::registerType()::{lambda()#1}::_FUN()
[2018-08-30 13:29:45.079][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #4 0x61f058 lj_BC_FUNCC
[2018-08-30 13:29:45.081][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #5 0x62134b gc_call_finalizer
[2018-08-30 13:29:45.083][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #6 0x621427 gc_finalize
[2018-08-30 13:29:45.086][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #7 0x622146 gc_onestep
[2018-08-30 13:29:45.102][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #8 0x62264b lj_gc_step
[2018-08-30 13:29:45.104][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #9 0x60d0c4 lua_pushstring
[2018-08-30 13:29:45.106][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #10 0x603d58 Envoy::Extensions::HttpFilters::Lua::HeaderMapWrapper::luaGet()
[2018-08-30 13:29:45.106][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #11 0x61f058 lj_BC_FUNCC
[2018-08-30 13:29:45.108][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #12 0x606fa7 Envoy::Extensions::Filters::Common::Lua::Coroutine::resume()
[2018-08-30 13:29:45.110][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #13 0x5fca10 Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper::start()
[2018-08-30 13:29:45.112][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #14 0x6015c7 Envoy::Extensions::HttpFilters::Lua::Filter::doHeaders()
[2018-08-30 13:29:45.114][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #15 0x6016bd Envoy::Extensions::HttpFilters::Lua::Filter::encodeHeaders()
[2018-08-30 13:29:45.116][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #16 0x93d8c9 Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeHeaders()
[2018-08-30 13:29:45.119][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #17 0x9b3c84 Envoy::Http::Utility::sendLocalReply()
[2018-08-30 13:29:45.121][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #18 0x9369c0 Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply()
[2018-08-30 13:29:45.123][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #19 0x936c27 Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter::sendLocalReply()
[2018-08-30 13:29:45.125][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #20 0x47a4cf Envoy::Http::Istio::AuthN::AuthenticationFilter::rejectRequest()
[2018-08-30 13:29:45.127][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #21 0x48259e Envoy::Http::Istio::AuthN::AuthenticationFilter::decodeHeaders()
[2018-08-30 13:29:45.129][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #22 0x93c7c4 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 13:29:45.131][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #23 0x93cf6d Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 13:29:45.133][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #24 0x97b649 Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete()
[2018-08-30 13:29:45.135][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #25 0x97ef78 Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase()
[2018-08-30 13:29:45.137][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #26 0x97efec Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::_FUN()
[2018-08-30 13:29:45.140][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #27 0x98207e http_parser_execute
[2018-08-30 13:29:45.142][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #28 0x97c5b0 Envoy::Http::Http1::ConnectionImpl::dispatchSlice()
[2018-08-30 13:29:45.144][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #29 0x97ea8a Envoy::Http::Http1::ConnectionImpl::dispatch()
[2018-08-30 13:29:45.146][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #30 0x93a59f Envoy::Http::ConnectionManagerImpl::onData()
[2018-08-30 13:29:45.148][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #31 0x7c1d36 Envoy::Network::FilterManagerImpl::onContinueReading()
[2018-08-30 13:29:45.150][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #32 0x7c03be Envoy::Network::ConnectionImpl::onReadReady()
[2018-08-30 13:29:45.152][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #33 0x7c0bad Envoy::Network::ConnectionImpl::onFileEvent()
[2018-08-30 13:29:45.154][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #34 0x7ba007 Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN()
[2018-08-30 13:29:45.156][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #35 0xabd184 event_process_active_single_queue.isra.29
[2018-08-30 13:29:45.158][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #36 0xabd8ce event_process_active
[2018-08-30 13:29:45.160][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #37 0xac0657 event_base_loop
[2018-08-30 13:29:45.163][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #38 0x7b5775 Envoy::Server::WorkerImpl::threadRoutine()
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #39 0xb0de0d Envoy::Thread::Thread::Thread()::{lambda()#1}::_FUN()
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<36> obj</lib/x86_64-linux-gnu/libpthread.so.0>
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #40 0x7fd108dca6b9 start_thread
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<36> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<36> #41 0x7fd1087f741c (unknown)
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 36
2018-08-30T13:29:45.194548Z	warn	Epoch 0 terminated with an error: signal: segmentation fault (core dumped)
2018-08-30T13:29:45.194630Z	warn	Aborted all epochs
2018-08-30T13:29:45.194721Z	info	Epoch 0: set retry delay to 200ms, budget to 9
2018-08-30T13:29:45.395197Z	info	Reconciling configuration (budget 9)
2018-08-30T13:29:45.395313Z	info	Epoch 0 starting
2018-08-30T13:29:45.396136Z	info	Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.34.4.18~istio-ingressgateway-67989599df-8tqm8.istio-ingress-internal~istio-ingress-internal.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only]
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:183] initializing epoch 0 (hot restart version=10.200.16384.256.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=4882536)
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:185] statically linked extensions:
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:187]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:190]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,istio_authn,jwt-auth,mixer
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:193]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:196]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy,mixer
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:198]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:200]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:203]   transport_sockets.downstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:206]   transport_sockets.upstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 13:29:45.416][43][info][config] external/envoy/source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-30 13:29:45.418][43][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, no healthy upstream
[2018-08-30 13:29:45.418][43][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:41] Unable to establish new stream
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:60] loading 0 listener(s)
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:94] loading tracing configuration
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:103]   loading tracing driver: envoy.zipkin
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:116] loading stats sink configuration
[2018-08-30 13:29:45.419][43][info][main] external/envoy/source/server/server.cc:410] starting main dispatch loop
[2018-08-30 13:29:45.454][43][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:129] cm init: initializing cds

Repro steps:

  1. Use example configuration that comes with Envoy (google proxy) and add the following Lua code:
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:headers():add("foo", "bar")
                end
                function envoy_on_response(response_handle)
                  response_handle:requestInfo():dynamicMetadata()
                end
  1. Make a few requests to cause a crash (in my case, Envoy crashes consistently after 5-10 requests)

Config:

Here is the minimal Envoy configuration to reproduce the issue:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite: www.google.com
                  cluster: service_google
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:headers():add("foo", "bar")
                end
                function envoy_on_response(response_handle)
                  response_handle:requestInfo():dynamicMetadata()
                end
          - name: envoy.router
  clusters:
  - name: service_google
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts:
      - socket_address:
          address: google.com
          port_value: 443
    tls_context: { sni: www.google.com }

Call Stack:

[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:191] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:193] statically linked extensions:
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:195]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:198]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:201]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:204]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.rbac,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:206]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:208]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:211]   transport_sockets.downstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 14:45:22.339][8][info][main] source/server/server.cc:214]   transport_sockets.upstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 14:45:22.345][8][info][config] source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-30 14:45:22.345][8][info][config] source/server/configuration_impl.cc:56] loading 1 cluster(s)
[2018-08-30 14:45:22.346][8][info][config] source/server/configuration_impl.cc:61] loading 1 listener(s)
[2018-08-30 14:45:22.348][8][info][config] source/server/configuration_impl.cc:95] loading tracing configuration
[2018-08-30 14:45:22.348][8][info][config] source/server/configuration_impl.cc:117] loading stats sink configuration
[2018-08-30 14:45:22.349][8][info][main] source/server/server.cc:435] starting main dispatch loop
[2018-08-30 14:45:22.356][8][info][upstream] source/common/upstream/cluster_manager_impl.cc:134] cm init: all clusters initialized
[2018-08-30 14:45:22.356][8][info][main] source/server/server.cc:404] all clusters initialized. initializing init manager
[2018-08-30 14:45:22.356][8][info][config] source/server/listener_manager_impl.cc:907] all dependencies initialized. starting workers
[2018-08-30 14:45:36.211][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x18
[2018-08-30 14:45:36.212][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<16> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-08-30 14:45:36.215][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #0 0x51ea66 lua_rawgeti
[2018-08-30 14:45:36.218][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #1 0x52cbc2 luaL_unref
[2018-08-30 14:45:36.221][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #2 0x510c4f Envoy::Extensions::HttpFilters::Lua::RequestInfoWrapper::~RequestInfoWrapper()
[2018-08-30 14:45:36.223][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #3 0x510ce6 Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<>::registerType()::{lambda()#1}::_FUN()
[2018-08-30 14:45:36.225][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #4 0x530228 lj_BC_FUNCC
[2018-08-30 14:45:36.228][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #5 0x53251b gc_call_finalizer
[2018-08-30 14:45:36.231][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #6 0x5325f7 gc_finalize
[2018-08-30 14:45:36.234][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #7 0x533316 gc_onestep
[2018-08-30 14:45:36.237][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #8 0x53381b lj_gc_step
[2018-08-30 14:45:36.240][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #9 0x51e7f8 lua_newuserdata
[2018-08-30 14:45:36.242][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #10 0x511d23 Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper::luaHeaders()
[2018-08-30 14:45:36.242][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #11 0x530228 lj_BC_FUNCC
[2018-08-30 14:45:36.244][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #12 0x518177 Envoy::Extensions::Filters::Common::Lua::Coroutine::resume()
[2018-08-30 14:45:36.246][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #13 0x50db90 Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper::start()
[2018-08-30 14:45:36.248][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #14 0x512737 Envoy::Extensions::HttpFilters::Lua::Filter::doHeaders()
[2018-08-30 14:45:36.250][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #15 0x51289d Envoy::Extensions::HttpFilters::Lua::Filter::decodeHeaders()
[2018-08-30 14:45:36.251][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #16 0x8764a4 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 14:45:36.253][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #17 0x876c65 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 14:45:36.256][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #18 0x8b6b49 Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete()
[2018-08-30 14:45:36.258][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #19 0x8ba53b Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase()
[2018-08-30 14:45:36.260][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #20 0x8ba5ac Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::_FUN()
[2018-08-30 14:45:36.262][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #21 0x8bd67e http_parser_execute
[2018-08-30 14:45:36.265][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #22 0x8b7ab0 Envoy::Http::Http1::ConnectionImpl::dispatchSlice()
[2018-08-30 14:45:36.268][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #23 0x8ba04a Envoy::Http::Http1::ConnectionImpl::dispatch()
[2018-08-30 14:45:36.270][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #24 0x8749ff Envoy::Http::ConnectionManagerImpl::onData()
[2018-08-30 14:45:36.272][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #25 0x6f50a6 Envoy::Network::FilterManagerImpl::onContinueReading()
[2018-08-30 14:45:36.275][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #26 0x6f36be Envoy::Network::ConnectionImpl::onReadReady()
[2018-08-30 14:45:36.276][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #27 0x6f3ead Envoy::Network::ConnectionImpl::onFileEvent()
[2018-08-30 14:45:36.278][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #28 0x6ed057 Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN()
[2018-08-30 14:45:36.280][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #29 0xa02604 event_process_active_single_queue.isra.29
[2018-08-30 14:45:36.282][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #30 0xa02d4e event_process_active
[2018-08-30 14:45:36.285][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #31 0xa05ad7 event_base_loop
[2018-08-30 14:45:36.287][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #32 0x6e8785 Envoy::Server::WorkerImpl::threadRoutine()
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #33 0xa53ead Envoy::Thread::Thread::Thread()::{lambda()#1}::_FUN()
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<16> obj</usr/glibc-compat/lib/libpthread.so.0>
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<16> #34 0x7fd0d9737d2b (unknown)
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<16> obj</usr/glibc-compat/lib/libc.so.6>
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<16> #35 0x7fd0d90e7e3e (unknown)
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 16
@dio
Copy link
Member

dio commented Aug 30, 2018

@yskopets thanks for this.

This happens when the ~RequestInfoWrapper is called (when the gc kicks in). Need to check the sequence.

@dio
Copy link
Member

dio commented Aug 30, 2018 •
edited
Loading

It's missing resetting dynamic_metadata_wrapper_ when RequestInfoWrapper receives onMarkDead. Thinking about tests now.

@dio dio mentioned this issue Aug 30, 2018
Merged
@yskopets
Copy link
Member Author

@dio Thanks a lot for the quick feedback!

I have another question about correctness of my approach.

As you can see, I'm trying to change response entity inside envoy_on_response callback.

Since respond() function is not available for use inside envoy_on_response, I have to fallback to response_handle:headers():replace(":status", "301"), which seems very brittle:

  • nothing in documentation of Lua filter indicates that it could possibly work
  • even though I manage to replace HTTP Status code, I can't remove/change the original response body

So, I'm wondering if I'm using Lua filter the right way.

Do you have any plans to add support for respond() inside envoy_on_response ? Or is it problematic due to the reasons I'm not seeing ?

@yskopets
Copy link
Member Author

yskopets commented Aug 31, 2018 •
edited
Loading

I've notice that the issue is also reproducible when function envoy_on_request(request_handle) calls request_handle:requestInfo():dynamicMetadata()

Repro steps:

  1. Use example configuration that comes with Envoy (google proxy) and add the following Lua code:
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:requestInfo():dynamicMetadata()
                end
          - name: envoy.router
  1. Make a few requests to cause a crash (in my case, Envoy crashes consistently after 5-10 requests)

Config:

Here is the minimal Envoy configuration to reproduce the issue:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite: www.google.com
                  cluster: service_google
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:requestInfo():dynamicMetadata()
                end
          - name: envoy.router
  clusters:
  - name: service_google
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts:
      - socket_address:
          address: google.com
          port_value: 443
    tls_context: { sni: www.google.com }

Call Stack:

[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:202] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:204] statically linked extensions:
[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:206]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:209]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:212]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:215]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.rbac,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:217]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:219]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:222]   transport_sockets.downstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:225]   transport_sockets.upstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-31 15:53:30.032][8][info][config] source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-31 15:53:30.032][8][info][config] source/server/configuration_impl.cc:56] loading 1 cluster(s)
[2018-08-31 15:53:30.034][8][info][config] source/server/configuration_impl.cc:61] loading 1 listener(s)
[2018-08-31 15:53:30.035][8][info][lua] source/extensions/filters/http/lua/lua_filter.cc:449] envoy_on_response() function not found. Lua filter will not hook responses.
[2018-08-31 15:53:30.035][8][info][config] source/server/configuration_impl.cc:95] loading tracing configuration
[2018-08-31 15:53:30.035][8][info][config] source/server/configuration_impl.cc:117] loading stats sink configuration
[2018-08-31 15:53:30.036][8][info][main] source/server/server.cc:446] starting main dispatch loop
[2018-08-31 15:53:30.043][8][info][upstream] source/common/upstream/cluster_manager_impl.cc:134] cm init: all clusters initialized
[2018-08-31 15:53:30.043][8][info][main] source/server/server.cc:415] all clusters initialized. initializing init manager
[2018-08-31 15:53:30.043][8][info][config] source/server/listener_manager_impl.cc:907] all dependencies initialized. starting workers
[2018-08-31 15:53:36.920][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x98
[2018-08-31 15:53:36.920][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<17> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-08-31 15:53:36.923][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #0 0x51eaa4 lua_rawgeti
[2018-08-31 15:53:36.925][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #1 0x52cc02 luaL_unref
[2018-08-31 15:53:36.928][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #2 0x510c8f Envoy::Extensions::HttpFilters::Lua::RequestInfoWrapper::~RequestInfoWrapper()
[2018-08-31 15:53:36.930][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #3 0x510d26 Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<>::registerType()::{lambda()#1}::_FUN()
[2018-08-31 15:53:36.932][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #4 0x530268 lj_BC_FUNCC
[2018-08-31 15:53:36.934][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #5 0x53255b gc_call_finalizer
[2018-08-31 15:53:36.936][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #6 0x532637 gc_finalize
[2018-08-31 15:53:36.938][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #7 0x533356 gc_onestep
[2018-08-31 15:53:36.942][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #8 0x53385b lj_gc_step
[2018-08-31 15:53:36.944][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #9 0x51e838 lua_newuserdata
[2018-08-31 15:53:36.948][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #10 0x5125b6 Envoy::Extensions::HttpFilters::Lua::Filter::doHeaders()
[2018-08-31 15:53:36.950][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #11 0x5128dd Envoy::Extensions::HttpFilters::Lua::Filter::decodeHeaders()
[2018-08-31 15:53:36.952][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #12 0x879de4 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-31 15:53:36.954][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #13 0x87a5a5 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-31 15:53:36.956][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #14 0x8ba489 Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete()
[2018-08-31 15:53:36.959][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #15 0x8bde7b Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase()
[2018-08-31 15:53:36.962][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #16 0x8bdeec Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::_FUN()
[2018-08-31 15:53:36.965][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #17 0x8c03ee http_parser_execute
[2018-08-31 15:53:36.967][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #18 0x8bb3f0 Envoy::Http::Http1::ConnectionImpl::dispatchSlice()
[2018-08-31 15:53:36.970][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #19 0x8bd98a Envoy::Http::Http1::ConnectionImpl::dispatch()
[2018-08-31 15:53:36.972][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #20 0x87833f Envoy::Http::ConnectionManagerImpl::onData()
[2018-08-31 15:53:36.975][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #21 0x6f89e6 Envoy::Network::FilterManagerImpl::onContinueReading()
[2018-08-31 15:53:36.978][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #22 0x6f6ffe Envoy::Network::ConnectionImpl::onReadReady()
[2018-08-31 15:53:36.982][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #23 0x6f77ed Envoy::Network::ConnectionImpl::onFileEvent()
[2018-08-31 15:53:36.984][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #24 0x6f0997 Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN()
[2018-08-31 15:53:36.987][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #25 0xa05f44 event_process_active_single_queue.isra.29
[2018-08-31 15:53:36.990][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #26 0xa0668e event_process_active
[2018-08-31 15:53:36.993][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #27 0xa09417 event_base_loop
[2018-08-31 15:53:36.996][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #28 0x6ec0c5 Envoy::Server::WorkerImpl::threadRoutine()
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #29 0xa577ed Envoy::Thread::Thread::Thread()::{lambda()#1}::_FUN()
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<17> obj</usr/glibc-compat/lib/libpthread.so.0>
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<17> #30 0x7f9304102d2b (unknown)
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<17> obj</usr/glibc-compat/lib/libc.so.6>
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<17> #31 0x7f9303ab2e3e (unknown)
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 17

@yskopets yskopets changed the title Envoy Lua filter causes a crash when code inside function envoy_on_response(response_handle) calls response_handle:requestInfo():dynamicMetadata() Envoy Lua filter causes a crash when code in either envoy_on_request or envoy_on_response calls requestInfo():dynamicMetadata() Aug 31, 2018
@dio
Copy link
Member

dio commented Aug 31, 2018 •
edited
Loading

@yskopets yes, thanks to you that I can reproduce the issue well. Would you mind building envoy with this patch: #4312 and see if that fixes yours? (If you have a problem on building envoy from the source you can ping me via slack (@dio at https://envoyslack.cncf.io/) or email). Thanks!

@yskopets
Copy link
Member Author

yskopets commented Sep 1, 2018

@dio Thanks a lot!

I confirm that your changes fixed my problem.

@dnoe dnoe closed this as completed in #4312 Sep 4, 2018
dnoe pushed a commit that referenced this issue Sep 4, 2018
@dio @dnoe
lua: make sure resetting dynamic metadata wrapper when request info i…
1485a13 
…s marked dead

This PR makes sure we reset dynamic metadata wrapper when its parent (request info wrapper) is marked dead.

Risk Level: Low
Testing: Add integration testing
Docs Changes: N/A
Release Notes: N/A
Fixes #4305

Signed-off-by: Dhi Aurrahman <[email protected]>
@yskopets yskopets mentioned this issue Sep 11, 2018
Closed
@ceastman-ibm
Copy link

@yskopets after this patch was your code for redirecting on status code 401 working?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dio dio

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dio @yskopets @ceastman-ibm