Envoy with Postgres filter, envoy is crashing #35908
Comments
|
Is there any update on this? |
|
Did it crash again? |
|
Yes, it is crashing everyday.
Thanks
Rohit Kanchan
…On Thu, Sep 5, 2024 at 4:16 PM Christoph Pakulski ***@***.***> wrote:
Did it crash again?
—
Reply to this email directly, view it on GitHub
<#35908 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4XMFPVMGUJ7NWJBJ3QRODZVDQ3JAVCNFSM6AAAAABNK4TR6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZSHA2TSNRXGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
@rohitkanchan Can you try to narrow it to a specific SQL query which causes the crash? |
|
Hi Chris,We started seeing crash without even running any sql, server is up on virtual machine for 4-5 hours, without any new connection or sql query, it just crashes.-RohitSent from my iPhoneOn Sep 9, 2024, at 5:32 PM, Christoph Pakulski ***@***.***> wrote:
@rohitkanchan Can you try to narrow it to a specific SQL query which causes the crash?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@rohitkanchan I suspect that non-postgres traffic is received by postgres filter. I added some protection code to validate if a packet is legitimate postgres request, but maybe it is not sufficient. The security posture of this filter is for "trusted downstream" only, so in general you should not expose it to wider audience. |
|
Sure, I can try what you are asking but there is no traffic on that
instance. No one is calling that server.
Thanks
Rohit Kanchan
…On Tue, Sep 10, 2024 at 7:45 AM Christoph Pakulski ***@***.***> wrote:
@rohitkanchan <https://github.com/rohitkanchan> I suspect that
non-postgres traffic is received by postgres filter. I added some
protection code to validate if a packet is legitimate postgres request, but
maybe it is not sufficient. The security posture of this filter is for
"trusted downstream" only, so in general you should not expose it to wider
audience.
There are several methods we could use to check if non-postgres traffic is
received. Since you claim that it crashes without any queries, can you
remove postgres filter and check stats of backend_cluster? I assume that
backend_cluster is used only by postgres filter chain. You should see
that there is no traffic received by backend_cluster. If it is and you do
not generate any SQL queries, it means that some other app sends something
to port 5432. WDYT?
—
Reply to this email directly, view it on GitHub
<#35908 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4XMFNIREGT7PRRKTEAS7TZV4AZPAVCNFSM6AAAAABNK4TR6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBRGA3TSMRRHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
|
No, it is not working, issue is stale but it is still crashing.
Thanks
Rohit Kanchan
…On Thu, Oct 10, 2024 at 5:04 PM github-actions[bot] < ***@***.***> wrote:
This issue has been automatically marked as stale because it has not had
activity in the last 30 days. It will be closed in the next 7 days unless
it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank
you for your contributions.
—
Reply to this email directly, view it on GitHub
<#35908 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4XMFP2ITO6UIVUHQ4FZ6DZ24IXVAVCNFSM6AAAAABNK4TR6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBWGI2TINBSGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
github-actions
bot
removed
the
stale
|
@rohitkanchan Can you check if it generates the same backtrace each time it crashes? |
|
It is the same stack trace each time.
Thanks
Rohit Kanchan
…On Tue, Oct 15, 2024 at 4:19 PM Christoph Pakulski ***@***.***> wrote:
@rohitkanchan <https://github.com/rohitkanchan> Can you check if it
generates the same backtrace each time it crashes?
—
Reply to this email directly, view it on GitHub
<#35908 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4XMFPZ4UFGZW4ESCA3NC3Z3WPJZAVCNFSM6AAAAABNK4TR6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJVGMZDSMBRGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
OK. Let us try one more thing. Can you modify config and change the listener's port from 5432 to something different, like 12345 for example? Instead of change it to
Thanks. |
|
Add Sibin from my team to take over the hit and trial which @chris is
asking.
@sibin,
Can you please change what Chris is asking and see if envoy is still
crashing.
Thanks
Rohit Kanchan
…On Wed, Oct 16, 2024 at 1:26 PM Christoph Pakulski ***@***.***> wrote:
OK. Let us try one more thing. Can you modify config and change the
listener's port from 5432 to something different, like 12345 for example?
Instead of
port_value: 5432 # Frontend port
change it to
port_value: 12345 # Frontend port
Thanks.
—
Reply to this email directly, view it on GitHub
<#35908 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4XMFO3XELCPXF3PQCKDD3Z33DWRAVCNFSM6AAAAABNK4TR6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXHA4DENRSG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
[email protected] where the issue will be triaged appropriately.
Title: Envoy with Postgres is crashing
Description:
Repro steps:
Admin and Stats Output:
Config:
Config:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 5432 # Frontend port
filter_chains:
- filters:
- name: envoy.filters.network.postgres_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.postgres_proxy.v3alpha.PostgresProxy
stat_prefix: imperva
terminate_ssl: true
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: tcp
cluster: backend_cluster
transport_socket:
name: envoy.transport_sockets.starttls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.starttls.v3.StartTlsConfig
tls_socket_config:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/fullchain.pem"
private_key:
filename: "/etc/envoy/privkey.pem"
clusters:
- name: backend_cluster
connect_timeout: 0.25s
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: backend_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address:
port_value: 5432
Logs:
Call Stack:
I added logs already in logs section.
The text was updated successfully, but these errors were encountered: