Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ips handling/v24 #7359

Closed
wants to merge 13 commits into from
Closed

Conversation

victorjulien
Copy link
Member

Implements a new way of dealing with memcap hits: a per subsystem memcap-policy which can be drop-packet, drop-flow, pass-packet, pass-flow, bypass or ignore. Where ignore is essentially the current behavior. Looking for a better name and general feedback on the approach.

In OISF/suricata-verify#818 there are tests for most options in combination with stream reassembly memcap failure simulation.

continues work in #7234
(includes #7356)

suricata-verify-pr: 818

Set correct direction for PORT mode, where the server connects
to the client.

The direction is not also strictly enforced. No data in the wrong
direction will be accepted to setup the file or to be added to the
file after setup.

This also fixes files getting closed twice.

Adds some general cleanups.

Bug: OISF#3542.
Now that spurious retransmissions don't propegate into the reassembly
code, error handling can be simplified.
If stream session or reassembly memcaps are hit call the memcap
policy on the packet and flow.
Apply policy when memcap is reached and no flow could be freed up.
@codecov
Copy link

codecov bot commented May 2, 2022

Codecov Report

Merging #7359 (2adbc1f) into master (323fe1c) will decrease coverage by 0.03%.
The diff coverage is 62.73%.

@@            Coverage Diff             @@
##           master    #7359      +/-   ##
==========================================
- Coverage   75.84%   75.80%   -0.04%     
==========================================
  Files         656      657       +1     
  Lines      190093   190235     +142     
==========================================
+ Hits       144172   144213      +41     
- Misses      45921    46022     +101     
Flag Coverage Δ
fuzzcorpus 60.36% <35.05%> (-0.12%) ⬇️
suricata-verify 51.87% <63.27%> (+0.12%) ⬆️
unittests 60.98% <20.67%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@victorjulien
Copy link
Member Author

On further thought I think the MemcapPolicy thing could even be a more general ExceptionPolicy, where possible exceptions include memcap hits, other resource limit hits, app errors and probably more

@@ -899,7 +918,7 @@ static inline void PacketSetAction(Packet *p, const uint8_t a)

#define PACKET_ACCEPT(p) PACKET_SET_ACTION(p, ACTION_ACCEPT)

#define PACKET_DROP(p) PACKET_SET_ACTION(p, ACTION_DROP)
//#define PACKET_DROP(p) PACKET_SET_ACTION(p, ACTION_DROP)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove if not used as this is a non-WIP commit.

Comment on lines +75 to +76
} else if ((strcmp(value_str, "bypass") == 0) ||
(strcmp(value_str, "bypass-flow") == 0)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both names required? Or can we just use one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can do one. We use bypass in various places in config, but here I used flow-(packet|flow) pattern, so it seemed to make sense to have both.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Being a new configuration option, I'd rather not see an alias right from the start.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

went with just bypass

Comment on lines +466 to +467
stream_config.ssn_memcap_policy = MemcapPolicyParse("stream.memcap-policy");
stream_config.reassembly_memcap_policy = MemcapPolicyParse("stream.reassembly.memcap-policy");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or should policies be in their own configuration file section:

policy:
  stream-memcap: ...
  stream-reassembly-memcap: ...

@jasonish
Copy link
Member

jasonish commented May 2, 2022

On further thought I think the MemcapPolicy thing could even be a more general ExceptionPolicy, where possible exceptions include memcap hits, other resource limit hits, app errors and probably more

Yes, I agree. I also think it would be more user friendly if this exception policy was configured in one place, rather than having it sprinkled among the various modules.

@victorjulien
Copy link
Member Author

On further thought I think the MemcapPolicy thing could even be a more general ExceptionPolicy, where possible exceptions include memcap hits, other resource limit hits, app errors and probably more

Yes, I agree. I also think it would be more user friendly if this exception policy was configured in one place, rather than having it sprinkled among the various modules.

Tough one. I feel like it makes sense to have them close to the memcap settings, but I also see the point of grouping them. Ultimately I feel most ppl would never touch these. They would get the implied values (ignore for IDS, drop-flow for IPS). The actual settings would be available for expert tuning.

@suricata-qa
Copy link

ERROR:

ERROR: QA failed on tlpw1_files_sha256.

field test baseline %
tlpr1_stats_chk
.app_layer.error.ftp-data.parser 0 395 0.0%

Pipeline 7353

@victorjulien victorjulien mentioned this pull request May 9, 2022
@victorjulien
Copy link
Member Author

continued in #7393

@victorjulien victorjulien deleted the ips-handling/v24 branch May 8, 2023 18:55
jasonish added a commit to jasonish/suricata that referenced this pull request Oct 30, 2024
Only call ThreadInit and ThreadDeinit for custom eve filetypes if they
exist. They are not required by all filetypes.

Ticket: OISF#7359
victorjulien pushed a commit to victorjulien/suricata that referenced this pull request Nov 1, 2024
Only call ThreadInit and ThreadDeinit for custom eve filetypes if they
exist. They are not required by all filetypes.

Ticket: OISF#7359
AkakiAlice pushed a commit to AkakiAlice/suricata that referenced this pull request Nov 7, 2024
Only call ThreadInit and ThreadDeinit for custom eve filetypes if they
exist. They are not required by all filetypes.

Ticket: OISF#7359
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants