Flatters WAF config API for better flexibility in setting memory limits and enabling BodyAccess #503

M4tteoP · 2022-11-09T14:31:56Z

Following #499, I'm working on avoiding body-related actions when the bodies themself are not accessible (see corazawaf/coraza-proxy-wasm#76). I'm facing a problem with WithRequestBodyAccess because it always enforces the body access, not taking into account any related user directive.
This PR proposes to remove the enforcement. WithRequestBodyAccess and WithResponseBodyAccess would become a way to set memory limits, demanding to the traditional directives (SecRequestBodyAccess and SecResponseBodyAccess) the decision of enabling or not the body access.

I see value in slitting the enforcement from the configuration also considering actions like ctl:requestBodyAccess: for specific transactions the body could become accessible and would be okay to already have customized configurations in place if it happens.

Thanks for the feedback and alternative ideas/proposals!

P.S: WithRequestBodyAccess and WithResponseBodyAccess names would become not completely consistent, should I change them into something like WithRequestBodyConfig even if it will break implementations?

anuraaga · 2022-11-11T23:10:32Z

Sorry for the late review. We want all the settings to be programmatically configurable without any text directives so don't think we can lose that ability for body access.

I think we could add an Enabled method to the body configs, or just flatten them instead of having separate sub-structs. Leaning towards latter, what do you think?

codecov-commenter · 2022-11-12T19:10:57Z

Codecov Report

Base: 78.21% // Head: 78.27% // Increases project coverage by +0.06% 🎉

Coverage data is based on head (9bd1aa8) compared to base (fb52935).
Patch coverage: 51.28% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           v3/dev     #503      +/-   ##
==========================================
+ Coverage   78.21%   78.27%   +0.06%     
==========================================
  Files         145      145              
  Lines        6876     6872       -4     
==========================================
+ Hits         5378     5379       +1     
+ Misses       1264     1256       -8     
- Partials      234      237       +3

Flag	Coverage Δ
default	`73.53% <51.28%> (+0.05%)`	⬆️
examples	`27.27% <20.51%> (+0.07%)`	⬆️
ftw	`55.53% <33.33%> (+0.09%)`	⬆️
tinygo	`71.59% <51.28%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
internal/corazawaf/waf.go	`91.60% <ø> (ø)`
types/waf.go	`70.73% <ø> (ø)`
waf.go	`40.00% <18.18%> (-5.77%)`	⬇️
config.go	`40.00% <41.17%> (+6.00%)`	⬆️
actions/ctl.go	`71.50% <100.00%> (ø)`
internal/corazawaf/body_buffer.go	`71.69% <100.00%> (ø)`
internal/corazawaf/transaction.go	`79.94% <100.00%> (ø)`
internal/seclang/directives.go	`67.97% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

M4tteoP · 2022-11-12T19:13:30Z

config.go

-
-type responseBodyConfig struct {
-	limit         int
-	inMemoryLimit int


responseBodyConfig had inMemoryLimit and WithInMemoryLimit, but then it was not populated nor called, nor the WAF struct has a corresponding default value and field for it. It is okay to assume that it was not used, and not needed?

waf.go

M4tteoP · 2022-11-12T20:34:10Z

Thanks @anuraaga! I removed both RequestBodyConfig and ResponseBodyConfig structures and refactored the API making each element individually configurable. PTAL 🙏

anuraaga · 2022-11-15T00:33:06Z

Thanks! Mostly LGTM

M4tteoP · 2022-11-17T00:00:27Z

Sorry for the delay! Addressed the review and updated the title to a more coherent one with the PR evolution

M4tteoP · 2022-11-21T11:54:03Z

Could you please take another look at this one? @jptosso @fzipi
Thanks! 🙏

jptosso

LGTM, has a lot of methods, but it's easier to understand.

fzipi

In essence, why don't we use uint instead of int? I don't expect sizes to be negative in the future, but maybe someone wants reaaaly big files 😄

config.go

M4tteoP · 2022-11-21T17:22:03Z

Internally these limits are defined as int64. I can't see a reason for negative sizes, but to avoid conversions we may stick with int64 also for the exported function. I went for this solution in the last pushed commit.
Otherwise, we can swap everything to unsigned, even if I feel that int64 is already enough in terms of super big files.
Done! Added "Bytes" for all the exported function names.

jcchavezs · 2022-11-21T17:54:56Z

Another consideration here is that we access to those content types that are being inspected and only buffer when inspection.

…

On Mon, 21 Nov 2022, 18:22 Matteo Pace, ***@***.***> wrote: 1. Internally these limits are defined as int64. I can't see a reason for negative sizes, but to avoid conversions we may stick with int64 also for the exported function. I went for this solution in the last pushed commit. Otherwise, we can swap everything to unsigned, even if I feel that int64 is already enough in terms of super big files. 2. Done! Added "Bytes" for all the exported function names. — Reply to this email directly, view it on GitHub <#503 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAR3M64YSCGIY6ZCMUDWJOVUNANCNFSM6AAAAAAR3P365A> . You are receiving this because your review was requested.Message ID: ***@***.***>

config.go

fzipi · 2022-11-30T11:59:18Z

So, in summary:

people here don't care about big sizes, because "most of the times you don't need them".
we don't mind about negative numbers
we just care most of all about simplicity and usage. People use int, and despite being good or bad, is what people use so we stick to that to avoid conversion.

Can we document these decisions somewhere else where they can be reached by more people? Also, for future Coraza developers :)

anuraaga · 2022-11-30T12:39:14Z

Yup except for this

people here don't care about big sizes

We basically assume 64-bit usage where int are big sizes.

Would be good to document but don't think it's needed for this PR

M4tteoP · 2022-12-11T22:41:08Z

Moved from int64 to int and tried to properly comment about it. @fzipi PTAL 🙏

fzipi · 2022-12-12T11:34:27Z

In general looks good. But... are we considering the practical implications of an integer overflow here?
Can we generate an overflow in: l := len(data) + br.length? If data is near max int, and br.length adds enough to make l a very small int?

M4tteoP · 2022-12-13T10:40:50Z

@fzipi that's a legit concern. I feel that the fundamental question is: are 32-bit machines a thing (to be supported)? If so, and We basically assume 64-bit usage where int are big sizes can not be a strong assumption, I think we should explicitly use int64 internally and just keep the int exposed to the API. The agreed limits will still be in place, but internally no problems like this one should happen anymore. @jptosso

fzipi · 2022-12-13T11:41:52Z

What I want to point is that assumptions are the "enemy" of security, in general. We make things explicit so we are sure. Making this explicit is about:

documenting it properly and clearly.
the limit is clearly checked. For a signed int, this implies checking that the file is between 0 and max int. We should explicitly take action if the size is negative (but not -1, because we are overriding that with Unset)

Hopefully, it is clear now.

fzipi · 2022-12-28T11:54:40Z

Can we fix conflicts here @M4tteoP? Thx!

M4tteoP · 2023-01-24T13:53:12Z

internal/corazawaf/waf.go

@@ -100,6 +100,7 @@ type WAF struct {
 	// UploadDir is the directory where the uploaded files will be stored
 	UploadDir string

+	// Request body in memory limit excluding the size of any files being transported in the request.
 	RequestBodyNoFilesLimit int64


Actually, I think that the value is parsed and stored, but never used. It should be implemented or documented (also in the coraza.conf-recommended)

config.go

jcchavezs · 2023-01-25T08:31:16Z

config.go

+	// int is a signed integer type that is at least 32 bits in size (platform-dependent size).
+	// While, the theoretical settable upper limit for 32-bit machines is 2GiB,
+	// it is recommended to keep this value as low as possible.
+	WithRequestBodyBytesLimit(limit int) WAFConfig


Do we really need the work Bytes?

I don't have strong feelings about it. It makes the function more explicit but makes the name even more verbose. It has been added addressing a previous review #503 (comment).

I think given the directive is SecRequestBodyLimit it is better to align with it, also Bytes seem redundant, specifying the units is something that I like when in doubt but in this case and in general in golang ecosystem, bytes is the unit for bodies.

…ts and enabling BodyAccess (corazawaf#503) * Removes BodyAccess by default when setting WithBodyAccess * flatters config removing RequestBodyConfig and ResponseBodyConfig * sets default waf config limits to -1 * changes to int64, renames exported functions names * address review clarifying config variable initialization * moves memory limits from int64 to int * fix responseBodyLimit * wip merge * wip merge * makes UnsetLimit private * address review, removes world Byte

Removes BodyAccess by default when setting WithBodyAccess

947e7eb

M4tteoP requested a review from a team as a code owner November 9, 2022 14:31

M4tteoP force-pushed the not_enforce_body_access branch from d5f2cd9 to 0494ee2 Compare November 12, 2022 19:08

M4tteoP commented Nov 12, 2022

View reviewed changes

M4tteoP force-pushed the not_enforce_body_access branch from 0494ee2 to a611181 Compare November 12, 2022 19:15

flatters config removing RequestBodyConfig and ResponseBodyConfig

0c97aff

M4tteoP force-pushed the not_enforce_body_access branch from a611181 to 0c97aff Compare November 12, 2022 20:24

M4tteoP commented Nov 12, 2022

View reviewed changes

waf.go Outdated Show resolved Hide resolved

sets default waf config limits to -1

07c0508

M4tteoP changed the title ~~Removes BodyAccess by default when setting WithBodyAccess~~ Flatters WAF config API for better flexibility in setting memory limits and enabling BodyAccess Nov 16, 2022

anuraaga approved these changes Nov 17, 2022

View reviewed changes

jptosso approved these changes Nov 21, 2022

View reviewed changes

fzipi reviewed Nov 21, 2022

View reviewed changes

config.go Show resolved Hide resolved

config.go Outdated Show resolved Hide resolved

M4tteoP added 2 commits November 21, 2022 17:37

Merge branch 'v3/dev' into not_enforce_body_access

25a9ae3

changes to int64, renames exported functions names

d1b256b

M4tteoP added 3 commits November 22, 2022 00:13

Merge branch 'v3/dev' into not_enforce_body_access

50a6e86

address review clarifying config variable initialization

1264a82

Merge branch 'v3/dev' into not_enforce_body_access

ad3dbea

anuraaga reviewed Nov 30, 2022

View reviewed changes

config.go Outdated Show resolved Hide resolved

moves memory limits from int64 to int

8df057d

Merge branch 'v3/dev' into not_enforce_body_access

9bd1aa8

anuraaga approved these changes Dec 12, 2022

View reviewed changes

fzipi closed this Dec 12, 2022

fzipi reopened this Dec 12, 2022

fix responseBodyLimit

53d3715

M4tteoP mentioned this pull request Dec 22, 2022

Write Body buffer: adds Limit check, overflow check and limits comments #539

Merged

jcchavezs mentioned this pull request Dec 28, 2022

Body buffer rework defensive #556

Closed

M4tteoP added 2 commits January 23, 2023 18:57

merge

ad3f66a

wip merge

072b863

anuraaga approved these changes Jan 24, 2023

View reviewed changes

wip merge

ce740e4

M4tteoP commented Jan 24, 2023

View reviewed changes

jcchavezs reviewed Jan 25, 2023

View reviewed changes

config.go Outdated Show resolved Hide resolved

jcchavezs reviewed Jan 25, 2023

View reviewed changes

M4tteoP added 2 commits January 25, 2023 19:55

makes UnsetLimit private

c594301

address review, removes world Byte

6ac6a89

jcchavezs approved these changes Jan 25, 2023

View reviewed changes

jcchavezs merged commit bb55a14 into corazawaf:v3/dev Jan 25, 2023

M4tteoP deleted the not_enforce_body_access branch January 25, 2023 23:59

M4tteoP mentioned this pull request Jan 27, 2023

address Flattered limits in Coraza corazawaf/coraza-proxy-wasm#135

Merged

fzipi mentioned this pull request May 9, 2024

fix: use uint for (no_)expect_ids coreruleset/ftw-tests-schema#8

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flatters WAF config API for better flexibility in setting memory limits and enabling BodyAccess #503

Flatters WAF config API for better flexibility in setting memory limits and enabling BodyAccess #503

M4tteoP commented Nov 9, 2022

anuraaga commented Nov 11, 2022

codecov-commenter commented Nov 12, 2022 •

edited

Loading

M4tteoP Nov 12, 2022 •

edited

Loading

M4tteoP commented Nov 12, 2022

anuraaga commented Nov 15, 2022

M4tteoP commented Nov 17, 2022

M4tteoP commented Nov 21, 2022

jptosso left a comment

fzipi left a comment

M4tteoP commented Nov 21, 2022

jcchavezs commented Nov 21, 2022 via email

fzipi commented Nov 30, 2022

anuraaga commented Nov 30, 2022

M4tteoP commented Dec 11, 2022

fzipi commented Dec 12, 2022 •

edited

Loading

M4tteoP commented Dec 13, 2022 •

edited

Loading

fzipi commented Dec 13, 2022

fzipi commented Dec 28, 2022

M4tteoP Jan 24, 2023 •

edited

Loading

jcchavezs Jan 25, 2023

M4tteoP Jan 25, 2023

jcchavezs Jan 25, 2023

Flatters WAF config API for better flexibility in setting memory limits and enabling BodyAccess #503

Flatters WAF config API for better flexibility in setting memory limits and enabling BodyAccess #503

Conversation

M4tteoP commented Nov 9, 2022

anuraaga commented Nov 11, 2022

codecov-commenter commented Nov 12, 2022 • edited Loading

Codecov Report

M4tteoP Nov 12, 2022 • edited Loading

Choose a reason for hiding this comment

M4tteoP commented Nov 12, 2022

anuraaga commented Nov 15, 2022

M4tteoP commented Nov 17, 2022

M4tteoP commented Nov 21, 2022

jptosso left a comment

Choose a reason for hiding this comment

fzipi left a comment

Choose a reason for hiding this comment

M4tteoP commented Nov 21, 2022

jcchavezs commented Nov 21, 2022 via email

fzipi commented Nov 30, 2022

anuraaga commented Nov 30, 2022

M4tteoP commented Dec 11, 2022

fzipi commented Dec 12, 2022 • edited Loading

M4tteoP commented Dec 13, 2022 • edited Loading

fzipi commented Dec 13, 2022

fzipi commented Dec 28, 2022

M4tteoP Jan 24, 2023 • edited Loading

Choose a reason for hiding this comment

jcchavezs Jan 25, 2023

Choose a reason for hiding this comment

M4tteoP Jan 25, 2023

Choose a reason for hiding this comment

jcchavezs Jan 25, 2023

Choose a reason for hiding this comment

codecov-commenter commented Nov 12, 2022 •

edited

Loading

M4tteoP Nov 12, 2022 •

edited

Loading

fzipi commented Dec 12, 2022 •

edited

Loading

M4tteoP commented Dec 13, 2022 •

edited

Loading

M4tteoP Jan 24, 2023 •

edited

Loading