Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

&max-size will overread and only raise error afterwards #1815

Closed
0xxon opened this issue Jul 31, 2024 · 1 comment · Fixed by #1816
Closed

&max-size will overread and only raise error afterwards #1815

0xxon opened this issue Jul 31, 2024 · 1 comment · Fixed by #1816
Assignees
Labels
Bug Something isn't working Runtime Library Issues related to the HILTI or Spicy runtime libraries

Comments

@0xxon
Copy link
Member

0xxon commented Jul 31, 2024

The &max-size attribute will happily allow inner units to over-read the data from the input stream till they are done, including raising hooks (or zeek events) - only raising an error after all parsing is finished.

Example:

module Test;

public type Top = unit {
  innerdata: Inner &max-size=2;
};

type Inner = unit {
   data: bytes &size=20;
   on %done { print self; }
};
$ printf '\x0b\x00\x00\x01\x00\x0e\xec\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' | spicy-driver test2.spicy
[$data=b"\x0b\x00\x00\x01\x00\x0e\xec\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"]
[error] terminating with uncaught exception of type spicy::rt::ParseError: parsing not done within &max-size bytes (test2.spicy:4:20-4:30)

This obviously is a problem, especially in cases when one parses a protocol that specifies unit-lengths itself.

To give a real-world example: This bug was found while examining a test-failure fo the TLS testsuite. In that case, an inner field specifies a much-too-large-length, exceeding the size of the outer fields by several kilobytes. As this overread seems to exceed the amount of data remaining in this connection, no error is ever raised.

@bbannier bbannier added the Bug Something isn't working label Jul 31, 2024
@bbannier
Copy link
Member

bbannier commented Jul 31, 2024

What happens here is that we set up two dependent limited views into the input stream:

  1. a View view1 with size=3 to parse innerdata from (one larger than the &max-size value to accommodate a sentinel byte for detecting overruns)
  2. a View view2 from view1 with size=20 to parse data from

This fails since when calling limit on an already limited view but with a larger size, the original limit is overriden. This is an issue in the implementation of hilti::rt::stream::View::limit, and can reproduced with this unit test:

SUBCASE("limited view inherits limit") {
    auto input = "1234567890"_b;
    auto stream = Stream(input);
    auto view = stream.view();

    auto limit1 = view.limit(5U);
    REQUIRE_EQ(limit1.size(), 5U);

    auto limit2 = limit1.limit(input.size());
    REQUIRE_EQ(limit2.size(), 5U);
}

The fix would probably be to only ever decrease the size of a View with limit. The only downside of that would be that parsing would then fail since data cannot read enough data, and we do not explicitly trigger an error pointing out&max-size anymore. With that approach we also seem to fail a number of tests, so we might have assumptions about limit being able to override elsewhere.

@bbannier bbannier added Good first issue Good for newcomers Runtime Library Issues related to the HILTI or Spicy runtime libraries and removed Good first issue Good for newcomers labels Jul 31, 2024
@bbannier bbannier self-assigned this Aug 1, 2024
bbannier added a commit that referenced this issue Aug 1, 2024
The documented semantics of `View::limit` are that it creates a new view
with equal or smaller size. In contrast to that we would still have
allowed to expand views with more calls `limit` again as well.

This patch changes the implementation of `View::limit` so it can only
ever make a `View` smaller.

Closes #1815.
bbannier added a commit that referenced this issue Aug 2, 2024
The documented semantics of `View::limit` are that it creates a new view
with equal or smaller size. In contrast to that we would still have
allowed to expand views with more calls `limit` again as well.

This patch changes the implementation of `View::limit` so it can only
ever make a `View` smaller.

We also tweak the implementation of the check for consumed `&size` when
used together with `&eod`: if the `&size` was already nested in a
limited view a larger `&size` value could previously extend the view so
the `&eod` effectively was ignored. Since we now do not extend the
`View` anymore we need to only activate the check for consumed `&size`
if `&eod` was not specified since in this case the user communicated that
they are fine with consuming less data.

Closes #1815.
bbannier added a commit that referenced this issue Aug 2, 2024
The documented semantics of `View::limit` are that it creates a new view
with equal or smaller size. In contrast to that we would still have
allowed to expand views with more calls `limit` again as well.

This patch changes the implementation of `View::limit` so it can only
ever make a `View` smaller.

We also tweak the implementation of the check for consumed `&size` when
used together with `&eod`: if the `&size` was already nested in a
limited view a larger `&size` value could previously extend the view so
the `&eod` effectively was ignored. Since we now do not extend the
`View` anymore we need to only activate the check for consumed `&size`
if `&eod` was not specified since in this case the user communicated that
they are fine with consuming less data.

Closes #1815.

(cherry picked from commit 44d87b2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Runtime Library Issues related to the HILTI or Spicy runtime libraries
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants