Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add shrink timeout #488

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add shrink timeout #488

wants to merge 1 commit into from

Conversation

tbidne
Copy link

@tbidne tbidne commented May 23, 2023

Resolves #476.

There are some design decisions to make, so this is intended more to jumpstart the conversation than it is to present a finished implementation. Some notes:

  1. I chose to add a new field, ShrinkTimeLimit :: Int independent of the current ShrinkLimit :: Int. This way is backwards compatible, and it allows one to specify both a total number of shrinks and a time limit. That said, it is arguable that combining these choices into a single option makes for a friendlier interface as e.g. someone might specify a time limit of 30s only to be unexpectedly thwarted by the default ShrinkLimit of 1,000.
  2. Bikeshedding: ShrinkTimeLimit could also be named ShrinkTimeout.
  3. The field is Int microseconds to make interop with timeout simpler, though I could imagine choosing something less implementation-derived, say, seconds (and maybe a different type e.g. Natural).
  4. In case of a timeout, we want to present the latest shrink, so I modified the shrink loop to take in an "update" parameter that saves the current result in an IORef iff ShrinkTimeLimit is set. If ShrinkTimeLimit is not set then the "update" logic is const (pure ()). I mention this in case it can affect performance, as an alternative would be to have two totally different loops, at the cost of code reuse.
  5. I added some tests, though the timeout test is naturally pretty weak as it is non-deterministic. If anyone has ideas here, I'm all ears.

Thanks!

@tbidne tbidne force-pushed the shrink-time branch 2 times, most recently from 58a6512 to 620ca5f Compare May 23, 2023 05:43
@ChickenProp
Copy link
Contributor

Not a maintainer myself, but I think this is neat!

More name bikeshedding: I'd suggest adding "micros" (e.g. withShrinkTimeMicros, ShrinkTimeLimitMicros) to make it clear what the units are.

I added some tests, though the timeout test is naturally pretty weak as it is non-deterministic. If anyone has ideas here, I'm all ears.

Given a known shrink tree, you could have a test deliberately hang on a given input, e.g. a timeout of 100 microseconds and hang for 200 after five shrinks.

@TysonMN
Copy link
Member

TysonMN commented May 23, 2023

My two cents:

I like the naming format xInUnits. In this case, the name would be withShrinkTimeInMicroseconds. I admit that the name can get a bit long though.

@tbidne tbidne force-pushed the shrink-time branch 2 times, most recently from 1393063 to 8419e26 Compare May 25, 2023 21:47
@tbidne
Copy link
Author

tbidne commented May 25, 2023

@ChickenProp Given a known shrink tree, you could have a test deliberately hang on a given input, e.g. a timeout of 100 microseconds and hang for 200 after five shrinks.

Thanks for the idea! I added such a test here. I made the timeout longer (1 second) as lower values would often cause at least one ci job to fail (usually MacOS on an older GHC), due to the test timing out before it reached the generated values I specifically wanted it to get stuck on.

I also had a "sanity-check" style test that verified the timeout indeed cancels shrinking in the expected wall-clock time.

-- Time limit of 2 seconds. Verifies that withShrinkTime indeed cancels
-- shrinking within the time limit we want.
prop_ShrinkTimeLimitClock :: Property
prop_ShrinkTimeLimitClock =
  property $ do
    startTime <- liftIO $ Clock.getMonotonicTime
    annotateShow startTime
    _ <- checkModPropGen delay30s (withShrinkTime 2000000)
    endTime <- liftIO $ Clock.getMonotonicTime
    annotateShow endTime
    let timeElapsed = endTime - startTime
    annotateShow timeElapsed
    -- should be around 2
    diff timeElapsed (>=) 1.5
    diff timeElapsed (<=) 2.5
  where
    delay30s x = when (x == 13) (liftIO $ CC.threadDelay 30000000)

Unfortunately this relies on GHC.Clock, which requires a newer GHC (8.4.1) than the oldest on CI (8.0.2). I suppose I could conditionally add it with cpp, or maybe this test isn't so important anyway.

@TysonMN

As far as the name goes, I agree adding the units is a good idea. Yet ShrinkTimeInMicroseconds is a bit grim, considering the corresponding functions/fields e.g. propertyShrinkTimeInMicroseconds. Perhaps ShrinkTimeoutMicros? I am fine with either InMicro... vs. Micro..., but I think Microseconds should probably be abbreviated (Micro, Micros, Microsec), and ShrinkTimeout sounds a little better to me than ShrinkTimeLimit.

@tbidne
Copy link
Author

tbidne commented May 25, 2023

Also it is probably a good idea to mention withShrinks in the documentation for withShrinkTime e.g.

-- | Set the timeout -- in microseconds -- after which the test runner gives up
-- on shrinking and prints the best counterexample. Note that shrinking can be
-- cancelled before the timeout if the 'ShrinkLimit' is reached
-- (defaults to 1,000). See 'withShrinks'.
--
withShrinkTime :: ShrinkTimeLimit -> Property -> Property

@tbidne tbidne force-pushed the shrink-time branch 2 times, most recently from c74f125 to c247881 Compare May 26, 2023 05:17
@tbidne
Copy link
Author

tbidne commented May 26, 2023

I went ahead and made those changes:

  • Renamed ShrinkTimeLimit --> ShrinkTimeoutMicros.
  • Added wall clock test w/ cpp.

Add withShrinkTimeoutMicros to allow configuring shrink behavior in
terms of a timeout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Shrink with a timeout
3 participants