Refined failed in the community build #530
Comments
|
Alright, I understand the problem. The failing property looks like this: def createProperty[A: Arbitrary: Min: NonNegShift](implicit num: Numeric[A]): Prop = {
import num.{abs, gteq, lt, plus, zero}
forAll { a: A =>
gteq(a, zero) ==> (NonNegShift[A].shift(a) == a)
} &&
forAll { a: A =>
lt(a, zero) ==> (NonNegShift[A].shift(a) == plus(a, abs(Min[A].min)))
}
}The issue is that the two I think it's likely that changes around deterministic testing are causing this problem. Either that, or we are not updating seeds appropriately. I'll dig into this a bit and report back. cc @ashawley |
|
I figured it out. It's a bug with the interaction between |
|
Ok, glad you figured it out. I will quit bisecting then, which I didn't even get a chance to start. ^_^ I didn't think we had any changes to seeds or deterministic testing in this release, but I wasn't aware that it could be a side effect of using Because the failing test in refined was about numbers, the only other thing I thought of were changes #380 and #451, but those deal with floating point. |
|
I'm about to submit a PR to fix this with a test case. Definitely take a look and see what you think. |
Here's the basic issue: When using viewSeed, we need to put an initialSeed in the Gen.Parameters used to evaluate the Prop, so we can recover the seed later and display it. So far, so good. In some cases, we need to "slide" the seed that's embedded in the parameters, so we don't reuse it. Any time we need to evaluate several properties and don't want identical RNG state (which could lead to identical inputs) we need to slide. We were missing a "slide" in flatMap -- we were reusing the same parameters in both cases. Since flatMap was used to define && (via for-comprehension) that meant that when you said p0 && p1, you'd use the same seed for both if viewSeed was set. Refined's property was unusual, but valid. Basically, one property had (x >= 0) ==> and one had (x < 0) ==>. So no single x value would satisfy both, but on average you'd satisfy each 50% of the time, and together you'd get a non-discarded case about 25% of the time. Previously, this worked OK. But since 1.14.0 we started always displaying seeds. This meant that the bug associated with failing to slide manifested, and we always generated the same x value for both sides of the property. This meant we ended up discarding every x generated. The fix was to slide correctly in this case. I also added a toString to Gen.Parameters which was missing, since this helped with debugging.
|
Yes, I can confirm that your change in #531 fixed it. I'm still trying to follow the reasoning behind your fix and the refined test, so thanks for providing the long write-up in the commit message. This entire thing reminds me of something I wrote about about
It's a long discussion that is difficult to follow. A lot of the back and forth was trying to find idiomatic examples, minimize and generally clarify the issue(s). There's a good chance that your change in #531 addresses those problems somehow. |
|
Alright, I'm going to merge #531 then. Thanks for confirming the fix. |
|
I'm closing this -- I'll try to head over to #425 to chime in there, although it's unlikely I'll be able to do that today. |
|
Yeah, fine to merge. We need to keep the community build process going. |
|
Any idea if |
Extracting the error from CI we get:
It sounds like @ashawley wasn't able to reproduce these. I'm going to read these specific tests tests to see if I can figure out what's going on. Given that the tests were discarded (not failed) it's possible a change to our generation logic may have caused problems.
The text was updated successfully, but these errors were encountered: