Fix Random Seed Functionality in TestFramework

[NightOwl888]

The random seed that is set to StringHelper.GOOD_FAST_HASH_SEED through the tests.seed system property in Java currently is the same as the seed that NUnit uses in NUnit.Framework.TestContext.CurrentContext.Random. If we continue using NUnit's Random property, more research is needed to figure out how to set that seed the way the NUnit designers intended so it can be documented for end users.

During testing, I noticed that NUnit does set the seed to a consistent pseudo-random value, so the same test can be run over and over with exactly the same random setup. However, the seed is re-generated when the code in the project under test is changed. I noted that NUnit writes the seed to a file named nunit_random_seed.tmp in the bin folder of the test project when run in Visual Studio.

A Typical Testing Scenario in Lucene

1. Test fails
2. Test failure generates an error message that includes the random seed as a hexadecimal string (that represents a long)
3. Developer sets the tests.seed system property to the same hexadecimal string that caused the test to fail
4. Debugging can take place because the pseudo-randomly generated conditions are exactly the same as they were when the test failed
5. A fix is devised

Testing in .NET

All we have been able to do is get the same test to fail multiple times, but that breaks as soon as any code is changed. We could probably revert back if we save a copy of the nunit_random_seed.tmp, but this is very complicated to do in addition to debugging, and not very intuitive.

We are missing:

1. The test framework reporting the seed that it was using when the test failed
2. The ability to set the random seed

I suspect that although setting the seed cannot be done publicly, we can hack it by using .NET Reflection to set an internal field. There was some discussion about setting it through an attribute, but it doesn't sound like that has been implemented.

I haven't looked into what it will take to extend NUnit to attach the random seed to the test message that is reported. One option (but not a very good one) would be to change the Lucene.Net.TestFramework.Assert class to add the random seed to all of test messages. The main thing we need is to read the seed value. The only place it seems to be exposed is in the bin/nunit_random_seed.tmp file, but it would be preferable to read it through a property within the test session.

Testing Lucene.NET Itself

For end users, the above would solve all of the issues. However, for Lucene.NET we often need to generate the same conditions as Java to determine where the execution paths diverge. This is a problem because the NUnit Random class (probably) isn't the same implementation as the Java Random class. Also, in Java the random seed is a long, but in .NET, it is an int.

We have ported the Java Random class in J2N, named Randomizer. Ideally, we would use this implementation during testing. However, NUnit doesn't seem to have a way to inject a custom implementation of Random nor does it have a way to read the seed it uses to seed a new instance of Randomizer for the test framework or a way to override that setting manually during debugging.

Ideal Solution

1. By default, the test framework uses a seed generated by the same hook that NUnit generates their seed. Alternatively, we could use System.DateTime.UtcNow.Ticks.
2. The seed generated would be a long.
3. The generated seed would be used to seed the J2N.Randomizer class, which the getter of the LuceneTestFramework.Random property would provide and cache.
4. The test framework would ensure the seed is always output along with failure test messages, as a hexadecimal string.
5. The "System Properties" feature allows tests:seed to be set to a hexadecimal string, which if set would override the auto-generated random seed used in step 3.

Note that the ideal solution doesn't necessarily involve NUnit in the equation. With Java's Random class ported and "System Properties" solved in a way that doesn't involve hacking the system's environment variables when debugging, we are much closer to fixing this to make it compatible with Java Lucene so we can test apples to apples.

The main thing we are missing is writing the random seed out into the test message with hexadecimal formatting (the same string that is used in Lucene). Other than that, setting up the system property to override the automatically generated seed should be fairly straightforward.

[rclabo]

I'd like to tackle this one. Should be a good learning exercise.

[NightOwl888]

Great. I have assigned it to you.