[entropy_src] measurement of entropy is needed to configure health checks

[vsukhoml]

Description

NIST 800-90B provides guidance how APT & RCT health checks shall be configured depending on required probability of detecting error and measured entropy.
For the approved continuous health tests, the false positive probability $\alpha$ is recommended to be between $2^{−20}$ and $2^{−40}$. Lower probability values are acceptable. The submitter shall specify and document a false positive probability suitable for their application

AIS.31 defines parameters for other tests.

NIST Entropy Assessment toolkit includes set of test, documented in NIST 800-90B to measure entropy for use in configuring health checks appropritately.

In my experience measurements would be in the range ~0.8-0.85 per bit even for very high quality entropy source, or, say DRBG output due to compression test nuances on 1-bit streams (alphabet of entropy source).

So, action items are:

1. Measure entropy on different samples, take the minimum value (real tests involves environmental tests at different points in operational range)
2. Calculate configuration of health checks
3. Make sure that health checks are configured properly before using entropy.

@vogelpi @jadephilipoom

[vogelpi]

This is a dupliate of #2111 and #19392.

[vsukhoml]

Do we have dump of logs with entropy measurement somewhere? I see issues, but can't find how they are resolved.

NIST entropy tests produce output like below and shall be executed multiple times at different environmental points:

Loaded 1000000 samples of 2 distinct 1-bit-wide symbols

Running non-IID tests...

Running Most Common Value Estimate...
Literal MCV Estimate: mode = 500238, p-hat = 0.50023799999999996, p_u = 0.50152591514982692
	Most Common Value Estimate = 0.995604 / 1 bit(s)

Running Entropic Statistic Estimates (bit strings only)...
Literal Collision Estimate: X-bar = 2.4995388341627653, sigma-hat = 0.50000041221288927, p = 0.53533662114698544
	Collision Test Estimate = 0.901482 / 1 bit(s)
Literal Markov Estimate: P_0 = 0.49976199999999998, P_1 = 0.50023799999999996, P_0,0 = 0.5005132453312684, P_0,1 = 0.4994867546687316, P_1,0 = 0.49901047101579649, P_1,1 = 0.50098952898420346, p_max = 3.7793324099521518e-39
	Markov Test Estimate = 0.997165 / 1 bit(s)
Literal Compression Estimate: X-bar = 5.2192586826314926, sigma-hat = 1.013874035750848, p = 0.027190262043963354
	tCompression Test Estimate = 0.866794 / 1 bit(s)

Running Tuple Estimates...
Literal t-Tuple Estimate: t = 16, p-hat_max = 0.52755690708950331, p_u = 0.52884286485529719
Literal LRS Estimate: u = 17, v = 39, p-hat = 0.50128460831368271, p_u = 0.5025725193587246
	T-Tuple Test Estimate = 0.919089 / 1 bit(s)
	LRS Test Estimate = 0.992596 / 1 bit(s)

Running Predictor Estimates...
Literal MultiMCW Prediction Estimate: N = 999937, Pglobal' = 0.50123745267875108 (C = 499918) Plocal can't affect result (r = 19)
	Multi Most Common in Window (MultiMCW) Prediction Test Estimate = 0.996434 / 1 bit(s)
Literal Lag Prediction Estimate: N = 999999, Pglobal' = 0.50050341356991546 (C = 499215) Plocal can't affect result (r = 20)
	Lag Prediction Test Estimate = 0.998548 / 1 bit(s)
Literal MultiMMC Prediction Estimate: N = 999998, Pglobal' = 0.50190291683940447 (C = 500614) Plocal can't affect result (r = 17)
	Multi Markov Model with Counting (MultiMMC) Prediction Test Estimate = 0.994520 / 1 bit(s)
Literal LZ78Y Prediction Estimate: N = 999983, Pglobal' = 0.50115842399842625 (C = 499862) Plocal can't affect result (r = 18)
	LZ78Y Prediction Test Estimate = 0.996661 / 1 bit(s)

H_original: 0.866794
Minimal entropy 0.866794
Opening file: '/tmp/ea/trng_output_restart'
Loaded 1000000 samples made up of 2 distinct 1-bit-wide symbols.
H_I: 0.866794
ALPHA: 5.0251553006530614e-06, X_cutoff: 617
X_max: 569

Restart Sanity Check Passed...

Running non-IID tests...

Running Most Common Value Estimate...
Literal MCV Estimate: mode = 500768, p-hat = 0.50076799999999999, p_u = 0.50205591377644465
	Most Common Value Estimate (Rows) = 0.994080 / 1 bit(s)
Literal MCV Estimate: mode = 500768, p-hat = 0.50076799999999999, p_u = 0.50205591377644465
	Most Common Value Estimate (Cols) = 0.994080 / 1 bit(s)

Running Entropic Statistic Estimates (bit strings only)...
Literal Collision Estimate: X-bar = 2.4994313793611953, sigma-hat = 0.50000030152907682, p = 0.53608851711200045
	Collision Test Estimate (Rows) = 0.899457 / 1 bit(s)
Literal Collision Estimate: X-bar = 2.4999850000749997, sigma-hat = 0.50000062477304874, p = 0.53202631140467849
	Collision Test Estimate (Cols) = 0.910430 / 1 bit(s)
Literal Markov Estimate: P_0 = 0.50076799999999999, P_1 = 0.49923200000000001, P_0,0 = 0.50116820563614284, P_0,1 = 0.49883179436385716, P_1,0 = 0.50036756531545512, P_1,1 = 0.49963243468454488, p_max = 3.9586080948949002e-39
	Markov Test Estimate (Rows) = 0.996642 / 1 bit(s)
Literal Markov Estimate: P_0 = 0.50076799999999999, P_1 = 0.49923200000000001, P_0,0 = 0.50150369033165054, P_0,1 = 0.49849630966834946, P_1,0 = 0.50003104775144169, P_1,1 = 0.49996895224855831, p_max = 4.309744037351884e-39
	Markov Test Estimate (Cols) = 0.995684 / 1 bit(s)
Literal Compression Estimate: X-bar = 5.217106041939239, sigma-hat = 1.0163907231372382, p = 0.029793851221070922
	Compression Test Estimate (Rows) = 0.844807 / 1 bit(s)
Literal Compression Estimate: X-bar = 5.2187618310457076, sigma-hat = 1.0139480373818941, p = 0.027823914855976351
	Compression Test Estimate (Cols) = 0.861255 / 1 bit(s)

Running Tuple Estimates...
Literal t-Tuple Estimate: t = 15, p-hat_max = 0.51808453453604919, p_u = 0.51937160712968733
Literal LRS Estimate: u = 16, v = 41, p-hat = 0.51935546186299164, p_u = 0.52064241180346316
Literal t-Tuple Estimate: t = 15, p-hat_max = 0.52065048083654542, p_u = 0.52193729722018178
Literal LRS Estimate: u = 16, v = 39, p-hat = 0.50243644406537435, p_u = 0.5037243440702166
	T-Tuple Test Estimate (Rows) = 0.945161 / 1 bit(s)
	T-Tuple Test Estimate (Cols) = 0.938052 / 1 bit(s)
	LRS Test Estimate (Rows) = 0.941635 / 1 bit(s)
	LRS Test Estimate (Cols) = 0.989294 / 1 bit(s)

Running Predictor Estimates...
Literal MultiMCW Prediction Estimate: N = 999937, Pglobal' = 0.50171848251277584 (C = 500399) Plocal can't affect result (r = 21)
	Multi Most Common in Window (MultiMCW) Prediction Test Estimate (Rows) = 0.995050 / 1 bit(s)
Literal MultiMCW Prediction Estimate: N = 999937, Pglobal' = 0.50108244281091086 (C = 499763) Plocal can't affect result (r = 22)
	Multi Most Common in Window (MultiMCW) Prediction Test Estimate (Cols) = 0.996880 / 1 bit(s)
Literal Lag Prediction Estimate: N = 999999, Pglobal' = 0.50153641602912746 (C = 500248) Plocal can't affect result (r = 19)
	Lag Prediction Test Estimate (Rows) = 0.995574 / 1 bit(s)
Literal Lag Prediction Estimate: N = 999999, Pglobal' = 0.50100441544916496 (C = 499716) Plocal can't affect result (r = 21)
	Lag Prediction Test Estimate (Cols) = 0.997105 / 1 bit(s)
Literal MultiMMC Prediction Estimate: N = 999998, Pglobal' = 0.50168691697157541 (C = 500398) Plocal can't affect result (r = 20)
	Multi Markov Model with Counting (MultiMMC) Prediction Test Estimate (Rows) = 0.995141 / 1 bit(s)
Literal MultiMMC Prediction Estimate: N = 999998, Pglobal' = 0.501786916940265 (C = 500498) Plocal can't affect result (r = 19)
	Multi Markov Model with Counting (MultiMMC) Prediction Test Estimate (Cols) = 0.994853 / 1 bit(s)
Literal LZ78Y Prediction Estimate: N = 999983, Pglobal' = 0.50172843323195404 (C = 500432) Plocal can't affect result (r = 20)
	LZ78Y Prediction Test Estimate (Rows) = 0.995021 / 1 bit(s)
Literal LZ78Y Prediction Estimate: N = 999983, Pglobal' = 0.50134142714530527 (C = 500045) Plocal can't affect result (r = 22)
	LZ78Y Prediction Test Estimate (Cols) = 0.996135 / 1 bit(s)

H_r: 0.844807
H_c: 0.861255
H_I: 0.866794

Validation Test Passed...

min(H_r, H_c, H_I): 0.844807