Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some unit tests fail #316

Open
dbogdanov opened this issue Nov 9, 2015 · 6 comments
Open

Some unit tests fail #316

dbogdanov opened this issue Nov 9, 2015 · 6 comments
Assignees
@palonso
Labels
algorithms QA bug
Milestone
2.1

Comments

@dbogdanov
Copy link
Member

Some regression tests are failing for some algorithms. Review and update the ground truth annotation.

PitchYinFFT

ERR FAIL: testARealCase (tonal.test_pitchyinfft.TestPitchYinFFT)
ERR ----------------------------------------------------------------------
ERR Traceback (most recent call last):
ERR   File "/data/go-agent/workspace/pipelines/afep-deps-libessentia/dependencies/libessentia/work/essentia-2.1_beta2/test/src/unittest/tonal/test_pitchyinfft.py", line 131, in testARealCase
ERR     self.assertAlmostEqualVector(pitch, expected_pitch)
ERR   File "/data/go-agent/workspace/pipelines/afep-deps-libessentia/dependencies/libessentia/work/essentia-2.1_beta2/test/src/unittest/essentia_test.py", line 115, in assertAlmostEqualVector
ERR     self.assertAlmostEqual(val1, val2, precision)
ERR   File "/data/go-agent/workspace/pipelines/afep-deps-libessentia/dependencies/libessentia/work/essentia-2.1_beta2/test/src/unittest/essentia_test.py", line 97, in assertAlmostEqual
ERR     self.assert_(diff <= precision, 'Difference is %e while allowed precision is %e' % (diff, precision))
ERR AssertionError: Difference is 1.141042e-02 while allowed precision is 1.000000e-07

Panning

ERR FAIL: testRegression (standard.test_panning.TestPanning)
ERR ----------------------------------------------------------------------
ERR Traceback (most recent call last):
ERR   File "/data/go-agent/workspace/pipelines/afep-deps-libessentia/dependencies/libessentia/work/essentia-2.1_beta2/test/src/unittest/standard/test_panning.py", line 78, in testRegression
ERR     self.assertAlmostEqualVectorFixedPrecision(expected[i], output[0], 2)
ERR   File "/data/go-agent/workspace/pipelines/afep-deps-libessentia/dependencies/libessentia/work/essentia-2.1_beta2/test/src/unittest/essentia_test.py", line 102, in assertAlmostEqualVectorFixedPrecision
ERR     self.assertAlmostEqualFixedPrecision(val1, val2, digits)
ERR   File "/data/go-agent/workspace/pipelines/afep-deps-libessentia/dependencies/libessentia/work/essentia-2.1_beta2/test/src/unittest/essentia_test.py", line 88, in assertAlmostEqualFixedPrecision
ERR     BaseTestCase.assertAlmostEqual(self, found, expected, digits)
ERR AssertionError: -49.7533 != -49.766972 within 2 places

RhythmExtractor (outdated algorithm, #3 )

FAIL: testImpulseTrain (rhythm.test_rhythmextractor.TestRhythmExtractor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/essentia/test/src/unittest/rhythm/test_rhythmextractor.py", line 308, in testImpulseTrain
    self.assertVectorWithinVector(result[1], expectedTicks, 0.03)
  File "/home/dbogdanov/essentia/test/src/unittest/rhythm/test_rhythmextractor.py", line 110, in assertVectorWithinVector
    self.assertAlmostEqual(found[i], expected[j-1], precision)
  File "/home/dbogdanov/essentia/test/src/unittest/essentia_test.py", line 97, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed precision is %e' % (diff, precision))
AssertionError: Difference is 3.712006e-02 while allowed precision is 3.000000e-02
@dbogdanov dbogdanov mentioned this issue Jan 27, 2016
Open
@alastair alastair mentioned this issue Jul 25, 2016
Closed
@dbogdanov
Copy link
Member Author

Tests failing on OSX:

  • AudioWriter seems to be broken with the new LibAv
  • We need to decide what to do with the issue of consistency of MD5 values in AudioLoader
  • tests for outdated RhythmExtractor fail, perhaps we should not support it anymore
  • some regression tests fail because we need to review and improve them
======================================================================
ERROR: testOgg (io.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/io/test_audiowriter_streaming.py", line 223, in testOgg
    self.encoderTest('audiowritertest.ogg', precision=5e-6);
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/io/test_audiowriter_streaming.py", line 183, in encoderTest
    run(gen)
  File "test/src/unittest/all_tests.py", line 190, in runResetRun
  File "/Users/dbogdanov/mtg/essentia/build/python/essentia/__init__.py", line 148, in run
    return _essentia.run(gen)
RuntimeError: Error while writing delayed audio frame

======================================================================
FAIL: testRegression (io.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/io/test_audiowriter_streaming.py", line 69, in testRegression
    self.compare(pool['audio'], signal)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/io/test_audiowriter_streaming.py", line 34, in compare
    self.assertAlmostEqual(result[i][1] - expected[i][1], 0, 0.5/32767)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 3.051758e-05 while allowed deviation from the expected value is 1.525925e-03 %

======================================================================
FAIL: testSaturation (io.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/io/test_audiowriter_streaming.py", line 140, in testSaturation
    self.compare(pool['audio'], expected)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/io/test_audiowriter_streaming.py", line 33, in compare
    self.assertAlmostEqual(result[i][0] - expected[i][0], 0, 0.5/32767)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 3.051758e-05 while allowed deviation from the expected value is 1.525925e-03 %

======================================================================
FAIL: testMD5 (io.test_audioloader_streaming.TestAudioLoader_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/io/test_audioloader_streaming.py", line 241, in testMD5
    self.assertEqual(md5_ogg, "a87dad40fea0966cc5b967d5412e8868")
AssertionError: 'd90386f75405eb12c7555d2ea8fd0f16' != 'a87dad40fea0966cc5b967d5412e8868'

======================================================================
FAIL: testImpulseTrain (rhythm.test_rhythmextractor.TestRhythmExtractor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/rhythm/test_rhythmextractor.py", line 308, in testImpulseTrain
    self.assertVectorWithinVector(result[1], expectedTicks, 0.03)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/rhythm/test_rhythmextractor.py", line 110, in assertVectorWithinVector
    self.assertAlmostEqual(found[i], expected[j-1], precision)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 3.711977e-02 while allowed deviation from the expected value is 3.000000e+00 %

======================================================================
FAIL: testImpulse (rhythm.test_superfluxextractor.TestSuperFluxExtractor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/rhythm/test_superfluxextractor.py", line 48, in testImpulse
    self.assertAlmostEqualVectorAbs(result, expected, 256./44100)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 159, in assertAlmostEqualVectorAbs
    self.assertAlmostEqualAbs(val1, val2, precision)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 154, in assertAlmostEqualAbs
    self.assert_(diff <= precision, 'Difference is %e while allowed absolute error is %e' % (diff, precision))
AssertionError: Difference is 2.979591e-02 while allowed absolute error is 5.804989e-03

======================================================================
FAIL: testRealCase (spectral.test_mfcc.TestMFCC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/spectral/test_mfcc.py", line 97, in testRealCase
    self.assertAlmostEqualVector(mean(pool['mfcc'], 0), expected, 1.0e-5)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 129, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 1.266321e-05 while allowed deviation from the expected value is 1.000000e-03 %

======================================================================
FAIL: testRegression (standard.test_panning.TestPanning)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/standard/test_panning.py", line 83, in testRegression
    self.assertAlmostEqualVectorFixedPrecision(expected[i], output[0], 2)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 107, in assertAlmostEqualVectorFixedPrecision
    self.assertAlmostEqualFixedPrecision(val1, val2, digits)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 102, in assertAlmostEqualFixedPrecision
    BaseTestCase.assertAlmostEqual(self, found, expected, digits)
AssertionError: -149.452 != -149.87411 within 2 places


======================================================================
FAIL: testRegression (tonal.test_dissonance.TestDissonance)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/tonal/test_dissonance.py", line 168, in testRegression
    self.assertAlmostEqual(dissAvg, 0.475740219278, 1e-6)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 1.619066e-06 while allowed deviation from the expected value is 1.000000e-04 %

======================================================================
FAIL: testARealCase (tonal.test_pitchyin.TestPitchYin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/tonal/test_pitchyin.py", line 131, in testARealCase
    self.assertAlmostEqualVector(pitch, expected_pitch)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 129, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 6.245615e-01 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testARealCase (tonal.test_pitchyinfft.TestPitchYinFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/tonal/test_pitchyinfft.py", line 131, in testARealCase
    self.assertAlmostEqualVector(pitch, expected_pitch)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 129, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/Users/dbogdanov/mtg/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 1.141042e-02 while allowed deviation from the expected value is 1.000000e-05 %

@dbogdanov dbogdanov added this to the 2.1 milestone Sep 23, 2016
@dbogdanov dbogdanov added the bug label Sep 23, 2016
This was referenced Nov 7, 2016
Closed
Closed
@dbogdanov
Copy link
Member Author

Test fails in SpectralContrast:

======================================================================
FAIL: testCompare (spectral.test_spectralcontrast.TestSpectralContrast)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittest/spectral/test_spectralcontrast.py", line 80, in testCompare
    self.assertTrue(numpy.mean(sc0[0]) < numpy.mean(sc1[0]))
AssertionError: False is not true

======================================================================
FAIL: testRegression (spectral.test_spectralcontrast.TestSpectralContrast)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittest/spectral/test_spectralcontrast.py", line 46, in testRegression
    self.assertAlmostEqual(numpy.mean(sc), -0.604606057431, 1e-5)
  File "/home/dbogdanov/code/essentia/test/src/unittest/essentia_test.py", line 116, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 5.639607e-02 while allowed deviation from the expected value is 1.000000e-03 %

======================================================================
FAIL: testSpectrumSizeSmallerThanNumberOfBands (spectral.test_spectralcontrast.TestSpectralContrast)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittest/spectral/test_spectralcontrast.py", line 108, in testSpectrumSizeSmallerThanNumberOfBands
    self.assertAlmostEquals(numpy.mean(sc[0]), -2.7182817459)
AssertionError: -2.4319015 != -2.7182817459 within 7 places

@dbogdanov
Copy link
Member Author

Tests failing with the latest master on Linux:

======================================================================
FAIL: testCompare (unittests.spectral.test_spectralcontrast.TestSpectralContrast)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/spectral/test_spectralcontrast.py", line 80, in testCompare
    self.assertTrue(numpy.mean(sc0[0]) < numpy.mean(sc1[0]))
AssertionError: False is not true

======================================================================
FAIL: testRegression (unittests.spectral.test_spectralcontrast.TestSpectralContrast)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/spectral/test_spectralcontrast.py", line 46, in testRegression
    self.assertAlmostEqual(numpy.mean(sc), -0.604606057431, 1e-5)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 125, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 5.639607e-02 while allowed deviation from the expected value is 1.000000e-03 %

======================================================================
FAIL: testSpectrumSizeSmallerThanNumberOfBands (unittests.spectral.test_spectralcontrast.TestSpectralContrast)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/spectral/test_spectralcontrast.py", line 108, in testSpectrumSizeSmallerThanNumberOfBands
    self.assertAlmostEquals(numpy.mean(sc[0]), -2.7182817459)
AssertionError: -2.4319015 != -2.7182817459 within 7 places

======================================================================
FAIL: testOgg (unittests.io.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 223, in testOgg
    self.encoderTest('audiowritertest.ogg', precision=5e-6);
  File "/home/dbogdanov/code/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 197, in encoderTest
    self.assertAlmostEqual(max(pool['audio']), max(sine), precision)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 125, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 1.044999e-02 while allowed deviation from the expected value is 5.000000e-04 %

======================================================================
FAIL: testRegression (unittests.io.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 69, in testRegression
    self.compare(pool['audio'], signal)
  File "/home/dbogdanov/code/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 34, in compare
    self.assertAlmostEqual(result[i][1] - expected[i][1], 0, 0.5/32767)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 125, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 3.051758e-05 while allowed deviation from the expected value is 1.525925e-03 %

======================================================================
FAIL: testSaturation (unittests.io.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 140, in testSaturation
    self.compare(pool['audio'], expected)
  File "/home/dbogdanov/code/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 33, in compare
    self.assertAlmostEqual(result[i][0] - expected[i][0], 0, 0.5/32767)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 125, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 3.051758e-05 while allowed deviation from the expected value is 1.525925e-03 %

======================================================================
FAIL: testMD5 (unittests.io.test_audioloader_streaming.TestAudioLoader_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/io/test_audioloader_streaming.py", line 235, in testMD5
    self.assertEqual(md5_ogg, "a87dad40fea0966cc5b967d5412e8868")
AssertionError: 'd90386f75405eb12c7555d2ea8fd0f16' != 'a87dad40fea0966cc5b967d5412e8868'
- d90386f75405eb12c7555d2ea8fd0f16
+ a87dad40fea0966cc5b967d5412e8868


======================================================================
FAIL: testARealCase (unittests.tonal.test_pitchyinfft.TestPitchYinFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/tonal/test_pitchyinfft.py", line 131, in testARealCase
    self.assertAlmostEqualVector(pitch, expected_pitch)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 138, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 125, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 1.141042e-02 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testARealCase (unittests.tonal.test_pitchyin.TestPitchYin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/tonal/test_pitchyin.py", line 131, in testARealCase
    self.assertAlmostEqualVector(pitch, expected_pitch)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 138, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 125, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 5.787395e-01 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testImpulseTrain (unittests.rhythm.test_rhythmextractor.TestRhythmExtractor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/rhythm/test_rhythmextractor.py", line 286, in testImpulseTrain
    self.assertVectorWithinVectorFixedPrecision(result[1], expectedTicks, 1)
  File "/home/dbogdanov/code/essentia/test/src/unittests/rhythm/test_rhythmextractor.py", line 119, in assertVectorWithinVectorFixedPrecision
    self.assertAlmostEqualFixedPrecision(found[i], expected[j-1], precision)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 111, in assertAlmostEqualFixedPrecision
    BaseTestCase.assertAlmostEqual(self, found, expected, digits)
AssertionError: 10.369769 != 10.0 within 1 places

======================================================================
FAIL: testImpulse (unittests.rhythm.test_superfluxextractor.TestSuperFluxExtractor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/rhythm/test_superfluxextractor.py", line 48, in testImpulse
    self.assertAlmostEqualVectorAbs(result, expected, 256./44100)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 168, in assertAlmostEqualVectorAbs
    self.assertAlmostEqualAbs(val1, val2, precision)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 163, in assertAlmostEqualAbs
    self.assert_(diff <= precision, 'Difference is %e while allowed absolute error is %e' % (diff, precision))
AssertionError: False is not true : Difference is 2.979591e-02 while allowed absolute error is 5.804989e-03

======================================================================
FAIL: testWhiteNoise (unittests.synthesis.test_sprmodel_streaming.TestSprModel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/synthesis/test_sprmodel_streaming.py", line 207, in testWhiteNoise
    self.assertAlmostEqualMatrix(freqs, zerofreqs)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 157, in assertAlmostEqualMatrix
    self.assertAlmostEqual(val1, val2, precision)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 125, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 2.039191e+04 while allowed deviation from the expected value is 1.000000e-05 %

----------------------------------------------------------------------
Ran 1146 tests in 415.941s

FAILED (failures=12)

@dbogdanov
Copy link
Member Author

Resample errors

======================================================================
FAIL: testDouble (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 57, in testDouble
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 88200 != 88199

======================================================================
FAIL: testHalf (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 78, in testHalf
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 22050 != 22049

======================================================================
FAIL: testOneAndHalf (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 64, in testOneAndHalf
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 66150 != 66149

======================================================================
FAIL: testOneQuarter (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 92, in testOneQuarter
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 11025 != 11024

======================================================================
FAIL: testSingle (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 50, in testSingle
    self.assertAlmostEqualVector(self.resample([1], 2), [1], 5e-2)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 136, in assertAlmostEqualVector
    self.assertEqual(len(found), len(expected))
AssertionError: 2 != 1

======================================================================
FAIL: testThreeQuarters (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 85, in testThreeQuarters
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 33075 != 33074

----------------------------------------------------------------------
Ran 8 tests in 0.949s

FAILED (failures=6)


Running tests with compute/reset/compute
----------------------------------------------------------------------
Running tests:
['resample_streaming']
testDouble (unittests.standard.test_resample_streaming.TestResample_Streaming) ... FAIL
testEmpty (unittests.standard.test_resample_streaming.TestResample_Streaming) ... ok
testHalf (unittests.standard.test_resample_streaming.TestResample_Streaming) ... FAIL
testOne (unittests.standard.test_resample_streaming.TestResample_Streaming) ... ok
testOneAndHalf (unittests.standard.test_resample_streaming.TestResample_Streaming) ... FAIL
testOneQuarter (unittests.standard.test_resample_streaming.TestResample_Streaming) ... FAIL
testSingle (unittests.standard.test_resample_streaming.TestResample_Streaming) ... FAIL
testThreeQuarters (unittests.standard.test_resample_streaming.TestResample_Streaming) ... FAIL

======================================================================
FAIL: testDouble (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 57, in testDouble
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 88200 != 88199

======================================================================
FAIL: testHalf (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 78, in testHalf
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 22050 != 22049

======================================================================
FAIL: testOneAndHalf (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 64, in testOneAndHalf
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 66150 != 66149

======================================================================
FAIL: testOneQuarter (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 92, in testOneQuarter
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 11025 != 11024

======================================================================
FAIL: testSingle (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 50, in testSingle
    self.assertAlmostEqualVector(self.resample([1], 2), [1], 5e-2)
  File "/home/dbogdanov/code/essentia/test/src/unittests/essentia_test.py", line 136, in assertAlmostEqualVector
    self.assertEqual(len(found), len(expected))
AssertionError: 2 != 1

======================================================================
FAIL: testThreeQuarters (unittests.standard.test_resample_streaming.TestResample_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 85, in testThreeQuarters
    self.assertResults(input, expected, factor)
  File "/home/dbogdanov/code/essentia/test/src/unittests/standard/test_resample_streaming.py", line 43, in assertResults
    self.assertEqual(len(result), len(expected))
AssertionError: 33075 != 33074

----------------------------------------------------------------------
Ran 8 tests in 1.458s

FAILED (failures=6)

@palonso palonso mentioned this issue Jul 24, 2019
Merged
@palonso
Copy link
Contributor

palonso commented Jul 30, 2019 •
edited by dbogdanov
Loading

Thi is the all_tests.py log after #885:

======================================================================
FAIL: testOgg (unittests.io.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablo/reps/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 243, in testOgg
    self.encoderTest('audiowritertest.ogg', precision=5e-6)
  File "/home/pablo/reps/essentia/test/src/unittests/io/test_audiowriter_streaming.py", line 213, in encoderTest
    self.assertAlmostEqual(max(pool['audio']), max(sine), precision)
  File "/home/pablo/reps/essentia/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 1.044999e-02 while allowed deviation from the expected value is 5.000000e-04 %

======================================================================
FAIL: testMD5 (unittests.io.test_audioloader_streaming.TestAudioLoader_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablo/reps/essentia/test/src/unittests/io/test_audioloader_streaming.py", line 235, in testMD5
    self.assertEqual(md5_ogg, "a87dad40fea0966cc5b967d5412e8868")
AssertionError: 'd90386f75405eb12c7555d2ea8fd0f16' != 'a87dad40fea0966cc5b967d5412e8868'
- d90386f75405eb12c7555d2ea8fd0f16
+ a87dad40fea0966cc5b967d5412e8868


----------------------------------------------------------------------
Ran 1269 tests in 574.118s

FAILED (failures=2)

Both failures are known to be due to FFmpeg and it was decided to left them untouched so we can track them in the future.

  • testOgg fails depending on the Vorbis codec in use. libVorbis makes the test fail, probably due to the uncapability of the FFmpeg wrapper to decode the same number of samples as in the original audio. The internal Vorbis codec (which is the one used for our static build scripts) makes it pass. However, it is considered expermental and libVorbis is prefered. Also libVorbis is the one shipped with libavcodec-dev and for this reason the test passes or fails depending on if Essentia is compiled with static or dynamic dependencies.

  • testMD5was know to fail from a long time ago Incorrect md5 for ogg files after ffmpeg version upgrade #400. Recently, the test was passing again on Ubuntu 19.04 (details on the FFmpeg version to be added).

@dbogdanov
Copy link
Member Author

Excellent results, we are converging towards closing this issue 👏

After merging #885, running on MacOS:

  • Segmentation fault running testOgg in audiowriter_streaming test (crash_audiowriter.txt)
  • tests for MFCC fail:
======================================================================
FAIL: testRealCase (unittests.spectral.test_mfcc.TestMFCC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/code/essentia/test/src/unittests/spectral/test_mfcc.py", line 278, in testRealCase
    self.assertAlmostEqualVector(mean(pool['mfcc'], 0), expected, 1.e-8)
  File "/Users/dbogdanov/mtg/code/essentia/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/Users/dbogdanov/mtg/code/essentia/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 1.607066e-07 while allowed deviation from the expected value is 1.000000e-06 %

======================================================================
FAIL: testRegression (unittests.spectral.test_mfcc.TestMFCC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dbogdanov/mtg/code/essentia/test/src/unittests/spectral/test_mfcc.py", line 67, in testRegression
    self.assertAlmostEqualVector(mfccs, expected, 1.e-6)
  File "/Users/dbogdanov/mtg/code/essentia/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/Users/dbogdanov/mtg/code/essentia/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: False is not true : Difference is 1.635205e-06 while allowed deviation from the expected value is 1.000000e-04 %

----------------------------------------------------------------------
Ran 1278 tests in 825.412s

FAILED (failures=2)

We need to look more in detail on the difference in computed values attributed to different architectures (e.g. see #179), and define the difference value thresholds in tests accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@palonso palonso

Labels
algorithms QA bug
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
2 participants
@dbogdanov @palonso