phetioCompareAPI is failing. #301

pixelzoom · 2021-07-07T16:22:12Z

In #294, I followed all of the steps in https://github.com/phetsims/phet-io/blob/master/doc/phet-io-instrumentation-technical-guide.md#post-publication-steps for setting up phetioDesigned: true. Everything is commited and pushed.

When I run NS (master) in Studio with phetioCompareAPI, it's failing with:

Assertion failed: Designed API changes detected, please roll them back or revise the reference API:
.... lots of json ....
at window.assertions.assertFunction (assert.js:25)
at XMLHttpRequest. (phetioEngine.js:312)

The json part of the console output is way too long to be examined manually.

I've tried re-running grunt generate-phet-io-api, and no changes are generated, so there's nothing for me to diff.

@samreid @zepumph suggestions on how to proceed?

The text was updated successfully, but these errors were encountered:

pixelzoom · 2021-07-07T16:26:11Z

I'm also wondering why this error is not showing up in CT. I added phetioDesigned: true on 7/6/21 @ 6:32pm, and CT has run 6 test cycles since then. Is this test actually being run by CT?

zepumph · 2021-07-07T16:48:06Z

I'll take a look!

zepumph · 2021-07-07T17:02:56Z

I pulled all, and then ran:

http://localhost:8080/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&ea&debugger

Then I parsed the horrible json-ish output by replacing Expected: and , actual: with newlines, and I got this:


naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.22165343022406,"y":48.54714899884378,"z":12.684829323916581},"hopStartPosition":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234},"hopTime":0.4357117786067223,"restTime":2.9807053366829397},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.7018245396764424,"hopTime":0.37145791805325357,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-16.780560589326512,"y":32.560992241519045,"z":-9.734973426342497},"hopStartPosition":{"x":-59.77427630950419,"y":-62.26770743671871,"z":206.59843884492193}}}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234}}
{"value":{"x":-59.77427630950419,"y":-62.26770743671871,"z":206.59843884492193},"validValues":null,"units":null}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype.xDirectionProperty._data.initialState differs:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"}
{"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-16.17943842560941,"y":49.17372834079743,"z":-6.723558180699183},"hopStartPosition":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044},"hopTime":0.4105757432869891,"restTime":1.733928075687837},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.3926442195784587,"hopTime":0.3664566424200951,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-13.345672919838014,"y":32.62236026723778,"z":10.377831239320455},"hopStartPosition":{"x":-25.143955421198655,"y":-77.12669201353901,"z":184.30996197969148}}}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044}}
{"value":{"x":-25.143955421198655,"y":-77.12669201353901,"z":184.30996197969148},"validValues":null,"units":null}

naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs:
{"_private":{"speed":158.03332201218853}}
{"_private":{"speed":170.958872349204}}

naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-125.23828095416042,"y":-58.25886076740691,"z":212.61170884888963}}
{"value":{"x":-71.00571623876567,"y":-54.996686657376905,"z":217.50497001393464},"validValues":null,"units":null}

naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.xDirectionProperty._data.initialState differs:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"}
{"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.026184316645049,"y":41.21830940413096,"z":-10.247822414681844},"hopStartPosition":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693},"hopTime":0.40510814444208754,"restTime":1.926434494549918},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.806020437221202,"hopTime":0.43276774069209145,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":17.63867517890751,"y":49.090402481554236,"z":-8.40494483084945},"hopStartPosition":{"x":103.34636259832627,"y":-50.670095616209345,"z":223.994856575686}}}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693}}
{"value":{"x":103.34636259832627,"y":-50.670095616209345,"z":223.994856575686},"validValues":null,"units":null}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-14.756223648971094,"y":47.44263166490198,"z":7.335895397398229},"hopStartPosition":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763},"hopTime":0.4596950485642879,"restTime":1.9745883803442004},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.2176901511580194,"hopTime":0.32057176331792103,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":15.733891888178658,"y":39.70201648770933,"z":-2.475136734138804},"hopStartPosition":{"x":71.37289231164004,"y":-78.85318525008829,"z":181.72022212486758}}}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763}}
{"value":{"x":71.37289231164004,"y":-78.85318525008829,"z":181.72022212486758},"validValues":null,"units":null}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.xDirectionProperty._data.initialState differs:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"LEFT"}
{"value":"RIGHT","validValues":["LEFT","RIGHT"],"units":null}

naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs:
{"_private":{"speed":161.4491569500465}}
{"_private":{"speed":189.15268002265276}}

naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-260.3667461546996,"y":-13.013580371909939,"z":280.4796294421351}}
{"value":{"x":178.88171902020912,"y":-9.68208837704151,"z":285.47686743443774},"validValues":null,"units":null}

All of the above look to be values of PhET-iO element's initial state that are based on a seeded random, and so are different between runtimes. This seems like something to try to manage, but I'm not sure off the top of my head the best solution. I'll create a general issue.

pixelzoom · 2021-07-07T17:10:24Z

Any idea why this is not showing up as an error in CT?

zepumph · 2021-07-07T17:16:10Z

Hmm, not totally sure. Looks like it isn't run very often, but I guess it should have failed when it ran that one time:

zepumph · 2021-07-07T17:29:24Z

After the above formatting commits, this is what the assertion message now looks like off the bat:

Assertion failed: Designed API changes detected, please roll them back or revise the reference API:

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs. 
Expected:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.22165343022406,"y":48.54714899884378,"z":12.684829323916581},"hopStartPosition":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234},"hopTime":0.4357117786067223,"restTime":2.9807053366829397},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
 actual:
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.7929579531085924,"hopTime":0.4044680057386113,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":19.846291550894136,"y":43.770404575613156,"z":-1.5172937110237914},"hopStartPosition":{"x":139.36410382280886,"y":-82.16378669842732,"z":176.75431995235903}}}


naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":null,"value":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234}}
 actual:
{"value":{"x":139.36410382280886,"y":-82.16378669842732,"z":176.75431995235903},"validValues":null,"units":null}


naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs. 
Expected:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-16.17943842560941,"y":49.17372834079743,"z":-6.723558180699183},"hopStartPosition":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044},"hopTime":0.4105757432869891,"restTime":1.733928075687837},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
 actual:
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.216618591386859,"hopTime":0.3102949295904416,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-16.28015246843647,"y":38.86503891787034,"z":1.768838080914106},"hopStartPosition":{"x":-52.238887323577785,"y":-9.744994988234037,"z":285.38250751764895}}}


naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":null,"value":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044}}
 actual:
{"value":{"x":-52.238887323577785,"y":-9.744994988234037,"z":285.38250751764895},"validValues":null,"units":null}


naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs. 
Expected:
{"_private":{"speed":158.03332201218853}}
 actual:
{"_private":{"speed":197.3664738102289}}


naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":null,"value":{"x":-125.23828095416042,"y":-58.25886076740691,"z":212.61170884888963}}
 actual:
{"value":{"x":119.24290492209968,"y":-58.093556786655654,"z":212.85966482001652},"validValues":null,"units":null}


naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.xDirectionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"}
 actual:
{"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null}


naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs. 
Expected:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.026184316645049,"y":41.21830940413096,"z":-10.247822414681844},"hopStartPosition":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693},"hopTime":0.40510814444208754,"restTime":1.926434494549918},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
 actual:
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.4965507520181018,"hopTime":0.4575100400118636,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-12.829774489733234,"y":42.61902453026255,"z":-10.258552183046708},"hopStartPosition":{"x":255.24723725659146,"y":-26.555662155718473,"z":260.1665067664223}}}


naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":null,"value":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693}}
 actual:
{"value":{"x":255.24723725659146,"y":-26.555662155718473,"z":260.1665067664223},"validValues":null,"units":null}


naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype.xDirectionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"}
 actual:
{"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null}


naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs. 
Expected:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-14.756223648971094,"y":47.44263166490198,"z":7.335895397398229},"hopStartPosition":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763},"hopTime":0.4596950485642879,"restTime":1.9745883803442004},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
 actual:
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.021831147444063,"hopTime":0.42458830499204303,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":17.842566644128816,"y":35.56028711778421,"z":-8.173613873384644},"hopStartPosition":{"x":-299.1660441608122,"y":-3.3060569794218586,"z":295.0409145308672}}}


naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":null,"value":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763}}
 actual:
{"value":{"x":-299.1660441608122,"y":-3.3060569794218586,"z":295.0409145308672},"validValues":null,"units":null}


naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.xDirectionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"LEFT"}
 actual:
{"value":"RIGHT","validValues":["LEFT","RIGHT"],"units":null}


naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs. 
Expected:
{"_private":{"speed":161.4491569500465}}
 actual:
{"_private":{"speed":134.9222695517728}}


naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":null,"value":{"x":-260.3667461546996,"y":-13.013580371909939,"z":280.4796294421351}}
 actual:
{"value":{"x":168.88009675579448,"y":-80.42750768111614,"z":179.3587384783258},"validValues":null,"units":null}


naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype.xDirectionProperty._data.initialState differs. 
Expected:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"}
 actual:
{"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null}

zepumph · 2021-07-07T17:39:59Z

I found https://github.com/phetsims/phet-io/issues/1765, noting that the generated API file is consistent as it pertains to randomness. From this I don't feel like this needs to block publication. The issue is in our ability to maintain master and prevent changes from occurring that change the API on master.

pixelzoom · 2021-07-07T17:50:10Z

OK, I think I understand why this doesn't need to be blocking -- it's relevant to master, not a release branch.

But if these types of errors persist, how will we identify real errors? And will these types of errors occur in CT?

zepumph · 2021-07-07T18:01:13Z

This is certainly something to fix, and to not let linger.

Any idea why this is not showing up as an error in CT?

I poked around and tried to see if CT was seeding anything, purposefully or accidentally. I couldn't find anything. More investigation to be done in https://github.com/phetsims/phet-io/issues/1799

zepumph · 2021-07-11T18:21:57Z

My guess is that something is broken with this test for NS, and it is breaking something in CT, because this feels unlikely to be probabilistically possible:

zepumph · 2021-07-11T18:41:15Z

I recommend closing this issue after @pixelzoom notes that locally this link will not fail:

http://localhost:8080/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&randomSeed=12345&ea&debugger

And changing the randomSeed to anything else will cause an assertion error.

We will continue the CT trouble over in https://github.com/phetsims/phet-io/issues/1799.

pixelzoom · 2021-07-12T21:37:09Z

And changing the randomSeed to anything else will cause an assertion error.

I can use any number for randomSeed and it does not cause an error. For example http://localhost/~cmalley/GitHub/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&randomSeed=789&ea&debugger

You've also duplicated the API-generation seed "12345" in two places: generatePhetioMacroAPI.js, listContinuousTests.js. This should be factored out, not duplicated in string literals.

Back to @zepumph.

zepumph · 2021-07-13T14:04:49Z

I can use any number for randomSeed and it does not cause an error. For example http://localhost/~cmalley/GitHub/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&randomSeed=789&ea&debugger

With that link I get

I see that I am pushed, pulled, and entirely on master.

You've also duplicated the API-generation seed "12345" in two places: generatePhetioMacroAPI.js, listContinuousTests.js. This should be factored out, not duplicated in string literals.

Yes, it is duplicated in more than just those two spots. I cannot think of a place to factor it out to, see https://github.com/phetsims/phet-io/issues/1798#issuecomment-877845936 for discussion. Conclusion still TBD.

pixelzoom · 2021-07-13T14:18:23Z

Hmm.... It seems to be working as expected now. Maybe I didn't have something pulled, or had a cacheing problem. Sorry for the reassignment.

Closing - because this pertains to master, it does not need to be verified by QA for phetsims/qa#662.

pixelzoom added type:bug Something isn't working dev:phet-io status:blocks-sim-publication labels Jul 7, 2021

pixelzoom assigned samreid and zepumph Jul 7, 2021

pixelzoom mentioned this issue Jul 7, 2021

Add phetioDesigned:true #294

Closed

zepumph added a commit to phetsims/chipper that referenced this issue Jul 7, 2021

add newlines to problem string, phetsims/natural-selection#301

d4a8054

zepumph removed the status:blocks-sim-publication label Jul 7, 2021

samreid removed their assignment Jul 8, 2021

zepumph added the status:ready-for-review label Jul 11, 2021

zepumph assigned pixelzoom and unassigned zepumph Jul 11, 2021

pixelzoom assigned zepumph and unassigned pixelzoom Jul 12, 2021

pixelzoom removed the status:ready-for-review label Jul 12, 2021

zepumph assigned pixelzoom and unassigned zepumph Jul 13, 2021

pixelzoom closed this as completed Jul 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

phetioCompareAPI is failing. #301

phetioCompareAPI is failing. #301

pixelzoom commented Jul 7, 2021 •

edited

Loading

pixelzoom commented Jul 7, 2021

zepumph commented Jul 7, 2021

zepumph commented Jul 7, 2021 •

edited by pixelzoom

Loading

pixelzoom commented Jul 7, 2021

zepumph commented Jul 7, 2021

zepumph commented Jul 7, 2021 •

edited

Loading

zepumph commented Jul 7, 2021

pixelzoom commented Jul 7, 2021

zepumph commented Jul 7, 2021

zepumph commented Jul 11, 2021

zepumph commented Jul 11, 2021

pixelzoom commented Jul 12, 2021 •

edited

Loading

zepumph commented Jul 13, 2021 •

edited by pixelzoom

Loading

pixelzoom commented Jul 13, 2021 •

edited

Loading

phetioCompareAPI is failing. #301

phetioCompareAPI is failing. #301

Comments

pixelzoom commented Jul 7, 2021 • edited Loading

pixelzoom commented Jul 7, 2021

zepumph commented Jul 7, 2021

zepumph commented Jul 7, 2021 • edited by pixelzoom Loading

pixelzoom commented Jul 7, 2021

zepumph commented Jul 7, 2021

zepumph commented Jul 7, 2021 • edited Loading

zepumph commented Jul 7, 2021

pixelzoom commented Jul 7, 2021

zepumph commented Jul 7, 2021

zepumph commented Jul 11, 2021

zepumph commented Jul 11, 2021

pixelzoom commented Jul 12, 2021 • edited Loading

zepumph commented Jul 13, 2021 • edited by pixelzoom Loading

pixelzoom commented Jul 13, 2021 • edited Loading

pixelzoom commented Jul 7, 2021 •

edited

Loading

zepumph commented Jul 7, 2021 •

edited by pixelzoom

Loading

zepumph commented Jul 7, 2021 •

edited

Loading

pixelzoom commented Jul 12, 2021 •

edited

Loading

zepumph commented Jul 13, 2021 •

edited by pixelzoom

Loading

pixelzoom commented Jul 13, 2021 •

edited

Loading