Re-run notebooks that don't need human attention

[michaelosthege]

I'm using the following script, located in pymc-examples/scripts and accompanied by todo.txt, error.txt and success.txt.
The todo.txt has a line with the absolute path to each notebook for which there is currently no other open pull request.

import pathlib
import rerun
import logging


logging.basicConfig(level=logging.INFO)
_log = logging.getLogger(__file__)


DP_SCRIPTS = pathlib.Path(__file__).absolute().parent
DP_REPO = DP_SCRIPTS.parent
FP_TODO = DP_SCRIPTS / "todo.txt"
FP_SUCCESS = DP_SCRIPTS / "success.txt"
FP_ERROR = DP_SCRIPTS / "error.txt"


def run(fp: pathlib.Path) -> bool:
    success = True
    success = success and rerun.apply_replacements(fp)
    success = success and rerun.run_precommit(fp)
    success = success and rerun.execute_notebook(fp)
    success = success and rerun.run_precommit(fp)
    success = success and rerun.commit(fp, "rerun-quickies")
    success = success and rerun.push("rerun-quickies", "mine")

    if success:
        _log.info("✔ All steps succeeded.")
        return True
    else:
        _log.error("❌ Manual investigation needed.")
        return False


if __name__ == "__main__":
    for fp in FP_TODO.open("r").read().split("\n"):
        already = {
            *FP_SUCCESS.open("r").read().split("\n"),
            *FP_ERROR.open("r").read().split("\n"),
        }
        if fp in already:
            _log.info("Skipping %s because it was already processed.", fp)
            continue

        if run(pathlib.Path(fp)):
            FP_SUCCESS.open("a").write(str(fp) + "\n")
        else:
            FP_ERROR.open("a").write(str(fp) + "\n")

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:11Z
----------------------------------------------------------------

We need to change one of the scripts I think so that the pymc3 tags do not get added, I did that with a replace also inside of .py files.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:12Z
----------------------------------------------------------------

The deep learning link won't work.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:13Z
----------------------------------------------------------------

Also has deeplearning.net links.

michaelosthege commented on 2022-01-09T15:57:31Z
----------------------------------------------------------------

True, but out of scope. Broken Hyperlinks on Website · Issue #165 · pymc-devs/pymc-examples (github.com)

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:13Z
----------------------------------------------------------------

That didn't work.

michaelosthege commented on 2022-01-09T15:54:30Z
----------------------------------------------------------------

I have no idea why it was committed then. It must have had a 0 exit code.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:15Z
----------------------------------------------------------------

Maybe use ConstantData.

michaelosthege commented on 2022-01-09T16:01:11Z
----------------------------------------------------------------

ConstantData was not part of 4.0.0b1 yet.

Expecting that we'll have to do more automated re-runs in the near future, I'd like to prioritize determining which notebooks (don't) need manual inspection..

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:17Z
----------------------------------------------------------------

pm.ConstantData()

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:18Z
----------------------------------------------------------------

pm.ConstantData

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2022-01-09T15:39:18Z
----------------------------------------------------------------

pm.ConstantData()

[michaelosthege]

I have no idea why it was committed then. It must have had a 0 exit code.

---

View entire conversation on ReviewNB

[michaelosthege]

True, but out of scope. Broken Hyperlinks on Website · Issue #165 · pymc-devs/pymc-examples (github.com)

---

View entire conversation on ReviewNB

[michaelosthege]

ConstantData was not part of 4.0.0b1 yet.

Expecting that we'll have to do more automated re-runs in the near future, I'd like to prioritize determining which notebooks (don't) need manual inspection..

---

View entire conversation on ReviewNB

[michaelosthege]

Ready for review.
All other notebooks resulted in errors of varying severity.

[OriolAbril]

Thanks, this is even worse than I expected 😅 and I thought that getting a 50% auto updated would be a huge success. We landed at 10/95

Related to #56 #217 #214 #80 #113 #90 #120 #121 #268

[OriolAbril]

will merge in a few minutes after merging some other PRs and releasing a snapshot

[michaelosthege]

Many also failed because I didn't have Bambi installed.
Also there might be a few more OK ones with the fixed pre-commit now. (I didn't restart from the beginning.)

Let's make a list of what we need to do manually - some of that might be candidates for the migration script, or for reversing unnecessary breaks at the API level.