[INTERRUPTION 1d] `ds.publish()` doesn't work well with ad-hoc interactive use of notebooks

[lukemarsden]

The original issue text was:

calling ds.publish() twice should report the same run twice - the fact that it clears after publish breaks a common Jupyter workflow where you re-run some but not all of your cells.

yes, there's an edge case here where someone runs only some of the cells after restarting the kernel, but this change makes it at least less broken :)

However, Alaric had Principle Objections (see comments below).

An idea to fix this nicely came up and has been recorded in issue #6, but we need to try it and see if it works nicely in practice.

In the meantime, as a temporary fix, we should change the python Library like so:

• Add a ds.start_run([optional description]) that starts a run, setting the description if provided. The implementation actually clears the buffered run state.
• Make ds.publish([optional description]) NOT clear the buffered run state any more.
• Change the documentation to tell users to bracket their runs with ds.start_run()...ds.publish(), so if there's multiple runs, each one is cleared after the previous publish due to the ds.start_run(), thereby isolating them.
• If you start to do things that record metadata before calling ds.start_run(), you get reminded to call ds.start_run() first. This needn't be a fatal error.
• ds.start_run() is really just a clear operation - but by calling it ds.start_run() we gently move users away from thinking about python library internal state and more about cleanly declaring their intentions...

[alaric-dotmesh]

I'm nervous about this change. It depends on the user being somewhat mindful of the (invisible) in-memory state of the Python library to save themselves from outputting a run with half the data missing.

It means that:

• People need to be careful to call clear() after publish() when they have notebooks with multiple runs in them (well, we'd give them a publish_and_clear() but they'd still need to remember to use it) or they'll get smushed together.
• People with multiple runs in the notebook also need to be VERY careful to remember to clear() and re-run all the cells in the new run if they go to run a single cell in a run after having published a different run, or they'll end up with a partial smush of the previous run with a single cell of the new run.
• The user needs to remember if they've removed something from the cell they run - not read a file they previously read, for example - and take some steps (remove_input(filename)?) to remove that from the in-memory state, otherwise it'll still be there. We might be able to fix that somewhat by having an explicit concept of starting and ending a cell in the Python library - call ds.start_cell('name of cell') at the start of a cell and ds.end_cell() at the end, so if a cell is re-run, the Python library can remove the metadata generated by that cell's previous execution.
• People need to make sure that restarting Jupyter, which will implicitly clear the in-memory state, is followed by running all the cells to get the system ready before they do any publish().
• People who go and re-run some cells in their notebook need to remember to go and do the publish() cell as well before hitting save, to make a run of their changes, or they'll be in the notebook and in the files but not in a run and will just end up in a (partial, as it'll be lacking metadata) derived run when Jupyter ends, or merged into the next run as a correction; the plugin/committer integration should help here by presenting warnings of an uncommitted state in the meantime.

I think the best way to make this work would be with ds.start_cell('run name','cell name'), with the name pairs having to be unique within the scope of that ds instance. The python library could keep a list of cells in memory, and when publish() is called, any runs that have cells that have been modified since the last run will get written out. Having to name the run and cell before doing any other calls will ensure that metadata from the previous run of that cell doesn't get kept if the next run fails to overwrite it, and will ensure that all cells end up in the correct runs.

It might help to make the python library interact with the Jupyter plugin and say what state it has - in particular, which runs have changed cells in and need publishing and saving.

It would also be possible to store that transient state of the python library on disk, so it survives Jupyter restarts.

Command runs would never call start_cell at all (perhaps we should require the use of an explicit ds.command_run() flag, so we can warn if neither is called, to prevent mistakes), and would bypass all this stuff.

So, what are the alternatives?

• Encourage people to "run all cells" before saving.
• Encourage people to have one run per cell, with its own publish at the bottom of each cell.

[alaric-dotmesh]

@satsumas can you have a think about this from a user perspective, please? I want to know what the underlying desire really is, so I can suggest a better fix for the problem than this (IMHO) terrible and error-prone hack :-)

[alaric-dotmesh]

Oh, and I've thought of another issue: this could make a mockery of reproducability. Imagine a run has three steps (A, B, C); steps A and B do some data science, with B using the output of step A. Step C writes it all up nicely as a Dotscience run, referencing data in variables assigned to by A and B to find the summary stats.

Now, imagine you change something in step A and run that cell on its own. Satisfied with the result, you re-run step C to publish the updated run. The python library's state contains the result of the original run of A, then of B, then the second run of A. Assuming that the second run of A totally overwrites the data from the old run as we just write new values to the same summary stats, what we save is still "stats from old B, stats from new A" - but if we ran all cells, we'd re-run B with the input from the new A, generating different values. There's nothing recording the fact that the steps required are "old A, B, new A" to get to this state again.

[satsumas]

@alaric-dotmesh sorry, I missed the notification when you first tagged me on this.

If I understand you correctly, you raise at least two problems. First, that a second run might not overwrite all variables set by a first run, so the second call to publish would capture lingering metadata:

The user needs to remember if they've removed something from the cell they run - not read a file they previously read, for example - and take some steps (remove_input(filename)?) to remove that from the in-memory state, otherwise it'll still be there.

And secondly, issues with running cells out of order and then calling publish.

I think the second problem here is something that is just a general, often overlooked, pain when using notebooks. You do need to keep state in your head if you re-run cells out of order, and by varying the order you can get different objects assigned to variables, different outputs and so on. But I don't think it is a problem for Dotscience specifically, and in any case is something Jupyter users will be wary of. There are extensions that try to solve this by specifying cell run orders, but my feeling is that people just tend to run cells from top to bottom or hit Run All.

Regarding the first problem -- it shouldn't crop up too often, given what I think are the main use cases for multiple runs per notebook. There are two use cases that I had in mind for this, and both just involve changing the hyperparameter values (rather than data).

In one case, its handy to load in some training data, train your model (given some param values) and get some validation score, then (new cell) copy the training model code but change the param values, and get new validation scores, and so on.

In a more efficient improvement on this practice, you'd use a method ((like scikit-learn's grid search cross-validation([https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html]) for re-running on all combinations possible of given ranges of given hyperparameters.

For both use cases, you want the input_file to be remembered, so clearing cell outputs between publishings would make more work for the user (I'm not even sure how grid search cross validation would work with Dotscience if the input file wasn't persistent)

What do you think? My thoughts are that on balance, we shouldn't clear outputs between calls to publish.

[alaric-dotmesh]

We have a CUNNING PLAN to break this impasse! It's Luke's suggestion:

• Don't store state in-memory in the python library, because the history of that in-memory state is the dynamic flow of execution of Jupyter cells which may have nothing to do with their order in the notebook, leading to the problems expounded above.
• Instead, every time you call a metadata-registration function like ds.input(), it should output a machine-readable tag at that very point.
• ds.publish() outputs an "end of this run" tag
• the parser (be it notebook or command-output) reads the tags from top to bottom, building up in-memory state in notebook lexical order and outputting a run and clearing its in-memory state at the "end of this run" tag
• Therefore, the assignment of actions to runs is based purely on the lexical structure, not the dynamic structure.
• For extra niceness, in Jupyter mode, we can output the markers inside "Jupyter widgets" that control their display (rather than plain text) so they're less obtrusive and prettier