jupyter realtime sync is not sufficiently robust

[williamstein]

• On Firefox we disabled Jupyter entirely, temporarily (see jupyter python2 notebook may truncate in firefox #1537).

• We have had regular reports from users of issues over the years, and no amount of rewrites or subtle bug fixes has 100% eliminated these.

• Upgrading to new versions of Jupyter is terrifying since the API changes, so things randomly break (for our very brittle use of Jupyter).

• We are massively abusing and monkey patching Jupyter as it is to get sync to work.

The only viable solution I can think of is to completely rewrite Jupyter from scratch, except the backend kernel functionality. Thus this issue is labeled "insanely hard".

[williamstein]

Work happening here: https://github.com/sagemathinc/smc/commits/jupyter

[williamstein]

[edited] My goal is to finish and launch this by April 12.

[jasongrout]

I just noticed this. To clarify, is your goal to write a new Jupyter frontend?

[williamstein]

Yes. Similar to what the nteract project has done, but with different constraints, including (1) realtime sync, (2) compatibility with the official Jupyter UI (so don't change anything significantly unless necessary).

[jasongrout]

Thanks. Did you look at building a notebook component with SMC syncing based on the JupyterLab notebook component? It may be much easier to get sync working in the JupyterLab notebook component - the codebase there is in much better shape, especially for extensibility (still alpha or beta-level, though) - CC @blink1073, @ian-r-rose. See the jlab notebook package: https://github.com/jupyterlab/jupyterlab/tree/master/packages/notebook (this was just factored out of jupyterlab in the last week or so and published as https://www.npmjs.com/package/@jupyterlab/notebook).

Of course, you may still decide (or have already decided) it's worth it to have a react-based in-house Jupyter frontend implementation anyway.

[jasongrout]

much easier to get sync working in the JupyterLab notebook

(much easier than the classic Jupyter notebook...)

[jasongrout]

Also, it would be much appreciated if you would please let us know of any warts or other difficulties you find on our end that we can improve.

[haraldschilly]

Also, it would be much appreciated if you would please let us know of any warts or other difficulties you find on our end that we can improve.

Is there some sort of test suite for the whole range of possible messages in the protocol? The main point would be to check if the front-end is able to properly interpret what's coming in, etc. I don't know that much about the protocol or what to really look after, though.

My very first naive thought about this is to have an artificial test-kernel and the client asks to evaluate strings like test(1), test(2), ...