Jupyter syntax highlighting

[max-sixty]

As a follow-up to #44, I wonder whether it's possible to get syntax highlighting in the jupyter cells.

Currently it inherits from python:

There's a discussion here on this, which links to jupyter-lsp/jupyterlab-lsp#553, which looks good, and not too involved, to get %%bigquery working. They have the advantage that there's already available syntax for SQL. I would hope that we could reuse one of our many existing definitions, but I've said that a few times before about syntax definitions :)

[rbpatt2019]

Perhaps it's just too early and I need more coffee, but the above approach isn't twigging for me. Do we need a separate typescript plugin to handle this?

[max-sixty]

Yes, unfortunately I think this will be in typescript. And I'm also not sure how the typescript file is linked to (does it rely on other code in jupyterlab-lsp??)...

[max-sixty]

I think this would be great to do, I may give this a swing soon. Unfortunately it involve JS, my great weakness...

Another reference: https://twitter.com/Mbussonn/status/636626829269442562 from @Carreau, suggests it should be possible.

And catherinedevlin/ipython-sql#85

[max-sixty]

OK, this is a bigger project than I imagined. Here's the thread for Stan: jupyterlab/jupyterlab#5504. So it's possible, but not easy, and hardly documented.

(and that's for jupyterlab, it's not completely clear that would work for the notebook. And it might be really hard for Colab: googlecolab/colabtools#2039)

It looks like it depends on at least PRQL/prql#1102

It's a shame this is difficult — this could be a first-class PRQL experience — and under our control, rather than relying on another tool. Without syntax highlighting, it's going to be difficult to build the momentum to add auto-complete / LSP / etc here.

So if anyone would be up for making some contributions here — e.g. the Lezer grammar, or a better understanding of what's required overall — that would be very welcome.

[max-sixty]

The Lezer grammar is coming along (thanks to @vanillajonathan ) — including packaging & tests.

So I'll take another look at doing this, despite my very poor JS. (unless anyone else would like to take a swing...)

Here is the JupySQL version, which has recently been added: https://github.com/ploomber/jupysql-plugin/blob/1e058e7f93d3085b7a7c0eab4f5c4864000b44eb/src/editor/editor.ts

And another, from a while ago: https://github.com/WardBrian/jupyterlab-stan-highlight/blob/master/lib/stan.js