Move .spy playground files in examples and test all examples

[paugier]

Fixes #416

[paugier]

This is a first step but it might be enough for now. At least, it can be useful to have feedback on this strategy.

[antocuni]

thank you a lot for this!
I generally like the approach and I'm super happy to add this functionality.
I have added some minor comments.

One additional discussion point: have you considered to embed the expected output directly in the example files? Something like this:

def main() -> None:
    print("hello")

# ==== expected output ====
# hello

This would make the update logic slightly more complex but it has the advantage of avoiding creating a lot of new tiny files.

I'm not 100% sure it's a good idea though, happy to hear your thoughts about it

[antocuni]

this is a bit fragile (we should have a better repr for the dict, but that's a different story) because it contains a hardcoded address, which happens to be always the same because malloc on WASM is very deterministic.

If we want to be more robust, we probably want to support some kind of regexp/glob syntax in these "expected" files.

Anyway, not a big deal and not a merge blocker, just something to keep in mind for the future.

[paugier]

Done with a regexp (line = re.sub(r"0x[0-9a-fA-F]+", "<ADDR>", line)).

[antocuni]

could you use spy.util.print_diff to print a readable diff here?

[paugier]

done

[antocuni]

this works, but I propose a different UI.
What about introducing an option pytest --update-examples or similar?

This way:

1. by default, test_example checks the output against the expected output
2. with --update-examples, it SAVES the new expected output

This has some nice properties like the builtin ability to update only a single test by using pytest standard filtering mechanism.

[paugier]

I did that and I agree it's better!

[antocuni]

how is it possible that this test passes in CI? Doesn't it print a different path? 🤔

[paugier]

The tests were not run in the CI because of

 [tool.pytest.ini_options]
testpaths = ["spy"]

I added an explicit step in the CI for the examples.

[paugier]

Note that I didn't change testpaths = ["spy"], meaning that just running pytest in the root directory does not run the examples. It seems to me that it is a reasonable behavior. To run the example tests locally:

pytest examples
# or
cd examples
pytest

[antocuni]

I think it's fine to modify the example to print less numbers here :)

[paugier]

Done

[paugier]

One additional discussion point: have you considered to embed the expected output directly in the example files?

I don't love the idea. I think the advantage (avoiding few small files in examples/expected_output) is not that interesting. This is only one file per examples, and I don't see how these files would cause any issue to anyone.

Then I thought about the "advantage" of having the expected output directly in the examples in terms of pedagogy. However, it is not a very nice way to present the output, in particular when the examples will be used in the doc. We could of course trim this part for the doc but it seems to me that it's going to be hacky.

It is also not unreasonable to have in future examples with relatively long and not very interesting output, which is not a problem if the output is not embedded in the source file.

Therefore, I propose that we keep it simple with standard golden files in examples/expected_output. Would it be fine for you?