Where should the helper functions go in an exported Python script?

[kz930]

When a workflow is exported to Python, the script defines every helper function it calls before the workflow code starts. These helpers handle things like number parsing, type casts and null handling. For a small workflow the helpers take up most of the file. With the export series in #8325 applied, a CSV source followed by one Filter exports as 93 lines. 72 of them are seven helper functions, and the workflow itself is 16 lines at the bottom.

I would like to agree on a layout before going further. These are the options I see.

A. Keep everything in one script (current). The script is self-contained and runs with nothing extra. The cost is length, and the workflow logic is hard to find.

The export modal for a CSV, Filter, Sort, Limit and Bar Chart workflow. The first screen is all helper code.

B. Download a zip with a helper module. The zip holds script.py and texera_support.py. The script starts with from texera_support import ... and only names the helpers it uses. The module comes from the same Texera that wrote the script, so the two always match, and nothing needs to be installed. The cost is a second file, so a single copy and paste of the script no longer runs. There is a draft prototype in #9240. Below is the same workflow with it, and the button left of copy downloads the zip.

C. Publish a PyPI package. The script starts with pip install texera-support and imports from it. This gives the shortest script. It needs an Apache release to publish from, a PPMC vote, and a decision on the package name. Each script would also need the package version that matches the Texera that exported it.

B and C are compatible. A script written for B keeps the same import line if the package is published later, and only the zip step goes away.

#9240 stays a draft until the export series in #8325 has merged. The outcome here decides whether it moves forward or gets closed.

I'd appreciate hearing everyone's thoughts on these options. Any feedback or suggestions are welcome. Thank you!

[chenlica]

My thoughts:

1. No, as the UDF is getting unnecessarily long.
2. Yes. It enables a good user experience. A minor comment: we can find a name better than "texera_support.py".
3. No. "pip install" requires network access, which may not be available due to security reasons.

[kz930]

Thank you for your feedback! I would think of a better file name for Option B.

[zyratlo]

I prefer option B. Maybe something like texera_helpers.py as the name is better instead.

My only concern is the copy button. With B the modal still offers copy, but the copied script won't run without the zip, so someone pasting it into a notebook gets an ImportError and has to figure out why. Could copy give the self-contained version instead, so copy always runs and download gives you the cleaner two-file layout?

Related to that, if the problem is that the workflow is hard to find, there's another option that keeps one file. Wrap the workflow code in a main() function at the top and put the helpers below it with if __name__ == "__main__": main() at the end. This way the workflow ends up at the top of the file and the script still runs on its own. That could be the copy version, and B stays the download, but then the copy and download give different formats which might not be ideal.

[kz930]

That's a good point. I was initially thinking that once users have downloaded the ZIP with texera_helpers.py, they could simply copy the workflow code for future use without needing all the helpers again. But I can also see how this might be confusing for users who just want to copy and run the code directly without downloading anything first. I'm not sure which behavior would be more intuitive.

[zyratlo]

Here are the options I can think of for Copy:

1. Keep copy, and it gives the script without the helpers.
   Copy gives the same script as in the ZIP, so it won't run on its own. This works for the case you mentioned where someone already has texera_helpers.py, but for anyone exporting for the first time, copy is a dead end. They'd have to download the ZIP anyway, so they might as well have downloaded in the first place.

2. Keep copy, and include the helpers so it runs out of the box.
   This could use the main() layout from my earlier comment, or something else. The user can run a copied script right away, but the format is different from the download.

3. Remove copy and only allow download.
   This avoids both problems (different formats, or a copied script that doesn't run), but users can no longer paste the script straight into a notebook or an existing file.

I personally support 2. I think it's fine for copy and download to give different formats, since they're for different uses, but I would like to hear what other people think too.

[kunwp1]

I also prefer B. Having a very long Python script can feel overwhelming, especially if it includes auto-generated code that the user didn't write.