Texcoord unit region support for image sheet-based effects

[ggcrunchy]

This is NOT yet ready to integrate, but the basic features seem to be working, per this test: uv_fixup.zip

For the moment an Android AAR is also available.

---

The basic idea

The frames in an image sheet are normalized to [0, 1] by the time an effect sees them: it's more or less getting x / width and y / height. "Normal" images and rects will only see 0 or 1, and other objects like circles or polygons will have, for instance, x = distance-from-left / width for a given vertex.)

Values like this are fine for just fetching colors out of an image. However, some effects assume a nice [0, 1] range; as an example, calculating a light distance with normal maps. This is very likely to be the issue here and in Anderoth's more general assessment. (Digression: Composite)

With this PR, an effect can request a [0, 1] unit region, alongside the texcoords, for image sheet-based objects. Other objects are mostly already in [0, 1], so the unit coordinates and texcoords are the same.

---

About the test example

The example takes an image sheet from Solar's sample code and runs through all 64 frames.

There are two effects: "effect1" doesn't request a unit region, "effect2" does.

The effects currently render the unit coords as colors. (The commented-out bit instead uses them to draw the full sheet, as a way to confirm they were encoded properly.)

There are seven objects from left to right:

• The sprite itself, without an effect, for reference.
• Another instance of the sprite with "effect1"; the colors are all over the place as the frames change.
• Again with "effect1", plus rect distortion; just a sanity check.
• A circle, i.e. a non-image sheet object, with "effect1"; another sanity check.
• An instance of the sprite with "effect2"; the colors hold steady in the [0, 1] range.
• Another with "effect2", plus rect distortion; colors again hold steady, so the underlying encoding seems robust against some of the needed texcoord shenanigans.
• A circle with "effect2"; sanity check for a non-image sheet object in the presence of the decode logic.

@kan6868 has a use case with normal maps, so hoping to hear word on that too. Also crossing my fingers for a wavy cloth-type one.

---

Usage

To request the unit region, you add wantUnitRegion = true to your graphics.defineEffect() params table.

In a vertex kernel the unit coordinates are available as CoronaLocalTexCoord.

In the fragment kernel it's a macro instead, CoronaLocalTexCoord(texCoord). This will default to texCoord if a unit region wasn't requested. (Maybe it should just error? This was mostly for the next item mentioned...)

With display.setDefault( "isUnitRegionWantedForBuiltinEffects", true ), various builtin effects can opt-in to the feature. (This is for compatibility; however, if this were treated as a bugfix a few simplifications could be made.)

---

Details

When the effect is set up, a few flags are assigned. If an object is about to be drawn, and is seen to both have coordinates originating from an image sheet and to be using such an effect, those coordinates will get run through an encoding; otherwise the values will just pass through.

Basically, the frame is reinterpreted as having a one-time mirrored repeat: values less than 0 are actually -x, or 2 - x when > 1. This is forgiving of any values close to 0 and 1, and is dead simple to decode in the vertex part of the shell. (Another digression.)

The texcoord z defines in the shell weren't being used (I'm guessing somebody got hung up on an #if vs. #ifdef thing), so I hijacked that for the WANT_UNIT_REGION define used by the effect, avoiding the need to rearrange any component strings in the shader source.

---

TODOs

Composites with an image sheet aren't yet tested. (I assumed any image sheet coordinates must come from "paint1", as currently implemented; is this wrong if only "paint2" has an image sheet paint?)

None of the builtins are yet handled, nor obviously the accompanying testing for compatibility (those would have wantUnitRegion = "builtin"). This is in part owing to the "is this a bug that we should outright fix?" question. 🙂

The Vulkan shell still needs doing, but I'm expecting it's fairly straightforward.

---

Gotchas

You can set texcoords outside [0, 1] with meshes, e.g. with setUV(), and I've actually found this useful. I don't see any sane way to make the encoding account for this, so even with compatibility those meshes would "now" break with builtins, and also of course any new effect requesting a unit region. I suspect nobody is really depending on this setup, though.

The [1, 2] range (for encoded values > 1) is a separate floating point range than [0, 1], with a narrowed accuracy. It's possible this would accentuate some right-hand-side "shimmer" problems. I do have a more sophisticated fallback in mind if that ever amounts to anything, but for now would like to avoid it. 😄

---

NOTES

Composite fills

I do have working code for something like walter describes in that link—mentioned a while ago in #dev on Discord—for composites with one image already in place. As he said, it's a bit awkward to find a good fit; that's held it back so far. I'll try to tidy that up in the near future.

Shell transforms

With 3713+, the "shell" used by effects can be modified, although this currently entails knowing about how it's written and doing text substitution.

This is obviously suboptimal—it was an expedient to get those features written at all—especially in light of shell-modifying PRs like this one.

@kan6868 and I have been discussing ideas. Probably the current graphics.defineShellTransform() would be deprecated ("fortunately" it's not in the docs yet, although the tests are out there), and a 2.0 version have a more pinpoint, data-driven kind of interface that would reproduce the default shell if given no input.

Fallback encoding

The texcoords use a float for u and v, on the CPU side, and only the [0, 1] range really matters for most cases. Without breaking the vertex format, those values could be reinterpreted as unsigned short[2] and use normalized 16-bit values to pass in [0, 1] values. Presumably those would be a nice tight fit.

This looks like a lot of work, though. It would involve switching vertex formats (so not as batching-friendly); multiple VAOs if those are in use; consume another attribute; etc. But anyhow, it's there if needed. 🙂