Skip to content

feat: decode tagged struct payloads generically and report undecoded ones - #24

Open
FredZvt wants to merge 3 commits into
wilddogjp:mainfrom
FredZvt:feat/decode-coverage-and-undecoded-reporting
Open

feat: decode tagged struct payloads generically and report undecoded ones#24
FredZvt wants to merge 3 commits into
wilddogjp:mainfrom
FredZvt:feat/decode-coverage-and-undecoded-reporting

Conversation

@FredZvt

@FredZvt FredZvt commented Jul 29, 2026

Copy link
Copy Markdown

Title:

What changed

Two independent changes plus one test fix.

1. Decode any tagged struct payload, not just allowlisted ones

decodeKnownStructFromReader only attempted a tagged-property decode for structs
named in isKnownTaggedStructDecodeCandidate, or whose type string contained
(/game/ or (/engine/. Native C++ struct types matched neither, so a project
struct such as

ArrayProperty(StructProperty(FMyLayer(/Script/MyPlugin)))

returned one opaque rawBase64 blob for the whole array even though every element
was fully tagged inside.

This now attempts the tagged decode for every unknown struct type. Drops
isLikelyTaggedAssetStructType, whose heuristic is subsumed.

No type allowlist is needed because decodeTaggedStructFromReader already
validates and rewinds: the tag stream must parse with zero warnings, EndOffset
must stay within the payload, and a zero-property parse is rejected unless the
type is explicitly allowlisted. A natively serialized payload fails those checks
and falls back to raw bytes exactly as before. Array elements self-terminate, so
decodeTopLevelArrayProperty needs no change.

2. Report undecoded payloads instead of silent rawBase64

A payload that could not be interpreted was preserved as rawBase64 with nothing
saying so. A consumer could not distinguish "this struct decoded" from "this is 44
opaque bytes" without inspecting which keys happened to be present.

New pkg/uasset/undecoded.go adds, on the value: decodeStatus
(undecoded / partial), undecodedReason naming the type and cause, and
undecodedBytes. At the CLI layer, a decodeReport block with counts, a per-type
histogram, and one items entry per payload, plus lines appended to warnings.

decodeReport is emitted only when something failed, so a fully decoded read keeps
its exact previous shape and the absence of the key is the clean signal. Wired into
prop list and exportReadInfo (which also covers blueprint info, metadata).

Annotation runs at the CLI read layer, deliberately not inside
Asset.DecodePropertyValue. pkg/edit consumes decoded maps as an intermediate
representation when rewriting bytes; annotating at the core leaks the marker keys
into property_set and name_ref_remap and changes written output.
TestOperationEquivalence catches it against the UE golden fixtures. Documented in
the DecodePropertyValue doc comment so it is not re-attempted.

3. Windows test fix

buildBPXBinaryForTest wrote the compiled binary with no extension, which Windows
refuses to execute. Six widget tests failed with "executable file not found"
despite a successful build. Appends .exe on runtime.GOOS == "windows".
Test-only.

Why

Reading a DataAsset whose payload is an array of plain project structs is a core
use case, and it returned a base64 blob. The data was fully labeled on disk the
whole time; only the allowlist gate stood in the way.

The reporting half exists because expanding decode coverage does not remove the
undecodable cases, it just changes which ones remain. Those remaining cases need to
be visible: silence is indistinguishable from success.

Measured effect

On a UE 5.8 project, 101 assets, all exports: values preserved as raw bytes drop
from 162 to 132. Every one of the remaining 132 is a struct with
WithSerializer = true (FExpressionInput and the material input derivatives,
FFontData). All plain USTRUCTs now decode.

Compatibility impact

Please treat this as an output-shape change, and decide whether it warrants a
BREAKING CHANGE marker:

  • Values that previously returned {"structType": ..., "rawBase64": ...} may now
    return decoded fields. A consumer keyed on rawBase64 being present will see
    different output.
  • Responses gain decodeReport and extra warnings entries, but only when
    something failed to decode.

No write-path behavior changes. docs/commands.md updated with the new
"Undecoded Payload Reporting" section.

How it was tested

gofmt -l . clean, go vet ./... clean, go test ./....

  • pkg/uasset, pkg/edit, pkg/validate pass. TestOperationEquivalence and
    TestManifestIntegrity in particular, since both would catch write-path drift.
  • New unit tests in pkg/uasset/undecoded_test.go cover annotation of raw and
    partial payloads, non-annotation of decoded values, and nested path collection.
  • Verified the intermediate commit builds and passes on its own, so the history is
    bisectable.
  • internal/cli has 8 failures on my machine, all pre-existing and Windows-only:
    find_assets_parse_recursive, find_summary_parse_recursive,
    blueprint_scan_functions_parse_recursive, and ..._aggregate, per engine root.
    Those fixtures record forward-slash paths while bpx emits filepath
    separators, so they pass only on Linux. Present before this branch; untouched
    here, since normalizing separators would change output for every user. Happy to
    open a separate issue.

Fixture regeneration — needs your call

28 expected_output fixtures (14 per engine root) recorded output predating these
changes: 6 because the decode got deeper, 22 because they now carry the markers and
the report. I regenerated the recorded expected payloads with
scripts/refresh_decode_fixtures (included) and updated their manifest entries.

The .uasset inputs are untouched and the argv are unchanged — only bpx's own
recorded output moved. But TestFieldAccuracy enforces oracle: "ue-fixture"
precisely to keep these UE-verified rather than self-generated, and I have no
UE 5.6/5.7 editor to regenerate them properly.

FredZvt added 3 commits July 29, 2026 13:51
buildBPXBinaryForTest wrote the compiled test binary to a path with no
extension. Windows only executes files carrying an executable extension, so
every test that shells out to that binary failed with "executable file not
found in %PATH%" even though the build itself succeeded.

Append .exe when runtime.GOOS is "windows". Unblocks six widget tests:

- TestBuildWidgetBlueprintEntryIncludesRichTextBlockStyleSetSummary
- TestBuildWidgetBlueprintEntryIncludesRichTextBlockDecoratorSummary
- TestRunBlueprintWidgetWriteRichTextStyleProperties
- TestRunBlueprintWidgetWriteRichTextStyleSetProperty
- TestRunBlueprintWidgetWriteRichTextDecoratorClassesProperty
- TestRunBlueprintWidgetWriteRichTextDecoratorClassesPropertyMultiple

Test-only change; no runtime behavior is affected.
…ones

A USTRUCT without a native Serialize() writes its own property-tag stream
terminated by a None tag. That payload is self-describing on disk, so it can be
read with no reflection and no knowledge of the C++ type.

decodeKnownStructFromReader only exploited that for structs already named in
isKnownTaggedStructDecodeCandidate, or whose type string contained "(/game/" or
"(/engine/" (Blueprint-defined structs). Native C++ struct types matched
neither, so a project struct such as

    ArrayProperty(StructProperty(FMyLayer(/Script/MyPlugin)))

returned one opaque rawBase64 blob for the whole array even though every
element was fully labeled inside.

Attempt the tagged decode for every unknown struct type instead. This is safe
without a type allowlist because decodeTaggedStructFromReader already validates
the stream and rewinds on failure:

- the tag stream must parse with zero warnings
- EndOffset must stay within the payload
- a zero-property parse is rejected unless the type is explicitly allowlisted

A natively serialized payload fails those checks and falls back to raw bytes
exactly as before. Array elements self-terminate, so decodeTopLevelArrayProperty
needs no change; a misparsed element leaves trailing bytes and the whole array
falls back.

Drops isLikelyTaggedAssetStructType, whose "(/game/" and "(/engine/" heuristic
is now subsumed by attempting every type.

Measured on a UE 5.8 project (101 assets, all exports): values preserved as raw
bytes drop from 162 to 132, and every remaining one is a struct with
WithSerializer = true (ExpressionInput and the material input types, FontData).
All plain USTRUCTs now decode.

Refreshes three expected-output fixtures per engine root whose recorded output
was the previous shallower decode of StructVariableDescription. The .uasset
inputs are untouched; only bpx's decode depth changed. scripts/refresh_decode_fixtures
re-records those fixtures and their manifest entries after an intentional
decode-coverage change.
A payload bpx cannot interpret is preserved as a rawBase64 blob. Nothing said
so. A caller reading the JSON could not distinguish "this struct decoded" from
"this struct is 44 opaque bytes" without inspecting which keys happened to be
present, and a summary built from such a value silently omitted whatever was
inside the blob.

Make the gap explicit at two levels.

On the value (pkg/uasset/undecoded.go):

- decodeStatus: "undecoded" for a raw fallback, "partial" when a decoder left
  trailing bytes unconsumed
- undecodedReason: names the struct/array/map/set element type and why it
  failed, e.g. "struct ScalarMaterialInput has no built-in decoder and its
  payload is not tagged-property data (likely a native Serialize())"
- undecodedBytes: size of the preserved payload, or the unconsumed tail

On the response (internal/cli):

- decodeReport with undecoded/partial counts, a per-type histogram, and one
  items entry per payload (path, status, type, reason, bytes)
- a summary line plus one line per item appended to warnings

decodeReport is emitted only when something failed, so a fully decoded read
keeps its exact previous output shape and the absence of the key is the clean
signal. Wired into prop list and exportReadInfo, which also covers
blueprint info and metadata.

Annotation runs at the CLI read layer, deliberately NOT inside
Asset.DecodePropertyValue. pkg/edit consumes decoded maps as an intermediate
representation when rewriting bytes; annotating at the core leaks the marker
keys into property_set and name_ref_remap and changes written output.
TestOperationEquivalence catches it against the UE golden fixtures.

Refreshes the expected-output fixtures whose assets contain undecodable
payloads and therefore now carry the markers and the report.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant