feat: decode tagged struct payloads generically and report undecoded ones - #24
Open
FredZvt wants to merge 3 commits into
Open
feat: decode tagged struct payloads generically and report undecoded ones#24FredZvt wants to merge 3 commits into
FredZvt wants to merge 3 commits into
Conversation
buildBPXBinaryForTest wrote the compiled test binary to a path with no extension. Windows only executes files carrying an executable extension, so every test that shells out to that binary failed with "executable file not found in %PATH%" even though the build itself succeeded. Append .exe when runtime.GOOS is "windows". Unblocks six widget tests: - TestBuildWidgetBlueprintEntryIncludesRichTextBlockStyleSetSummary - TestBuildWidgetBlueprintEntryIncludesRichTextBlockDecoratorSummary - TestRunBlueprintWidgetWriteRichTextStyleProperties - TestRunBlueprintWidgetWriteRichTextStyleSetProperty - TestRunBlueprintWidgetWriteRichTextDecoratorClassesProperty - TestRunBlueprintWidgetWriteRichTextDecoratorClassesPropertyMultiple Test-only change; no runtime behavior is affected.
…ones
A USTRUCT without a native Serialize() writes its own property-tag stream
terminated by a None tag. That payload is self-describing on disk, so it can be
read with no reflection and no knowledge of the C++ type.
decodeKnownStructFromReader only exploited that for structs already named in
isKnownTaggedStructDecodeCandidate, or whose type string contained "(/game/" or
"(/engine/" (Blueprint-defined structs). Native C++ struct types matched
neither, so a project struct such as
ArrayProperty(StructProperty(FMyLayer(/Script/MyPlugin)))
returned one opaque rawBase64 blob for the whole array even though every
element was fully labeled inside.
Attempt the tagged decode for every unknown struct type instead. This is safe
without a type allowlist because decodeTaggedStructFromReader already validates
the stream and rewinds on failure:
- the tag stream must parse with zero warnings
- EndOffset must stay within the payload
- a zero-property parse is rejected unless the type is explicitly allowlisted
A natively serialized payload fails those checks and falls back to raw bytes
exactly as before. Array elements self-terminate, so decodeTopLevelArrayProperty
needs no change; a misparsed element leaves trailing bytes and the whole array
falls back.
Drops isLikelyTaggedAssetStructType, whose "(/game/" and "(/engine/" heuristic
is now subsumed by attempting every type.
Measured on a UE 5.8 project (101 assets, all exports): values preserved as raw
bytes drop from 162 to 132, and every remaining one is a struct with
WithSerializer = true (ExpressionInput and the material input types, FontData).
All plain USTRUCTs now decode.
Refreshes three expected-output fixtures per engine root whose recorded output
was the previous shallower decode of StructVariableDescription. The .uasset
inputs are untouched; only bpx's decode depth changed. scripts/refresh_decode_fixtures
re-records those fixtures and their manifest entries after an intentional
decode-coverage change.
A payload bpx cannot interpret is preserved as a rawBase64 blob. Nothing said so. A caller reading the JSON could not distinguish "this struct decoded" from "this struct is 44 opaque bytes" without inspecting which keys happened to be present, and a summary built from such a value silently omitted whatever was inside the blob. Make the gap explicit at two levels. On the value (pkg/uasset/undecoded.go): - decodeStatus: "undecoded" for a raw fallback, "partial" when a decoder left trailing bytes unconsumed - undecodedReason: names the struct/array/map/set element type and why it failed, e.g. "struct ScalarMaterialInput has no built-in decoder and its payload is not tagged-property data (likely a native Serialize())" - undecodedBytes: size of the preserved payload, or the unconsumed tail On the response (internal/cli): - decodeReport with undecoded/partial counts, a per-type histogram, and one items entry per payload (path, status, type, reason, bytes) - a summary line plus one line per item appended to warnings decodeReport is emitted only when something failed, so a fully decoded read keeps its exact previous output shape and the absence of the key is the clean signal. Wired into prop list and exportReadInfo, which also covers blueprint info and metadata. Annotation runs at the CLI read layer, deliberately NOT inside Asset.DecodePropertyValue. pkg/edit consumes decoded maps as an intermediate representation when rewriting bytes; annotating at the core leaks the marker keys into property_set and name_ref_remap and changes written output. TestOperationEquivalence catches it against the UE golden fixtures. Refreshes the expected-output fixtures whose assets contain undecodable payloads and therefore now carry the markers and the report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title:
What changed
Two independent changes plus one test fix.
1. Decode any tagged struct payload, not just allowlisted ones
decodeKnownStructFromReaderonly attempted a tagged-property decode for structsnamed in
isKnownTaggedStructDecodeCandidate, or whose type string contained(/game/or(/engine/. Native C++ struct types matched neither, so a projectstruct such as
ArrayProperty(StructProperty(FMyLayer(/Script/MyPlugin)))
returned one opaque
rawBase64blob for the whole array even though every elementwas fully tagged inside.
This now attempts the tagged decode for every unknown struct type. Drops
isLikelyTaggedAssetStructType, whose heuristic is subsumed.No type allowlist is needed because
decodeTaggedStructFromReaderalreadyvalidates and rewinds: the tag stream must parse with zero warnings,
EndOffsetmust stay within the payload, and a zero-property parse is rejected unless the
type is explicitly allowlisted. A natively serialized payload fails those checks
and falls back to raw bytes exactly as before. Array elements self-terminate, so
decodeTopLevelArrayPropertyneeds no change.2. Report undecoded payloads instead of silent rawBase64
A payload that could not be interpreted was preserved as
rawBase64with nothingsaying so. A consumer could not distinguish "this struct decoded" from "this is 44
opaque bytes" without inspecting which keys happened to be present.
New
pkg/uasset/undecoded.goadds, on the value:decodeStatus(
undecoded/partial),undecodedReasonnaming the type and cause, andundecodedBytes. At the CLI layer, adecodeReportblock with counts, a per-typehistogram, and one
itemsentry per payload, plus lines appended towarnings.decodeReportis emitted only when something failed, so a fully decoded read keepsits exact previous shape and the absence of the key is the clean signal. Wired into
prop listandexportReadInfo(which also coversblueprint info,metadata).Annotation runs at the CLI read layer, deliberately not inside
Asset.DecodePropertyValue.pkg/editconsumes decoded maps as an intermediaterepresentation when rewriting bytes; annotating at the core leaks the marker keys
into
property_setandname_ref_remapand changes written output.TestOperationEquivalencecatches it against the UE golden fixtures. Documented inthe
DecodePropertyValuedoc comment so it is not re-attempted.3. Windows test fix
buildBPXBinaryForTestwrote the compiled binary with no extension, which Windowsrefuses to execute. Six widget tests failed with "executable file not found"
despite a successful build. Appends
.exeonruntime.GOOS == "windows".Test-only.
Why
Reading a DataAsset whose payload is an array of plain project structs is a core
use case, and it returned a base64 blob. The data was fully labeled on disk the
whole time; only the allowlist gate stood in the way.
The reporting half exists because expanding decode coverage does not remove the
undecodable cases, it just changes which ones remain. Those remaining cases need to
be visible: silence is indistinguishable from success.
Measured effect
On a UE 5.8 project, 101 assets, all exports: values preserved as raw bytes drop
from 162 to 132. Every one of the remaining 132 is a struct with
WithSerializer = true(FExpressionInputand the material input derivatives,FFontData). All plain USTRUCTs now decode.Compatibility impact
Please treat this as an output-shape change, and decide whether it warrants a
BREAKING CHANGEmarker:{"structType": ..., "rawBase64": ...}may nowreturn decoded fields. A consumer keyed on
rawBase64being present will seedifferent output.
decodeReportand extrawarningsentries, but only whensomething failed to decode.
No write-path behavior changes.
docs/commands.mdupdated with the new"Undecoded Payload Reporting" section.
How it was tested
gofmt -l .clean,go vet ./...clean,go test ./....pkg/uasset,pkg/edit,pkg/validatepass.TestOperationEquivalenceandTestManifestIntegrityin particular, since both would catch write-path drift.pkg/uasset/undecoded_test.gocover annotation of raw andpartial payloads, non-annotation of decoded values, and nested path collection.
bisectable.
internal/clihas 8 failures on my machine, all pre-existing and Windows-only:find_assets_parse_recursive,find_summary_parse_recursive,blueprint_scan_functions_parse_recursive, and..._aggregate, per engine root.Those fixtures record forward-slash paths while
bpxemitsfilepathseparators, so they pass only on Linux. Present before this branch; untouched
here, since normalizing separators would change output for every user. Happy to
open a separate issue.
Fixture regeneration — needs your call
28
expected_outputfixtures (14 per engine root) recorded output predating thesechanges: 6 because the decode got deeper, 22 because they now carry the markers and
the report. I regenerated the recorded
expectedpayloads withscripts/refresh_decode_fixtures(included) and updated their manifest entries.The
.uassetinputs are untouched and theargvare unchanged — only bpx's ownrecorded output moved. But
TestFieldAccuracyenforcesoracle: "ue-fixture"precisely to keep these UE-verified rather than self-generated, and I have no
UE 5.6/5.7 editor to regenerate them properly.