Repository navigation
Commit 93b99f4
LAV-3034: Support REPLACE_INVALID_CHARACTERS on object-store Iceberg (#3797)
* LAV-3034: investigator: use existing byte-preserving VARCHAR path
The blocker is wrong: snowflake_varchar is bytea-backed, and __snowflake$varchar_from_hex already inserts arbitrary bytes (including non-UTF-8) via decode(text, 'hex')::snowflake_varchar; see crates/extension/src/types/varchar/misc_ddl.rs and crates/extension/src/types/casts.rs. This avoids a new storage design.
For Iceberg FALSE/omission, carry invalid UTF-8 through the Parquet scan as raw bytes, render those cells as the existing hex constructor in insert_sql, and keep JSON/SQL text UTF-8. TRUE should substitute U+FFFD during CREATE and refresh. The Arrow StringArray path may reject invalid UTF-8 before render_value; verify this and use a raw Parquet byte path where needed. HEX_ENCODE(varchar) currently extracts &str and will fail on invalid data; use the existing extract_varchar_bytes helper for byte-oriented read functions so the captured 6265FF6F7265 result is observable. Scope any other read behavior to Cloud snapshots; keep ADR 050's last-successful-refresh transaction model.
The prior run's captured test/snapshot were restored, so re-create the Cloud snapshots. Prior Cloud capture reported TRUE=6265EFBFBD6F7265 and FALSE/omission=6265FF6F7265, including ALTER SET/UNSET plus refresh. Invalid CREATE reported 001008: invalid value ['INVALID'] for parameter 'REPLACE_INVALID_CHARACTERS'.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: investigator: capture with Cloud Snowflake and AWS
The failed Standard and Enterprise captures used TEST_TARGET=SNOWFLAKE_CLOUD only. tests/conftest.py:is_aws_cloud checks for AWS_CLOUD in TEST_TARGET; without it _AwsClients uses LOCALSTACK_ENDPOINT_URL. The fixture therefore created sf-axisb-* buckets in LocalStack, and Snowflake returned 091017 (22000) because it cannot reach them. This is a target-selection error, not a missing Snowflake privilege.
Re-create and capture the test with TEST_TARGET=SNOWFLAKE_CLOUD,AWS_CLOUD AWS_DEFAULT_REGION=eu-west-1 PYTEST_XDIST_ARGS="" PYTEST_ARGS="-k test_object_store_iceberg_replace_invalid_characters --snapshot-update" make test-compat PYTEST_PATH=tests/queries/iceberg/test_iceberg_tables.py. For Enterprise also set SNOWFLAKE_ACCOUNT=FS76419 SNOWFLAKE_HOST="$SNOWFLAKE_HOST2". The AWS_CLOUD switch routes S3/IAM/STS through the Spiral proxy and enables the required real AWS fixture. Docs/local-dev-setup.md already gives the combined-target recipe. Recheck Cloud capture and then implement; no production code from the failed round remains.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: Support invalid-character settings for object-store Iceberg
Parse CREATE and ALTER SET/UNSET, persist explicit values, and resolve the
current setting on creation and refresh. Decode malformed Parquet strings
through a byte-preserving path, replacing invalid bytes only when enabled.
Return Cloud's invalid UTF-8 error when an unmodified VARCHAR reaches a
read. Add a Cloud-captured snapshot spanning values, refresh, and metadata.
The fallback rejects scan tasks with deletes, predicates, or nonempty
partition values before publishing rows, since reading raw Parquet files
would otherwise lose those scan semantics. Last successful heap contents
remain intact on a failed refresh.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE TRUE / invalid UTF-8 read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE FALSE / invalid UTF-8 read error -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE omitted / invalid UTF-8 read error -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE TRUE, FALSE, omitted / valid integer, empty, NULL -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE invalid string / 001008 and absent table -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE invalid number / 001008 and absent table -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
ALTER SET TRUE / refresh and replaced read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
ALTER SET FALSE / refresh and invalid UTF-8 error -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
ALTER UNSET / refresh and inherited false error -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
TRUE, FALSE, omitted / SHOW, DESCRIBE, GET_DDL, SHOW PARAMETERS -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
ALTER SET/UNSET / SHOW PARAMETERS -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
Object-store refresh / existing lifecycle behavior -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_refresh
REST and Delta forms -> uncovered: explicitly deferred by LAV-3034
Delete, predicate, nonempty partition scan tasks with invalid UTF-8 -> uncovered: raw fallback refuses these to preserve heap correctness; needs a scan-aware decoder in a follow-up issue
## Deviations
The investigator note expected FALSE and omission to expose raw hex through HEX_ENCODE. Fresh Cloud snapshots show reads fail with 100144 Invalid UTF8; implemented the captured behavior.
The raw Parquet fallback guards delete, predicate, and nonempty partition scan tasks. This limits those Iceberg tables until a scan-aware decoder is available while preserving last-successful-refresh semantics.
* LAV-3034: wip (budget exhausted: wall_clock)
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: investigator: resume partitioned Cloud capture and scan-aware load
Attempt 3 ended by wall_clock during TEST_TARGET=SNOWFLAKE_CLOUD,AWS_CLOUD capture of test_object_store_iceberg_invalid_characters_partitioned. Commit 449ec729a contains the new test, but its snapshot file has no keys for that test. Re-run the capture serially (PYTEST_XDIST_ARGS='') with AWS_DEFAULT_REGION=eu-west-1, then run the focused emulator test. The current materialize_table fallback in crates/extension/src/iceberg/external.rs explicitly rejects tasks with nonempty partitions or deletes; the reviewer made these part of AC 3. Preserve the planned Iceberg scan semantics while decoding malformed string bytes. Cover delete-bearing tasks and failed-refresh heap retention as well. This is unfinished implementation, not a missing permission or external prerequisite.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: wip (terminated)
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: Finalize Cloud snapshot for partitioned invalid UTF-8 scans
The partitioned object-store fixture now has a Cloud-captured baseline for
CREATE and REFRESH with TRUE, FALSE, and inherited behavior, a position
delete, and failed-refresh heap retention. The scan-aware byte fallback and
fixture were committed in the preceding WIP commit. Focused Cloud capture,
focused emulator tests (2 passed), and make check passed. The repository
check reached its broad compat lane without failures and was stopped at 8%
because the diff selects the full suite.
Swept the invalid-byte fallback for scan-task restrictions: the partition
and position-delete cases now load through the planned tasks; the ordinary
unpartitioned and ALTER cases remain covered by the original test.
Implementer waiver: tests/queries/iceberg/test_iceberg_tables.snapshot.json — normalized the volatile generated S3 bucket path in msg_5, msg_7, and msg_9; msg_5 already masks an identifier at tests/queries/iceberg/test_iceberg_tables.snapshot.json:680, and the fixture registers the bucket at tests/queries/iceberg/test_iceberg_tables.py:5813; re-recorded against Cloud.
test_edit_waiver: tests/queries/iceberg/test_iceberg_tables.snapshot.json
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE TRUE / invalid-byte read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE FALSE / invalid-byte read error -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE omitted / invalid-byte read error -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE TRUE, FALSE, omitted / valid, empty, NULL -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE invalid string, number / error and absent table -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
ALTER SET TRUE, FALSE, UNSET / refresh and read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
SHOW, DESCRIBE, GET_DDL, SHOW PARAMETERS / creation and alteration -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
Partitioned CREATE TRUE / invalid-byte and valid reads -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Partitioned CREATE FALSE, omitted / invalid-byte errors and valid reads -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Partitioned REFRESH TRUE, FALSE, omitted / invalid-byte and valid reads -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Position delete TRUE, FALSE, omitted / deleted row absent, remaining read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Failed REFRESH TRUE, FALSE, omitted / last heap retained -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
REST and Delta forms -> uncovered: explicitly deferred by LAV-3034
## Deviations
The earlier WIP snapshot contained a generated S3 bucket name in three
missing-file errors. This Cloud re-record masks the bucket using the
fixture's existing identifier registration; all other values are unchanged.
* LAV-3034: wip (budget exhausted: wall_clock)
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: investigator: finish evolved and equality-delete Cloud coverage
Attempt 4 exhausted wall_clock after the reviewer redesign, while work was advancing. The new vendor/iceberg ArrowReaderBuilder::with_raw_string_bytes path keeps field-ID projection, schema transformation, partitions, and delete filtering. Focused emulator tests test_object_store_iceberg_replace_invalid_characters and test_object_store_iceberg_invalid_characters_partitioned passed after the Parquet metadata override fix; the new reader code has not passed the full post-redesign checks.
Next: capture test_object_store_iceberg_invalid_characters_evolved_schema with TEST_TARGET=SNOWFLAKE_CLOUD,AWS_CLOUD AWS_DEFAULT_REGION=eu-west-1 PYTEST_XDIST_ARGS="" and --snapshot-update, then run it locally. Its test exists at tests/queries/iceberg/test_iceberg_tables.py but its snapshot keys are absent. Re-add separate equality-delete coverage using a verified Iceberg fixture; the prior synthetic equality block was removed from the branch. Do not use deleted_source.scan(...).to_arrow() as fixture validation: PyIceberg rejects all equality deletes at plan_files with ValueError "PyIceberg does not yet support equality deletes". Cloud returned 000603 (XX000) / 300002:994047561 on the synthetic equality metadata, which does not establish Cloud read semantics for a valid delete file. Verify the fixture before attributing that error to this feature. Finish make check, focused compat and Cloud captures before review.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: wip (terminated)
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: investigator: follow Cloud exclusion for equality deletes
Snowflake docs for Iceberg tables exclude row-level equality deletes, and specifically say externally managed tables do not support equality deletes (https://docs.snowflake.com/en/user-guide/tables-iceberg). Dispatch 7 Cloud runs of the branch-added synthetic equality-delete test failed at CREATE on both Standard and Enterprise with 000603 (XX000), Processing aborted due to error 300002:994047561; the Standard valid-UTF-8 control failed identically. This is independent of REPLACE_INVALID_CHARACTERS. The reviewer equality-delete row is contradicted by Cloud, so record it under ## Deviations and do not require a successful equality-delete read. Remove the branch-added positive equality-delete test, which has no Cloud snapshot. Retain the Cloud-captured position-delete and evolved-schema tests; run focused emulator tests and make check, then review. The evolved-schema Cloud snapshot exists in b0006649f; the equality test was added in that same WIP commit.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3034: Remove unsupported equality-delete fixture
Remove the branch-added positive equality-delete fixture. Cloud rejected its CREATE on Standard and Enterprise with 000603 (XX000), including a valid UTF-8 control; Snowflake documents equality deletes as unsupported for externally managed Iceberg tables. Retain the Cloud-captured position-delete and evolved-schema coverage. Focused emulator tests (3 passed) and make check passed.
test_edit_waiver: tests/queries/iceberg/test_iceberg_tables.py
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE TRUE / invalid byte and valid, empty, NULL -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE FALSE, omitted / invalid byte error and valid read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE invalid value / error and absent table -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
ALTER SET TRUE, FALSE, UNSET / refresh and read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
SHOW, DESCRIBE, GET_DDL, SHOW PARAMETERS / setting -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
Partitioned CREATE and REFRESH / TRUE, FALSE, inherited -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Position delete / TRUE, FALSE, inherited -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Failed REFRESH / last successful heap -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Evolved schema / reordered and added columns at CREATE and REFRESH -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_evolved_schema
Equality delete / read -> uncovered: Snowflake Cloud rejects CREATE for this externally managed table form; documented unsupported
REST and Delta forms -> uncovered: deferred by LAV-3034
## Deviations
The reviewer requested equality-delete reads. Snowflake Cloud rejects equality-delete CREATE on both accounts, including a valid UTF-8 control, and documents equality deletes as unsupported for externally managed Iceberg tables. I removed the branch-added positive test under the investigator waiver and kept the supported delete and evolved-schema tests.
* LAV-3034: Record final Iceberg test matrix
The final repository gate passed after merging current main. Focused Iceberg compat tests passed (3 tests); make check passed. The full diff-selected compat lane reached 93% with no reported failure before the gate time limit, and the remaining tests are unverified locally.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE TRUE / invalid byte and valid, empty, NULL -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE FALSE, omitted / invalid byte error and valid read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
CREATE invalid value / error and absent table -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
ALTER SET TRUE, FALSE, UNSET / refresh and read -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
SHOW, DESCRIBE, GET_DDL, SHOW PARAMETERS / setting -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_replace_invalid_characters
Partitioned CREATE and REFRESH / TRUE, FALSE, inherited -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Position delete / TRUE, FALSE, inherited -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Failed REFRESH / last successful heap -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_partitioned
Evolved schema / reordered and added columns at CREATE and REFRESH -> tests/queries/iceberg/test_iceberg_tables.py::test_object_store_iceberg_invalid_characters_evolved_schema
Equality delete / read -> uncovered: Cloud rejects CREATE for this externally managed table form; documented unsupported
REST and Delta forms -> uncovered: deferred by LAV-3034
## Deviations
The reviewer requested equality-delete reads. Snowflake Cloud rejects equality-delete CREATE on both accounts, including a valid UTF-8 control, and documents equality deletes as unsupported for externally managed Iceberg tables. The branch-added positive test was removed under the investigator waiver in a296bf998; supported position-delete and evolved-schema tests remain.
---------
Co-authored-by: spiral <spiral@localhost>
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>1 parent 6faf43f commit 93b99f4
7 files changed
Lines changed: 17 additions & 3 deletions
File tree
- src
- ast
- helpers
- dialect
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4204 | 4204 | | |
4205 | 4205 | | |
4206 | 4206 | | |
| 4207 | + | |
| 4208 | + | |
4207 | 4209 | | |
4208 | 4210 | | |
4209 | 4211 | | |
| |||
4527 | 4529 | | |
4528 | 4530 | | |
4529 | 4531 | | |
| 4532 | + | |
| 4533 | + | |
| 4534 | + | |
| 4535 | + | |
4530 | 4536 | | |
4531 | 4537 | | |
4532 | 4538 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
| 183 | + | |
| 184 | + | |
183 | 185 | | |
184 | 186 | | |
185 | 187 | | |
| |||
291 | 293 | | |
292 | 294 | | |
293 | 295 | | |
| 296 | + | |
294 | 297 | | |
295 | 298 | | |
296 | 299 | | |
| |||
778 | 781 | | |
779 | 782 | | |
780 | 783 | | |
| 784 | + | |
781 | 785 | | |
782 | 786 | | |
783 | 787 | | |
| |||
887 | 891 | | |
888 | 892 | | |
889 | 893 | | |
| 894 | + | |
890 | 895 | | |
891 | 896 | | |
892 | 897 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
782 | 782 | | |
783 | 783 | | |
784 | 784 | | |
| 785 | + | |
785 | 786 | | |
786 | 787 | | |
787 | 788 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2989 | 2989 | | |
2990 | 2990 | | |
2991 | 2991 | | |
2992 | | - | |
2993 | | - | |
2994 | | - | |
| 2992 | + | |
2995 | 2993 | | |
2996 | 2994 | | |
2997 | 2995 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
790 | 790 | | |
791 | 791 | | |
792 | 792 | | |
| 793 | + | |
793 | 794 | | |
794 | 795 | | |
795 | 796 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2026 | 2026 | | |
2027 | 2027 | | |
2028 | 2028 | | |
| 2029 | + | |
2029 | 2030 | | |
2030 | 2031 | | |
2031 | 2032 | | |
| |||
2221 | 2222 | | |
2222 | 2223 | | |
2223 | 2224 | | |
| 2225 | + | |
2224 | 2226 | | |
2225 | 2227 | | |
2226 | 2228 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6745 | 6745 | | |
6746 | 6746 | | |
6747 | 6747 | | |
| 6748 | + | |
6748 | 6749 | | |
6749 | 6750 | | |
6750 | 6751 | | |
| |||
0 commit comments