Conversation
e-kotov
marked this pull request as ready for review
September 12, 2026 13:30
Collaborator
Author
|
Hi @Cidree ! This is a quick fix to further improve compatibility with https://github.com/r-spatial/geoarrowDeckglLayers and https://github.com/r-spatial/geoarrowWidget cc @tim-salabim, PR to geoarrowDeckglLayers soon too |
|
r-spatial/deckglgeoarrow#19 merged. |
`as_nanoarrow_array_stream(native = TRUE)` rebuilt the result with `as.list(tab)` and `arrow::record_batch()`. `as.list()` on an Arrow table converts every column to an R vector, and rebuilding from those re-infers the types, so a DuckDB BIGINT came back as int32 whenever its values happened to fit in 32 bits: largest id value source emitted 1,000 BIGINT int32 2,000,001,000 BIGINT int32 3,000,001,000 BIGINT int64 The emitted schema therefore depended on the data rather than the table: the same query over different rows could produce a different column type. Consumers that key on the schema (browser code, caches, tests) could see a column change type for no visible reason. Replace only the geometry column and keep every other column as an Arrow array, which also removes an unnecessary Arrow -> R -> Arrow round trip for all non-geometry columns. Verified unchanged for inputs without int64 columns: the nc fixture emits an identical schema, the same geoarrow.multipolygon extension and the same 1924-character CRS metadata before and after. Adds a regression test that fails before this change and passes after. Full suite: exit 0, no failures, no new warnings.
`native = TRUE` combined the WKB column into a single Arrow array with `as_arrow_array()` while every other column kept the chunk layout Arrow produced when fetching (1e6 rows per chunk). Each record batch after the first therefore held a geometry child that was a slice with a nonzero offset, and the Arrow IPC writer rejects those: ArrowIpcWriterWriteArrayStream() failed: Cannot encode arrays with nonzero offset So any result over one chunk could not be serialised to Arrow IPC at all: rows before after 10,000 ok ok 500,000 ok ok 2,000,000 IPC error ok 5,000,000 IPC error ok (120,626,848 bytes) Convert the WKB column one chunk at a time and rebuild it as a ChunkedArray, so its chunk boundaries match the other columns and every batch starts at offset 0. Adds a regression test (skipped on CRAN, 2e6 rows) that fails before this change with the error above and passes after. Full suite: exit 0, no failures, no new warnings.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #121.
Summary
nanoarrow::as_nanoarrow_array_stream(..., native = TRUE)so WKB geometry is converted to native GeoArrow layouts instead of remaininggeoarrow.wkb.geoarrow.point,geoarrow.linestring, andgeoarrow.multipolygon.Verification
Rscript -e 'devtools::test(filter = "^nanoarrow_methods$")': 17 passed.\n-Rscript -e 'devtools::test()': 3,033 passed, 0 failed, 3 skipped.