Skip to content

Added support for passing list of parquet files in insert. #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
subkanthi wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Added support for passing list of parquet files in insert. #77

subkanthi wants to merge 1 commit into from

Conversation

@subkanthi
Copy link
Collaborator

@subkanthi subkanthi commented Nov 18, 2025

closes: #73

@subkanthi subkanthi linked an issue Nov 18, 2025 that may be closed by this pull request
Add bulk insert feature #73
Open
@subkanthi
Copy link
Collaborator Author

subkanthi commented Dec 11, 2025 

insert iris.flowers --from-file=filelist --thread-count=2    
2025-12-11 12:48:07 [61304908-37] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
2025-12-11 12:48:07 [61304908-31] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/iris/tables/flowers
2025-12-11 12:48:07 [61304908-34] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/iris/tables/flowers
2025-12-11 12:48:07 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: processing
2025-12-11 12:48:07 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: processing
2025-12-11 12:48:08 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: copying to s3://bucket1/iris/flowers/data/1765478887832-f075b667730a0938354e59ab3f148503e15a114e8f227ef675719bb63b17dcd1.parquet
2025-12-11 12:48:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: copying to s3://bucket1/iris/flowers/data/1765478887832-31e515659a185bbdc820b3b19c90384ca5a77530f265cdaa2313b8e0c6f9bd80.parquet
2025-12-11 12:48:08 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: adding data file (copy took 0s)
2025-12-11 12:48:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: adding data file (copy took 0s)
2025-12-11 12:48:08 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 4069011466165178630 (MergeAppend)
2025-12-11 12:48:08 [61304908-31] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous POST v1/namespaces/iris/tables/flowers/metrics
2025-12-11 12:48:08 [61304908-34] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/iris/tables/flowers
2025-12-11 12:48:08 [61304908-31] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous POST v1/namespaces/iris/tables/flowers

@subkanthi
Copy link
Collaborator Author

subkanthi commented Dec 11, 2025 

insert iris.flowers --from-file=filelist --thread-count=3
2025-12-11 12:50:58 [61304908-33] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
2025-12-11 12:50:58 [61304908-50] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/iris/tables/flowers
2025-12-11 12:50:58 [61304908-32] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/iris/tables/flowers
2025-12-11 12:50:58 [61304908-50] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous POST v1/namespaces/iris/tables/flowers/metrics
2025-12-11 12:50:58 [-6-thread-3] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: processing
2025-12-11 12:50:58 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: processing
2025-12-11 12:50:58 [-6-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris3.parquet: processing
2025-12-11 12:50:58 [-6-thread-3] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: copying to s3://bucket1/iris/flowers/data/1765479058769-f075b667730a0938354e59ab3f148503e15a114e8f227ef675719bb63b17dcd1.parquet
2025-12-11 12:50:58 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: copying to s3://bucket1/iris/flowers/data/1765479058769-31e515659a185bbdc820b3b19c90384ca5a77530f265cdaa2313b8e0c6f9bd80.parquet
2025-12-11 12:50:59 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: adding data file (copy took 0s)
2025-12-11 12:50:59 [-6-thread-3] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: adding data file (copy took 0s)
2025-12-11 12:50:59 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
	at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:227)
	at com.altinity.ice.cli.Main.insert(Main.java:432)
	at com.altinity.ice.cli.Main.lambda$main$1(Main.java:663) [9 skipped]
	at com.altinity.ice.cli.Main.main(Main.java:670) [1 skipped]
Caused by: java.io.IOException: Error processing file://iris3.parquet
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$2(Insert.java:212)
 [4 skipped]
Caused by: org.apache.iceberg.exceptions.NotFoundException: Failed to read file: iris3.parquet
	at org.apache.iceberg.Files$LocalInputFile.newStream(Files.java:125)
	at com.altinity.ice.cli.internal.iceberg.parquet.Metadata.read(Metadata.java:36)
	at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:392)
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$2(Insert.java:190)
	... 4 common frames omitted
Caused by: java.io.FileNotFoundException: iris3.parquet (No such file or directory)
	at org.apache.iceberg.Files$LocalInputFile.newStream(Files.java:123) [4 skipped]
	at com.altinity.ice.cli.internal.iceberg.parquet.Metadata.read(Metadata.java:36)
	at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:392)
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$2(Insert.java:190)
 [4 skipped]
	... 7 common frames omitted
(devbox) scratch % echo $?
1

@subkanthi
Copy link
Collaborator Author

subkanthi commented Dec 11, 2025 

insert nyc.taxis --from-file=../../filelist_taxis --thread-count=3
2025-12-11 12:59:34 [61304908-50] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
2025-12-11 12:59:34 [61304908-59] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/nyc/tables/taxis
2025-12-11 12:59:34 [61304908-33] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/nyc/tables/taxis
2025-12-11 12:59:34 [-5-thread-3] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-03.parquet: processing
2025-12-11 12:59:34 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet: processing
2025-12-11 12:59:34 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-02.parquet: processing
2025-12-11 12:59:48 [-5-thread-3] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-03.parquet: copying to s3://bucket1/nyc/taxis/data/1765479574233-5d56cf5214c2c9c9a1d163202e67d3107334dd488cb257ec51f8c28302526e1c.parquet
2025-12-11 12:59:48 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet: copying to s3://bucket1/nyc/taxis/data/1765479574233-5526da589bc474f953669f343505ec94f0074bfc701eb046184de439bfe0485f.parquet
2025-12-11 12:59:52 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-02.parquet: copying to s3://bucket1/nyc/taxis/data/1765479574233-55216470e6c7eb20a5ab2a3a33291334e2d0d5ed0103c09fab0ba3e7a0b217da.parquet
2025-12-11 12:59:54 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet: adding data file (copy took 6s)
2025-12-11 12:59:55 [-5-thread-3] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-03.parquet: adding data file (copy took 8s)
2025-12-11 12:59:58 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-02.parquet: adding data file (copy took 6s)
2025-12-11 12:59:58 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 5332098937313335168 (MergeAppend)
2025-12-11 12:59:58 [61304908-59] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous POST v1/namespaces/nyc/tables/taxis/metrics
2025-12-11 12:59:58 [61304908-33] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/nyc/tables/taxis
2025-12-11 12:59:58 [61304908-59] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous POST v1/namespaces/nyc/tables/taxis

filelist_taxis

https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-02.parquet
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-03.parquet

@subkanthi subkanthi requested a review from shyiko December 15, 2025 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shyiko shyiko Awaiting requested review from shyiko

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add bulk insert feature

2 participants

@subkanthi