feat: Add iceberg partition transform evaluator and utility #15440

PingLiuPing · 2025-11-07T15:20:07Z

Introduces infrastructure for evaluating Iceberg partition transforms in Velox.
Added TransformExprBuilder which converts Iceberg partition specifications into Velox expressions
Added TransformEvaluator which evaluates multiple transform expressions in a single pass using compiled ExprSet. It automatically register Iceberg function in first call.
Added IcebergPartitionPath which converts partition keys to their string representation for directory paths following Iceberg specification (handles dates, timestamps, binary data encoding)
Added test cases to cover the transform evaluation result.
Update iceberg CMake file to link Iceberg functions library.

netlify · 2025-11-07T15:20:13Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`8047d9b`
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/690fa7794b77fa000894725d

PingLiuPing · 2025-11-07T15:30:09Z

@mbasmanova This PR continues from #15035 (review).
In next PR, I plan to include IcebergPartitionIdGenerator and do some refactor on Hive PartitionIdGenerator, I roughly count the lines which should be also around 700 lines. After that, all the preliminary work should be complete, and I’ll submit another PR to integrate them with IcebergDataSink.

mbasmanova · 2025-11-07T17:18:36Z

Ci is red

mbasmanova · 2025-11-07T17:20:22Z

velox/connectors/hive/iceberg/IcebergPartitionPath.h

+
+/// Converts a partition value to its string representation for use in
+/// partition directory path. The format follows the Iceberg specification
+/// for partition path encoding.


Do you have a link to the spec? Would be nice to add here.

Thanks for the comment.
I cannot find an explicit document about this. I will revise the comment.
I followed the Iceberg java code to implement this.

mbasmanova · 2025-11-07T17:22:50Z

velox/connectors/hive/iceberg/IcebergPartitionPath.h

+  explicit IcebergPartitionPath(TransformType transformType)
+      : transformType_(transformType) {}
+
+  ~IcebergPartitionPath() override = default;


mbasmanova · 2025-11-07T17:23:32Z

velox/connectors/hive/iceberg/IcebergPartitionPath.h

+
+  using HivePartitionUtil::toPartitionString;
+
+  std::string toPartitionString(int32_t value, const TypePtr& type)


perhaps, document the format of the result

mbasmanova · 2025-11-07T17:25:13Z

velox/connectors/hive/iceberg/IcebergPartitionPath.h

+/// Converts a partition value to its string representation for use in
+/// partition directory path. The format follows the Iceberg specification
+/// for partition path encoding.
+class IcebergPartitionPath : public HivePartitionUtil {


Would it be possible to extract this class into a separate PR and add a test?

Thank you for reviewing.

I will separate this to a new PR.

mbasmanova · 2025-11-07T17:25:59Z

velox/connectors/hive/iceberg/TransformExprBuilder.h

+  /// type, and optional parameter (e.g., bucket count, truncate width).
+  /// @param inputFieldName Name of the source column in the input RowVector.
+  /// @return Typed expression representing the transform.
+  static core::TypedExprPtr toExpression(


Do we need to expose this API?

Thanks. I moved it to anonymous namespace.

mbasmanova · 2025-11-07T17:27:32Z

velox/connectors/hive/iceberg/TransformEvaluator.cpp

+
+namespace {
+
+constexpr char const* kIcebergFunctionPrefix{"iceberg_"};


move this next to its only use

mbasmanova · 2025-11-07T17:28:21Z

velox/connectors/hive/iceberg/tests/TransformTest.cpp

+class TransformTest : public test::IcebergTestBase {
+ protected:
+  template <typename T>
+  VectorPtr createFlatVector(


Why not use VectorMaker or helper methods from VectorTestBase?

Thanks. Changed to VectorMaker

mbasmanova · 2025-11-07T17:29:43Z

velox/connectors/hive/iceberg/tests/TransformTest.cpp

+
+    // Create expected vector and verify results.
+    auto expectedVector = createFlatVector<OUT>(expectedValues);
+    ASSERT_EQ(resultVector[0]->size(), expectedValues.size());


use assertEqualVectors or similar from VectorTestBase.h

PingLiuPing · 2025-11-07T19:48:17Z

velox/connectors/hive/iceberg/PartitionSpec.h

        case TransformType::kTruncate:
          return type;
      }
+      VELOX_UNREACHABLE("Unknown transform type");


@mbasmanova Have to add this to fix the building break on Linux. Macos is fine without this change.

mbasmanova · 2025-11-07T21:20:29Z

velox/connectors/hive/iceberg/tests/TransformTest.cpp

+      const std::vector<std::optional<IN>>& inputValues,
+      const std::vector<std::optional<OUT>>& expectedValues,


Can we replace these arguments with RowVectorPtr and use makeFlatVector and similar in the callers?

Thanks. Just updated.

mbasmanova · 2025-11-07T22:54:04Z

velox/connectors/hive/iceberg/tests/TransformTest.cpp

+ protected:
+  void testTransform(
+      const IcebergPartitionSpec::Field& field,
+      const RowVectorPtr& inputRowVector,


nit: inputRowVector -> input, expectedRowVector -> expected

mbasmanova · 2025-11-07T22:54:17Z

velox/connectors/hive/iceberg/tests/TransformTest.cpp

+    auto transformExprs = TransformExprBuilder::toExpressions(
+        spec,
+        std::vector<column_index_t>{0},
+        asRowType(inputRowVector->type()));


input->rowType()

mbasmanova · 2025-11-07T22:54:39Z

velox/connectors/hive/iceberg/tests/TransformTest.cpp

+      const RowVectorPtr& inputRowVector,
+      const RowVectorPtr& expectedRowVector) {
+    // Build and evaluate transform expressions.
+    auto spec = std::make_shared<const IcebergPartitionSpec>(


Why do we test single field specs only? Why not test multi-field specs?

Thank you for the comment.
I added those test one by one when I implement the transformation logic one by one, and type by type.
I will refactor it to support test multiple transforms.

mbasmanova · 2025-11-07T22:56:25Z

velox/connectors/hive/iceberg/tests/TransformTest.cpp

+          {makeFlatVector<StringView>(
+              {StringView("\x01\x02\x03", 3), StringView("", 0)},
+              VARBINARY())}),
+      makeRowVector({makeFlatVector<StringView>(


drop StringView or use std::string

makeFlatVector<std::string>({"foo", "bar"})

PingLiuPing requested a review from majetideepak as a code owner November 7, 2025 15:20

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 7, 2025

PingLiuPing requested review from mbasmanova and removed request for majetideepak November 7, 2025 15:20

mbasmanova reviewed Nov 7, 2025

View reviewed changes

PingLiuPing force-pushed the lp_iceberg_transform_eval branch from ff2e9e0 to ae18463 Compare November 7, 2025 19:47

PingLiuPing commented Nov 7, 2025

View reviewed changes

mbasmanova reviewed Nov 7, 2025

View reviewed changes

PingLiuPing force-pushed the lp_iceberg_transform_eval branch from ae18463 to d007247 Compare November 7, 2025 22:15

mbasmanova reviewed Nov 7, 2025

View reviewed changes

Add iceberg partition transform evaluator and utility

8047d9b

PingLiuPing force-pushed the lp_iceberg_transform_eval branch from d007247 to 8047d9b Compare November 8, 2025 20:26


		using HivePartitionUtil::toPartitionString;

		std::string toPartitionString(int32_t value, const TypePtr& type)


		namespace {

		constexpr char const* kIcebergFunctionPrefix{"iceberg_"};

		const std::vector<std::optional<IN>>& inputValues,
		const std::vector<std::optional<OUT>>& expectedValues,

feat: Add iceberg partition transform evaluator and utility #15440

Are you sure you want to change the base?

feat: Add iceberg partition transform evaluator and utility #15440

Uh oh!

Conversation

PingLiuPing commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for meta-velox canceled.

Uh oh!

PingLiuPing commented Nov 7, 2025

Uh oh!

mbasmanova commented Nov 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbasmanova Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PingLiuPing Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PingLiuPing commented Nov 7, 2025 •

edited

Loading

netlify bot commented Nov 7, 2025 •

edited

Loading

mbasmanova Nov 7, 2025 •

edited

Loading

PingLiuPing Nov 8, 2025 •

edited

Loading