feat: Support table format: Iceberg, Delta, and Hudi (#5650)

* add support for table format such as Iceberg, Delta, Hudi etc.

Signed-off-by: HaoXuAI <[email protected]>

* linting

Signed-off-by: HaoXuAI <[email protected]>

* linting

Signed-off-by: HaoXuAI <[email protected]>

* add tests

Signed-off-by: HaoXuAI <[email protected]>

* fix tests

Signed-off-by: HaoXuAI <[email protected]>

* fix tests

Signed-off-by: HaoXuAI <[email protected]>

* linting

Signed-off-by: HaoXuAI <[email protected]>

* add tableformat proto

Signed-off-by: hao-xu5 <[email protected]>

* update

Signed-off-by: hao-xu5 <[email protected]>

* update doc

Signed-off-by: hao-xu5 <[email protected]>

* fix linting

Signed-off-by: hao-xu5 <[email protected]>

* fix test

Signed-off-by: hao-xu5 <[email protected]>

---------

Signed-off-by: HaoXuAI <[email protected]>
Signed-off-by: hao-xu5 <[email protected]>
Co-authored-by: hao-xu5 <[email protected]>

Author: HaoXuAI

docs/SUMMARY.md

@@ -82,6 +82,7 @@
 * [Type System](reference/type-system.md)
 * [Data sources](reference/data-sources/README.md)
   * [Overview](reference/data-sources/overview.md)
+  * [Table formats](reference/data-sources/table-formats.md)
   * [File](reference/data-sources/file.md)
   * [Snowflake](reference/data-sources/snowflake.md)
   * [BigQuery](reference/data-sources/bigquery.md)

docs/reference/data-sources/spark.md

@@ -4,13 +4,17 @@
 
 Spark data sources are tables or files that can be loaded from some Spark store (e.g. Hive or in-memory). They can also be specified by a SQL query.
 
+**New in Feast:** SparkSource now supports advanced table formats including **Apache Iceberg**, **Delta Lake**, and **Apache Hudi**, enabling ACID transactions, time travel, and schema evolution capabilities. See the [Table Formats guide](table-formats.md) for detailed documentation.
+
 ## Disclaimer
 
 The Spark data source does not achieve full test coverage.
 Please do not assume complete stability.
 
 ## Examples
 
+### Basic Examples
+
 Using a table reference from SparkSession (for example, either in-memory or a Hive Metastore):
 
 ```python
@@ -51,8 +55,77 @@ my_spark_source = SparkSource(
 )
 ```
 
+### Table Format Examples
+
+SparkSource supports advanced table formats for modern data lakehouse architectures. For detailed documentation, configuration options, and best practices, see the **[Table Formats guide](table-formats.md)**.
+
+#### Apache Iceberg
+
+```python
+from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource
+from feast.table_format import IcebergFormat
+
+iceberg_format = IcebergFormat(
+    catalog="my_catalog",
+    namespace="my_database"
+)
+
+my_spark_source = SparkSource(
+    name="user_features",
+    path="my_catalog.my_database.user_table",
+    table_format=iceberg_format,
+    timestamp_field="event_timestamp"
+)
+```
+
+#### Delta Lake
+
+```python
+from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource
+from feast.table_format import DeltaFormat
+
+delta_format = DeltaFormat()
+
+my_spark_source = SparkSource(
+    name="transaction_features",
+    path="s3://my-bucket/delta-tables/transactions",
+    table_format=delta_format,
+    timestamp_field="transaction_timestamp"
+)
+```
+
+#### Apache Hudi
+
+```python
+from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource
+from feast.table_format import HudiFormat
+
+hudi_format = HudiFormat(
+    table_type="COPY_ON_WRITE",
+    record_key="user_id",
+    precombine_field="updated_at"
+)
+
+my_spark_source = SparkSource(
+    name="user_profiles",
+    path="s3://my-bucket/hudi-tables/user_profiles",
+    table_format=hudi_format,
+    timestamp_field="event_timestamp"
+)
+```
+
+For advanced configuration including time travel, incremental queries, and performance tuning, see the **[Table Formats guide](table-formats.md)**.
+
+## Configuration Options
+
 The full set of configuration options is available [here](https://rtd.feast.dev/en/master/#feast.infra.offline_stores.contrib.spark_offline_store.spark_source.SparkSource).
 
+### Table Format Options
+
+- **IcebergFormat**: See [Table Formats - Iceberg](table-formats.md#apache-iceberg)
+- **DeltaFormat**: See [Table Formats - Delta Lake](table-formats.md#delta-lake)
+- **HudiFormat**: See [Table Formats - Hudi](table-formats.md#apache-hudi)
+
 ## Supported Types
 
 Spark data sources support all eight primitive types and their corresponding array types.