s3_output: add parquet format support #11312
Conversation
kalavt
requested review from
a team,
celalettin1286,
cosmo0920,
edsiper,
fujimotos,
niedbalski and
patrick-stephens
as code owners
📝 WalkthroughWalkthroughAdds Parquet encoder support and Arrow/Parquet detection; rewrites out_s3 into modular multipart/blob/queue/stream/auth subsystems with SDS buffer usage, presigned-URL flows, DB-backed recovery, new S3 queue/store/stream APIs, compression enum adjustments (snappy/zstd), and a large suite of unit and runtime tests. Changes
Sequence Diagram(s)sequenceDiagram
participant Plugin as Out S3 plugin
participant DB as Blob DB
participant Store as Local fstore
participant Auth as Auth service
participant S3 as AWS S3
Plugin->>DB: register file & parts
Plugin->>Store: write chunk / stream file
Plugin->>Auth: request presigned URL (CREATE)
Auth-->>Plugin: presigned create URL
Plugin->>S3: initiate multipart (presigned URL)
S3-->>Plugin: upload_id
loop for each part
Plugin->>Auth: request presigned URL (UPLOAD_PART)
Auth-->>Plugin: presigned part URL
Plugin->>S3: upload part (PUT to presigned URL)
S3-->>Plugin: ETag
Plugin->>DB: mark part uploaded / store ETag
end
Plugin->>Auth: request presigned URL (COMPLETE)
Auth-->>Plugin: presigned complete URL
Plugin->>S3: complete multipart (with parts/ETags)
S3-->>Plugin: success/failure
Plugin->>DB: notify delivery and cleanup
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
b4c0a26 to
a821fec
Compare
kalavt
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (10)
src/CMakeLists.txt (1)
525-528: Clarify SASL library linking scope.The comments indicate "KAFKA SPECIFIC" for SASL library linking, but the linking appears in both the shared library and binary targets. This suggests it's a general requirement when
FLB_SASL_ENABLEDis true, not specific to this Parquet PR.Consider updating the comment to be more accurate:
🔎 Suggested comment improvement
- # KAFKA SPECIFIC - Link SASL library for shared library + # Link SASL library when SASL support is enabled (used by Kafka and other plugins) if(FLB_SASL_ENABLED) target_link_libraries(fluent-bit-shared sasl2) endif()- # KAFKA SPECIFIC - Link SASL library for binary + # Link SASL library when SASL support is enabled (used by Kafka and other plugins) if(FLB_SASL_ENABLED) target_link_libraries(fluent-bit-bin sasl2) endif()Also applies to: 600-603
examples/s3_output/run-s3-parquet-test.sh (1)
38-39: Add error checking for fluent-bit binary and configuration.The script assumes the Fluent Bit binary and configuration file exist without verification, which could lead to confusing errors.
🔎 Proposed pre-flight checks
+# Verify fluent-bit binary exists +if [ ! -f ./build/bin/fluent-bit ]; then + echo "Error: fluent-bit binary not found at ./build/bin/fluent-bit" + echo "Please build Fluent Bit first:" + echo " cd ../../build && cmake .. && make" + exit 1 +fi + +# Verify configuration file exists +if [ ! -f fluent-bit-s3-parquet.conf ]; then + echo "Error: Configuration file fluent-bit-s3-parquet.conf not found" + exit 1 +fi + +# Check AWS credentials +if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then + echo "Warning: AWS credentials not set. Fluent Bit may fail to upload to S3." + echo "Set credentials with: export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=..." + echo "" +fi + # Run Fluent Bit ./build/bin/fluent-bit -c fluent-bit-s3-parquet.confexamples/s3_output/README.md (1)
112-114: Consider adding language identifiers to fenced code blocks.Static analysis detected some fenced code blocks without language specifiers, which improves syntax highlighting and readability.
🔎 Minor markdown improvements
-``` +```text Upload Trigger = upload_timeout reached OR total_file_size reached```diff -``` +```text s3://s3-bucket-kafka-sink/fluent-bit-logs/ ├── putobject/app/day=YYYYMMDD/ # Case 1: Small files via PutObject └── multipart/access/day=YYYYMMDD/ # Case 2: Large files via Multipart Upload```diff -``` +```text [parquet] Data quality summary for 53206 records: [parquet] Missing non-nullable fields (defaults used): [parquet] field='level' count=52506 [parquet] field='timestamp' count=52506</details> Also applies to: 177-179, 270-272 </blockquote></details> <details> <summary>examples/s3_output/athena-queries.sql (2)</summary><blockquote> `38-45`: **Use placeholder bucket names in example documentation.** The hardcoded bucket name `s3-bucket-kafka-sink` appears throughout the file. Consider using a placeholder like `<your-bucket-name>` or `my-fluent-bit-bucket` to make it clearer that users should substitute their own bucket name. --- `99-99`: **Document the placeholder date format.** The hardcoded date `20251224` is used in all example queries. Consider adding a note at the top of the query section (around line 83) explaining that users should replace this with their actual date, similar to the note at line 346. </blockquote></details> <details> <summary>cmake/parquet.cmake (2)</summary><blockquote> `151-163`: **Consider allowing default paths as fallback.** Using `NO_DEFAULT_PATH` exclusively may miss Arrow/Parquet installations in standard system locations not explicitly listed. Consider removing `NO_DEFAULT_PATH` or adding a second search without it as a fallback to improve detection success on systems with non-standard installations. <details> <summary>🔎 Suggested approach</summary> ```diff find_path(ARROW_INCLUDE_DIR NAMES arrow/api.h PATHS ${SEARCH_PATHS} PATH_SUFFIXES ${INCLUDE_SUFFIXES} include/arrow - NO_DEFAULT_PATH )Or add a fallback search without
NO_DEFAULT_PATHif the first search fails.
223-226: Unicode characters in CMake messages may cause issues.The checkmark (✓) and cross (✗) Unicode characters in status messages may not render correctly on all terminals or CI systems. Consider using ASCII alternatives like
[OK]and[MISSING].🔎 Proposed fix
if(ARROW_FOUND AND PARQUET_FOUND) - message(STATUS "✓ Arrow found: ${ARROW_LIBRARIES}") + message(STATUS "[OK] Arrow found: ${ARROW_LIBRARIES}") message(STATUS " Include dirs: ${ARROW_INCLUDE_DIRS}") - message(STATUS "✓ Parquet found: ${PARQUET_LIBRARIES}") + message(STATUS "[OK] Parquet found: ${PARQUET_LIBRARIES}") message(STATUS " Include dirs: ${PARQUET_INCLUDE_DIRS}") else() - message(STATUS "✗ Arrow/Parquet not found") + message(STATUS "[MISSING] Arrow/Parquet not found")examples/s3_output/fluent-bit-s3-parquet.conf (1)
145-145: Very long schema_str line reduces readability.The
schema_strvalue is a single long JSON line that's difficult to read and maintain. Consider adding a comment above it with a formatted version of the schema for documentation purposes, or noting in the README that multi-line values would need escaping.src/flb_parquet_impl.cpp (1)
250-251: Appending nullptr to BinaryBuilder may cause undefined behavior.The
append_default_valuefor BINARY type passesnullptrwith size 0. While this may work, it's safer to use an empty byte array explicitly.🔎 Proposed fix
case arrow::Type::BINARY: - return static_cast<arrow::BinaryBuilder*>(builder)->Append(static_cast<const uint8_t*>(nullptr), 0).ok(); + return static_cast<arrow::BinaryBuilder*>(builder)->Append(reinterpret_cast<const uint8_t*>(""), 0).ok();plugins/out_s3/s3.c (1)
49-56: Duplicate forward declarations.These forward declarations at lines 49-56 duplicate those at lines 86-91. Remove the duplicates to avoid confusion.
🔎 Proposed fix
-/* Forward declarations */ -static struct multipart_upload *get_upload(struct flb_s3 *ctx, - const char *tag, int tag_len); -static struct multipart_upload *create_upload(struct flb_s3 *ctx, - const char *tag, int tag_len, - time_t file_first_log_time); -static flb_sds_t flb_pack_msgpack_extract_log_key(void *out_context, const char *data, - uint64_t bytes, struct flb_config *config); -
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (19)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/parquet.cmakeexamples/s3_output/README.mdexamples/s3_output/athena-queries.sqlexamples/s3_output/fluent-bit-s3-parquet.confexamples/s3_output/run-s3-parquet-test.shinclude/fluent-bit/aws/flb_aws_compress.hinclude/fluent-bit/flb_parquet.hplugins/out_s3/s3.cplugins/out_s3/s3.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/compression/arrow/CMakeLists.txtsrc/aws/compression/arrow/compress.csrc/aws/compression/arrow/compress.hsrc/aws/flb_aws_compress.csrc/flb_parquet_impl.cpp
💤 Files with no reviewable changes (5)
- src/aws/compression/CMakeLists.txt
- src/aws/compression/arrow/compress.c
- src/aws/compression/arrow/compress.h
- src/aws/CMakeLists.txt
- src/aws/compression/arrow/CMakeLists.txt
🧰 Additional context used
🧠 Learnings (16)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
examples/s3_output/README.mdinclude/fluent-bit/flb_parquet.h
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
.github/workflows/unit-tests.yamlcmake/parquet.cmake
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtcmake/parquet.cmakeCMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtcmake/parquet.cmakeCMakeLists.txt
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtsrc/aws/flb_aws_compress.cCMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
.github/workflows/unit-tests.yamlsrc/CMakeLists.txtCMakeLists.txt
📚 Learning: 2025-12-22T05:39:02.291Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: lib/ripser-1.2.1/Makefile:1-18
Timestamp: 2025-12-22T05:39:02.291Z
Learning: In fluent/fluent-bit, the Makefile at lib/ripser-1.2.1/Makefile is imported from upstream Ripser and is not used in the actual build process. The project uses CMake for building (lib/ripser-1.2.1/CMakeLists.txt), so changes to the imported Makefile are not necessary.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
src/CMakeLists.txtsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-09-14T09:46:09.531Z
Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T07:28:37.083Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10832
File: src/wasm/CMakeLists.txt:112-131
Timestamp: 2025-09-04T07:28:37.083Z
Learning: In fluent-bit CMake files, the user cosmo0920 prefers treating Git as a command rather than a package, emphasizing that Git is not a pkg-config retrievable package but just a command.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3.c
🧬 Code graph analysis (3)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
src/aws/flb_aws_compress.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (1)
s3_put_object(1663-1800)
🪛 Cppcheck (2.19.0)
plugins/out_s3/s3.c
[error] 791-791: failed to expand 'flb_plg_error', it is invalid to use a preprocessor directive as macro parameter
(syntaxError)
🪛 LanguageTool
examples/s3_output/README.md
[grammar] ~56-~56: Ensure spelling is correct
Context: ..., 201, 404, 500, 401) - Response times (2ms to 5000ms) - Client information (IP, us...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~465-~465: Ensure spelling is correct
Context: ...les uploaded when reaching 50MB (before 60min timeout) Verification Steps: 1. R...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
examples/s3_output/README.md
112-112: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
177-177: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
270-270: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (14)
include/fluent-bit/flb_parquet.h (1)
82-85: LGTM! Well-documented API with clear ownership semantics.The function signature and documentation are comprehensive and clear. The API design properly communicates:
- Memory ownership (caller must free with flb_free)
- Error handling (NULL on failure)
- Schema format expectations (Arrow JSON schema)
- Supported types and examples
include/fluent-bit/aws/flb_aws_compress.h (2)
33-46: Clear deprecation notice and migration guidance.The documentation clearly separates true compression algorithms from format converters and provides excellent guidance:
- ARROW marked as deprecated with rationale
- PARQUET (5) directs users to use
format=parquetinstead- Clear distinction between compression and format conversion
This will help users migrate away from the deprecated patterns.
30-31: No backward compatibility issue—this renumbering is safe.The compression constants are internal implementation details used only at runtime. Configuration files use string values (
compression=gzip,compression=snappy,compression=zstd) which are mapped to numeric constants viaflb_aws_compression_get_type(). Since the numeric constant values are never persisted to storage and users never interact with them directly, renumbering fromFLB_AWS_COMPRESS_ZSTD=4toFLB_AWS_COMPRESS_ZSTD=3does not break backward compatibility.Likely an incorrect or invalid review comment.
CMakeLists.txt (1)
284-286: LGTM! Consistent encoder enablement pattern.When
FLB_ALLis enabled, both Avro and Parquet encoders are automatically enabled, following the existing pattern for optional features. This provides a consistent user experience.src/CMakeLists.txt (1)
157-166: LGTM! Proper C++ integration for Parquet implementation.The Parquet encoder support is correctly integrated:
- Source file added conditionally based on
FLB_PARQUET_ENCODER- Explicit CXX language property set for the C++ implementation file
- Follows the existing pattern for optional features like Avro encoder
src/aws/flb_aws_compress.c (2)
31-37: LGTM! Correct adapter pattern for Snappy compression.The wrapper function properly adapts
flb_snappy_compress(which useschar*pointers) to the standard compression interface (which usesvoid*pointers). This follows the same pattern as the existingflb_gzip_compressandflb_zstd_compressfunctions shown in the relevant code snippets.
46-76: Excellent documentation and clean separation of concerns.The updated comments clearly distinguish:
- True compression algorithms (gzip, snappy, zstd)
- Deprecated format converters (ARROW, PARQUET)
- Guidance to use
format=parquetinstead ofcompression=parquetThe removal of Arrow/Parquet compression options from this module aligns with the PR's goal of decoupling format conversion from the compression subsystem.
examples/s3_output/README.md (1)
1-518: Exceptional documentation! Comprehensive and production-ready guidance.This README provides outstanding documentation that covers:
- Clear test scenarios with realistic use cases
- Detailed explanation of upload triggers (
upload_timeoutvstotal_file_size)- Schema design best practices with nullable/non-nullable guidance
- Data quality monitoring and troubleshooting
- Performance testing methodology
- Athena integration examples
The documentation will significantly help users understand Parquet encoder usage in production scenarios.
plugins/out_s3/s3.h (1)
234-235: LGTM - API signature updated for SDS-based buffers.The signature change from
char* bodytoflb_sds_t bodyaligns with the SDS-based buffering approach used throughout the refactored code, ensuring consistent memory management.examples/s3_output/fluent-bit-s3-parquet.conf (1)
104-117: LGTM - Well-documented PutObject configuration.The configuration clearly documents the upload strategy, timeout behavior, and use case. The comments explaining the difference between
upload_timeoutandtotal_file_sizeare helpful for users.src/flb_parquet_impl.cpp (1)
696-710: LGTM - Exception handling with proper cleanup.The exception handlers correctly destroy the msgpack unpacked result and free the output buffer if allocated, preventing memory leaks on error paths.
plugins/out_s3/s3.c (3)
752-774: LGTM - Deprecated compression values handled gracefully.The code properly warns users about deprecated
arrow/parquetcompression values while still supporting them by mapping to the newformat=parquetapproach. The build-time guard ensures helpful error messages when Parquet isn't available.
1583-1630: LGTM - Parquet conversion with proper error handling.The Parquet conversion path validates
schema_stris present, handles conversion failures gracefully, and properly manages memory by converting the Parquet buffer toflb_sds_tfor consistent lifecycle management.
3951-3962: Storing raw msgpack improves efficiency for Parquet.Storing raw msgpack data and deferring format conversion to upload time is the right approach for columnar formats like Parquet, which benefit from batching records together for better compression and schema inference.
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/out_s3/s3.c (1)
3701-3725: Leak ofchunkSDS inunit_test_flush()path.In
unit_test_flush()you now treatchunkasflb_sds_t, but never destroy it:s3_store_buffer_put(..., chunk, ...); ret = construct_request_buffer(ctx, chunk, upload_file, &buffer, &buffer_size); ... ret = upload_data(..., buffer, ...); flb_sds_destroy(buffer); FLB_OUTPUT_RETURN(ret);
construct_request_buffer()only reads fromchunkand does not take ownership, sounit_test_flush()shouldflb_sds_destroy(chunk)once that call returns. As written, every test‑mode flush leaks the SDS buffer.Suggested minimal fix
static void unit_test_flush(void *out_context, struct s3_file *upload_file, const char *tag, int tag_len, flb_sds_t chunk, int chunk_size, struct multipart_upload *m_upload_file, time_t file_first_log_time) { int ret; flb_sds_t buffer = NULL; size_t buffer_size; struct flb_s3 *ctx = out_context; s3_store_buffer_put(ctx, upload_file, tag, tag_len, chunk, (size_t) chunk_size, file_first_log_time); - ret = construct_request_buffer(ctx, chunk, upload_file, &buffer, &buffer_size); + ret = construct_request_buffer(ctx, chunk, upload_file, &buffer, &buffer_size); + flb_sds_destroy(chunk); if (ret < 0) { flb_plg_error(ctx->ins, "Could not construct request buffer for %s", upload_file->file_path); FLB_OUTPUT_RETURN(FLB_RETRY); } ret = upload_data(ctx, upload_file, m_upload_file, buffer, buffer_size, tag, tag_len); flb_sds_destroy(buffer); FLB_OUTPUT_RETURN(ret); }
♻️ Duplicate comments (7)
examples/s3_output/run-s3-parquet-test.sh (1)
6-16: Logic issue: directories created then immediately cleaned.The script creates directories (lines 8-11) and then immediately removes their contents (line 15). This sequence is confusing and redundant. If the intent is to ensure clean directories exist, reverse the order or use
mkdir -palone (which is idempotent).🔎 Proposed fix
# Create temporary directories STORE_DIR="/tmp/fluent-bit-test" + +# Clean old data first (if exists) +if [ -d "$STORE_DIR" ]; then + rm -rf "$STORE_DIR" +fi + +# Create fresh directories mkdir -p "$STORE_DIR/parquet-gzip-simple" mkdir -p "$STORE_DIR/json-snappy-complex-low" mkdir -p "$STORE_DIR/json-gzip-complex-medium" mkdir -p "$STORE_DIR/json-zstd-complex-high" echo "✓ Created store directories" - -# Clean old data -rm -rf "$STORE_DIR"/*/ -echo "✓ Cleaned old data"This also addresses the safety concern from the previous review about
rm -rfon a potentially unset variable, by checking directory existence first.plugins/out_s3/s3.h (1)
123-124: Unusedapply_outer_compressionfield instruct flb_s3.
apply_outer_compressionis not referenced anywhere inplugins/out_s3/s3.c(compression behavior is driven byctx->format != FLB_S3_FORMAT_PARQUETinstead), so this field is dead state and adds confusion. Consider removing it or wiring it into the compression path; otherwise future readers might assume it actually controls outer compression.CMakeLists.txt (1)
202-202: Update Parquet C++ standard from C++11 to C++17.The Parquet encoder block enables C++ and forces
CMAKE_CXX_STANDARD 11, but current Apache Arrow/Parquet C++ releases require at least C++17. Keeping this at 11 will either fail the build or subtly misconfigure Arrow compilation whenFLB_PARQUET_ENCODERis ON. Please bump this to 17 (and consider aligning the earlier simdutfCMAKE_CXX_STANDARDas well).You should double‑check against the Arrow/Parquet version you target (their build docs list the minimum C++ standard) and confirm that setting
CMAKE_CXX_STANDARD 17is required whenFLB_PARQUET_ENCODERis enabled.Also applies to: 278-287, 1311-1316
plugins/out_s3/s3.c (1)
4210-4217:compressionconfig map still writes a string into anintfield.The
compressionentry is declared asFLB_CONFIG_MAP_STRbut its offset points tostruct flb_s3::compression(anint):{ FLB_CONFIG_MAP_STR, "compression", NULL, 0, FLB_TRUE, offsetof(struct flb_s3, compression), ... },During
flb_output_config_map_set(), this causes the parser to store achar *into anintslot, corruptingcompressionand adjacent fields. You now parsecompressionexplicitly viaflb_output_get_property()incb_s3_init(), so this map binding is both unsafe and unnecessary.Recommend either:
- Removing the struct offset (
FLB_FALSE, 0) and treatingcompressionas a pure property, or- Introducing a
char *compression_strfield and mapping the string there.src/flb_parquet_impl.cpp (3)
138-151: Add overflow handling for uint64 → int64 inconvert_to_int64().
MSGPACK_OBJECT_POSITIVE_INTEGERvalues are cast directly fromuint64_ttoint64_t:if (obj->type == MSGPACK_OBJECT_POSITIVE_INTEGER) { return builder->Append(static_cast<int64_t>(obj->via.u64)); }Values greater than
INT64_MAXwill silently wrap, corrupting data in the Parquet file. This mirrors the earlierint32bug you already guarded against.You should add an overflow check similar to
convert_to_int32()and either clamp, append a default, or return an error so the caller can account for it.Please confirm how Arrow/Parquet is expected to handle out‑of‑range
uint64values (e.g., whether to clamp, treat as error, or promote to unsigned logical type) based on the versions you target, and align this converter with that behavior.
213-226: Timestamp conversion ignores schema time unit.
convert_to_timestamp()always treats the input as seconds and multiplies by 1,000,000 to get microseconds:return builder->Append(static_cast<int64_t>(obj->via.u64) * 1000000);However, when parsing the schema you support
"unit": "s"|"ms"|"us"|"ns"and buildarrow::timestamp(time_unit)accordingly. The converter never looks at that unit, so a schema declaringmsornswill still receive values scaled as if they were seconds→µs.You likely want to:
- Downcast
type(or the builder’s type) toarrow::TimestampType,- Inspect its
unit(),- Apply the appropriate multiplier/divider so the stored values actually match the declared unit.
Double‑check the Arrow
TimestampTypesemantics for units (SECOND,MILLI,MICRO,NANO) and ensure the scaling here matches how your upstream logs encode timestamps (seconds vs ms vs ns).Also applies to: 386-424, 505-555
301-321: Storingmsgpack_objectinstances acrossmsgpack_unpack_next()iterations may rely on msgpack zone semantics.You collect
msgpack_objectinstances into astd::vectorand process them after the unpack loop:std::vector<msgpack_object> records; ... while (msgpack_unpack_next(&result, ...) == MSGPACK_UNPACK_SUCCESS) { ... records.push_back(array.ptr[i]); // or result.data } ... for (auto &record : records) { ... }These
msgpack_objects contain pointers into themsgpack_unpackedzone. This is fine only ifmsgpack_unpack_next()never invalidates existing zone allocations while reusing the samemsgpack_unpackedacross iterations; some msgpack documentation/patterns assume you consume each object immediately.If the zone is reused or compacted between iterations,
recordscould end up with dangling pointers once more data is unpacked.Please verify against the msgpack‑c docs for
msgpack_unpackedandmsgpack_unpack_next()whether it is safe to keepmsgpack_objectcopies after subsequentmsgpack_unpack_next()calls, or whether you must either:
- Process each record inside the loop, or
- Deep‑copy the record into your own storage (including map/array/string contents) before the next unpack.
Also applies to: 474-555
🧹 Nitpick comments (9)
examples/s3_output/run-s3-parquet-test.sh (1)
38-39: Relative paths may cause execution failures.The script uses relative paths for both the binary (
./build/bin/fluent-bit) and config file (fluent-bit-s3-parquet.conf). This will fail unless run from the repository root.Consider adding a working directory check or using
$SCRIPT_DIRto make paths relative to the script location.🔎 Proposed improvement
+# Get script directory for relative paths +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" + # Run Fluent Bit -./build/bin/fluent-bit -c fluent-bit-s3-parquet.conf +"$REPO_ROOT/build/bin/fluent-bit" -c "$SCRIPT_DIR/fluent-bit-s3-parquet.conf"examples/s3_output/athena-queries.sql (2)
38-44: Hardcoded bucket name - consider adding placeholder instructions.The S3 bucket
s3-bucket-kafka-sinkis hardcoded throughout. While acceptable for examples, consider adding a comment at the top noting that users should replace this with their own bucket name.
99-101: Hardcoded date filter in example queries.All queries use
day='20251224'which matches "today's date" per the context. For a reusable example, consider using a dynamic date or adding a note about updating this value.examples/s3_output/README.md (3)
112-114: Add language specifier to fenced code block.Per markdownlint, fenced code blocks should have a language specified. This block shows a logical expression.
🔎 Proposed fix
-``` +```text Upload Trigger = upload_timeout reached OR total_file_size reached</details> --- `177-181`: **Add language specifier to fenced code block.** Per markdownlint, fenced code blocks should have a language specified for the directory tree. <details> <summary>🔎 Proposed fix</summary> ```diff -``` +```text s3://s3-bucket-kafka-sink/fluent-bit-logs/ ├── putobject/app/day=YYYYMMDD/ # Case 1: Small files via PutObject └── multipart/access/day=YYYYMMDD/ # Case 2: Large files via Multipart Upload</details> --- `270-275`: **Add language specifier to fenced code block.** Per markdownlint, fenced code blocks should have a language specified for log output examples. <details> <summary>🔎 Proposed fix</summary> ```diff -``` +```text [parquet] Data quality summary for 53206 records: [parquet] Missing non-nullable fields (defaults used): [parquet] field='level' count=52506 [parquet] field='timestamp' count=52506</details> </blockquote></details> <details> <summary>cmake/parquet.cmake (1)</summary><blockquote> `150-163`: **Consider adding default path fallback for Arrow discovery.** The `NO_DEFAULT_PATH` option prevents CMake from searching standard system paths. This might miss Arrow installations in non-standard but common locations not listed in `SEARCH_PATHS`. Consider removing `NO_DEFAULT_PATH` or adding a secondary search without it as a fallback. <details> <summary>🔎 Proposed improvement</summary> ```diff find_path(ARROW_INCLUDE_DIR NAMES arrow/api.h PATHS ${SEARCH_PATHS} PATH_SUFFIXES ${INCLUDE_SUFFIXES} include/arrow - NO_DEFAULT_PATH ) find_library(ARROW_LIBRARY NAMES ${ARROW_LIB_NAMES} PATHS ${SEARCH_PATHS} PATH_SUFFIXES ${LIB_SUFFIXES} - NO_DEFAULT_PATH )Alternatively, keep
NO_DEFAULT_PATHfor the first search and add a fallback search without it if the first fails.include/fluent-bit/flb_parquet.h (2)
23-25: Consider removing unused include.The header includes
<fluent-bit/flb_sds.h>but the function signature doesn't useflb_sds_t. If this is not used elsewhere in the implementation exposed by this header, consider removing it to reduce compilation dependencies.#!/bin/bash # Check if flb_sds types are used in the parquet implementation or header rg -n "flb_sds" src/flb_parquet_impl.cpp include/fluent-bit/flb_parquet.h
82-85: Missing extern "C" guard for C++ compatibility.Since this header will be included by C++ code (
flb_parquet_impl.cpp), consider addingextern "C"guards to ensure proper C linkage when compiled as C++.🔎 Proposed fix
+#ifdef __cplusplus +extern "C" { +#endif + void *flb_msgpack_raw_to_parquet(const void *in_buf, size_t in_size, const char *schema_str, int compression, size_t *out_size); +#ifdef __cplusplus +} +#endif + #endif
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (19)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/parquet.cmakeexamples/s3_output/README.mdexamples/s3_output/athena-queries.sqlexamples/s3_output/fluent-bit-s3-parquet.confexamples/s3_output/run-s3-parquet-test.shinclude/fluent-bit/aws/flb_aws_compress.hinclude/fluent-bit/flb_parquet.hplugins/out_s3/s3.cplugins/out_s3/s3.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/compression/arrow/CMakeLists.txtsrc/aws/compression/arrow/compress.csrc/aws/compression/arrow/compress.hsrc/aws/flb_aws_compress.csrc/flb_parquet_impl.cpp
💤 Files with no reviewable changes (5)
- src/aws/compression/arrow/compress.h
- src/aws/compression/arrow/CMakeLists.txt
- src/aws/compression/CMakeLists.txt
- src/aws/CMakeLists.txt
- src/aws/compression/arrow/compress.c
🚧 Files skipped from review as they are similar to previous changes (2)
- .github/workflows/unit-tests.yaml
- examples/s3_output/fluent-bit-s3-parquet.conf
🧰 Additional context used
🧠 Learnings (15)
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtsrc/flb_parquet_impl.cppplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtplugins/out_s3/s3.cinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
src/CMakeLists.txtCMakeLists.txtcmake/parquet.cmake
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
src/CMakeLists.txtCMakeLists.txtcmake/parquet.cmake
📚 Learning: 2025-12-22T05:39:02.291Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: lib/ripser-1.2.1/Makefile:1-18
Timestamp: 2025-12-22T05:39:02.291Z
Learning: In fluent/fluent-bit, the Makefile at lib/ripser-1.2.1/Makefile is imported from upstream Ripser and is not used in the actual build process. The project uses CMake for building (lib/ripser-1.2.1/CMakeLists.txt), so changes to the imported Makefile are not necessary.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
src/CMakeLists.txtCMakeLists.txt
📚 Learning: 2025-09-14T09:46:09.531Z
Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T07:28:37.083Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10832
File: src/wasm/CMakeLists.txt:112-131
Timestamp: 2025-09-04T07:28:37.083Z
Learning: In fluent-bit CMake files, the user cosmo0920 prefers treating Git as a command rather than a package, emphasizing that Git is not a pkg-config retrievable package but just a command.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_parquet.hexamples/s3_output/README.md
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3.c
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
cmake/parquet.cmake
🧬 Code graph analysis (4)
src/aws/flb_aws_compress.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
include/fluent-bit/flb_parquet.h (1)
src/flb_parquet_impl.cpp (2)
flb_msgpack_raw_to_parquet(265-711)flb_msgpack_raw_to_parquet(265-268)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (1)
s3_put_object(1663-1800)
🪛 Cppcheck (2.19.0)
plugins/out_s3/s3.c
[error] 791-791: failed to expand 'flb_plg_error', it is invalid to use a preprocessor directive as macro parameter
(syntaxError)
🪛 LanguageTool
examples/s3_output/README.md
[grammar] ~56-~56: Ensure spelling is correct
Context: ..., 201, 404, 500, 401) - Response times (2ms to 5000ms) - Client information (IP, us...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~465-~465: Ensure spelling is correct
Context: ...les uploaded when reaching 50MB (before 60min timeout) Verification Steps: 1. R...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
examples/s3_output/README.md
112-112: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
177-177: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
270-270: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (19)
src/aws/flb_aws_compress.c (2)
31-37: LGTM! Wrapper function correctly adapts the signature.The wrapper properly casts between
void*andchar*to adaptflb_snappy_compress's signature to the uniform compression interface. This follows the same pattern used for gzip and zstd.
45-76: Good refactoring to clarify compression algorithms vs format converters.The updated comments clearly distinguish true compression algorithms (gzip, snappy, zstd) from format converters (Arrow/Parquet). Removing ARROW/PARQUET from the runtime
compression_optionstable while keeping deprecated macros in the header is a clean backward-compatible approach.One minor observation: the comment on line 59 ("FLB_AWS_COMPRESS_NONE which is 0 is reserved for array footer") is now orphaned after the blank line. Consider moving it closer to the footer entry on line 77.
include/fluent-bit/aws/flb_aws_compress.h (2)
33-46: Good deprecation strategy with clear documentation.The comments clearly explain that ARROW is deprecated and PARQUET should use
format=parquetinstead. Keeping the macros prevents compilation errors for any downstream code that references them.
28-31: No backward compatibility issue from ZSTD renumbering.Compression type configuration uses string keywords ("gzip", "snappy", "zstd"), not numeric values. The numeric constants (0, 1, 2, 3, etc.) are internal implementation details used only at runtime for comparisons and branching. The
flb_aws_compression_get_type()function converts configuration strings to these numeric types at startup. Since numeric values are never persisted, serialized, or exposed to external systems, renumbering internal constants has no impact on user configurations or backward compatibility.cmake/parquet.cmake (1)
222-226: Good: Clear status reporting with emoji indicators.The status messages with ✓ and ✗ symbols provide clear visual feedback during CMake configuration. The inclusion of library paths and include directories aids debugging.
src/CMakeLists.txt (3)
157-166: LGTM! Parquet encoder integration follows existing patterns.The Parquet encoder source is conditionally added and marked as C++ correctly. This follows the same pattern as the AVRO encoder block above.
525-528: SASL linking additions appear correct for Kafka integration.The SASL library is conditionally linked when
FLB_SASL_ENABLEDis set, properly handling both the shared library and binary targets. This follows standard CMake patterns.Also applies to: 600-605
398-405: The code is correctly guarded. Whenif(FLB_PARQUET_ENCODER)at line 399 is true, it means the parentCMakeLists.txtverified that bothARROW_FOUNDandPARQUET_FOUNDwere true (lines 1320-1328), and it would have setFLB_PARQUET_ENCODER OFFotherwise. The variablesARROW_LIBRARIESandPARQUET_LIBRARIESare set and exported withPARENT_SCOPEincmake/parquet.cmake, ensuring they are available in the subdirectory scope.include/fluent-bit/flb_parquet.h (1)
27-85: Excellent API documentation with comprehensive schema examples.The documentation thoroughly covers:
- Parameter descriptions with format specifications
- Supported type mappings
- Usage example with proper memory management
- Links to Arrow documentation
This level of documentation is exemplary for a public API.
plugins/out_s3/s3.h (1)
197-205: Format fields, macros, ands3_put_objectprototype look consistent.
format/schema_strfields,FLB_S3_FORMAT_JSON/PARQUETmacros, and the updateds3_put_objectprototype (usingflb_sds_t buffer) all match their usage inplugins/out_s3/s3.cand cleanly expose JSON vs Parquet behavior.Also applies to: 234-235
plugins/out_s3/s3.c (8)
192-199: Correctly skippingContent-Encodingfor Parquet payloads.Conditioning the
Content-Encodingheader on(compression == gzip|zstd) && format != FLB_S3_FORMAT_PARQUETensures Parquet objects are not wrapped in an outer HTTP compression layer while still advertising compression for JSON payloads. This matches the “internal only” compression model for Parquet and looks good.Also applies to: 227-239
803-823: Parquet format path: good separation of internal vs outer compression, but hard‑fails withoutschema_str.The Parquet branch in
construct_request_buffer()correctly:
- Avoids outer HTTP compression and uses
ctx->compressiononly as the internal Parquet codec.- Requires
schema_strand bails out early with a clear error if it’s missing.- Converts the Arrow buffer to
flb_sds_tand frees the Arrow‑allocated memory.This behavior is sound: format‑specific requirements are enforced once at conversion time, and buffer ownership is cleanly transferred to Fluent Bit.
Also applies to: 1515-1631
1217-1277: SDS compression path inupload_data()is generally correct, but relies solely onformatto skip outer compression.The new
upload_data()signature and logic look good overall:
bodyis nowflb_sds_t, withbody_to_send/size_to_sendcorrectly switched to the compressed SDS when compression succeeds.- The compression block is gated by
ctx->compression != FLB_AWS_COMPRESS_NONE && ctx->format != FLB_S3_FORMAT_PARQUET, so Parquet payloads are never double‑compressed.- All error paths that allocate
compressed_bodyalso destroy it, and both PutObject and multipart paths clean upcompressed_bodyafter use.No functional issues seen here; the behavior matches the intended “outer compression only for non‑Parquet” design.
Also applies to: 1325-1392
1442-1513: SDS buffer lifecycle aroundconstruct_request_buffer()/upload_data()is consistent (except unit tests).
put_all_chunks(),send_upload_request(), andcb_s3_upload()all:
- Call
construct_request_buffer()to obtain anflb_sds_t buffer.- Pass
bufferintos3_put_object()/upload_data().- Always
flb_sds_destroy(buffer)afterward, regardless of success/failure.This gives a single clear owner for each formatted buffer and avoids leaks in the main upload paths.
Also applies to: 1991-2015, 3440-3488, 3701-3724
1515-1661: Error handling and locking inconstruct_request_buffer()look solid.
- Handles the illegal
(new_data == NULL && chunk == NULL)combination explicitly.- For file‑backed chunks, locks the
s3_fileonly after a successful read, and reliably unlocks on every error path (realloc failure, Parquet schema error, conversion failure).- Frees the intermediate msgpack buffer in all paths (success and failure).
- Ensures
formatted_data/formatted_sizeare both valid before returning and destroys the SDS on conversion failure.This is a good, consistent ownership pattern between the file store, msgpack buffer, and formatted SDS.
1663-1800:s3_put_object()SDS refactor is behavior‑preserving.Switching the body parameter to
flb_sds_tintegrates cleanly:
- MD5 calculation and HTTP request still operate on a contiguous buffer (
body,body_size).- URI/key construction, seq_index updates, error paths, and logging remain unchanged.
- No new ownership expectations are imposed on callers (they still manage the SDS lifetime).
3953-3963: Raw msgpack buffering incb_s3_flush()is a good fit for Parquet.Creating
chunkas an SDS copy of the raw msgpack and deferring JSON/Parquet conversion toconstruct_request_buffer():
- Avoids double conversion when retries occur.
- Makes it easy to plug in additional formats later, since the on‑disk representation stays msgpack.
- Keeps buffer ownership straightforward via
buffer_chunk()/send_upload_request().No issues spotted here.
Also applies to: 4036-4085
4219-4231: Newformat,schema_str,upload_timeout, and bearer token options are wired correctly.
formatis exposed as a documented string option and interpreted viaflb_output_get_property("format", ...).schema_stris mapped intostruct flb_s3::schema_strand used by the Parquet conversion path.upload_timeoutis now a properFLB_CONFIG_MAP_TIMEoption and is referenced consistently in timeout checks and timer setup.authorization_endpoint_bearer_tokenfollows the existing auth option patterns.These mappings and docs align with how the fields are used later in the file.
Also applies to: 4381-4386, 4406-4414
src/flb_parquet_impl.cpp (1)
265-299: Overall Parquet conversion pipeline and memory ownership look good.
- Parameter validation, compression mapping, schema parsing (via cJSON), and Arrow schema construction are all checked with clear error logs.
- Builders are created per field, and all failure paths free cJSON resources and destroy the msgpack zone before returning.
- Record conversion tracks data‑quality stats (missing fields, type failures, int32 overflow, float→int clamps) and emits a concise summary only when issues exist.
- Parquet writing uses
BufferOutputStream(with auto‑growing buffer), closes the writer, then copies the final Arrow buffer intoflb_malloc‑managed memory (output_buffer), which is freed on all exception paths.The high‑level control flow and ownership model are solid; the remaining issues are mainly around numeric edge cases and msgpack object lifetime.
Also applies to: 331-446, 449-611, 616-692
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
a821fec to
5e868fc
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (3)
CMakeLists.txt (1)
1311-1316: Parquet encoder should request C++17, not C++11Arrow/Parquet C++ now require at least C++17; forcing
CMAKE_CXX_STANDARD 11in the Parquet block will fail with current Arrow toolchains or silently misconfigure builds. Please bump this to 17 (and keep it consistent with any otherCMAKE_CXX_STANDARDoverrides).if(FLB_PARQUET_ENCODER) enable_language(CXX) set(CMAKE_CXX_STANDARD 17) include(cmake/parquet.cmake) ... endif()#!/bin/bash # Verify Arrow C++ minimum standard in your environment rg -n "CMAKE_CXX_STANDARD" CMakeLists.txt cmake -Scmake/parquet.cmake (1)
149-157: Fix Arrow include search (include/arrowsuffix still produces wrong include dir)In the manual search:
find_path(ARROW_INCLUDE_DIR NAMES arrow/api.h PATHS ${SEARCH_PATHS} PATH_SUFFIXES ${INCLUDE_SUFFIXES} include/arrow NO_DEFAULT_PATH )using
include/arrowas a suffix causesARROW_INCLUDE_DIRto resolve to${prefix}/include/arrow, so the later version check:if(EXISTS "${ARROW_INCLUDE_DIRS}/arrow/util/config.h")looks for
${prefix}/include/arrow/arrow/util/config.h, which is incorrect for standard Arrow layouts (${prefix}/include/arrow/...).Drop the
include/arrowsuffix and rely on${INCLUDE_SUFFIXES}(typically justinclude), e.g.:find_path(ARROW_INCLUDE_DIR NAMES arrow/api.h PATHS ${SEARCH_PATHS} PATH_SUFFIXES ${INCLUDE_SUFFIXES} NO_DEFAULT_PATH )The Parquet search’s
include/parquetsuffix is fine and should remain.Also applies to: 198-216
plugins/out_s3/s3.c (1)
744-775: Backward compatibility broken:compression=arrow|parquetnow fails instead of mapping toformat=parquetThe check at line 752 can never be true. When
flb_aws_compression_get_type(tmp)is called with"arrow"or"parquet", it returns-1because these strings are not in thecompression_optionstable inflb_aws_compress.c(which only includes gzip, snappy, and zstd). This causes the code to hit the "Unknown compression type" error path instead of handling these deprecated aliases.The constants
FLB_AWS_COMPRESS_ARROW(4) andFLB_AWS_COMPRESS_PARQUET(5) are defined but unreachable, contradicting the documented deprecated behavior and the inline comments claiming support.Handle these strings explicitly before calling
flb_aws_compression_get_type:tmp = flb_output_get_property("compression", ins); if (tmp) { if (strcasecmp(tmp, "arrow") == 0 || strcasecmp(tmp, "parquet") == 0) { flb_plg_warn(ctx->ins, "DEPRECATED: compression=%s is deprecated. Use format=parquet instead.", tmp); #ifdef FLB_HAVE_PARQUET_ENCODER ctx->format = FLB_S3_FORMAT_PARQUET; #else flb_plg_error(ctx->ins, "Parquet format is not supported in this build. " "Rebuild with -DFLB_PARQUET_ENCODER=On."); return -1; #endif } else { ret = flb_aws_compression_get_type(tmp); if (ret == -1) { flb_plg_error(ctx->ins, "Unknown compression type: %s", tmp); return -1; } ctx->compression = ret; } }
🧹 Nitpick comments (3)
plugins/out_s3/s3.c (2)
49-57: Remove duplicate static declarations forget_upload/create_uploadBoth functions are declared twice (once near the top as forward declarations and again before their definitions). This is harmless but noisy and can confuse readers; you only need one forward declaration (or none if you reorder definitions).
Also applies to: 86-92
1218-1277: SDS-based outer compression flow is sound, but pre/post compression logging is now misleadingThe refactor in
upload_datato:
- Use
body_to_send/size_to_send(possibly compressed) while leaving the originalbody/body_sizeunchanged, and- Wrap the compressed payload in an
flb_sds_t(compressed_body) with consistent cleanup,looks correct and avoids raw
malloc/freehandling in callers.However, the informational log in the "too small, using PutObject" branch still prints:
flb_plg_info(ctx->ins, "Pre-compression upload_chunk_size= %zu, After compression, chunk is only %zu bytes, ...", preCompress_size, body_size);while the actual payload on the wire is
size_to_send, notbody_size. For Parquet, this message can also trigger even though no outer compression runs.Consider:
- Basing the "after compression" size on
size_to_send, and- Gating this log on
ctx->format != FLB_S3_FORMAT_PARQUETto avoid implying any outer compression there.Also applies to: 1300-1314, 1330-1392
examples/s3_output/README.md (1)
63-72: Add language identifiers to fenced code blocks for better tooling supportThe README content is solid and matches the new S3/Parquet behavior, but several fenced blocks (config snippets, shell commands, JSON schemas, SQL) are missing language tags. Adding them will quiet markdownlint and improve rendering/IDE support, e.g.:
```ini [OUTPUT] Name s3 ...cmake .. make -j8{ "fields": [ ... ] }SELECT ...;Also applies to: 85-95, 144-148, 152-156, 160-164, 178-181, 209-218, 241-251, 296-314, 341-349, 361-365, 384-426, 432-438, 442-456, 460-466, 484-495
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (17)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/parquet.cmakeexamples/s3_output/README.mdexamples/s3_output/athena-queries.sqlexamples/s3_output/fluent-bit-s3-parquet.confexamples/s3_output/run-s3-parquet-test.shinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cplugins/out_s3/s3.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/compression/arrow/CMakeLists.txtsrc/aws/compression/arrow/compress.csrc/aws/compression/arrow/compress.hsrc/aws/flb_aws_compress.c
💤 Files with no reviewable changes (5)
- src/aws/compression/arrow/CMakeLists.txt
- src/aws/compression/arrow/compress.c
- src/aws/compression/arrow/compress.h
- src/aws/CMakeLists.txt
- src/aws/compression/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (4)
- examples/s3_output/athena-queries.sql
- examples/s3_output/fluent-bit-s3-parquet.conf
- examples/s3_output/run-s3-parquet-test.sh
- .github/workflows/unit-tests.yaml
🧰 Additional context used
🧠 Learnings (16)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
src/CMakeLists.txtcmake/parquet.cmakeCMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
src/CMakeLists.txtcmake/parquet.cmakeCMakeLists.txt
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
src/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-12-22T05:39:02.291Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: lib/ripser-1.2.1/Makefile:1-18
Timestamp: 2025-12-22T05:39:02.291Z
Learning: In fluent/fluent-bit, the Makefile at lib/ripser-1.2.1/Makefile is imported from upstream Ripser and is not used in the actual build process. The project uses CMake for building (lib/ripser-1.2.1/CMakeLists.txt), so changes to the imported Makefile are not necessary.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
src/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
src/CMakeLists.txtCMakeLists.txt
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
src/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
src/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
src/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
src/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hCMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-09-14T09:46:09.531Z
Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T07:28:37.083Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10832
File: src/wasm/CMakeLists.txt:112-131
Timestamp: 2025-09-04T07:28:37.083Z
Learning: In fluent-bit CMake files, the user cosmo0920 prefers treating Git as a command rather than a package, emphasizing that Git is not a pkg-config retrievable package but just a command.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
examples/s3_output/README.md
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
cmake/parquet.cmake
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3.c
🪛 Cppcheck (2.19.0)
plugins/out_s3/s3.c
[error] 791-791: failed to expand 'flb_plg_error', it is invalid to use a preprocessor directive as macro parameter
(syntaxError)
🪛 LanguageTool
examples/s3_output/README.md
[grammar] ~56-~56: Ensure spelling is correct
Context: ..., 201, 404, 500, 401) - Response times (2ms to 5000ms) - Client information (IP, us...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~465-~465: Ensure spelling is correct
Context: ...les uploaded when reaching 50MB (before 60min timeout) Verification Steps: 1. R...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
examples/s3_output/README.md
112-112: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
177-177: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
270-270: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (11)
plugins/out_s3/s3.h (1)
123-199: New format/schema fields and S3 format macros look consistent with implementationAppending
format,schema_str, the JSON/PARQUET macros, and the updateds3_put_objectprototype matches hows3.cnow routes JSON vs Parquet and SDS bodies. No structural or API consistency issues here.Also applies to: 201-204, 233-234
plugins/out_s3/s3.c (7)
192-239: Header logic correctly skips outer Content-Encoding for ParquetThe updated
create_headersnow omitsContent-Encodingwhenctx->format == FLB_S3_FORMAT_PARQUET, matching the fact that Parquet uses its own internal compression. This aligns headers with the new upload semantics.
777-823: Format selection and Parquet encoder guards are consistent with new designThe new
formatparsing and#ifdef FLB_HAVE_PARQUET_ENCODERchecks correctly:
- Default to JSON unless Parquet was set (including via legacy compression mapping once fixed).
- Reject
format=parquetbuilds that lackFLB_PARQUET_ENCODER.- For Parquet builds, keep
ctx->compressionfor internal Parquet compression only and skip outer compression.This matches the intended separation of JSON vs Parquet and internal-vs-outer compression.
1663-1800: Updateds3_put_objectsignature and SDS handling are consistentThe new
s3_put_objecttakingflb_sds_t body, size_t body_sizeis wired correctly:
- Callers pass SDS buffers and later destroy them (or the owning wrappers).
- MD5 computation and HTTP request use the SDS pointer as a regular
char *.- Error paths free
uriand preserveseq_indexsemantics via thedecrement_indexlabel.No memory lifetime issues are apparent around the new body type.
1991-2015: SDS buffer ownership insend_upload_requestandunit_test_flushis now explicitBoth paths:
- Create an SDS buffer via
construct_request_buffer,- Pass it into
upload_data,- Always
flb_sds_destroy(buffer)after the call,which removes the previous mixed ownership around raw buffers and avoids leaks in error paths.
Also applies to: 3706-3724
3953-4089: Flush path correctly preserves raw MsgPack and defers formatting
cb_s3_flushnow:
- Stores raw MsgPack into an SDS (
chunk),- Uses
s3_store_file_get/buffer_chunkto persist raw MsgPack,- Only converts to JSON/Parquet inside
construct_request_bufferright before upload.This is the right abstraction for supporting multiple output formats (JSON vs Parquet) without duplicating buffer logic.
4210-4231: Compression/format/schema/upload_timeout config docs largely match behavior, but note legacy compression caveatThe updated config map entries:
- Document
compressionasnone|gzip|snappy|zstdwith legacyarrow/parquetvalues deprecated,- Introduce
formatandschema_str(required forformat=parquet),- Add a documented
upload_timeoutparameter, and- Add
authorization_endpoint_bearer_token,which aligns with the new logic in
cb_s3_initand the Parquet encoder path—once the legacycompression=arrow|parquethandling is fixed as noted above.Also applies to: 4381-4413
1450-1660: Raw MsgPack buffering + deferred JSON/Parquet conversion is correctly implementedThe new
construct_request_buffer/put_all_chunksflow:
- Reads raw MsgPack from
s3_store_file_read,- Optionally appends in-memory MsgPack (
new_data) viaflb_realloc+memcpy,- Converts once per upload to either:
log_keyprojection,- JSON lines, or
- Parquet via
flb_msgpack_raw_to_parquet(withschema_strandctx->compression),- Properly cleans up all internal allocations (msgpack buffer, intermediate formatted data) on all error paths before returning,
- Never sets
*out_bufon error, ensuring callers receive only valid or NULL pointers, and- Returns an
flb_sds_twhich all four callers properly destroy on the success path.Chunk locking/unlocking on error paths is balanced throughout. Memory management is sound.
src/CMakeLists.txt (1)
157-166: Parquet encoder and SASL library wiring in src/CMakeLists.txt look correct
flb_parquet_impl.cppis only built whenFLB_PARQUET_ENCODERis enabled and is explicitly marked as C++, matching the Arrow/Parquet dependency.ARROW_LIBRARIES/PARQUET_LIBRARIESare only appended toFLB_DEPSunder the same flag, consistent withcmake/parquet.cmake.- SASL is linked into both
fluent-bit-sharedandfluent-bit-binonly whenFLB_SASL_ENABLEDis set, so non-Kafka or non-SASL builds remain unaffected.Also applies to: 398-405, 525-528, 600-603
include/fluent-bit/aws/flb_aws_compress.h (1)
23-47: Compression enum layout is fine; ensure callers don’t rely on ARROW/PARQUET inflb_aws_compression_get_typeThe new assignments (NONE=0, GZIP=1, SNAPPY=2, ZSTD=3, ARROW=4 [deprecated], PARQUET=5 [“use format=parquet instead”]) are internally consistent and safe as long as all code uses the macros, not raw integers.
Given
flb_aws_compress.cnow only recognizesgzip|snappy|zstd, ARROW/PARQUET are effectively compile-time constants for legacy conditionals; callers must not expectflb_aws_compression_get_type()to ever return these values anymore. The S3 plugin currently still does (and should be fixed as noted ins3.c).src/aws/flb_aws_compress.c (1)
24-38: SNAPPY support is wired correctly; note that ARROW/PARQUET are no longer valid compression types
flb_snappy_compress_wrappercorrectly adaptsflb_snappy_compressto the common(void **out_data, size_t *out_len)signature and is registered under"snappy"incompression_options.- The options table now only contains true compression algorithms (gzip/snappy/zstd); ARROW/PARQUET have been removed, so
flb_aws_compression_get_type("arrow")/"parquet"will now return-1.This matches the new separation between compression and format conversion, but requires callers (notably the S3 plugin) to treat
"arrow"/"parquet"as legacy format aliases rather than compression types—see the earlier comment ins3.c.Also applies to: 57-78
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
5e868fc to
2555471
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (3)
examples/s3_output/athena-queries.sql (1)
168-181: ReplacePERCENTILEwithapprox_percentilefor Athena compatibility.Athena does not support the
PERCENTILE(duration_ms, 0.95)syntax. Useapprox_percentile(duration_ms, 0.95)instead.🔎 Proposed fix
SELECT path, method, COUNT(*) as request_count, AVG(duration_ms) as avg_duration_ms, - PERCENTILE(duration_ms, 0.95) as p95_duration_ms, + approx_percentile(duration_ms, 0.95) as p95_duration_ms, MAX(duration_ms) as max_duration_ms FROM access_logs_parquetcmake/parquet.cmake (1)
149-170: Remove theinclude/arrowsuffix from Arrow header search.The
PATH_SUFFIXES include/arrowinfind_pathfor Arrow headers is incorrect. When searching forarrow/api.h, CMake would find it at${prefix}/include/arrow/arrow/api.hwhich doesn't exist. The correct include directory should be${prefix}/includeso that#include <arrow/api.h>resolves correctly.🔎 Proposed fix
# Search for Arrow if(NOT ARROW_FOUND) find_path(ARROW_INCLUDE_DIR NAMES arrow/api.h PATHS ${SEARCH_PATHS} - PATH_SUFFIXES ${INCLUDE_SUFFIXES} include/arrow + PATH_SUFFIXES ${INCLUDE_SUFFIXES} NO_DEFAULT_PATH )plugins/out_s3/s3.c (1)
744-824: Legacy compression path should explicitly default to GZIP.The comment at line 764 says "Keep default gzip compression" but
ctx->compressionremainsFLB_AWS_COMPRESS_NONE(set at line 745). This contradicts user expectations for backward compatibility.🔎 Suggested fix
#ifndef FLB_HAVE_PARQUET_ENCODER flb_plg_error(ctx->ins, "Parquet format is not supported in this build. " "Rebuild with -DFLB_PARQUET_ENCODER=On."); return -1; #else ctx->format = FLB_S3_FORMAT_PARQUET; - /* Keep default gzip compression */ + /* Default to gzip compression for backward compatibility */ + if (ctx->compression == FLB_AWS_COMPRESS_NONE) { + ctx->compression = FLB_AWS_COMPRESS_GZIP; + } #endif
🧹 Nitpick comments (3)
examples/s3_output/README.md (2)
112-114: Add language specifier to fenced code block.The code block at line 112 is missing a language specifier, which affects syntax highlighting and linting.
🔎 Proposed fix
-``` +```text Upload Trigger = upload_timeout reached OR total_file_size reached</details> --- `177-181`: **Add language specifier to fenced code block.** The code block showing S3 upload paths is missing a language specifier. <details> <summary>🔎 Proposed fix</summary> ```diff -``` +```text s3://s3-bucket-kafka-sink/fluent-bit-logs/ ├── putobject/app/day=YYYYMMDD/ # Case 1: Small files via PutObject └── multipart/access/day=YYYYMMDD/ # Case 2: Large files via Multipart Upload</details> </blockquote></details> <details> <summary>src/flb_parquet_impl.cpp (1)</summary><blockquote> `708-736`: **Memory management is correct, but ownership should be documented.** The function allocates output_buffer with flb_malloc (line 710) and copies Arrow data into it (line 716). Exception handlers properly clean up on error (line 726). **Suggestion:** Add a comment in the function documentation (or in the header file) explicitly stating that the caller must free the returned buffer with flb_free(). </blockquote></details> </blockquote></details> <details> <summary>📜 Review details</summary> **Configuration used**: defaults **Review profile**: CHILL **Plan**: Pro <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between 5e868fceaf3521954dee257936901981215314ea and 2555471660ff1c95ba1231130e46af6cf36d9017. </details> <details> <summary>📒 Files selected for processing (19)</summary> * `.github/workflows/unit-tests.yaml` * `CMakeLists.txt` * `cmake/parquet.cmake` * `examples/s3_output/README.md` * `examples/s3_output/athena-queries.sql` * `examples/s3_output/fluent-bit-s3-parquet.conf` * `examples/s3_output/run-s3-parquet-test.sh` * `include/fluent-bit/aws/flb_aws_compress.h` * `include/fluent-bit/flb_parquet.h` * `plugins/out_s3/s3.c` * `plugins/out_s3/s3.h` * `src/CMakeLists.txt` * `src/aws/CMakeLists.txt` * `src/aws/compression/CMakeLists.txt` * `src/aws/compression/arrow/CMakeLists.txt` * `src/aws/compression/arrow/compress.c` * `src/aws/compression/arrow/compress.h` * `src/aws/flb_aws_compress.c` * `src/flb_parquet_impl.cpp` </details> <details> <summary>💤 Files with no reviewable changes (5)</summary> * src/aws/compression/arrow/compress.h * src/aws/CMakeLists.txt * src/aws/compression/CMakeLists.txt * src/aws/compression/arrow/CMakeLists.txt * src/aws/compression/arrow/compress.c </details> <details> <summary>🚧 Files skipped from review as they are similar to previous changes (4)</summary> * include/fluent-bit/flb_parquet.h * src/aws/flb_aws_compress.c * examples/s3_output/run-s3-parquet-test.sh * examples/s3_output/fluent-bit-s3-parquet.conf </details> <details> <summary>🧰 Additional context used</summary> <details> <summary>🧠 Learnings (16)</summary> <details> <summary>📓 Common learnings</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like#ifdef FLB_HAVE_ZSTD. Unlike some other optional components such as ARROW/PARQUET (which use#ifdef FLB_HAVE_ARROWguards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.</details> <details> <summary>📚 Learning: 2025-08-07T10:15:46.187Z</summary>Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.**Applied to files:** - `cmake/parquet.cmake` - `.github/workflows/unit-tests.yaml` </details> <details> <summary>📚 Learning: 2025-08-31T12:46:11.940Z</summary>Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.**Applied to files:** - `cmake/parquet.cmake` - `.github/workflows/unit-tests.yaml` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-08-31T12:46:11.940Z</summary>Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka isFLB_PREFER_SYSTEM_LIB_KAFKA=ON.**Applied to files:** - `cmake/parquet.cmake` - `.github/workflows/unit-tests.yaml` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-08-29T06:25:27.250Z</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like#ifdef FLB_HAVE_ZSTD. Unlike some other optional components such as ARROW/PARQUET (which use#ifdef FLB_HAVE_ARROWguards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.**Applied to files:** - `.github/workflows/unit-tests.yaml` - `include/fluent-bit/aws/flb_aws_compress.h` - `plugins/out_s3/s3.c` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-08-29T06:24:55.855Z</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.**Applied to files:** - `.github/workflows/unit-tests.yaml` - `src/flb_parquet_impl.cpp` - `include/fluent-bit/aws/flb_aws_compress.h` - `plugins/out_s3/s3.c` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-08-29T06:25:27.250Z</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like#ifdef FLB_HAVE_ZSTD. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.**Applied to files:** - `.github/workflows/unit-tests.yaml` - `include/fluent-bit/aws/flb_aws_compress.h` - `plugins/out_s3/s3.c` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-08-29T06:24:26.170Z</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.**Applied to files:** - `.github/workflows/unit-tests.yaml` - `include/fluent-bit/aws/flb_aws_compress.h` - `plugins/out_s3/s3.c` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-08-29T06:24:44.797Z</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.**Applied to files:** - `.github/workflows/unit-tests.yaml` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-09-08T11:21:33.975Z</summary>Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.**Applied to files:** - `.github/workflows/unit-tests.yaml` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-08-29T06:25:02.561Z</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree atlib/zstd-1.5.7and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards likeFLB_HAVE_ZSTDand is always available. Headers like<fluent-bit/flb_zstd.h>can be included directly without guards.**Applied to files:** - `include/fluent-bit/aws/flb_aws_compress.h` - `plugins/out_s3/s3.c` - `src/CMakeLists.txt` - `CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-09-04T12:35:22.872Z</summary>Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.**Applied to files:** - `plugins/out_s3/s3.c` </details> <details> <summary>📚 Learning: 2025-12-22T05:39:02.291Z</summary>Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: lib/ripser-1.2.1/Makefile:1-18
Timestamp: 2025-12-22T05:39:02.291Z
Learning: In fluent/fluent-bit, the Makefile at lib/ripser-1.2.1/Makefile is imported from upstream Ripser and is not used in the actual build process. The project uses CMake for building (lib/ripser-1.2.1/CMakeLists.txt), so changes to the imported Makefile are not necessary.**Applied to files:** - `src/CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-09-14T09:46:09.531Z</summary>Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.**Applied to files:** - `src/CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-09-04T07:28:37.083Z</summary>Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10832
File: src/wasm/CMakeLists.txt:112-131
Timestamp: 2025-09-04T07:28:37.083Z
Learning: In fluent-bit CMake files, the user cosmo0920 prefers treating Git as a command rather than a package, emphasizing that Git is not a pkg-config retrievable package but just a command.**Applied to files:** - `src/CMakeLists.txt` </details> <details> <summary>📚 Learning: 2025-11-21T06:23:29.770Z</summary>Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.**Applied to files:** - `examples/s3_output/README.md` </details> </details><details> <summary>🧬 Code graph analysis (2)</summary> <details> <summary>src/flb_parquet_impl.cpp (1)</summary><blockquote> <details> <summary>include/fluent-bit/flb_mem.h (1)</summary> * `flb_free` (126-128) </details> </blockquote></details> <details> <summary>plugins/out_s3/s3.h (1)</summary><blockquote> <details> <summary>plugins/out_s3/s3.c (1)</summary> * `s3_put_object` (1663-1800) </details> </blockquote></details> </details><details> <summary>🪛 Cppcheck (2.19.0)</summary> <details> <summary>plugins/out_s3/s3.c</summary> [error] 791-791: failed to expand 'flb_plg_error', it is invalid to use a preprocessor directive as macro parameter (syntaxError) </details> </details> <details> <summary>🪛 LanguageTool</summary> <details> <summary>examples/s3_output/README.md</summary> [grammar] ~56-~56: Ensure spelling is correct Context: ..., 201, 404, 500, 401) - Response times (2ms to 5000ms) - Client information (IP, us... (QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1) --- [grammar] ~465-~465: Ensure spelling is correct Context: ...les uploaded when reaching 50MB (before 60min timeout) **Verification Steps:** 1. R... (QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1) </details> </details> <details> <summary>🪛 markdownlint-cli2 (0.18.1)</summary> <details> <summary>examples/s3_output/README.md</summary> 112-112: Fenced code blocks should have a language specified (MD040, fenced-code-language) --- 177-177: Fenced code blocks should have a language specified (MD040, fenced-code-language) --- 270-270: Fenced code blocks should have a language specified (MD040, fenced-code-language) </details> </details> </details> <details> <summary>🔇 Additional comments (28)</summary><blockquote> <details> <summary>.github/workflows/unit-tests.yaml (2)</summary><blockquote> `56-79`: **CI matrix configuration for Parquet encoder looks correct.** The Parquet encoder test configuration properly: - Uses the new `FLB_PARQUET_ENCODER` flag - Excludes Clang compiler (consistent with other specialized build options like coverage) - Aligns with the broader transition from Arrow GLib to C++ Arrow --- `117-125`: **Apache Arrow setup uses correct packages and official distribution source.** The configuration correctly: - Uses `packages.apache.org` as the official distribution source - Installs `libarrow-dev` and `libparquet-dev` (C++ libraries) instead of the GLib variants - Aligns the step name and condition with the new `FLB_PARQUET_ENCODER` flag </blockquote></details> <details> <summary>include/fluent-bit/aws/flb_aws_compress.h (2)</summary><blockquote> `33-46`: **Good documentation distinguishing compression from format conversion.** The comment block clearly explains: - The distinction between actual compression algorithms and file format conversions - Arrow's deprecation status with guidance not to use in new code - That Parquet should be used via `format=parquet` rather than `compression=parquet` This helps prevent user confusion about the difference between compression and format conversion. --- `24-46`: The concern about backward compatibility from renumbering constants is not applicable. The compression type configuration uses string-based names ("gzip", "snappy", "zstd", "parquet") via an internal lookup table, not numeric values. The enum constants are purely internal implementation details that never get serialized, persisted, or exposed to external code. Renumbering these internal values has no backward compatibility impact. </blockquote></details> <details> <summary>examples/s3_output/athena-queries.sql (1)</summary><blockquote> `1-80`: **Well-structured Athena table definitions and query examples.** The SQL file provides comprehensive examples covering: - Partition projection for automatic partition discovery - Both JSON and Parquet format table definitions - Realistic query patterns for log analysis This is valuable documentation for users implementing the new Parquet format support. </blockquote></details> <details> <summary>examples/s3_output/README.md (1)</summary><blockquote> `1-80`: **Excellent documentation for Parquet format support.** The README provides comprehensive coverage of: - Upload methods (PutObject vs Multipart) - Format and compression options - Schema definition and type handling with best practices - Troubleshooting guidance - Performance testing scenarios This is high-quality documentation that will help users adopt the new Parquet format feature. </blockquote></details> <details> <summary>plugins/out_s3/s3.h (2)</summary><blockquote> `195-203`: **Format and schema fields properly added for Parquet support.** The new struct members and constants are well-organized: - `format` field for output format selection - `schema_str` for user-defined Parquet schema - Clear format type constants with JSON=0 as default --- `233-234`: **Updated `s3_put_object` signature uses SDS buffer type.** The function signature change from raw buffer to `flb_sds_t` aligns with the SDS-based buffer handling throughout the S3 upload flow. </blockquote></details> <details> <summary>src/CMakeLists.txt (3)</summary><blockquote> `157-166`: **Parquet encoder C++ integration correctly configured.** The Parquet encoder integration: - Adds `flb_parquet_impl.cpp` under `FLB_PARQUET_ENCODER` guard - Properly marks the file as C++ using `set_source_files_properties` - Follows the existing pattern for conditional source inclusion --- `398-405`: **Parquet library dependencies properly linked.** The `ARROW_LIBRARIES` and `PARQUET_LIBRARIES` are correctly added to `FLB_DEPS` when `FLB_PARQUET_ENCODER` is enabled, ensuring proper linkage with the detected Arrow/Parquet libraries from `cmake/parquet.cmake`. --- `525-528`: **Verify SASL linking changes are intentionally part of this PR.** The SASL library linking for Kafka support (`sasl2`) is added here but appears unrelated to the Parquet encoder feature. Please confirm this change is intentional for this PR or if it should be in a separate commit. Also applies to: 600-605 </blockquote></details> <details> <summary>cmake/parquet.cmake (1)</summary><blockquote> `1-83`: **Well-designed cross-platform library detection module.** The parquet.cmake module provides robust library detection with: - Three fallback methods (pkg-config → CMake config → manual search) - Platform-specific path handling for Windows, macOS (including Apple Silicon), and Linux - Proper handling of both shared and static library targets - Clear status messaging </blockquote></details> <details> <summary>CMakeLists.txt (3)</summary><blockquote> `202-202`: **New `FLB_PARQUET_ENCODER` option properly defined.** The option follows the existing naming convention for encoders (`FLB_AVRO_ENCODER`) and is disabled by default, which is appropriate for an optional feature requiring external dependencies. --- `284-286`: **`FLB_ALL` properly enables both encoders.** The "Enable all features" mode now consistently enables both Avro and Parquet encoders. --- `1311-1329`: **Parquet encoder integration is well-structured.** The configuration block: - Enables C++ language support as required by Arrow/Parquet - Uses C++17 standard (addressing past review comment about Arrow 10.0+ requiring C++17) - Integrates with `cmake/parquet.cmake` for library detection - Gracefully disables the feature with warnings when dependencies are missing </blockquote></details> <details> <summary>src/flb_parquet_impl.cpp (6)</summary><blockquote> `1-40`: **LGTM: Headers and includes are properly organized.** The extern "C" wrapper correctly isolates C headers from C++ code, and all necessary dependencies are included. --- `41-94`: **LGTM: Converter class design is clean and comprehensive.** The statistics tracking for overflows and type conversions will help users identify data quality issues. --- `139-158`: **Overflow handling correctly implemented.** The uint64 to int64 overflow check at lines 142-146 properly addresses the previous review concern. Values exceeding INT64_MAX are clamped and tracked in statistics. --- `220-257`: **Timestamp unit handling correctly implemented.** The function now properly reads the target time unit from the builder's type (lines 221-223) and applies the appropriate multiplier. This addresses the previous review concern. **Note:** The code assumes input timestamps are in seconds (line 225). Ensure this is documented in the public API or consider accepting a parameter for input unit. --- `458-547`: **Excellent: Immediate record processing avoids dangling pointer issue.** The lambda processes each msgpack record immediately within the unpack loop (lines 469-546), which correctly addresses the past review comment about storing msgpack_object pointers that become invalid. The use of string_view for field lookup (line 493) also avoids unnecessary string copies. --- `332-443`: **Schema parsing is robust with good error messages.** The JSON schema parsing properly validates required fields (lines 365-377) and provides helpful error messages that include the problematic schema content (lines 337-338, 347-348). Timestamp unit parsing (lines 410-420) correctly handles all standard Arrow time units. </blockquote></details> <details> <summary>plugins/out_s3/s3.c (7)</summary><blockquote> `42-57`: **LGTM: Proper conditional compilation for Parquet support.** The FLB_HAVE_PARQUET_ENCODER guard ensures the header is only included when Parquet support is enabled. --- `195-229`: **Correct: Content-Encoding header skipped for Parquet.** Lines 196-197 and 228-229 properly skip the Content-Encoding header when `format=parquet`, since Parquet handles compression internally within the file format. This avoids double-compression overhead. --- `1244-1277`: **LGTM: Compression properly skipped for Parquet with correct memory management.** The check at line 1245 ensures Parquet format bypasses outer compression. The compressed data is properly converted to flb_sds_t (lines 1260-1272) for consistent memory management, and cleanup is handled in all code paths (lines 1333-1335, 1362-1364, 1377-1379, 1388-1391). --- `1582-1630`: **Parquet conversion properly integrated with good error handling.** The Parquet conversion block (lines 1582-1630) correctly: - Validates schema_str is provided (lines 1588-1596) - Calls the Parquet encoder with compression setting (lines 1598-1603) - Converts the result to flb_sds_t for consistent memory management (lines 1615-1627) - Frees the intermediate parquet_buffer (line 1617) - Handles all error paths with proper cleanup Verify that the schema_str requirement is documented in user-facing documentation and error messages are clear enough for users to understand the schema format. --- `1663-1664`: **LGTM: Signature change to flb_sds_t improves consistency.** The s3_put_object signature change to accept flb_sds_t (line 1664) aligns with the overall refactoring for consistent SDS-based buffer management. --- `3953-3962`: **Excellent architectural decision with clear documentation.** The comment at lines 3953-3956 clearly explains the design choice to store raw msgpack and defer format conversion until upload. This is particularly beneficial for Parquet, which can batch multiple records into optimized columnar storage. --- `4210-4231`: **Config map correctly defined with comprehensive documentation.** The configuration parameters are properly defined: - `compression` and `format` use offset 0 (lines 4211, 4220), indicating manual parsing in cb_s3_init, which is correct - `schema_str` is mapped to `ctx->schema_str` (line 4228) - Descriptions clearly document supported values, defaults, and usage (lines 4212-4217, 4221-4225, 4229-4231) **Note:** The past review comment about compression type mismatch is not applicable here since offset 0 means no automatic field mapping. </blockquote></details> </blockquote></details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
664cb4e to
eff671c
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
plugins/out_s3/s3.c (2)
1217-1324: Usesize_to_send(compressed size) for routing decisions and loggingInside
upload_data(), after optional compression you introduce:flb_sds_t body_to_send = body; size_t size_to_send = body_size; ... preCompress_size = body_size; body_to_send = compressed_body; size_to_send = temp_size;but the subsequent logic still uses the original
body_sizefor decisions and logging:
- Timeout/size checks for whether to
goto put_objectvsmultipart.- The “Pre-compression ... After compression ...” log still uses
body_sizefor the “after” value.- Existing upload path compares
body_sizetoMIN_CHUNKED_UPLOAD_SIZE.With compression enabled, this means routing decisions and logs are based on the uncompressed size, which diverges from the actual payload size now stored in
size_to_send. It also breaks the intent of the informational log (both values end up identical).Switching those uses of
body_sizetosize_to_sendpreserves the old semantics (decisions based on the payload actually sent) and fixes the log.Suggested diff
- else if (body_size >= ctx->file_size) { + else if (size_to_send >= ctx->file_size) { @@ - else if(body_size > MIN_CHUNKED_UPLOAD_SIZE) { + else if(size_to_send > MIN_CHUNKED_UPLOAD_SIZE) { @@ - else { - if ((ctx->use_put_object == FLB_FALSE && (ctx->compression == FLB_AWS_COMPRESS_GZIP || ctx->compression == FLB_AWS_COMPRESS_ZSTD))) { - flb_plg_info(ctx->ins, "Pre-compression upload_chunk_size= %zu, After compression, chunk is only %zu bytes, " - "the chunk was too small, using PutObject to upload", preCompress_size, body_size); - } + else { + if (ctx->use_put_object == FLB_FALSE && + (ctx->compression == FLB_AWS_COMPRESS_GZIP || + ctx->compression == FLB_AWS_COMPRESS_ZSTD)) { + flb_plg_info(ctx->ins, + "Pre-compression upload_chunk_size= %zu, " + "After compression, chunk is only %zu bytes, " + "the chunk was too small, using PutObject to upload", + preCompress_size, size_to_send); + } @@ - if (body_size < MIN_CHUNKED_UPLOAD_SIZE) { + if (size_to_send < MIN_CHUNKED_UPLOAD_SIZE) {
1450-1500: SDS lifetime is mostly correct; fix leak inunit_test_flushThe SDS refactor is generally consistent:
construct_request_buffer()now always returnsflb_sds_tand callers destroy buffers withflb_sds_destroy().send_upload_request(),put_all_chunks(), andcb_s3_upload()correctly treat the returned buffer as an SDS and free it onceupload_data()completes.upload_data()cleans up the optionalcompressed_bodySDS in all exit paths.One leak remains in
unit_test_flush():s3_store_buffer_put(ctx, upload_file, tag, tag_len, chunk, (size_t) chunk_size, file_first_log_time); ret = construct_request_buffer(ctx, chunk, upload_file, &buffer, &buffer_size); ... ret = upload_data(ctx, upload_file, m_upload_file, buffer, buffer_size, tag, tag_len); /* buffer destroyed, but chunk is never freed */Unlike
buffer_chunk()and other call sites,chunkis never destroyed here, so every unit‑test flush leaks one SDS buffer whenFLB_S3_PLUGIN_UNDER_TESTis set.You can mirror
buffer_chunk()and freechunkimmediately afterconstruct_request_buffer():Suggested diff
- s3_store_buffer_put(ctx, upload_file, tag, tag_len, - chunk, (size_t) chunk_size, file_first_log_time); - ret = construct_request_buffer(ctx, chunk, upload_file, &buffer, &buffer_size); - if (ret < 0) { + s3_store_buffer_put(ctx, upload_file, tag, tag_len, + chunk, (size_t) chunk_size, file_first_log_time); + ret = construct_request_buffer(ctx, chunk, upload_file, &buffer, &buffer_size); + flb_sds_destroy(chunk); + if (ret < 0) { flb_plg_error(ctx->ins, "Could not construct request buffer for %s", upload_file->file_path); FLB_OUTPUT_RETURN(FLB_RETRY); }Also applies to: 1991-2015, 3701-3724
🧹 Nitpick comments (9)
examples/s3_output/run-s3-parquet-test.sh (2)
1-5: Consider adding shell script safety options.For test scripts, adding safety options helps catch errors early:
🔎 Suggested improvement
#!/bin/bash +set -euo pipefail echo "=== Fluent Bit S3 Comprehensive Test ===" echo ""
37-39: Cleanup section appears incomplete.The script echoes "Cleaning up..." but doesn't perform any actual cleanup. If cleanup is intentionally omitted (e.g., to allow inspection of test artifacts), consider adding a comment explaining this. Otherwise, add the cleanup logic:
🔎 Suggested completion
# Cleanup echo "" echo "Cleaning up..." +# Uncomment below to remove test artifacts after run +# rm -rf "$STORE_DIR" +echo "Test artifacts preserved in $STORE_DIR for inspection"examples/s3_output/README.md (3)
112-114: Add language specifier to fenced code block.Per markdownlint MD040, fenced code blocks should have a language specified for proper syntax highlighting.
🔎 Suggested fix
-``` +```text Upload Trigger = upload_timeout reached OR total_file_size reached</details> --- `177-181`: **Add language specifier to fenced code block.** <details> <summary>🔎 Suggested fix</summary> ```diff -``` +```text s3://s3-bucket-kafka-sink/fluent-bit-logs/ ├── putobject/app/day=YYYYMMDD/ # Case 1: Small files via PutObject └── multipart/access/day=YYYYMMDD/ # Case 2: Large files via Multipart Upload</details> --- `270-275`: **Add language specifier to fenced code block.** <details> <summary>🔎 Suggested fix</summary> ```diff -``` +```text [parquet] Data quality summary for 53206 records: [parquet] Missing non-nullable fields (defaults used): [parquet] field='level' count=52506 [parquet] field='timestamp' count=52506</details> </blockquote></details> <details> <summary>cmake/parquet.cmake (1)</summary><blockquote> `149-171`: **Manual include-path search is fine but extra `parquet` suffix is probably unnecessary** The manual search for Parquet uses: ```cmake PATH_SUFFIXES ${INCLUDE_SUFFIXES} parquetGiven
INCLUDE_SUFFIXESis alreadyinclude, this will check both${prefix}/include/parquet/...(viainclude+ theparquet/api/...name) and${prefix}/parquet/..., which is a bit unusual. It’s not harmful, but simplifying to just${INCLUDE_SUFFIXES}would avoid probing odd${prefix}/parquettrees and be more in line with the Arrow search just above.Also applies to: 173-195
CMakeLists.txt (1)
679-687: Consider unifying the C++ standard to avoid C++11/C++17 mixThe simdutf block sets
CMAKE_CXX_STANDARD 11, while the Parquet block later sets it to 17. CMake will honor the last assignment (17 when Parquet is enabled), but this mix of settings is slightly confusing and could surprise future refactors.You might want to:
- Drop the explicit
set(CMAKE_CXX_STANDARD 11)in the simdutf block, or- Set it to 17 there as well when Parquet support is enabled.
Functionally it works today; this is mainly about clarity and avoiding subtle build surprises.
Also applies to: 1311-1316
plugins/out_s3/s3.c (2)
42-57: Avoid duplicate forward declarations forget_upload/create_upload
get_uploadandcreate_uploadare forward‑declared twice (once in the new “Forward declarations” block and again just before their definitions). This is harmless but unnecessary noise.You can keep the new declaration for
flb_pack_msgpack_extract_log_keyand drop the duplicatedget_upload/create_uploadprototypes to reduce clutter.Also applies to: 86-91
744-823: Format/compression config flow is solid; consider earlyschema_strvalidationThe three‑step flow in
cb_s3_init:
- Defaults
compressiontoNONE, parses thecompressionproperty, and treatsarrow|parquetas legacy values that map toformat=parquet(with compile‑time checks aroundFLB_HAVE_PARQUET_ENCODER).- Adds an explicit
formatoption (json/parquet) that overrides the legacy compression mapping.- Validates that
format=parquetis not allowed when Parquet support isn’t compiled in.This is a good cleanup and keeps legacy configs working.
However,
schema_stris only validated later insideconstruct_request_buffer()andflb_msgpack_raw_to_parquet, which means a misconfiguredformat=parquetwithoutschema_strwill only surface on first flush, failing every chunk.To fail fast and give clearer feedback, you could add an init‑time check like:
#ifdef FLB_HAVE_PARQUET_ENCODER if (ctx->format == FLB_S3_FORMAT_PARQUET && ctx->schema_str == NULL) { flb_plg_error(ctx->ins, "format=parquet requires 'schema_str' to be set"); return -1; } #endifright after the format/validation block.
Also applies to: 4210-4231
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (18)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/parquet.cmakeexamples/s3_output/README.mdexamples/s3_output/athena-queries.sqlexamples/s3_output/fluent-bit-s3-parquet.confexamples/s3_output/run-s3-parquet-test.shinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_kinesis_firehose/firehose.cplugins/out_s3/s3.cplugins/out_s3/s3.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/compression/arrow/CMakeLists.txtsrc/aws/compression/arrow/compress.csrc/aws/compression/arrow/compress.hsrc/aws/flb_aws_compress.c
💤 Files with no reviewable changes (5)
- src/aws/compression/arrow/compress.c
- src/aws/compression/arrow/compress.h
- src/aws/compression/CMakeLists.txt
- src/aws/compression/arrow/CMakeLists.txt
- src/aws/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (1)
- examples/s3_output/athena-queries.sql
🧰 Additional context used
🧠 Learnings (16)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
.github/workflows/unit-tests.yamlcmake/parquet.cmake
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/parquet.cmakesrc/CMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/parquet.cmakesrc/CMakeLists.txt
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
.github/workflows/unit-tests.yamlinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/CMakeLists.txtplugins/out_s3/s3.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
.github/workflows/unit-tests.yamlinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/CMakeLists.txtplugins/out_s3/s3.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
.github/workflows/unit-tests.yamlinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/CMakeLists.txtplugins/out_s3/s3.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
.github/workflows/unit-tests.yamlinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.cCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/CMakeLists.txtplugins/out_s3/s3.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
.github/workflows/unit-tests.yamlsrc/aws/flb_aws_compress.cCMakeLists.txtsrc/CMakeLists.txt
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtsrc/CMakeLists.txt
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
examples/s3_output/README.md
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
include/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.cCMakeLists.txtsrc/CMakeLists.txtplugins/out_s3/s3.c
📚 Learning: 2025-12-22T05:39:02.291Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: lib/ripser-1.2.1/Makefile:1-18
Timestamp: 2025-12-22T05:39:02.291Z
Learning: In fluent/fluent-bit, the Makefile at lib/ripser-1.2.1/Makefile is imported from upstream Ripser and is not used in the actual build process. The project uses CMake for building (lib/ripser-1.2.1/CMakeLists.txt), so changes to the imported Makefile are not necessary.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-14T09:46:09.531Z
Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T07:28:37.083Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10832
File: src/wasm/CMakeLists.txt:112-131
Timestamp: 2025-09-04T07:28:37.083Z
Learning: In fluent-bit CMake files, the user cosmo0920 prefers treating Git as a command rather than a package, emphasizing that Git is not a pkg-config retrievable package but just a command.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3.c
🧬 Code graph analysis (2)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (1)
s3_put_object(1663-1800)
plugins/out_s3/s3.c (6)
src/flb_output.c (1)
flb_output_get_property(1108-1111)src/aws/flb_aws_compress.c (2)
flb_aws_compression_get_type(80-97)flb_aws_compression_compress(99-116)src/flb_sds.c (2)
flb_sds_create_len(58-76)flb_sds_destroy(389-399)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_s3/s3_multipart.c (1)
upload_part(685-799)src/flb_pack.c (1)
flb_pack_msgpack_to_json_format(1169-1450)
🪛 Cppcheck (2.19.0)
plugins/out_s3/s3.c
[error] 791-791: failed to expand 'flb_plg_error', it is invalid to use a preprocessor directive as macro parameter
(syntaxError)
🪛 LanguageTool
examples/s3_output/README.md
[grammar] ~56-~56: Ensure spelling is correct
Context: ..., 201, 404, 500, 401) - Response times (2ms to 5000ms) - Client information (IP, us...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~465-~465: Ensure spelling is correct
Context: ...les uploaded when reaching 50MB (before 60min timeout) Verification Steps: 1. R...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
examples/s3_output/README.md
112-112: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
177-177: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
270-270: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (18)
plugins/out_kinesis_firehose/firehose.c (1)
496-501: LGTM! Documentation updated to reflect new compression options.The help text correctly lists the now-supported compression algorithms:
gzip,snappy, andzstd. This aligns with the changes inflb_aws_compress.cthat add Snappy support and remove Arrow/Parquet compression options.src/aws/flb_aws_compress.c (2)
31-37: LGTM! Snappy wrapper correctly adapts the function signature.The wrapper properly casts
void *tochar *as required byflb_snappy_compress, and the return value is passed through directly.
57-78: Well-structured separation of compression algorithms from format converters.The updated
compression_optionstable cleanly includes only true compression algorithms (gzip, snappy, zstd), with clear documentation explaining that Arrow/Parquet are format converters handled separately via theformatparameter.include/fluent-bit/aws/flb_aws_compress.h (1)
24-46: Well-documented deprecation and backward compatibility approach.The header cleanly separates compression algorithms from format converters, retains deprecated values for backward compatibility, and provides clear guidance to use
format=parquetinstead ofcompression=parquet. Based on learnings, this follows the established pattern where Arrow/Parquet use conditional compilation guards unlike ZSTD which is always available.examples/s3_output/fluent-bit-s3-parquet.conf (2)
143-145: Parquet schema correctly matches the access log data structure.The
schema_strdefines fields that align with the dummy access log inputs:timestamp,level,message,method,path,status,duration_ms,client_ip, anduser_agent. Usingint64fortimestampandstatus, andfloat64forduration_msare appropriate choices for avoiding overflow issues as noted in the README.
1-117: Comprehensive and well-documented test configuration.The configuration provides excellent coverage of both upload scenarios (PutObject vs Multipart) with realistic traffic patterns. The inline comments clearly explain the rationale for each setting, making this a valuable reference for users.
.github/workflows/unit-tests.yaml (2)
56-57: LGTM! Build flag renamed to reflect Parquet encoder focus.The change from
-DFLB_ARROW=Onto-DFLB_PARQUET_ENCODER=Oncorrectly reflects the shift from Arrow-based compression to Parquet encoder support.
117-125: Apache Arrow C++ package names are correct.The packages
libarrow-devandlibparquet-devare the correct C++ development packages for Apache Arrow and Parquet, providing the required headers and libraries for theflb_parquet_impl.cppimplementation.plugins/out_s3/s3.h (3)
196-199: New Parquet format fields properly integrated into struct.The
formatandschema_strfields provide clean support for runtime format selection and user-defined Parquet schemas. The placement at the end of the struct with clear section comments improves readability.
201-203: LGTM! Format type macros clearly defined.The
FLB_S3_FORMAT_JSON(0) andFLB_S3_FORMAT_PARQUET(1) macros provide a clean enum-like interface for format selection.
233-235: Function signature change verified as consistent across the codebase.All
s3_put_objectcall sites (s3.c:1330 and s3.c:1494) have been properly updated to use the newflb_sds_t bufferparameter. The declaration in s3.h and implementation in s3.c both align with the new signature.src/CMakeLists.txt (2)
157-166: Parquet encoder source and dependency wiring look consistent
flb_parquet_impl.cppis only added whenFLB_PARQUET_ENCODERis enabled and is explicitly marked as C++ viaset_source_files_properties, with Arrow/Parquet libs added toFLB_DEPSunder the same flag. This keeps Parquet support fully optional and avoids impacting non-Parquet builds.Also applies to: 398-405
525-528: SASL linkage for Kafka/MSK is correctly scoped to SASL-enabled buildsConditionally linking
sasl2intofluent-bit-sharedandfluent-bit-binunderFLB_SASL_ENABLEDmatches how Kafka/MSK SASL is detected and avoids dragging SASL into builds that don’t need it.Also applies to: 600-606
CMakeLists.txt (1)
200-202: Parquet encoder option and C++17 enabling are wired correctly
FLB_PARQUET_ENCODERis now a dedicated option, enabled byFLB_ALL, and the Parquet block:
- Calls
enable_language(CXX)and setsCMAKE_CXX_STANDARD 17when the encoder is requested.- Delegates Arrow/Parquet discovery to
cmake/parquet.cmake.- Only defines
FLB_HAVE_PARQUET_ENCODERand adds include directories when both Arrow and Parquet are found, otherwise it warns and flipsFLB_PARQUET_ENCODERback toOFF.This keeps Parquet strictly opt‑in and prevents partial configuration when dependencies are missing.
Also applies to: 278-287, 1311-1329
plugins/out_s3/s3.c (3)
97-121: Header handling for compression and Parquet looks correctThe new
zstdContent-Encodingheader and the guards increate_headers()to skipContent-Encodingwhenctx->format == FLB_S3_FORMAT_PARQUETalign with the intended behavior:
- GZIP/ZSTD still get the appropriate header for JSON output.
- Parquet relies on its internal codec only, without an outer content‑encoding header.
Also applies to: 183-239
1516-1661: SDS-based request/flush pipeline is a nice cleanupThe move to
flb_sds_tfor all request buffers (construct_request_buffer(),upload_data(),send_upload_request(),put_all_chunks(),cb_s3_upload(),cb_s3_flush()) significantly simplifies memory management:
- All conversions (JSON / log_key / Parquet) produce SDS and are freed in a single place per call chain.
construct_request_buffer()now cleanly owns and frees the intermediate msgpack buffer and unlocks chunks on all error paths.- Callers always destroy the returned SDS (
buffer) afterupload_data()/s3_put_object()regardless of success or failure, avoiding mixedfree()vsflb_sds_destroy()semantics.This should make future format additions (or tweaks to Parquet encoding) much less error‑prone.
Also applies to: 1943-2015, 3440-3488, 3953-4068
4210-4231: Config map docs for compression/format/schema/upload_timeout/auth look coherentThe updated config map entries:
- Clarify that
compressionsupportsnone|gzip|snappy|zstdand thatarrow|parquetare deprecated legacy values.- Introduce an explicit
formatoption (json/parquet) andschema_strfor Parquet, with docs that match the runtime behavior.- Document
upload_timeoutsemantics more clearly and addauthorization_endpoint_bearer_token.These descriptions match the new logic in
cb_s3_init()andconstruct_request_buffer().Also applies to: 4382-4413
cmake/parquet.cmake (1)
249-258: No action needed—the URLs in the code already match the official Apache Arrow documentation.The URLs currently use
https://packages.apache.org/artifactory/arrow/…, which is the correct format per official Apache Arrow installation guides. The/artifactory/arrow/segment is not obsolete; it is the official repository path for both APT and RPM packages. Removing it would break the URLs.Likely an incorrect or invalid review comment.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
examples/s3_output/README.md (1)
112-114: Consider adding language specifiers to fenced code blocks.Several code blocks in this documentation lack language specifiers, which can affect syntax highlighting. Consider adding appropriate language identifiers (e.g.,
text,ini,json) for better readability.tests/internal/parquet.c (1)
27-31: Consider including the header instead of redefining compression constants.These compression type constants are duplicated from
flb_aws_compress.h. If the values change in the source header, these tests will silently use incorrect values.🔎 Proposed fix
-/* Compression types from flb_aws_compress.h */ -#define FLB_AWS_COMPRESS_NONE 0 -#define FLB_AWS_COMPRESS_GZIP 1 -#define FLB_AWS_COMPRESS_SNAPPY 2 -#define FLB_AWS_COMPRESS_ZSTD 3 +#include <fluent-bit/aws/flb_aws_compress.h>If the header cannot be included due to dependencies, consider adding a comment noting that these values must be kept in sync with the source file.
src/flb_parquet_impl.cpp (2)
178-197: Unnecessary try/catch block around strtol.The
try/catchblock on lines 195-197 is unnecessary sincestrtolis a C function that does not throw exceptions. Theerrnoandendptrchecks already handle all error cases. The same pattern appears in other string parsing functions.🔎 Proposed fix
else if (obj->type == MSGPACK_OBJECT_STR) { - try { std::string str(obj->via.str.ptr, obj->via.str.size); char* endptr; errno = 0; long val = std::strtol(str.c_str(), &endptr, 10); if (endptr == str.c_str() || *endptr != '\0' || errno == ERANGE) { return arrow::Status::Invalid("Cannot parse string to int32"); } if (val > INT32_MAX || val < INT32_MIN) { int32_overflow_by_field[field_name]++; return builder->Append(val > INT32_MAX ? INT32_MAX : INT32_MIN); } string_parsed_to_int_by_field[field_name]++; return builder->Append(static_cast<int32_t>(val)); - } catch (...) { - return arrow::Status::Invalid("Cannot parse string to int32"); - } }Note: The
std::stringconstructor can throwstd::bad_alloc, so if you want to be defensive against memory allocation failures, you could keep a narrower catch or let it propagate to the outer try/catch inflb_msgpack_raw_to_parquet.
619-622: Prefer ValueUnsafe() or check status before ValueOrDie().
ValueOrDie()will terminate the program if the status is not OK, but here the status is already checked on line 618. UsingValueUnsafe()orMoveValueUnsafe()would be more semantically correct since you've already verified success.🔎 Proposed fix
if (!schema_result.ok()) { flb_error("[parquet] Failed to parse schema: %s\n", schema_result.status().ToString().c_str()); return NULL; } - auto schema = schema_result.ValueOrDie(); + auto schema = std::move(schema_result).ValueUnsafe();This pattern should be applied to other
ValueOrDie()calls where the status is already checked (lines 635, 859, 877, 899, 922).
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
CMakeLists.txtcmake/kafka.cmakeexamples/s3_output/README.mdlib/avro/CMakeLists.txtlib/jansson-e23f558/CMakeLists.txtsrc/flb_parquet_impl.cpptests/internal/CMakeLists.txttests/internal/parquet.c
🧰 Additional context used
🧠 Learnings (11)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
examples/s3_output/README.md
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
CMakeLists.txtcmake/kafka.cmake
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
CMakeLists.txtcmake/kafka.cmakesrc/flb_parquet_impl.cpp
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
CMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
CMakeLists.txtcmake/kafka.cmake
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
CMakeLists.txtcmake/kafka.cmakesrc/flb_parquet_impl.cpp
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
CMakeLists.txt
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
CMakeLists.txt
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
CMakeLists.txt
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
CMakeLists.txt
🧬 Code graph analysis (1)
tests/internal/parquet.c (1)
src/flb_parquet_impl.cpp (2)
flb_msgpack_raw_to_parquet(579-952)flb_msgpack_raw_to_parquet(579-582)
🪛 LanguageTool
examples/s3_output/README.md
[grammar] ~56-~56: Ensure spelling is correct
Context: ..., 201, 404, 500, 401) - Response times (2ms to 5000ms) - Client information (IP, us...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~737-~737: Ensure spelling is correct
Context: ...les uploaded when reaching 50MB (before 60min timeout) Verification Steps: 1. R...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
examples/s3_output/README.md
112-112: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
177-177: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
286-286: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
294-294: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
305-305: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
315-315: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
328-328: Tables should be surrounded by blank lines
(MD058, blanks-around-tables)
344-344: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
362-362: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
436-436: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
447-447: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
542-542: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (10)
lib/avro/CMakeLists.txt (1)
19-19: CMake version bump aligns with project-wide requirements.The minimum CMake version increase to 3.12 is consistent with modern CMake practices and aligns with the Parquet/Arrow integration requirements introduced in this PR.
lib/jansson-e23f558/CMakeLists.txt (1)
567-569: Test target disabled with documented rationale.Commenting out the
checktarget to avoid conflicts with other libraries is acceptable. The comment explains the rationale clearly.cmake/kafka.cmake (1)
35-42: SASL configuration logic is well-structured.The changes correctly enable built-in SASL features (PLAIN/SCRAM/OAUTHBEARER) unconditionally while making Cyrus-SASL optional for GSSAPI/Kerberos. The comments clearly explain the rationale for each setting.
tests/internal/parquet.c (1)
74-103: Comprehensive test coverage with good structure.The test suite is well-organized into logical categories with good coverage of edge cases, type conversions, and error handling. The consistent pattern of setup/execute/verify/cleanup is followed throughout.
CMakeLists.txt (2)
284-287: Good documentation for FLB_PARQUET_ENCODER exclusion from FLB_ALL.The comment explaining that Parquet encoder requires explicit enabling due to external dependencies is helpful for users and maintainers.
1337-1355: Parquet encoder integration is well-structured.The Parquet encoder block properly:
- Enables C++ with C++17 standard (required for Arrow/Parquet)
- Delegates discovery to
cmake/parquet.cmake, which comprehensively handles detection across platforms with three fallback methods (pkg-config, CMake Config, manual search)- Gracefully disables the feature with clear warnings if dependencies are not found
The
cmake/parquet.cmakemodule correctly exports all required variables (ARROW_FOUND,PARQUET_FOUND,ARROW_INCLUDE_DIRS,PARQUET_INCLUDE_DIRS,ARROW_LIBRARIES,PARQUET_LIBRARIES) to the parent scope, with robust platform-specific library search paths, version validation, and helpful installation instructions.src/flb_parquet_impl.cpp (4)
39-42: Forward declaration approach is reasonable.The forward declaration of
flb_msgpack_to_json_strto avoid C++ compilation conflicts with mpack.h is a pragmatic solution. Consider adding a brief comment explaining why the full header cannot be included.
650-727: Record processing logic is well-designed.The on-the-fly processing approach addresses the previous review comment about dangling pointers. The lambda properly handles:
- Non-map records by appending nulls
- Missing fields with nullable vs non-nullable handling
- Conversion failures with appropriate fallbacks
- Statistics tracking for data quality monitoring
939-951: Exception handling properly cleans up resources.The catch blocks correctly free
output_bufferif it was allocated before the exception. However, note thatoutput_bufferis only assigned on line 926, so the null check is appropriate but the variable could be initialized tonullptrfor clarity (which it is on line 586).
863-865: RecordBatch creation uses correct row count.The
record_countvariable correctly tracks the number of processed records and is used when creating the RecordBatch, ensuring consistency between the arrays and the batch metadata.
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
2 times, most recently
from
a409e9f to
37a602a
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (2)
examples/s3_output/README.md (1)
486-486: Typo already flagged in previous review.This issue was previously identified: "2020-203c" should be "2020-2030".
tests/internal/CMakeLists.txt (1)
136-141: Missing Parquet library linkage will cause linker errors.The Parquet test file is correctly added to
UNIT_TESTS_FILES, but theprepare_unit_testsfunction (lines 234-236) only links Avro libraries whenFLB_AVRO_ENCODERis enabled. There's no corresponding linkage forARROW_LIBRARIESandPARQUET_LIBRARIESwhenFLB_PARQUET_ENCODERis enabled.🔎 Proposed fix in prepare_unit_tests function
Add the following after line 236:
if(FLB_AVRO_ENCODER) target_link_libraries(${source_file_we} avro-static jansson) endif() + + if(FLB_PARQUET_ENCODER) + target_link_libraries(${source_file_we} ${ARROW_LIBRARIES} ${PARQUET_LIBRARIES}) + endif() add_test(NAME ${source_file_we}
🧹 Nitpick comments (5)
examples/s3_output/README.md (2)
465-465: Fix spacing in time reference.Line 465 uses "60min" which should be "60-minute" or "60 minutes" for consistency with similar references elsewhere in the document (e.g., "5 minutes" at line 127).
🔎 Proposed fix
- **Fallback**: `upload_timeout=60m` (acts as safety net) + **Fallback**: `upload_timeout=60m` (acts as safety net for 60 minutes)
112-112: Add language identifiers to fenced code blocks.Markdown linting expects language specifications on all code blocks for syntax highlighting and consistency. Add language identifiers to the three blocks without them:
- Line 112: Plain-text block showing a formula/logic
- Line 177: S3 file path tree structure
- Line 270: Log output example
🔎 Proposed fixes
At line 112, specify a language (or use plain):
-``` +```text Upload Trigger = upload_timeout reached OR total_file_size reached -``` +```At line 177, specify a language for the tree structure:
-``` +``` s3://s3-bucket-kafka-sink/fluent-bit-logs/ ├── putobject/app/day=YYYYMMDD/ # Case 1: Small files via PutObject └── multipart/access/day=YYYYMMDD/ # Case 2: Large files via Multipart Upload -``` +```At line 270, specify a language for log output:
-``` +```log [parquet] Data quality summary for 53206 records: [parquet] Missing non-nullable fields (defaults used): [parquet] field='level' count=52506 [parquet] field='timestamp' count=52506 -``` +```Also applies to: 177-177, 270-270
include/fluent-bit/flb_parquet.h (1)
23-25: Consider removing unused includes.The
flb_sds.hinclude appears unnecessary since the API usesvoid*andsize_trather thanflb_sds_t. Themsgpack.hinclude is also not required in the public header since input is typed asconst void*. Onlystdlib.h(forsize_t) is strictly needed.Removing unused includes reduces header dependencies and compile times.
🔎 Proposed fix
-#include <fluent-bit/flb_sds.h> -#include <msgpack.h> #include <stdlib.h>cmake/parquet.cmake (1)
222-228: Unicode symbols may not render on all terminals.The checkmarks (
✓,✗) in status messages are visually helpful but may not display correctly on terminals without UTF-8 support or in CI logs. Consider using ASCII alternatives like[OK]/[FAIL]for broader compatibility.🔎 Alternative using ASCII
if(ARROW_FOUND AND PARQUET_FOUND) - message(STATUS "✓ Arrow found: ${ARROW_LIBRARIES}") + message(STATUS "[OK] Arrow found: ${ARROW_LIBRARIES}") message(STATUS " Include dirs: ${ARROW_INCLUDE_DIRS}") - message(STATUS "✓ Parquet found: ${PARQUET_LIBRARIES}") + message(STATUS "[OK] Parquet found: ${PARQUET_LIBRARIES}") message(STATUS " Include dirs: ${PARQUET_INCLUDE_DIRS}") else() - message(STATUS "✗ Arrow/Parquet not found") + message(STATUS "[NOT FOUND] Arrow/Parquet not found")src/flb_parquet_impl.cpp (1)
51-61: Consider documenting the buffer size parameter.The
256parameter inflb_msgpack_to_json_strappears to be an initial buffer size hint. Adding a brief comment or using a named constant would improve code clarity.🔎 Suggested improvement
std::string msgpack_object_to_json_string(const msgpack_object* obj) { /* Use Fluent Bit's built-in msgpack to JSON converter */ - char *json_str = flb_msgpack_to_json_str(256, obj, FLB_FALSE); + /* Initial buffer size hint for JSON conversion */ + char *json_str = flb_msgpack_to_json_str(256, obj, FLB_FALSE); if (!json_str) { return "{}";
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (25)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/kafka.cmakecmake/parquet.cmakeexamples/s3_output/README.mdexamples/s3_output/athena-queries.sqlexamples/s3_output/fluent-bit-s3-parquet.confexamples/s3_output/run-s3-parquet-test.shinclude/fluent-bit/aws/flb_aws_compress.hinclude/fluent-bit/flb_parquet.hlib/avro/CMakeLists.txtlib/jansson-e23f558/CMakeLists.txtplugins/out_kinesis_firehose/firehose.cplugins/out_s3/s3.cplugins/out_s3/s3.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/compression/arrow/CMakeLists.txtsrc/aws/compression/arrow/compress.csrc/aws/compression/arrow/compress.hsrc/aws/flb_aws_compress.csrc/flb_parquet_impl.cpptests/internal/CMakeLists.txttests/internal/parquet.c
💤 Files with no reviewable changes (5)
- src/aws/CMakeLists.txt
- src/aws/compression/arrow/CMakeLists.txt
- src/aws/compression/arrow/compress.c
- src/aws/compression/arrow/compress.h
- src/aws/compression/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (6)
- examples/s3_output/fluent-bit-s3-parquet.conf
- .github/workflows/unit-tests.yaml
- examples/s3_output/athena-queries.sql
- plugins/out_kinesis_firehose/firehose.c
- src/CMakeLists.txt
- examples/s3_output/run-s3-parquet-test.sh
🧰 Additional context used
🧠 Learnings (13)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_parquet.hexamples/s3_output/README.md
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
src/aws/flb_aws_compress.cinclude/fluent-bit/aws/flb_aws_compress.hcmake/kafka.cmakeplugins/out_s3/s3.cCMakeLists.txt
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
src/aws/flb_aws_compress.cinclude/fluent-bit/aws/flb_aws_compress.hsrc/flb_parquet_impl.cppplugins/out_s3/s3.cCMakeLists.txt
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
src/aws/flb_aws_compress.cinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txt
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
src/aws/flb_aws_compress.cinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txt
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
src/aws/flb_aws_compress.cinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_s3/s3.cCMakeLists.txt
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
src/aws/flb_aws_compress.cCMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
cmake/kafka.cmakeCMakeLists.txtcmake/parquet.cmake
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
cmake/kafka.cmakeCMakeLists.txtcmake/parquet.cmake
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
CMakeLists.txt
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
cmake/parquet.cmake
🧬 Code graph analysis (6)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (1)
s3_put_object(1663-1800)
include/fluent-bit/flb_parquet.h (1)
src/flb_parquet_impl.cpp (2)
flb_msgpack_raw_to_parquet(579-952)flb_msgpack_raw_to_parquet(579-582)
src/aws/flb_aws_compress.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3.c (5)
src/aws/flb_aws_compress.c (2)
flb_aws_compression_get_type(80-97)flb_aws_compression_compress(99-116)src/flb_sds.c (2)
flb_sds_create_len(58-76)flb_sds_destroy(389-399)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_s3/s3_multipart.c (1)
upload_part(685-799)src/flb_pack.c (1)
flb_pack_msgpack_to_json_format(1169-1450)
tests/internal/parquet.c (2)
src/flb_parquet_impl.cpp (2)
flb_msgpack_raw_to_parquet(579-952)flb_msgpack_raw_to_parquet(579-582)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
🪛 Cppcheck (2.19.0)
plugins/out_s3/s3.c
[error] 791-791: failed to expand 'flb_plg_error', it is invalid to use a preprocessor directive as macro parameter
(syntaxError)
🪛 LanguageTool
examples/s3_output/README.md
[grammar] ~56-~56: Ensure spelling is correct
Context: ..., 201, 404, 500, 401) - Response times (2ms to 5000ms) - Client information (IP, us...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~465-~465: Ensure spelling is correct
Context: ...les uploaded when reaching 50MB (before 60min timeout) Verification Steps: 1. R...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
examples/s3_output/README.md
112-112: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
177-177: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
270-270: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (23)
examples/s3_output/README.md (1)
1-512: Excellent comprehensive documentation for S3 Parquet testing.This new README provides a thorough, well-structured guide covering realistic test scenarios (two input sources, two output configurations), upload trigger mechanics, schema design best practices, data quality considerations, and Athena integration. The pedagogical tone helps users understand why decisions matter (e.g., nullable vs non-nullable fields, PutObject vs Multipart). The inclusion of expected behavior verification steps and troubleshooting guidance adds practical value. This fulfills the
docs-requiredlabel requirement.src/aws/flb_aws_compress.c (3)
31-37: LGTM!The wrapper function correctly adapts the
flb_snappy_compresssignature to match the generic compression interface. The pointer casts are safe and the return value is properly propagated.
66-70: The FLB_AWS_COMPRESS_SNAPPY constant is properly defined.The constant is defined in
include/fluent-bit/aws/flb_aws_compress.h:30with value2, and theflb_snappy_compress_wrapperfunction is correctly defined insrc/aws/flb_aws_compress.c:32. The array entry is correctly integrated.
26-26: No action needed. The include statement is correct as written.Snappy is a core bundled dependency (located at
lib/snappy-fef67ac/) built unconditionally in CMakeLists.txt. Unlike Arrow/Parquet which required#ifdef FLB_HAVE_ARROWguards as optional components, Snappy has noFLB_HAVE_SNAPPYconditional guards anywhere in the codebase and is included unconditionally across 12 existing files (http_server, flb_snappy.c, multiple plugins, etc.). The include at line 26 follows the established pattern.lib/avro/CMakeLists.txt (1)
19-19: CMake version bump looks appropriate.Bumping to CMake 3.12 aligns with the broader build system changes in this PR and enables modern CMake features. This is a reasonable minimum version for 2024+ projects.
lib/jansson-e23f558/CMakeLists.txt (2)
1-1: CMake version bump consistent with the PR.Aligns with the 3.12 minimum version used across other CMake files in this PR.
567-569: Reasonable approach to avoid target name collisions.Disabling the
checktarget prevents conflicts with other libraries (e.g., nghttp2) that may define the same target name. The comment adequately explains the rationale.plugins/out_s3/s3.h (2)
195-203: Clean addition of format support.The new
formatfield andschema_strpointer cleanly separate output format concerns from compression. TheFLB_S3_FORMAT_JSON/FLB_S3_FORMAT_PARQUETconstants follow a clear naming convention consistent with Fluent Bit patterns.
233-234: Function prototype aligns with SDS-based buffer handling.The use of
flb_sds_t bufferis consistent with Fluent Bit's string handling patterns. Note that the parameter names differ slightly from the implementation in s3.c (create_timevsfile_first_log_time,buffervsbody), but this is cosmetic and doesn't affect functionality.include/fluent-bit/flb_parquet.h (1)
27-85: Excellent documentation and clean API design.The header provides comprehensive documentation including:
- Clear parameter descriptions
- Schema format specification with supported types
- Usage examples
- Memory ownership semantics (caller frees with
flb_free)This level of documentation is valuable for a public API.
cmake/parquet.cmake (1)
269-278: Variable exports look correct.All discovered variables are properly exported to the parent scope, including conditional export of
ARROW_VERSION. This ensures the discovery results are available to the main CMakeLists.txt.cmake/kafka.cmake (2)
9-48: Clean simplification of SASL configuration.The refactoring correctly recognizes that librdkafka has built-in support for SASL/PLAIN, SASL/SCRAM, and SASL/OAUTHBEARER without external dependencies. Only SASL/GSSAPI (Kerberos) requires cyrus-sasl.
Key improvements:
FLB_SASL_ENABLEDis always ON (correct for built-in mechanisms)FLB_SASL_OAUTHBEARER_ENABLEDis always ON (built into librdkafka)FLB_KAFKA_MSK_IAM_ENABLEDis always ON (depends on OAuth Bearer)WITH_SASL_CYRUSis conditional on finding cyrus-sasl (for Kerberos only)This removes unnecessary complexity while maintaining full functionality.
77-81: Helpful feature summary for build diagnostics.The summary block clearly shows which Kafka features are enabled, aiding troubleshooting of build configurations.
include/fluent-bit/aws/flb_aws_compress.h (1)
24-46: Well-documented enum reorganization with clear deprecation guidance.The separation of compression algorithms (NONE=0, GZIP=1, SNAPPY=2, ZSTD=3) from format conversions (ARROW=4, PARQUET=5) improves semantic clarity. The deprecation comments provide clear migration paths.
The codebase exclusively uses symbolic constants (
FLB_AWS_COMPRESS_*) rather than hardcoded numeric values, so the enum value changes are transparent to all callers and pose no breaking change risk.Likely an incorrect or invalid review comment.
CMakeLists.txt (2)
1091-1111: Well-designed Homebrew libyaml detection for macOS.The automatic detection of libyaml via Homebrew (
brew --prefix libyaml) is a helpful developer experience improvement for macOS users. The graceful fallback sequence (pkg-config → Homebrew → compile test) ensures broad compatibility.
1337-1355: Parquet encoder integration looks correct.The Parquet encoder setup properly:
- Enables C++ and sets C++17 standard (addressing past review feedback)
- Includes cmake/parquet.cmake for library detection
- Guards all configuration behind
FLB_PARQUET_ENCODER- Provides clear warnings when dependencies are missing
tests/internal/parquet.c (2)
33-66: Excellent test organization and coverage.The test suite provides comprehensive coverage across 38 tests in 9 well-documented categories. The structured organization with clear category headers makes the test suite maintainable and easy to navigate.
74-103: Consistent and correct test pattern.All tests follow a clean pattern: construct msgpack input, invoke the conversion function, assert success, and cleanup resources. The consistent use of
flb_freefor output buffers ensures no memory leaks.src/flb_parquet_impl.cpp (3)
649-727: Immediate record processing correctly avoids dangling pointers.Processing each record immediately during the msgpack unpack loop (rather than storing
msgpack_object*pointers for later) correctly addresses the lifetime issue flagged in previous reviews. Themsgpack_unpackedzone is reused on each iteration, so this approach ensures data integrity.
579-952: Robust implementation with comprehensive error handling.The main conversion function demonstrates solid engineering:
- Input validation and compression mapping
- Proper Arrow/Parquet API usage with status checking
- Immediate record processing to avoid lifetime issues
- Statistics tracking for data quality monitoring
- Clean exception handling with resource cleanup
- Correct memory management between Arrow and Fluent Bit allocators
775-847: Fix missing newlines in debug log statements.All
flb_debugcalls in the statistics output section are missing trailing\ncharacters, which will cause log messages to run together incorrectly. Eachflb_debugcall should end with a newline.🔎 Proposed fix for debug log formatting
if (has_issues) { - flb_debug("[parquet] Data quality summary for %zu records:", record_count); + flb_debug("[parquet] Data quality summary for %zu records:\n", record_count); if (!converter.complex_to_string_by_field.empty()) { - flb_debug("[parquet] Complex types serialized to JSON string:"); + flb_debug("[parquet] Complex types serialized to JSON string:\n"); for (const auto& pair : converter.complex_to_string_by_field) { - flb_debug("[parquet] field='%s' count=%zu", - pair.first.c_str(), pair.second); + flb_debug("[parquet] field='%s' count=%zu\n", + pair.first.c_str(), pair.second); } } if (!converter.string_parsed_to_int_by_field.empty()) { - flb_debug("[parquet] Strings parsed to integers:"); + flb_debug("[parquet] Strings parsed to integers:\n"); for (const auto& pair : converter.string_parsed_to_int_by_field) { - flb_debug("[parquet] field='%s' count=%zu", - pair.first.c_str(), pair.second); + flb_debug("[parquet] field='%s' count=%zu\n", + pair.first.c_str(), pair.second); } } /* Apply similar fixes to remaining flb_debug calls in this section */Apply this pattern to all remaining
flb_debugcalls through line 847.Likely an incorrect or invalid review comment.
plugins/out_s3/s3.c (2)
744-824: Verify legacy Parquet compression behavior.The legacy
compression=arrow/parquetpath setsformat=parquetbut leavesctx->compression = FLB_AWS_COMPRESS_NONE(initialized at line 745). A previous review comment suggested this should default to GZIP for backward compatibility. The comment at line 764 says "Keep default no compression," but if the old Arrow/Parquet path applied GZIP by default, this represents a breaking change.Please verify whether the legacy compression path should default to GZIP when
format=parquetis set via the deprecatedcompression=arrow/parquetvalues.Based on the previous review comment at lines 744-824.
42-44: LGTM: Parquet integration is well-implemented.The Parquet format support is thoughtfully designed:
Clean separation: Parquet uses internal compression (passed to
flb_msgpack_raw_to_parquet), while outer compression is correctly skipped for Parquet format throughout the codebase.Efficient buffering: The architectural change to store raw msgpack data (lines 3953-3962) instead of pre-converting to JSON is a smart optimization that:
- Avoids repeated conversions when buffering
- Enables efficient batching for columnar formats
- Defers format conversion until upload time (line 1598)
Proper memory management: All SDS buffers are consistently managed with
flb_sds_create_lenandflb_sds_destroy. Arrow-allocated Parquet buffers are correctly freed withflb_free(line 1617) after copying to SDS.Good error handling: Schema validation (lines 1588-1596), conversion error checking (lines 1605-1613), and cleanup on all error paths are properly implemented.
Also applies to: 1520-1661, 3953-3962
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
37a602a to
7f32dd4
Compare
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
77b9d4b to
5dc4ed1
Compare
There was a problem hiding this comment.
Actionable comments posted: 13
🤖 Fix all issues with AI agents
In @plugins/out_s3/s3_blob.c:
- Around line 100-102: The code assigns notification->path =
cfl_sds_create(file_path) without checking for NULL; detect allocation failure
by checking the return of cfl_sds_create and handle it (e.g., log an error via
process logger or plugin logger, set notification->path to NULL or a safe
sentinel, set notification->success to false, and return an error/cleanup)
before proceeding; update the block around notification->success and
notification->path (the cfl_sds_create call) to perform the NULL check and
proper cleanup/early return to avoid using a NULL notification->path later.
In @plugins/out_s3/s3_queue.c:
- Around line 683-695: When handling entry->needs_upload_creation fail path, add
a NULL check for ctx->blob_db.db before calling
flb_blob_db_file_set_aborted_state so we mirror the earlier safety check used
near the s3_initiate_multipart_upload success path; only call
flb_blob_db_file_set_aborted_state(&ctx->blob_db, entry->file_id, 1) if
ctx->blob_db.db is non-NULL, then proceed to call s3_queue_entry_destroy(ctx,
entry) and return S3_QUEUE_ENTRY_INVALID as before.
In @plugins/out_s3/s3_store.c:
- Around line 479-491: The s3_store_file_inactive function frees the s3_file
struct but never frees s3_file->stream_path, causing a memory leak; update
s3_store_file_inactive to free s3_file->stream_path (and any other allocated
members handled by s3_store_file_delete) before calling flb_free(s3_file), then
proceed to call flb_fstore_file_inactive(ctx->fs, fsf) as before; reference the
s3_store_file_inactive function and mirror the cleanup sequence used in
s3_store_file_delete for correct ordering.
- Around line 36-39: The CI detection currently returns true only if both
getenv("CI") and getenv("TRAVIS") are set (function is_running_in_ci), which
misses non-Travis CIs; update the logic to detect any CI by checking
getenv("CI") != NULL || getenv("TRAVIS") != NULL (or simply getenv("CI") != NULL
if you only care about generic CI), or if the intent was Travis-only, rename the
function to is_running_in_travis() and keep the TRAVIS check.
- Around line 360-376: create_stream_name uses gmtime(), which is not
thread-safe; replace it with a thread-safe alternative (use gmtime_r on POSIX
and gmtime_s on Windows) by declaring a local struct tm (e.g., struct tm tm_buf)
and passing its address to the thread-safe call, then use that local tm_buf for
strftime and flb_sds_create; ensure the existing conditional on
FLB_SYSTEM_WINDOWS switches to gmtime_s and the non-Windows path uses gmtime_r
so no static internal buffer is referenced.
- Around line 23-25: Remove the unnecessary #include <unistd.h> from
plugins/out_s3/s3_store.c (the other includes sys/types.h and sys/stat.h are
fine); this file does not use any symbols from unistd.h so simply delete that
include line. Also review plugins/out_s3/s3_multipart.c where usleep() is used
and ensure the inclusion of <unistd.h> there is wrapped with a platform guard
(e.g., #ifndef FLB_SYSTEM_WINDOWS ... #endif) so Windows builds won’t fail if
Windows support is required.
In @src/flb_parquet_impl.cpp:
- Around line 757-1067: The cleanup sequence after processing must clear the
"ready" flags and null out pointers immediately after releasing each C resource
to avoid double-free on exception: after calling
msgpack_unpacked_destroy(&result) set result_ready = false; after
msgpack_unpacker_destroy(&unpacker) set unpacker_ready = false; after
flb_free(read_buffer) set read_buffer = NULL; after fclose(msgpack_fp) set
msgpack_fp = NULL; update the block that currently calls
msgpack_unpacked_destroy, msgpack_unpacker_destroy, flb_free, fclose (and the
later manual cleanup section) to perform these flag/pointer updates right after
each corresponding destroy/free/close call.
- Around line 463-465: The BINARY branch uses Append(nullptr, 0) which should be
replaced with the documented AppendEmptyValue() call: update the
arrow::Type::BINARY case to call
static_cast<arrow::BinaryBuilder*>(builder)->AppendEmptyValue().ok() so it
mirrors the STRING pattern and uses the correct BinaryBuilder API.
In @tests/runtime/out_s3_error_handling.c:
- Around line 45-82: The test function flb_test_error_access_denied sets
FLB_S3_PLUGIN_UNDER_TEST but never clears it; update the cleanup after
flb_destroy(ctx) to also unset FLB_S3_PLUGIN_UNDER_TEST so the environment is
restored for other tests (i.e., add unsetenv("FLB_S3_PLUGIN_UNDER_TEST")
alongside the existing unsetenv("TEST_CREATE_MULTIPART_UPLOAD_ERROR") in
flb_test_error_access_denied).
- Around line 84-527: Each test sets FLB_S3_PLUGIN_UNDER_TEST but never unsets
it, causing cross-test pollution; for each test function
(flb_test_error_no_such_bucket, flb_test_error_no_such_upload,
flb_test_error_invalid_access_key, flb_test_error_signature_mismatch,
flb_test_error_slow_down_throttling, flb_test_network_connection_timeout,
flb_test_network_read_timeout, flb_test_retry_layer_auto_retry,
flb_test_retry_layer_part_delivery, flb_test_retry_layer_file_delivery,
flb_test_failure_cleanup_and_abort) add a corresponding
unsetenv("FLB_S3_PLUGIN_UNDER_TEST") in the cleanup section (after
flb_destroy(ctx) and alongside any existing unsetenv calls) so the environment
is restored for subsequent tests.
In @tests/runtime/out_s3_multipart.c:
- Around line 419-479: The test flb_test_upload_id_persistence is missing a
persistent DB config so upload_id cannot survive the two
flb_create()/flb_destroy() sessions; add a shared blob_database_file
configuration via flb_output_set for the S3 output (e.g., call
flb_output_set(ctx, out_ffd, "blob_database_file", "<shared-path>", NULL) before
flb_start in both the first and second session) so the multipart upload_id is
persisted to and recovered from the same file across sessions.
In @tests/runtime/out_s3_queue.c:
- Around line 26-325: Each queue test sets FLB_S3_PLUGIN_UNDER_TEST but never
clears it, breaking isolation; in each test function
(flb_test_queue_basic_operations, flb_test_queue_multiple_workers,
flb_test_queue_retry_mechanism, flb_test_queue_file_buffering,
flb_test_queue_preserve_ordering, flb_test_queue_timeout_handling,
flb_test_queue_concurrent_uploads, flb_test_queue_cleanup_on_shutdown) add a
call to unsetenv("FLB_S3_PLUGIN_UNDER_TEST") after flb_destroy(ctx) (just before
the function returns) so the environment variable is removed for subsequent
tests.
🧹 Nitpick comments (20)
plugins/out_s3/s3_store.c (1)
545-553: Minor: Format specifier mismatch foruint16_t.On line 550,
meta_lenisuint16_tbut the format specifier is%d. While this works in practice due to integer promotion, using%uis more precise.♻️ Suggested fix
if (meta_len > 0 && fseek(fp, meta_len, SEEK_CUR) != 0) { - flb_error("[s3_store] Failed to skip metadata (%d bytes): %s", + flb_error("[s3_store] Failed to skip metadata (%u bytes): %s", meta_len, chunk_path); fclose(fp); return NULL; }examples/out_s3/log_sink_s3.conf (1)
47-50: Consider using placeholder bucket name in example configurations.The bucket name
pl-exchange-dev-kafka-sinkappears to be a real development bucket. For example configurations that may be shared publicly, consider using a placeholder likeyour-bucket-nameorexample-bucketto avoid exposing internal infrastructure details and to make the example more generic for users.♻️ Suggested change
# S3 bucket name (required) - bucket pl-exchange-dev-kafka-sink + bucket your-bucket-nameApply similarly to all other output sections (lines 83, 106, 130, 157, 183).
examples/out_s3/blob_sink_s3.conf (1)
21-22: Use placeholder paths for broader applicability.The path
~/Downloads/*.dmgis specific to macOS and a particular user directory. Consider using a more generic placeholder like/path/to/files/*.extto make the example applicable across different operating systems and use cases.♻️ Suggested change
# File path pattern to monitor, supports wildcards - Path ~/Downloads/*.dmg + Path /path/to/files/*.extcmake/kafka.cmake (1)
53-58: Dead code in conditional block.Since
FLB_SASL_ENABLEDis now unconditionally set toONon line 38, theelsebranch on lines 56-58 will never execute. Consider removing the conditional or converting to a simpler status message.♻️ Suggested simplification
-if(FLB_SASL_ENABLED) - FLB_DEFINITION(FLB_HAVE_KAFKA_SASL) - message(STATUS "Kafka SASL authentication: ENABLED") -else() - message(STATUS "Kafka SASL authentication: DISABLED") -endif() +FLB_DEFINITION(FLB_HAVE_KAFKA_SASL) +message(STATUS "Kafka SASL authentication: ENABLED")Same applies to
FLB_SASL_OAUTHBEARER_ENABLEDcheck on lines 60-65.tests/runtime/out_s3_edge_cases.c (2)
290-324: Test name doesn't match test behavior.
flb_test_storage_path_permissionsuses a valid writable path (/tmp/flb-s3-test-edge) but doesn't actually test permission-related behavior (e.g., what happens with a read-only directory or non-existent parent). Consider renaming toflb_test_custom_store_diror extending to test actual permission scenarios.
27-60: Tests verify startup but lack outcome assertions.The tests confirm that
flb_start()succeeds but don't assert on expected outcomes (e.g., files created, data processed, errors logged). This limits their value for detecting regressions. Consider adding assertions or callbacks to verify expected behavior.tests/runtime/out_s3_error_handling.c (1)
351-411: Consider splitting into two separate tests.This test function creates and runs two complete test scenarios sequentially (auto_retry enabled vs disabled). Splitting these into separate test functions (
flb_test_retry_layer_auto_retry_enabledandflb_test_retry_layer_auto_retry_disabled) would improve test isolation and make failures easier to diagnose.tests/internal/parquet.c (3)
47-52: Consider using the header-defined constants instead of duplicating.These compression type definitions duplicate values from
include/fluent-bit/aws/flb_aws_compress.h. Using the header directly would ensure consistency if values change.♻️ Suggested change
-/* Compression types */ -#define FLB_AWS_COMPRESS_NONE 0 -#define FLB_AWS_COMPRESS_GZIP 1 -#define FLB_AWS_COMPRESS_SNAPPY 2 -#define FLB_AWS_COMPRESS_ZSTD 3 +#include <fluent-bit/aws/flb_aws_compress.h>
873-878: Consider usingflb_mallocfor consistency with Fluent Bit codebase.The test uses standard
mallocbut Fluent Bit typically usesflb_mallocfor memory allocations, which provides additional debugging support in test/fuzz builds.♻️ Suggested change
- char *long_str = (char *)malloc(long_str_size); + char *long_str = (char *)flb_malloc(long_str_size);And at line 901:
- free(long_str); + flb_free(long_str);
1922-1924: Potential undefined behavior with division by zero for NaN/Inf generation.The expressions
0.0 / 0.0and1.0 / 0.0rely on IEEE 754 semantics. While this works on most modern systems, consider using<math.h>constants for clarity and portability.♻️ Suggested change
+#include <math.h> + /* Special floating point values - NaN, +Infinity, -Infinity */ static void test_special_float_values(void) { ... - const double nan_val = 0.0 / 0.0; - const double inf_val = 1.0 / 0.0; - const double neg_inf_val = -1.0 / 0.0; + const double nan_val = NAN; + const double inf_val = INFINITY; + const double neg_inf_val = -INFINITY;include/fluent-bit/flb_parquet.h (1)
23-25: Consider removing unusedmsgpack.hinclude.The
msgpack.hheader is included but no msgpack types are used in the public API declarations. The implementation file likely needs it, but this header only usesconst char*,size_t, andinttypes.♻️ Suggested change
#include <fluent-bit/flb_sds.h> -#include <msgpack.h> #include <stdlib.h>plugins/out_s3/s3_blob.c (1)
165-167: Return value ofabort_multipart_uploadis ignored.While failure to abort may not be critical (the upload will eventually expire), logging would help with debugging stale upload issues.
♻️ Suggested improvement
if (part_count > 1) { - abort_multipart_upload(ctx, file_tag, file_path, file_remote_id); + int abort_ret = abort_multipart_upload(ctx, file_tag, file_path, file_remote_id); + if (abort_ret < 0) { + flb_plg_warn(ctx->ins, "Failed to abort multipart upload for stale file_id=%" PRIu64, file_id); + } }cmake/parquet.cmake (1)
222-228: Unicode characters may not display correctly on all terminals.The checkmark characters (✓, ✗) in status messages may not render correctly on some terminals or CI systems. Consider using ASCII alternatives like
[OK]and[MISSING].♻️ Suggested change
if(ARROW_FOUND AND PARQUET_FOUND) - message(STATUS "✓ Arrow found: ${ARROW_LIBRARIES}") + message(STATUS "[OK] Arrow found: ${ARROW_LIBRARIES}") message(STATUS " Include dirs: ${ARROW_INCLUDE_DIRS}") - message(STATUS "✓ Parquet found: ${PARQUET_LIBRARIES}") + message(STATUS "[OK] Parquet found: ${PARQUET_LIBRARIES}") message(STATUS " Include dirs: ${PARQUET_INCLUDE_DIRS}") else() - message(STATUS "✗ Arrow/Parquet not found") + message(STATUS "[MISSING] Arrow/Parquet not found")src/aws/flb_aws_util.c (2)
1058-1076: Consider consolidating cleanup in error path.The error cleanup manually destroys multiple SDS strings. A helper or goto-based cleanup pattern with a single exit point would reduce duplication and risk of leaks if new allocations are added later. However, the current implementation is correct.
1105-1109: Path sanitization is correct but could be documented.The loop skipping leading
.and/characters prevents directory traversal and ensures relative paths. Consider adding a brief inline comment explaining this security consideration./* Clean up file_path: skip leading dots and slashes */ + /* This prevents directory traversal and ensures we generate valid S3 keys */ valid_file_path = (char *) file_path; while (*valid_file_path == '.' || *valid_file_path == '/') { valid_file_path++; }plugins/out_s3/s3_queue.h (1)
63-69: Return value documentation could be clearer.The comment says "Returns: 1 on success, -1 on failure (will retry), 0 if invalid (removed)" but this may conflict with typical conventions where 0 is success. Consider using named constants for these return values.
Looking at the implementation in
s3_queue.c, I see constants likeS3_QUEUE_ENTRY_SUCCESS,S3_QUEUE_ENTRY_RETRY,S3_QUEUE_ENTRY_INVALIDare used. Consider documenting these in the header:/* * Process a queue entry (called by timer callback) - * Returns: 1 on success, -1 on failure (will retry), 0 if invalid (removed) + * Returns: S3_QUEUE_ENTRY_SUCCESS (1), S3_QUEUE_ENTRY_RETRY (-1), + * or S3_QUEUE_ENTRY_INVALID (0) if entry was removed */plugins/out_s3/s3_queue.c (1)
469-613: Potential memory leak in cleanup section - etags loop always iterates 10000 times.The cleanup section at lines 591-595 iterates through all 10000 possible etags regardless of actual part count. While not incorrect, this is inefficient. More importantly, the cleanup correctly frees all allocated resources.
♻️ Consider using part_count for cleanup efficiency
- for (i = 0; i < 10000; i++) { + for (i = 0; i < part_count && i < 10000; i++) { if (m_upload.etags[i]) { flb_sds_destroy(m_upload.etags[i]); } }src/flb_blob_db.c (1)
1496-1506: Memory management on realloc failure could be cleaner.The current logic on lines 1497-1500 is complex: if
new_Xsucceeded it points to the new buffer (which includes old data), but if it failedtmp_Xis still valid. The conditional freeing is correct but hard to follow.Consider a cleaner pattern that's easier to verify:
♻️ Suggested cleaner realloc pattern
uint64_t *new_db_ids = flb_realloc(tmp_db_ids, capacity * sizeof(uint64_t)); + if (!new_db_ids) { + flb_free(tmp_db_ids); + flb_free(tmp_nums); + flb_free(tmp_starts); + flb_free(tmp_ends); + sqlite3_clear_bindings(statement); + sqlite3_reset(statement); + flb_sqldb_unlock(context->db); + return FLB_BLOB_DB_ERROR_ALLOCATOR_FAILURE; + } + tmp_db_ids = new_db_ids; + uint64_t *new_nums = flb_realloc(tmp_nums, capacity * sizeof(uint64_t)); - off_t *new_starts = flb_realloc(tmp_starts, capacity * sizeof(off_t)); - off_t *new_ends = flb_realloc(tmp_ends, capacity * sizeof(off_t)); - - if (!new_db_ids || !new_nums || !new_starts || !new_ends) { - if (new_db_ids) flb_free(new_db_ids); else flb_free(tmp_db_ids); - if (new_nums) flb_free(new_nums); else flb_free(tmp_nums); - if (new_starts) flb_free(new_starts); else flb_free(tmp_starts); - if (new_ends) flb_free(new_ends); else flb_free(tmp_ends); - + if (!new_nums) { + flb_free(tmp_db_ids); + flb_free(tmp_nums); + /* ... similar for others ... */ sqlite3_clear_bindings(statement); sqlite3_reset(statement); flb_sqldb_unlock(context->db); return FLB_BLOB_DB_ERROR_ALLOCATOR_FAILURE; } + tmp_nums = new_nums; + /* ... continue for starts/ends ... */plugins/out_s3/s3_stream.c (1)
80-81: Minor:remaininginitialized to -1 when offset_end is 0 or negative.When
offset_end <= 0,remainingis set to -1, and the loop at line 84 checksif (remaining > 0). This works but relies on signed comparison. The intent is clear (read to EOF), but a comment would help.src/flb_parquet_impl.cpp (1)
876-883: Consider using std::string_view consistently for field lookup.The field_map uses
std::string_viewas key (line 826), and the lookup at line 887 usesfield->name()which returnsstd::string. The comparison should work due to implicit conversion, but explicit handling would be clearer.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (50)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/kafka.cmakecmake/parquet.cmakeexamples/out_s3/blob_sink_s3.confexamples/out_s3/kafka_sink_s3.confexamples/out_s3/log_sink_s3.confinclude/fluent-bit/aws/flb_aws_compress.hinclude/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hlib/avro/CMakeLists.txtlib/jansson-e23f558/CMakeLists.txtplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/flb_aws_compress.csrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
💤 Files with no reviewable changes (2)
- src/aws/CMakeLists.txt
- src/aws/compression/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (6)
- .github/workflows/unit-tests.yaml
- tests/internal/CMakeLists.txt
- lib/jansson-e23f558/CMakeLists.txt
- lib/avro/CMakeLists.txt
- src/CMakeLists.txt
- plugins/out_kinesis_firehose/firehose.c
🧰 Additional context used
🧠 Learnings (13)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_parquet.h
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
CMakeLists.txtcmake/parquet.cmakecmake/kafka.cmake
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
CMakeLists.txtcmake/parquet.cmakecmake/kafka.cmake
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
CMakeLists.txtcmake/kafka.cmakeinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
CMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
CMakeLists.txtsrc/flb_parquet_impl.cppinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hsrc/aws/flb_aws_compress.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
CMakeLists.txt
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
cmake/parquet.cmake
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
🧬 Code graph analysis (19)
plugins/out_s3/s3_auth.h (1)
plugins/out_s3/s3_auth.c (3)
s3_auth_init_endpoint(78-121)s3_auth_request_presigned_url(154-241)s3_auth_fetch_presigned_url(334-375)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1177)
plugins/out_s3/s3_blob.h (1)
plugins/out_s3/s3_blob.c (5)
s3_blob_register_parts(299-331)s3_blob_process_events(427-507)s3_blob_store_chunk(338-412)s3_blob_recover_state(123-142)s3_blob_notify_delivery(81-113)
include/fluent-bit/flb_parquet.h (1)
src/flb_parquet_impl.cpp (8)
flb_parquet_validate_schema(698-714)flb_parquet_validate_schema(698-700)flb_parquet_schema_create(716-747)flb_parquet_schema_create(716-718)flb_parquet_schema_destroy(749-755)flb_parquet_schema_destroy(749-749)flb_msgpack_to_parquet_streaming(757-1067)flb_msgpack_to_parquet_streaming(757-762)
plugins/out_s3/s3_queue.h (1)
plugins/out_s3/s3_queue.c (9)
s3_queue_remove(267-275)s3_queue_buffer_chunk(315-337)s3_queue_add_file(57-111)s3_queue_add_part(113-183)s3_queue_add_pending_file(185-232)s3_queue_process_entry(665-721)s3_queue_recover_all(762-806)s3_queue_recover_from_database(816-920)s3_queue_recover_from_fstore(993-1053)
plugins/out_s3/s3_store.c (1)
src/flb_fstore.c (4)
flb_fstore_file_create(153-190)flb_fstore_file_meta_set(74-108)flb_fstore_file_append(298-329)flb_fstore_create(479-525)
tests/internal/parquet.c (3)
src/flb_parquet_impl.cpp (1)
field_name(386-386)src/flb_pack.c (1)
flb_pack_json(530-535)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
tests/runtime/out_s3_config.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (4)
flb_create(143-225)flb_service_set(652-678)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_error_handling.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_format.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_store.h (1)
plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(520-556)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3_blob.c (9)
plugins/out_s3/s3_multipart.c (4)
s3_multipart_upload_create(391-440)s3_multipart_upload_destroy(360-385)s3_multipart_abort(691-756)s3_multipart_check_upload_exists(766-845)lib/cfl/src/cfl_sds.c (3)
cfl_sds_len(88-91)cfl_sds_create(113-125)cfl_sds_destroy(127-137)src/flb_sds.c (2)
flb_sds_create(78-90)flb_sds_destroy(389-399)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(334-375)include/fluent-bit/flb_mem.h (1)
flb_calloc(84-96)src/flb_blob_db.c (2)
flb_blob_db_lock(367-372)flb_blob_file_update_remote_id(530-561)plugins/out_s3/s3.c (1)
s3_initiate_multipart_upload(982-1087)src/flb_log_event_decoder.c (2)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)plugins/out_s3/s3_queue.c (1)
s3_queue_add_pending_file(185-232)
tests/runtime/out_s3_queue.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (5)
cb_s3_upload(1413-1528)s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)mock_s3_call(105-179)
src/flb_blob_db.c (3)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)src/flb_sqldb.c (2)
flb_sqldb_lock(173-178)flb_sqldb_unlock(180-185)lib/cfl/src/cfl_sds.c (2)
cfl_sds_create(113-125)cfl_sds_destroy(127-137)
plugins/out_s3/s3_queue.c (5)
plugins/out_s3/s3_store.c (5)
s3_store_buffer_put(207-242)s3_store_file_delete(493-518)s3_store_file_unlock(565-570)s3_store_file_inactive(479-491)s3_store_file_lock(558-563)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(334-375)plugins/out_s3/s3_multipart.c (4)
s3_multipart_upload_file_part(847-905)s3_multipart_complete(605-689)s3_multipart_upload_create(391-440)s3_multipart_upload_destroy(360-385)src/flb_blob_db.c (1)
flb_blob_db_file_fetch_all_parts(1440-1557)plugins/out_s3/s3.c (2)
s3_format_chunk(1266-1407)s3_upload_file(1089-1194)
plugins/out_s3/s3_auth.c (4)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_utils.c (1)
flb_utils_url_split(1531-1667)src/flb_http_client.c (4)
flb_http_add_header(1013-1045)flb_http_basic_auth(1373-1377)flb_http_bearer_auth(1385-1420)flb_http_client(864-909)src/flb_sds.c (3)
flb_sds_copy(260-281)flb_sds_create_len(58-76)flb_sds_destroy(389-399)
tests/runtime/out_s3_multipart.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Fix all issues with AI agents
In @include/fluent-bit/flb_aws_util.h:
- Around line 200-207: The fuzzer call to flb_get_s3_key in
tests/internal/fuzzers/aws_util_fuzzer.c is missing the newly added file_path
parameter; update the call to flb_get_s3_key(...) to pass an appropriate sixth
argument (use NULL if no file path is needed) so the call matches the updated
signature flb_get_s3_key(const char *format, time_t time, const char *tag, char
*tag_delimiter, uint64_t seq_index, const char *file_path).
In @plugins/out_s3/s3_auth.c:
- Around line 297-303: The abort-multipart branch using
S3_PRESIGNED_URL_ABORT_MULTIPART builds the wrong path: change the
flb_sds_printf call that writes "/multipart_upload_presigned_url/%s%s/%s" to use
the abort-specific endpoint (e.g. "/multipart_abort_presigned_url/%s%s/%s") so
the abort operation constructs the distinct URL; update the flb_sds_printf
invocation in the S3_PRESIGNED_URL_ABORT_MULTIPART case (which uses path,
ctx->bucket, encoded_key and encoded_id) accordingly.
- Around line 123-134: The get_port_from_url function currently calls
strtoul(port_str, NULL, 10) without checking for conversion errors; update it to
use a char *endptr and errno: set errno=0, call strtoul(port_str, &endptr, 10),
then verify endptr != port_str, *endptr == '\0', errno != ERANGE, and the
returned value is <= 65535 before casting to uint16_t; if any check fails, fall
back to the scheme-based defaults (443 for "https", otherwise 80) instead of
returning an incorrect/zero port.
In @plugins/out_s3/s3_multipart.c:
- Around line 766-820: The function s3_multipart_check_upload_exists currently
documents returning -1 on network/API errors but treats a NULL HTTP client (c ==
NULL) as a non-existent upload and returns 0; either make the implementation
match the docs or update the docs. Fix by: in s3_multipart_check_upload_exists
(check the c == NULL branch) change the return value to -1 for network/API
failures (and keep or adjust the flb_plg_warn message to reflect an error rather
than assuming invalid upload), or alternatively update the function comment
block to state that network failures return 0; ensure any callers of
s3_multipart_check_upload_exists get their expected semantics and update
tests/comments accordingly.
In @plugins/out_s3/s3_queue.c:
- Around line 854-865: The strcmp call comparing destination and ctx->endpoint
can dereference a NULL destination; before calling strcmp in the loop (where
file_id is processed), add a NULL-check for destination (and optionally
ctx->endpoint) and handle it the same way as an endpoint mismatch: log a debug
message referencing file_id, clean up file_path/destination/remote_id/tag (using
cfl_sds_destroy if non-NULL) and continue; ensure you check destination first
(e.g., if (!destination || strcmp(destination, ctx->endpoint) != 0) ...) so you
avoid the null pointer dereference.
In @plugins/out_s3/s3_store.c:
- Around line 36-39: The is_running_in_ci function is too restrictive by
requiring both CI and TRAVIS env vars; change it to detect any common CI
indicator instead. Update is_running_in_ci to return true if getenv("CI") !=
NULL || getenv("TRAVIS") != NULL || getenv("GITHUB_ACTIONS") != NULL ||
getenv("GITLAB_CI") != NULL || getenv("JENKINS_URL") != NULL (or similar common
CI vars), so the memory backend is selected when running under any typical CI
system.
In @plugins/out_s3/s3_stream.c:
- Around line 216-223: The temp filename construction using the input_path
pointer is unsafe across processes; update the tmp generation in the block where
flb_sds_printf is called so it includes a unique process/timestamp element
(e.g., use getpid() and a timestamp or nanosecond counter) or replace the
sprintf-based approach with a secure creation call (e.g., mkstemp) to avoid
collisions; specifically modify the flb_sds_printf invocation that writes to
ctx->temp_path (currently using s3_ctx->buffer_dir, (void*)input_path,
output_suffix) to incorporate getpid() and time(NULL) or use mkstemp to
atomically create a unique file and then assign the returned name to
ctx->temp_path, preserving existing error handling (flb_errno() and
flb_plg_error) on failure.
- Around line 424-429: flb_msgpack_to_json_str can fail and currently the code
logs a warning then returns 0 (success), silently dropping the record; change
the error handling in the block where json_str is checked so it returns -1
(propagate error) instead of 0 to match the other error paths (e.g., the write
failure that returns -1), ensuring callers see the failure; locate the check
using flb_msgpack_to_json_str and json_str in s3_stream.c and replace the return
value, or alternatively implement proc_ctx tracking if you prefer deferred
reporting, but do not leave the function returning 0 on JSON conversion failure.
In @src/flb_blob_db.c:
- Around line 1491-1511: The realloc error handling frees the wrong buffers and
can free the original tmp_* pointers; change the failure path so you NEVER free
the original tmp_db_ids/tmp_nums/tmp_starts/tmp_ends, and instead only free any
of the newly returned pointers (new_db_ids/new_nums/new_starts/new_ends) if they
are non-NULL when a failure occurs; implement this by allocating to the new_*
variables as you do now, and on failure loop/if-check each new_* and flb_free()
only those new_* that succeeded, then call sqlite3_clear_bindings(statement),
sqlite3_reset(statement), flb_sqldb_unlock(context->db) and return
FLB_BLOB_DB_ERROR_ALLOCATOR_FAILURE; only assign tmp_* = new_* after all new_*
allocations succeeded.
🧹 Nitpick comments (23)
tests/runtime/out_s3_error_handling.c (1)
46-82: Missing cleanup ofFLB_S3_PLUGIN_UNDER_TESTenvironment variable.The test sets
FLB_S3_PLUGIN_UNDER_TESTbut never unsets it. This pattern is repeated across all test functions in this file and could cause test pollution if tests run in sequence within the same process.Proposed fix
flb_stop(ctx); flb_destroy(ctx); unsetenv("TEST_CREATE_MULTIPART_UPLOAD_ERROR"); + unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); }include/fluent-bit/flb_parquet.h (1)
23-25: Unused includes may be removed.Neither
flb_sds.hnormsgpack.htypes appear in the function declarations. These headers are likely needed only in the implementation file (flb_parquet_impl.cpp), not in this public header.Proposed fix
-#include <fluent-bit/flb_sds.h> -#include <msgpack.h> #include <stdlib.h> +#include <stddef.h> /* for size_t */tests/runtime/out_s3_edge_cases.c (1)
27-60: Same environment variable cleanup issue as in error handling tests.
FLB_S3_PLUGIN_UNDER_TESTis set but never unset. Consider adding cleanup at the end of each test function.tests/runtime/out_s3_special_features.c (1)
172-213: Potential collision in database path generation.Using
time(NULL)for unique paths could cause collisions if multiple tests run within the same second. Consider adding PID or a random component.Proposed fix
- snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%d.db", (int)time(NULL)); + snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%d_%d.db", (int)getpid(), (int)time(NULL));plugins/out_s3/s3.h (2)
48-70: Add documentation for the upload queue structure.The
upload_queuestructure has many fields tracking multipart upload state, but lacks documentation explaining:
- Field ownership and lifecycle (especially
flb_sds_tstrings)- Relationship between file_id, part_db_id, and part_id
- Purpose of
needs_upload_creationflag- Threading constraints (given the mutex in
flb_s3)📝 Suggested documentation
+/* + * upload_queue: Tracks a single multipart upload part in the upload queue + * + * Threading: Access protected by flb_s3->upload_queue_lock + * Lifecycle: Created during upload initiation, destroyed after upload completion + */ struct upload_queue {
171-173: Document the lifecycle of cached_arrow_schema.The
cached_arrow_schemapointer lacks documentation about:
- When it's allocated/freed
- Whether it's shared or per-instance
- Thread safety guarantees (if any)
📝 Suggested documentation
- int format; - char *schema_str; - struct flb_parquet_schema *cached_arrow_schema; + int format; /* FLB_S3_FORMAT_JSON or FLB_S3_FORMAT_PARQUET */ + char *schema_str; /* JSON schema string for Parquet (owned) */ + struct flb_parquet_schema *cached_arrow_schema; /* Parsed Arrow schema (owned, lazily initialized) */examples/out_s3/kafka_sink_s3.conf (1)
190-202: Consider documenting Parquet compression trade-offs.While the comment mentions GZIP provides "good balance between compression ratio and query performance," it would be helpful to note:
- GZIP is slower than Snappy but offers better compression
- Snappy is faster for writes but less compression
- ZSTD offers best of both but may have broader compatibility considerations
This helps users make informed decisions based on their workload.
📝 Enhanced compression guidance
# GZIP compression (recommended for Parquet) - # Provides good balance between compression ratio and query performance + # Compression options: + # - gzip: Best compression ratio, moderate speed (recommended for analytics) + # - snappy: Faster writes, lower compression (recommended for high throughput) + # - none: No compression (use when data is already compressed) compression gziptests/runtime/out_s3_recovery.c (4)
38-38: Consider potential path collision in concurrent tests.Using
time(NULL)for the database path could cause collisions if multiple tests run in the same second. Consider usinggetpid()in addition totime()for better uniqueness:♻️ Suggested improvement
- snprintf(db_path, sizeof(db_path), "/tmp/s3_recovery_test_%d.db", (int)time(NULL)); + snprintf(db_path, sizeof(db_path), "/tmp/s3_recovery_test_%d_%d.db", (int)getpid(), (int)time(NULL));
120-120: Missing cleanup of store_dir after test.The test uses a fixed
store_dirpath but doesn't clean it up after completion, which could leave test artifacts and cause issues with subsequent test runs or concurrent execution.♻️ Add cleanup for store_dir
flb_stop(ctx); flb_destroy(ctx); + + /* Cleanup store directory */ + system("rm -rf /tmp/flb-s3-recovery-test"); }
170-170: Consider path uniqueness for concurrent test execution.Similar to the previous test, using only
time(NULL)could cause path collisions. Consider addinggetpid()for better uniqueness in concurrent scenarios.
27-414: Consider adding assertions to verify recovery behavior.The tests follow a good pattern of setup, execution, and cleanup. However, none of them verify that recovery actually occurred or that the expected behavior happened. Consider adding assertions or output verification to make these tests more robust.
For example:
- Check that blob DB file exists after first run
- Verify that recovery logs appear in second run
- Assert expected upload outcomes
tests/runtime/out_s3_queue.c (2)
160-160: Missing cleanup of store_dir.Similar to the recovery tests, the fixed
store_dirpath should be cleaned up after the test completes to avoid leaving test artifacts.♻️ Add cleanup
flb_stop(ctx); flb_destroy(ctx); + + /* Cleanup store directory */ + system("rm -rf /tmp/flb-s3-queue-test"); }
27-325: Consider adding assertions for queue behavior verification.Similar to the recovery tests, these queue tests would benefit from assertions that verify the expected behavior rather than just checking that execution completes without crashing. Consider adding checks for:
- Queue size/count after pushes
- Worker distribution in multi-worker tests
- Retry attempt counts
- Timeout enforcement
- Ordering preservation
plugins/out_s3/s3_blob.c (1)
338-412: Minor: Improve temp filename uniqueness.Line 356-357 uses
time(NULL)and the data pointer address to generate a unique filename. While this usually works, using the pointer address could lead to collisions if memory is reused rapidly. Consider adding a counter or using a more robust unique identifier.💡 Suggested improvement
+static _Atomic uint64_t chunk_counter = 0; + int s3_blob_store_chunk(struct flb_s3 *ctx, const char *tag, int tag_len, const char *data, size_t data_size, time_t file_first_log_time) { char temp_path[PATH_MAX]; FILE *fp = NULL; size_t written; int64_t file_id; int ret; int part_count; + uint64_t chunk_id; if (!ctx->blob_db.db) { flb_plg_error(ctx->ins, "chunk_store: database not configured"); return -1; } + chunk_id = atomic_fetch_add(&chunk_counter, 1); + /* Generate temporary file path */ - ret = snprintf(temp_path, sizeof(temp_path), "%s/chunk_%lu_%p.msgpack", - ctx->buffer_dir, (unsigned long)time(NULL), (void*)data); + ret = snprintf(temp_path, sizeof(temp_path), "%s/chunk_%lu_%lu.msgpack", + ctx->buffer_dir, (unsigned long)time(NULL), (unsigned long)chunk_id);cmake/parquet.cmake (2)
149-171: Consider allowing default search paths as a fallback.Using
NO_DEFAULT_PATHinfind_pathandfind_libraryprevents CMake from searching standard system paths if the explicitSEARCH_PATHSdon't contain the libraries. This could cause false negatives on systems where Arrow/Parquet are installed in non-standard locations that CMake would normally find.Suggested approach
Consider a two-stage search: first with
NO_DEFAULT_PATHfor explicit paths, then without it as a fallback:# Search for Arrow if(NOT ARROW_FOUND) find_path(ARROW_INCLUDE_DIR NAMES arrow/api.h PATHS ${SEARCH_PATHS} PATH_SUFFIXES ${INCLUDE_SUFFIXES} - NO_DEFAULT_PATH )Or keep the current approach if you specifically want to avoid picking up system-installed versions that might conflict.
201-217: Version detection may fail silently ifARROW_INCLUDE_DIRSis a list.If
ARROW_INCLUDE_DIRScontains multiple paths (e.g., from pkg-config), the path concatenation"${ARROW_INCLUDE_DIRS}/arrow/util/config.h"may not resolve correctly. Consider iterating over the include directories.Suggested fix
if(ARROW_FOUND) # Try to detect Arrow version - if(EXISTS "${ARROW_INCLUDE_DIRS}/arrow/util/config.h") - file(STRINGS "${ARROW_INCLUDE_DIRS}/arrow/util/config.h" + foreach(_arrow_inc_dir ${ARROW_INCLUDE_DIRS}) + if(EXISTS "${_arrow_inc_dir}/arrow/util/config.h") + file(STRINGS "${_arrow_inc_dir}/arrow/util/config.h" ARROW_VERSION_LINE REGEX "^#define ARROW_VERSION_STRING") + if(ARROW_VERSION_LINE) + break() + endif() + endif() + endforeach() + if(ARROW_VERSION_LINE) ...tests/runtime/out_s3_config.c (1)
365-401: Consider usingsizeofto derive loop count from the array.The loop count
4is hardcoded but could become out of sync if theclassesarray is modified.Suggested fix
const char *classes[] = { "STANDARD", "STANDARD_IA", "GLACIER", "INTELLIGENT_TIERING" }; - int i; + size_t i; + size_t num_classes = sizeof(classes) / sizeof(classes[0]); setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); - for (i = 0; i < 4; i++) { + for (i = 0; i < num_classes; i++) {plugins/out_s3/s3_queue.c (2)
591-595: Unnecessary iteration over full 10000-element array.The cleanup loop iterates through all 10000 possible etag slots even when fewer parts exist. This is inefficient but functionally correct.
Suggested optimization
- for (i = 0; i < 10000; i++) { + for (i = 0; i < part_count && i < 10000; i++) { if (m_upload.etags[i]) { flb_sds_destroy(m_upload.etags[i]); } }
645-657: Empty conditional branches appear to be placeholders.The
if (entry->file_id > 0)andelseblocks at lines 645-654 contain no meaningful logic. If these are intentional no-ops, consider adding a comment explaining why, or remove the conditionals entirely.Suggested cleanup
- if (entry->file_id > 0) { - /* Will retry */ - } - else { + if (entry->file_id == 0) { + /* Non-DB tracked entries: update legacy retry state */ if (entry->upload_file) { s3_store_file_lock(entry->upload_file); } ctx->retry_time += 2 * entry->retry_counter; ctx->upload_queue_success = FLB_FALSE; }src/flb_parquet_impl.cpp (3)
885-906: Limited observability for type conversion failures.When
convert_valuefails (Line 892), the code silently appends null (for nullable fields) or a default value (for non-nullable fields) without logging or tracking these conversion failures. Users won't know if their data is being silently transformed.Consider adding:
- A conversion failure counter to track and report the number of failed conversions
- Debug logging for each conversion failure to aid troubleshooting
- Optional strict mode that fails the entire operation on conversion errors
Example enhancement
// Track failures size_t conversion_failures = 0; // In the conversion block: if (!status.ok()) { conversion_failures++; flb_debug("[parquet] Conversion failed for field '%s': %s", field->name().c_str(), status.ToString().c_str()); if (field->nullable()) { (void)field_builders[i]->AppendNull(); } else { append_default_value(field_builders[i].get(), field->type()); } } // After processing: if (conversion_failures > 0) { flb_info("[parquet] %zu field conversions failed and were replaced with null/default", conversion_failures); }
431-434: Silent fallback to UTF-8 for unknown types may hide schema errors.When a type name is not found in
TYPE_FACTORY_MAP, the code silently defaults toarrow::utf8()without logging a warning. Users may not realize their specified type wasn't recognized.Add observability for unknown types
auto it = TYPE_FACTORY_MAP.find(type_name); - data_type = (it != TYPE_FACTORY_MAP.end()) ? it->second() : arrow::utf8(); + if (it != TYPE_FACTORY_MAP.end()) { + data_type = it->second(); + } else { + flb_warn("[parquet] Unknown type '%s' for field '%s', defaulting to utf8", + type_name.c_str(), field_name.c_str()); + data_type = arrow::utf8(); + }
757-762: Consider removing unusedtotal_file_sizeparameter.The
total_file_sizeparameter is marked as unused and not referenced in the function body. If it's not needed for the current or planned functionality, consider removing it to simplify the API.If this parameter was intended for future use (e.g., pre-allocating resources or progress reporting), consider documenting its purpose. Otherwise, remove it:
int flb_msgpack_to_parquet_streaming(const char *msgpack_file_path, flb_parquet_schema *schema, int compression, const char *output_file, - size_t *out_file_size, - size_t total_file_size __attribute__((unused))) + size_t *out_file_size)plugins/out_s3/s3_multipart.c (1)
971-1027: Inconsistent retry backoff strategies.The presigned URL fetch uses linear backoff (Lines 989-994: 1s, 2s, 3s...), while the part upload uses exponential backoff (Lines 1018-1025: 1s, 2s, 4s, 8s...). Both strategies work, but exponential backoff is generally preferred for retries to handle cascading failures more gracefully.
Consider using exponential backoff consistently for both operations:
Unified exponential backoff
/* Retry presigned URL fetch with backoff */ if (part_attempt < ctx->part_delivery_attempt_limit) { - backoff_ms = 1000 * part_attempt; /* 1s, 2s, 3s... */ + backoff_ms = 1000 * (1 << (part_attempt - 1)); /* 1s, 2s, 4s, 8s... */ + if (backoff_ms > 30000) { + backoff_ms = 30000; /* Cap at 30 seconds */ + } flb_plg_info(ctx->ins, "Retrying presigned URL fetch after %d ms", backoff_ms); usleep(backoff_ms * 1000);
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (50)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/kafka.cmakecmake/parquet.cmakeexamples/out_s3/blob_sink_s3.confexamples/out_s3/kafka_sink_s3.confexamples/out_s3/log_sink_s3.confinclude/fluent-bit/aws/flb_aws_compress.hinclude/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hlib/avro/CMakeLists.txtlib/jansson-e23f558/CMakeLists.txtplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/flb_aws_compress.csrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
💤 Files with no reviewable changes (2)
- src/aws/compression/CMakeLists.txt
- src/aws/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (10)
- lib/jansson-e23f558/CMakeLists.txt
- tests/internal/CMakeLists.txt
- .github/workflows/unit-tests.yaml
- examples/out_s3/blob_sink_s3.conf
- plugins/out_s3/s3_queue.h
- tests/internal/parquet.c
- src/flb_signv4.c
- tests/runtime/CMakeLists.txt
- examples/out_s3/log_sink_s3.conf
- tests/runtime/out_s3_format.c
🧰 Additional context used
🧠 Learnings (14)
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
src/aws/flb_aws_compress.cCMakeLists.txtcmake/kafka.cmakesrc/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
src/aws/flb_aws_compress.cCMakeLists.txtsrc/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_kinesis_firehose/firehose.csrc/flb_parquet_impl.cpp
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
src/aws/flb_aws_compress.cCMakeLists.txtsrc/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
src/aws/flb_aws_compress.cCMakeLists.txtsrc/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.hplugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
src/aws/flb_aws_compress.cCMakeLists.txtsrc/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
src/aws/flb_aws_compress.cCMakeLists.txtsrc/CMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
CMakeLists.txtcmake/parquet.cmakecmake/kafka.cmakesrc/CMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
CMakeLists.txtcmake/parquet.cmakecmake/kafka.cmakesrc/CMakeLists.txt
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
CMakeLists.txtsrc/CMakeLists.txtplugins/out_s3/s3_store.c
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_parquet.hinclude/fluent-bit/flb_aws_util.h
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
cmake/parquet.cmake
📚 Learning: 2025-09-04T07:28:37.083Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10832
File: src/wasm/CMakeLists.txt:112-131
Timestamp: 2025-09-04T07:28:37.083Z
Learning: In fluent-bit CMake files, the user cosmo0920 prefers treating Git as a command rather than a package, emphasizing that Git is not a pkg-config retrievable package but just a command.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-14T09:46:09.531Z
Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
🧬 Code graph analysis (24)
plugins/in_blob/blob_file.c (3)
plugins/in_blob/blob_db.c (1)
blob_db_file_insert(164-196)lib/cfl/src/cfl_sds.c (1)
cfl_sds_destroy(127-137)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
tests/runtime/out_s3_recovery.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_store.h (1)
plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(520-556)
plugins/out_s3/s3_auth.h (1)
plugins/out_s3/s3_auth.c (3)
s3_auth_init_endpoint(78-121)s3_auth_request_presigned_url(154-241)s3_auth_fetch_presigned_url(334-375)
tests/runtime/out_s3_config.c (1)
src/flb_lib.c (9)
flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
src/aws/flb_aws_compress.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
tests/runtime/out_s3_edge_cases.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_stop(1011-1055)flb_destroy(228-263)flb_lib_push(843-870)
tests/runtime/out_s3_multipart.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_blob.c (5)
plugins/out_s3/s3_multipart.c (4)
s3_multipart_upload_create(391-440)s3_multipart_upload_destroy(360-385)s3_multipart_abort(691-756)s3_multipart_check_upload_exists(766-845)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(334-375)src/flb_blob_db.c (1)
flb_blob_file_update_remote_id(530-561)plugins/out_s3/s3.c (1)
s3_initiate_multipart_upload(982-1087)plugins/out_s3/s3_queue.c (1)
s3_queue_add_pending_file(185-232)
include/fluent-bit/flb_parquet.h (1)
src/flb_parquet_impl.cpp (8)
flb_parquet_validate_schema(698-714)flb_parquet_validate_schema(698-700)flb_parquet_schema_create(716-747)flb_parquet_schema_create(716-718)flb_parquet_schema_destroy(749-755)flb_parquet_schema_destroy(749-749)flb_msgpack_to_parquet_streaming(757-1067)flb_msgpack_to_parquet_streaming(757-762)
plugins/out_s3/s3_auth.c (3)
src/flb_utils.c (1)
flb_utils_url_split(1531-1667)src/flb_upstream.c (1)
flb_upstream_create_url(416-481)src/flb_http_client.c (4)
flb_http_add_header(1013-1045)flb_http_basic_auth(1373-1377)flb_http_bearer_auth(1385-1420)flb_http_client(864-909)
plugins/out_s3/s3_queue.c (6)
plugins/out_s3/s3_store.c (3)
s3_store_buffer_put(207-242)s3_store_file_unlock(565-570)s3_store_file_lock(558-563)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(334-375)plugins/out_s3/s3_multipart.c (3)
s3_multipart_upload_file_part(847-905)s3_multipart_upload_create(391-440)s3_multipart_upload_destroy(360-385)src/flb_blob_db.c (4)
flb_blob_db_file_part_update_remote_id(1112-1143)flb_blob_db_lock(367-372)flb_blob_db_unlock(374-379)flb_blob_db_file_fetch_all_parts(1440-1557)plugins/out_s3/s3.c (3)
s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)plugins/out_s3/s3_blob.c (2)
s3_blob_notify_delivery(81-113)s3_blob_recover_state(123-142)
src/flb_blob_db.c (3)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)src/flb_sqldb.c (2)
flb_sqldb_lock(173-178)flb_sqldb_unlock(180-185)lib/cfl/src/cfl_sds.c (2)
cfl_sds_create(113-125)cfl_sds_destroy(127-137)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (5)
cb_s3_upload(1413-1528)s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)mock_s3_call(105-179)
include/fluent-bit/flb_blob_db.h (1)
src/flb_blob_db.c (7)
flb_blob_db_file_fetch_all_parts(1440-1557)flb_blob_db_file_get_next_pending(1563-1663)flb_blob_db_file_get_next_pending(1896-1905)flb_blob_db_file_part_check_uploaded(1669-1704)flb_blob_db_file_part_check_uploaded(1907-1912)flb_blob_db_reset_zombie_parts(1707-1731)flb_blob_db_reset_zombie_parts(1914-1917)
plugins/out_s3/s3_stream.h (1)
plugins/out_s3/s3_stream.c (4)
stream_compress_file(32-147)stream_process_msgpack_file(344-412)stream_json_processor(414-442)stream_log_key_processor(517-546)
tests/internal/aws_util.c (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1177)
plugins/out_s3/s3_stream.c (5)
src/aws/flb_aws_compress.c (1)
flb_aws_compression_compress(103-120)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_printf(336-387)plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(520-556)tests/internal/processor.c (1)
processor(66-126)
tests/runtime/out_s3_error_handling.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_multipart.c (6)
plugins/out_s3/s3.c (1)
write_seq_index(226-246)src/flb_sds.c (6)
flb_sds_create_len(58-76)flb_sds_create(78-90)flb_sds_create_size(92-95)flb_sds_destroy(389-399)flb_sds_printf(336-387)flb_sds_cat(120-141)include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)src/aws/flb_aws_util.c (3)
flb_get_s3_key(1083-1177)flb_aws_xml_get_val(623-655)flb_aws_print_xml_error(586-610)src/flb_http_client.c (1)
flb_http_client_destroy(1738-1745)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(334-375)
tests/runtime/out_s3_special_features.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1177)
tests/runtime/out_s3_queue.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_blob.h (1)
plugins/out_s3/s3_blob.c (5)
s3_blob_register_parts(299-331)s3_blob_process_events(427-507)s3_blob_store_chunk(338-412)s3_blob_recover_state(123-142)s3_blob_notify_delivery(81-113)
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
5dc4ed1 to
da774d2
Compare
There was a problem hiding this comment.
Actionable comments posted: 20
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/flb_blob_db.c (1)
1211-1331:file_createdcolumn not selected in SQL query
flb_blob_db_file_fetch_oldest_readyreadsfile_createdfrom column index 6 (line 1280), butSQL_GET_OLDEST_FILE_WITH_PARTS_CONCATonly selects 6 columns (indices 0–5):f.id,f.path,GROUP_CONCAT(p.part_id...),f.source,f.remote_id, andf.tag. Thef.createdcolumn is missing from the SELECT list and only appears in the ORDER BY clause. This causes the function to silently read an out-of-range column index, returningfile_created == 0instead of the actual timestamp.Add
f.createdto the SELECT list inSQL_GET_OLDEST_FILE_WITH_PARTS_CONCAT:"SELECT f.id, f.path, GROUP_CONCAT(p.part_id ORDER BY p.part_id ASC) AS part_ids, f.source, " \ " f.remote_id, f.tag, f.created "
🤖 Fix all issues with AI agents
In @include/fluent-bit/flb_parquet.h:
- Around line 20-90: Wrap the public C declarations in flb_parquet.h with
standard C++ linkage guards: add #ifdef __cplusplus extern "C" { #endif before
the typedef and function prototypes (including flb_parquet_validate_schema, the
typedef flb_parquet_schema, flb_parquet_schema_create,
flb_parquet_schema_destroy, and flb_msgpack_to_parquet_streaming) and close with
#ifdef __cplusplus } #endif after them, keeping the existing include guards and
comments intact.
In @plugins/out_s3/s3_blob.c:
- Around line 343-417: The temp filename generated in s3_blob_store_chunk via
snprintf(temp_path, ..., "%s/chunk_%lu_%p.msgpack", ...) is predictable and can
collide; replace this with a safe unique-temp creation (e.g., use mkstemp with a
template like "%s/chunk_XXXXXX.msgpack" or generate a UUID/atomic counter and
format that into temp_path) so the file is created atomically and uniquely, open
the file descriptor returned by mkstemp (or use the UUID to build a path and
open with O_CREAT|O_EXCL) for writing instead of fopen, check for errors, and
ensure you still unlink on failure and close the fd after fwrite; update
references in this function (s3_blob_store_chunk, temp_path usage, and
error/unlink branches) accordingly.
In @plugins/out_s3/s3_multipart.c:
- Around line 244-298: The build_complete_multipart_xml function currently
allows completing a multipart upload when some m_upload->etags[] are NULL by
skipping those parts; change this to enforce integrity: scan m_upload->etags in
build_complete_multipart_xml (or reuse valid_parts logic) and if any etag is
NULL for part indexes < m_upload->part_number then log an error via
flb_plg_error with context and return NULL (do not build or return a partial
XML); keep the early check for zero valid parts but replace the "skip and
continue" behavior around the loop that references m_upload->etags[i] so that
any NULL causes immediate failure instead of being omitted from the XML.
- Around line 947-1042: The retry backoff uses the POSIX-only usleep() in
s3_multipart_upload_file_parts (calls where backoff_ms is multiplied by 1000 and
passed to usleep), which breaks Windows builds; replace these usleep calls with
a cross-platform millisecond sleep wrapper (or conditional Sleep on Windows) and
call that wrapper (e.g., sleep_ms(backoff_ms)) in both retry locations where
backoff_ms is computed for presigned URL fetch and part upload retries; ensure
the wrapper maps to Sleep(backoff_ms) on _WIN32 and to usleep(backoff_ms * 1000)
on POSIX, and update includes/guards accordingly.
- Around line 1044-1132: The use of stat() in s3_multipart_upload_file with
struct stat/file_stat and file_stat.st_size is not portable to Windows; replace
the direct stat() call with a platform-agnostic wrapper (or Fluent Bit's
abstraction) so file sizing works on Windows (e.g., on Windows call _stat64 or
_wstat64 when _WIN32 is defined and map its st_size to the code's expected type,
or use flb_stat/flb_fstat if available). Update the s3_multipart_upload_file
function to perform the conditional platform-specific stat and ensure m_upload
comparisons use the correct 64-bit size type for st_size (reference symbols:
s3_multipart_upload_file, file_stat, stat/_stat64, st_size, m_upload->bytes).
- Around line 848-906: The s3_multipart_upload_file_part function calls
POSIX-only APIs (open, lseek, read, close) unconditionally; make these calls
platform-safe by either switching to Fluent Bit's cross-platform file I/O
helpers if available or adding conditional wrappers/macros that map to _open,
_lseek, _read, _close on Windows and include <io.h> (or use <unistd.h> on
POSIX). Update the calls in s3_multipart_upload_file_part (where open, lseek,
read, close are used) to use the chosen wrapper/macro names so compilation works
on Windows and POSIX without changing logic or error handling.
- Around line 1020-1027: The exponential backoff calculation can overflow:
compute the shift in a wider unsigned type, bound the exponent before shifting,
and apply the 30s cap before multiplying by 1000; specifically, replace the
direct 1000 * (1 << (part_attempt - 1)) with a safe sequence that (1) checks
part_attempt and limits the exponent (e.g., if part_attempt - 1 >= 34 treat as
max), (2) computes the shift into a 64-bit unsigned (e.g., temp = 1ULL <<
(part_attempt - 1)), (3) compares temp to the cap in seconds (30000/1000) and
uses the cap if exceeded, and (4) multiplies by 1000 into backoff_ms only after
ensuring no overflow, keeping existing variables part_attempt,
ctx->part_delivery_attempt_limit, and backoff_ms and preserving the
usleep(backoff_ms * 1000) behavior.
- Around line 768-846: s3_multipart_check_upload_exists is rebuilding the S3 key
using ctx->seq_index which may differ from the original key and causes false
NoSuchUpload results; fix by preserving and using the original S3 key instead of
regenerating it: modify s3_multipart_upload_create to save the generated s3_key
into the blob record (or return it) and change s3_multipart_check_upload_exists
to accept an s3_key parameter (or read the persisted s3_key from the blob DB)
and use that when calling build_s3_uri/request; update all callers to pass the
original s3_key and ensure blob DB schema/code is extended to persist and
retrieve s3_key alongside file_tag/file_path/upload_id.
In @plugins/out_s3/s3_multipart.h:
- Around line 77-80: The declaration and implementation of
s3_multipart_check_upload_exists use cfl_sds_t for tag, file_path, and
upload_id, which is inconsistent with other functions (e.g.,
s3_multipart_upload_file) that use const char *; change the function signature
to use const char * for those three parameters, update any callers to pass C
strings, and in the implementation convert to flb_sds_t/local buffers only where
needed (or use flb_* helpers that accept const char *) so all flb_ functions
receive the expected types and the header matches the rest of the API.
In @plugins/out_s3/s3_queue.c:
- Around line 345-419: upload_part_with_db_tracking leaks the SDS stored in
m_upload.etags for the uploaded part because after calling
flb_blob_db_file_part_update_remote_id the code never frees that string; after
the DB update, call flb_sds_destroy on m_upload.etags[m_upload.part_number - 1]
and set it to NULL (use the same bounds check you already have), and also ensure
any early-return failure paths that leave a populated etag call flb_sds_destroy
and NULL it so no SDS is leaked; references: upload_part_with_db_tracking,
m_upload.etags, flb_blob_db_file_part_update_remote_id, flb_sds_destroy,
check_and_complete_multipart.
In @src/aws/flb_aws_util.c:
- Around line 1100-1121: After stripping leading '.' and '/' from file_path
(valid_file_path), add an explicit check that valid_file_path is not an empty
string before performing replacements: if *valid_file_path == '\0' treat this as
invalid input and skip/abort the $FILE_PATH replacement (e.g., do not call
replace_uri_tokens with an empty value), return the original s3_key or NULL as
appropriate, and emit a warning so users know the input was only dots/slashes;
update the logic around the while loop and the branch that uses
FILE_PATH_STRING/replace_uri_tokens to bail out or skip replacement when
valid_file_path is empty to avoid unintended empty substitutions.
In @src/flb_blob_db.c:
- Around line 1911-1934: The no-SQLDB stub for flb_blob_db_file_part_get_next
has a mismatched signature (missing the trailing int *part_count) compared to
the header and real implementation; update the stub function
declaration/definition for flb_blob_db_file_part_get_next to include the final
parameter int *part_count and return FLB_BLOB_DB_ERROR_NO_BACKEND_AVAILABLE so
the prototype matches the header and SQLDB build.
- Around line 1574-1678: The function flb_blob_db_file_get_next_pending
currently calls sqlite3_step() once and always resets the prepared statement,
causing repeated calls to return the same row; change the control flow in
flb_blob_db_file_get_next_pending so that when sqlite3_step(statement) yields
SQLITE_ROW you do NOT call sqlite3_reset()/sqlite3_clear_bindings() (leave the
statement open so subsequent calls step to the next row), and only call
sqlite3_reset()/sqlite3_clear_bindings() when result == SQLITE_DONE or on error
(set context->last_error and reset/clear then). Keep the unlocking and existing
cleanup logic for the exists == -1 path but ensure you do not destroy or reset
the statement on successful SQLITE_ROW so callers looping over
context->stmt_get_next_pending_file advance through the result set.
In @src/flb_parquet_impl.cpp:
- Around line 167-215: convert_to_integer currently silently clamps out-of-range
values which can hide data loss; update convert_to_integer to either emit a
warning when clamping occurs or make clamping configurable: add a parameter
(e.g., bool allow_clamp or enum ClampPolicy) to convert_to_integer and, in the
POSITIVE_INTEGER / NEGATIVE_INTEGER / FLOAT cases and the parsed-string branch
where clamping happens, if allow_clamp is true log a warning including the
original value, target type T and clamped value (use the module's logger) before
calling builder->Append, otherwise return an arrow::Status::Invalid (or a new
Status indicating overflow) instead of silently clamping; ensure references to
parse_string_to_number and builder remain unchanged and propagate the new
parameter through callers.
- Around line 726-731: Add a runtime guard at the start of
flb_parquet_schema_create() that detects coroutine context (i.e., whether the
caller is running in a Fluent Bit coroutine) and fails fast when not on the main
thread: check the appropriate Fluent Bit runtime API used elsewhere in the
codebase (same pattern as plugins/out_s3/s3.c) and if a coroutine context is
detected, log an error via flb_error/processLogger and return NULL (or an error
code) instead of proceeding to call parse_schema_from_json()/yyjson_read().
Ensure the check is the first action in flb_parquet_schema_create() so misuse is
prevented at runtime.
In @tests/runtime/out_s3_config.c:
- Around line 368-457: In flb_test_storage_class_variations and
flb_test_canned_acl_options the FLB_S3_PLUGIN_UNDER_TEST env var is set once
before the loop but unset inside the loop, causing only the first iteration to
run in test mode; fix by ensuring the env var lifetime covers all
iterations—either move unsetenv("FLB_S3_PLUGIN_UNDER_TEST") to after the
for-loop (so the entire loop runs with test mode) or setenv at the start and
unsetenv at the end of each loop iteration; apply the same adjustment to any
other multi-scenario tests that call setenv once before a loop and unsetenv
inside it.
In @tests/runtime/out_s3_queue.c:
- Around line 142-182: The test function flb_test_queue_file_buffering leaves
the temporary store_dir (variable store_dir) on disk; after flb_destroy(ctx) and
before unsetenv call add explicit cleanup to remove the directory (use rmdir()
or equivalent recursive removal if needed) so the /tmp/flb-s3-queue-test
directory is deleted; ensure cleanup runs even if flb_stop/flb_destroy succeeded
and reference store_dir string when calling the removal function.
🧹 Nitpick comments (20)
src/flb_parquet_impl.cpp (3)
431-434: Silent fallback to utf8 for unknown types could mask schema errors.When an unrecognized type name is encountered, the code silently defaults to
utf8(line 433). This could lead to incorrect data types being used without any indication of a schema problem.Suggest logging a warning when falling back to the default type.
⚠️ Add warning for unknown types} else { auto it = TYPE_FACTORY_MAP.find(type_name); - data_type = (it != TYPE_FACTORY_MAP.end()) ? it->second() : arrow::utf8(); + if (it != TYPE_FACTORY_MAP.end()) { + data_type = it->second(); + } else { + flb_warn("[parquet] Unknown type '%s' for field '%s', defaulting to string", + type_name.c_str(), field_name.c_str()); + data_type = arrow::utf8(); + } }
634-647: Document the expected format for binary field validation.Lines 638-644 convert binary data to hex string format for comparison. Ensure that test code and documentation specify that
expected_valuefor binary fields must be a hex string (e.g., "48656c6c6f" for "Hello").
936-946: Document behavior when builder recreation fails mid-stream.If builder recreation fails after successfully writing a batch (line 942), the function throws an exception. This leaves the output file in a partially written state. While this is likely acceptable for streaming contexts, ensure callers understand that partial data may exist on failure.
tests/runtime/out_s3_queue.c (1)
27-338: Tests provide good smoke test coverage for queue operations.The test suite covers essential S3 queue scenarios (basic ops, workers, retries, buffering, ordering, timeouts, concurrency, shutdown). The pattern of verifying setup success and checking for crashes/hangs is consistent with other runtime tests in the codebase.
Optional: Consider more robust synchronization
Tests currently use
sleep()for timing, which can be flaky in resource-constrained CI environments. For more robust tests, consider:
- Using polling with timeouts to check for expected conditions
- Implementing test hooks or callbacks to signal completion
- Adding explicit result validation beyond "did not crash"
However, this pattern appears standard for runtime tests in this codebase, so this refactor can be deferred.
include/fluent-bit/flb_parquet.h (1)
68-88: Consider documenting unused parameter behavior.The
total_file_sizeparameter is marked as unused in the implementation but remains in the function signature. Consider adding a note in the documentation explaining its purpose or future use, or removing it if it's not needed.tests/runtime/out_s3_recovery.c (3)
36-106: Consider adding verification of recovery behavior.The test validates that the plugin starts and stops successfully, but doesn't verify that recovery actually occurred. Consider adding assertions to check:
- Database state after recovery (file entries, part states)
- Whether buffered data from the first run was uploaded in the second run
- Upload queue state
172-215: Multipart resume test lacks verification of resume behavior.The test doesn't verify that multipart upload was actually initiated or that resume would work. Consider:
- Checking database for multipart upload state
- Verifying upload_id was persisted
- Testing actual resume on second run
388-433: Concurrent access test doesn't verify thread safety.Pushing data in a loop doesn't guarantee concurrent DB access. The test should verify:
- No database corruption occurred
- All chunks were processed
- Lock contention was handled properly
Consider using threading primitives or checking DB consistency after the test.
tests/runtime/out_s3_special_features.c (2)
27-62: Tests don't verify compression is actually applied.Both Snappy compression tests start the plugin successfully but don't verify that:
- Output data is actually compressed
- Snappy format is valid
- Compressed size is less than uncompressed
Consider adding verification of the actual S3 output or compression state.
Also applies to: 64-100
381-465: Credential expiration tests lack simulation of expiration.Both credential expiration tests configure retry limits but don't:
- Simulate actual credential expiration
- Verify refresh logic is triggered
- Check that retries occur as expected
These are smoke tests that verify configuration is accepted, not actual credential refresh behavior.
src/aws/flb_aws_util.c (1)
1179-1183: Minor: Stale comment from previous code version.The comment block at lines 1179-1183 appears to be documentation for
flb_get_s3_keybut is placed after the function definition and beforeflb_aws_strftime_precision. This seems like a remnant from code refactoring.Consider removing or relocating this orphaned comment
-/* - * Constructs S3 object key as per the format. - * Supports variables: $TAG, $TAG[0-9], $UUID, $INDEX, time formatters, $FILE_PATH, $FILE_NAME - * file_path: optional log file path (can be NULL, e.g., nginx log file path) - used only when format contains $FILE_PATH or $FILE_NAME - */ size_t flb_aws_strftime_precision(char **out_buf, const char *time_format,plugins/out_s3/s3_store.c (2)
371-386: Inconsistent time format between Windows and POSIX.Windows uses
%H-%M-%S(hyphens) while POSIX uses%H:%M:%S(colons). This could cause issues when:
- Files are shared between platforms
- Logs are compared across environments
Consider using consistent separators if cross-platform compatibility matters.
Use consistent time format across platforms
#ifdef FLB_SYSTEM_WINDOWS /* Windows: gmtime_s(struct tm*, const time_t*) */ if (gmtime_s(&tm_buf, &now) != 0) { return NULL; } - strftime(tmp, sizeof(tmp) - 1, "%Y-%m-%dT%H-%M-%S", &tm_buf); + strftime(tmp, sizeof(tmp) - 1, "%Y-%m-%dT%H-%M-%S", &tm_buf); /* Use hyphens for Windows filesystem compatibility */ #else /* POSIX: gmtime_r(const time_t*, struct tm*) */ if (gmtime_r(&now, &tm_buf) == NULL) { return NULL; } - strftime(tmp, sizeof(tmp) - 1, "%Y-%m-%dT%H:%M:%S", &tm_buf); + strftime(tmp, sizeof(tmp) - 1, "%Y-%m-%dT%H-%M-%S", &tm_buf); /* Match Windows format for consistency */ #endif
35-42: CI environment detection switches to memory backend instead of filesystem.The function checks for CI environment variables (CI, TRAVIS, GITHUB_ACTIONS, GITLAB_CI, JENKINS_URL) and selects the memory backend at line 395. A runtime warning is already logged when this behavior is triggered. Consider adding static documentation (e.g., in a configuration guide or README) about this behavior, as users running in CI-like environments with
CI=truemay expect persistent filesystem storage by default.plugins/out_s3/s3.h (1)
72-88: Verify: etags array size matches S3_AWS_MAX_PARTS.Line 78 declares
flb_sds_t etags[10000]which should matchS3_AWS_MAX_PARTS(10000). Consider using the constant for clarity and maintainability.Use S3_AWS_MAX_PARTS constant for etags array
- flb_sds_t etags[10000]; + flb_sds_t etags[S3_AWS_MAX_PARTS];tests/runtime/out_s3_multipart.c (1)
429-493: Hardcoded path/tmp/test_s3_multipart_persistence.dbmay not be portable.The persistence test uses a hardcoded path that:
- May not exist on all systems (e.g., Windows)
- Could conflict with concurrent test runs
- Doesn't clean up the database file after test
Consider using a test-specific temporary directory or cleaning up the file.
Add cleanup of test database file
flb_stop(ctx); flb_destroy(ctx); unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); + + /* Cleanup test database file */ + unlink("/tmp/test_s3_multipart_persistence.db"); }plugins/out_s3/s3_stream.c (1)
251-280: Handle all msgpack unpacker error codes to avoid potential spin
process_unpacked_recordsonly distinguishes:
MSGPACK_UNPACK_SUCCESSMSGPACK_UNPACK_PARSE_ERRORMSGPACK_UNPACK_EXTRA_BYTES- Everything else falls through.
If
msgpack_unpacker_nextever returns other error codes (e.g.,MSGPACK_UNPACK_NOMEM_ERROR), the loop condition (!= MSGPACK_UNPACK_CONTINUE) combined with the fall-through can lead to repeated calls returning the same error without progress.Consider explicitly handling all non-success, non-
CONTINUEcases and treating them as fatal, e.g.:ret = msgpack_unpacker_next(&stream_ctx->unpacker, &stream_ctx->result); if (ret == MSGPACK_UNPACK_SUCCESS) { ... } else if (ret == MSGPACK_UNPACK_CONTINUE) { break; } else { flb_plg_error(ctx->ins, "Msgpack unpack error: %d", ret); return -1; }This makes the error behavior clearer and avoids any chance of a tight loop on unexpected errors.
tests/internal/parquet.c (1)
1905-1977: Use standard macros or functions for NaN/Inf instead of division-by-zeroIn
test_special_float_values, special values are generated via:const double nan_val = 0.0 / 0.0; const double inf_val = 1.0 / 0.0; const double neg_inf_val = -1.0 / 0.0;This behavior is only guaranteed by the C standard when the implementation defines
__STDC_IEC_559__(IEEE 754 compliance). On systems that do not claim IEEE 754 support, the behavior is implementation-defined and may raise floating-point exceptions or produce unexpected results.Replace with standard constants or functions:
#include <math.h> const double nan_val = NAN; const double inf_val = INFINITY; const double neg_inf_val = -INFINITY;For maximum portability across older C implementations,
nan()andHUGE_VALfunctions are even more widely supported than the macros.plugins/out_s3/s3_multipart.c (3)
300-360: Add input validation for the ctx parameter.The function dereferences
ctxmultiple times (lines 308, 311, 317, 333, etc.) without verifying it's non-NULL. While callers should pass valid pointers, defensive programming suggests validating inputs, especially for public API functions.🛡️ Proposed fix to add input validation
int s3_multipart_create_headers(struct flb_s3 *ctx, char *body_md5, struct flb_aws_header **headers, int *num_headers, int is_multipart) { int n = 0; int headers_len = 0; struct flb_aws_header *s3_headers = NULL; + + if (!ctx || !headers || !num_headers) { + return -1; + }
393-442: Add input validation for public API parameters.The function is part of the public API but doesn't validate input parameters. Consider adding checks for:
ctxis non-NULLtagis non-NULL whentag_len > 0tag_lenis non-negative🛡️ Proposed input validation
struct multipart_upload *s3_multipart_upload_create(struct flb_s3 *ctx, const char *tag, int tag_len, const char *path) { struct multipart_upload *m_upload; flb_sds_t s3_key; flb_sds_t tmp_sds; int ret; + + if (!ctx || !tag || tag_len < 0) { + if (ctx) { + flb_plg_error(ctx->ins, "Invalid parameters to s3_multipart_upload_create"); + } + return NULL; + }
300-360: Consider adding API documentation for public functions.The public API functions (e.g.,
s3_multipart_create_headers,s3_multipart_upload_destroy,s3_multipart_upload_create,s3_multipart_create,s3_multipart_upload_part,s3_multipart_complete,s3_multipart_abort,s3_multipart_check_upload_exists,s3_multipart_upload_file) lack documentation comments describing their purpose, parameters, return values, and error conditions.Adding comprehensive documentation would improve maintainability and help future developers understand the API contracts.
Also applies to: 362-387, 393-442, 444-506, 508-605, 607-691, 693-758, 768-846, 848-906, 1044-1132
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (41)
examples/out_s3/blob_sink_s3.confexamples/out_s3/kafka_sink_s3.confexamples/out_s3/log_sink_s3.confinclude/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/fuzzers/aws_util_fuzzer.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3.ctests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
🚧 Files skipped from review as they are similar to previous changes (10)
- examples/out_s3/kafka_sink_s3.conf
- tests/runtime/out_s3_edge_cases.c
- examples/out_s3/log_sink_s3.conf
- examples/out_s3/blob_sink_s3.conf
- tests/internal/CMakeLists.txt
- tests/internal/aws_util.c
- plugins/out_s3/s3_blob.h
- plugins/out_s3/s3_stream.h
- plugins/out_s3/CMakeLists.txt
- tests/runtime/out_s3_format.c
🧰 Additional context used
🧠 Learnings (8)
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_parquet.hinclude/fluent-bit/flb_aws_util.h
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_kinesis_firehose/firehose.csrc/flb_parquet_impl.cpp
📚 Learning: 2025-09-04T12:35:36.904Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:3275-3282
Timestamp: 2025-09-04T12:35:36.904Z
Learning: The out_s3 plugin intentionally uses a simple numeric comparison for retry_limit (chunk->failures >= ctx->ins->retry_limit) rather than the standard Fluent Bit pattern that checks for FLB_OUT_RETRY_UNLIMITED (-1). The maintainer wants to keep this current behavior for consistency within the plugin.
Applied to files:
plugins/out_s3/s3_stream.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_s3/s3_store.c
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
🧬 Code graph analysis (23)
plugins/in_blob/blob_file.c (3)
plugins/in_blob/blob_db.c (1)
blob_db_file_insert(164-196)lib/cfl/src/cfl_sds.c (1)
cfl_sds_destroy(127-137)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3_queue.h (1)
plugins/out_s3/s3_queue.c (9)
s3_queue_remove(267-275)s3_queue_buffer_chunk(315-337)s3_queue_add_file(57-111)s3_queue_add_part(113-183)s3_queue_add_pending_file(185-232)s3_queue_process_entry(665-724)s3_queue_recover_all(765-809)s3_queue_recover_from_database(819-923)s3_queue_recover_from_fstore(996-1056)
include/fluent-bit/flb_parquet.h (1)
src/flb_parquet_impl.cpp (8)
flb_parquet_validate_schema(697-713)flb_parquet_validate_schema(697-699)flb_parquet_schema_create(715-746)flb_parquet_schema_create(715-717)flb_parquet_schema_destroy(748-754)flb_parquet_schema_destroy(748-748)flb_msgpack_to_parquet_streaming(756-1072)flb_msgpack_to_parquet_streaming(756-761)
tests/internal/parquet.c (3)
src/flb_parquet_impl.cpp (1)
field_name(386-386)src/flb_pack.c (1)
flb_pack_json(530-535)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1177)
tests/runtime/out_s3_config.c (1)
src/flb_lib.c (9)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/internal/fuzzers/aws_util_fuzzer.c (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1177)
tests/runtime/out_s3_queue.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_multipart.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (5)
cb_s3_upload(1413-1528)s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)mock_s3_call(105-179)
plugins/out_s3/s3_auth.h (1)
plugins/out_s3/s3_auth.c (3)
s3_auth_init_endpoint(78-121)s3_auth_request_presigned_url(166-253)s3_auth_fetch_presigned_url(346-387)
plugins/out_s3/s3_auth.c (6)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_utils.c (1)
flb_utils_url_split(1531-1667)src/flb_upstream.c (1)
flb_upstream_create_url(416-481)src/flb_output.c (1)
flb_output_upstream_set(1578-1669)src/flb_http_client.c (5)
flb_http_add_header(1013-1045)flb_http_basic_auth(1373-1377)flb_http_bearer_auth(1385-1420)flb_http_client(864-909)flb_http_do(1622-1682)src/flb_sds.c (3)
flb_sds_copy(260-281)flb_sds_create_len(58-76)flb_sds_destroy(389-399)
src/aws/flb_aws_util.c (2)
src/flb_http_client.c (1)
flb_http_client_destroy(1738-1745)src/flb_sds.c (2)
flb_sds_create_len(58-76)flb_sds_destroy(389-399)
plugins/out_s3/s3_store.h (1)
plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(536-572)
tests/runtime/out_s3_error_handling.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)
plugins/out_s3/s3_multipart.h (1)
plugins/out_s3/s3_multipart.c (12)
flb_s3_calculate_optimal_part_size(116-168)s3_multipart_get_md5_base64(170-191)s3_multipart_create_headers(300-360)s3_multipart_upload_create(393-442)s3_multipart_upload_destroy(362-387)s3_multipart_create(444-506)s3_multipart_upload_part(508-605)s3_multipart_complete(607-691)s3_multipart_abort(693-758)s3_multipart_check_upload_exists(768-846)s3_multipart_upload_file_part(848-906)s3_multipart_upload_file(1044-1132)
plugins/out_s3/s3_queue.c (9)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)plugins/out_s3/s3_store.c (4)
s3_store_buffer_put(210-245)s3_store_file_unlock(581-586)s3_store_file_inactive(489-507)s3_store_file_lock(574-579)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(346-387)plugins/out_s3/s3_multipart.c (3)
s3_multipart_upload_file_part(848-906)s3_multipart_complete(607-691)s3_multipart_upload_create(393-442)src/flb_blob_db.c (1)
flb_blob_db_file_fetch_all_parts(1440-1572)plugins/out_s3/s3.c (3)
s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)plugins/out_s3/s3_blob.c (1)
s3_blob_notify_delivery(81-118)lib/cfl/src/cfl_sds.c (1)
cfl_sds_destroy(127-137)src/flb_fstore.c (1)
flb_fstore_file_inactive(238-256)
tests/runtime/out_s3_special_features.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_blob.c (4)
plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(346-387)src/flb_input_blob.c (1)
flb_input_blob_delivery_notification_destroy(222-231)src/flb_blob_db.c (2)
flb_blob_db_lock(367-372)flb_blob_file_update_remote_id(530-561)plugins/out_s3/s3_queue.c (1)
s3_queue_add_pending_file(185-232)
include/fluent-bit/flb_blob_db.h (1)
src/flb_blob_db.c (7)
flb_blob_db_file_fetch_all_parts(1440-1572)flb_blob_db_file_get_next_pending(1578-1678)flb_blob_db_file_get_next_pending(1911-1920)flb_blob_db_file_part_check_uploaded(1684-1719)flb_blob_db_file_part_check_uploaded(1922-1927)flb_blob_db_reset_zombie_parts(1722-1746)flb_blob_db_reset_zombie_parts(1929-1932)
src/flb_blob_db.c (3)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)src/flb_sqldb.c (2)
flb_sqldb_lock(173-178)flb_sqldb_unlock(180-185)lib/cfl/src/cfl_sds.c (2)
cfl_sds_create(113-125)cfl_sds_destroy(127-137)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3_multipart.c (3)
plugins/out_s3/s3.c (2)
write_seq_index(226-246)s3_plugin_under_test(73-79)src/aws/flb_aws_util.c (3)
flb_get_s3_key(1083-1177)flb_aws_xml_get_val(623-655)flb_aws_print_xml_error(586-610)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(346-387)
🔇 Additional comments (50)
plugins/out_kinesis_firehose/firehose.c (1)
495-501: Compression type list is complete and accurate.The documented compression types ('gzip', 'snappy', 'zstd') match the implementation in
src/aws/flb_aws_compress.c, which defines exactly these three algorithms in its compression options array. The removal of 'arrow' is correct—Arrow is a data format, not a compression algorithm, and Parquet support is handled separately via format configuration in the S3 plugin, not as a compression type. The help text "Defaults to no compression" correctly reflects the default behavior. No changes needed.tests/runtime/out_s3.c (1)
49-49: LGTM! Proper test cleanup ensures isolation.The addition of
unsetenv("FLB_S3_PLUGIN_UNDER_TEST")in all six test functions properly cleans up the test-mode environment variable, preventing pollution between tests.Also applies to: 86-86, 125-125, 162-162, 198-198, 234-234
src/flb_signv4.c (1)
646-677: LGTM! URI handling correctly implements service-specific encoding.The updated logic properly handles two URI encoding modes:
- S3: uses pre-encoded URIs to preserve special characters
- Other services: normalize then encode
Error handling and memory management are correct in both branches.
src/flb_parquet_impl.cpp (3)
48-53: LGTM! Well-chosen constants for streaming Parquet processing.The constants are appropriately sized:
- 64K batch size aligns with Arrow standards
- 512K row groups are optimized for S3 compatibility
- Buffer sizes are reasonable for memory efficiency
1053-1071: LGTM! Proper error cleanup with flag tracking.The error cleanup path correctly:
- Checks flags before cleanup to prevent double-free
- Updates flags immediately after each cleanup operation
- Releases resources in proper order
- Removes the failed output file
983-1008: Behavior confirmed: unlink() on error is correct for temporary files.Verification shows:
Partial Parquet files are unusable — Confirmed. The function unlinks on any write/close failures (writer->Close(), stream->Close(), stat() failures), and no recovery logic exists.
Callers don't expect file inspection — Confirmed. The S3 plugin unconditionally calls
unlink(temp_path)after conversion failures and handles empty output with a warning message guiding configuration rather than suggesting file inspection.Behavior matches design — This is defensive cleanup for temporary files. Both the function and caller clean up, which is appropriate and prevents orphaned files.
The implementation is correct. Consider documenting this file cleanup behavior in the API header (flb_parquet.h) if users need to understand error handling expectations.
include/fluent-bit/flb_aws_util.h (1)
200-207: All callers have been properly updated with the newfile_pathparameter. Verified across test cases, fuzzers, and production code (s3_multipart.c, s3.c).plugins/out_s3/s3_auth.h (1)
26-47: Code review checks are satisfied in current implementation.The presigned URL API is well-implemented:
Enum handling: All 4
s3_presigned_url_type_tvalues are handled in the switch statement (s3_auth.c lines 288-315) with a default error case.Return value checking: All callers consistently check
if (ret < 0)— s3_queue.c, s3_multipart.c, s3.c, and s3_blob.c each verify error returns. The convention is-1for error,0for success.part_number validation: Part numbers are derived from
entry->part_id + 1(s3_queue.c:360) following AWS 1-based numbering. Bounds validation (1-10000 per AWS spec) occurs when storing ETags (s3_queue.c:391-394) with the etags array sized at 10000 elements.tests/internal/fuzzers/aws_util_fuzzer.c (1)
73-73: LGTM! Fuzzer correctly updated for extended API signature.The addition of the NULL parameter aligns with the extended
flb_get_s3_keysignature and maintains the existing fuzzing behavior without file path augmentation.tests/runtime/CMakeLists.txt (1)
238-245: Excellent test coverage expansion.Breaking down the S3 output tests into focused areas (config, multipart, format, error handling, edge cases, special features, queue, recovery) improves test organization and maintainability.
plugins/in_blob/blob_file.c (1)
81-88: LGTM! Database operations now properly optional.The conditional guard correctly makes database insertion optional when
database_fileis not configured, while preserving error handling within the guarded block.include/fluent-bit/flb_parquet.h (1)
20-90: Excellent API design with clear separation of concerns.The API design demonstrates several strengths:
- Opaque type pattern provides good encapsulation
- Separate validation enables fail-fast at startup
- Pre-parsing schema in main thread avoids coroutine stack overflow (well-documented rationale)
- Clear documentation beyond typical Fluent Bit standards
plugins/out_s3/s3_auth.c (1)
255-344: LGTM - URL encoding and path construction.The function properly encodes the S3 key and upload_id before constructing presigned URL paths, preventing injection issues. Error handling and resource cleanup are appropriate.
tests/runtime/out_s3_special_features.c (1)
264-297: Good: Test correctly expects initialization failure.The test properly validates that an invalid compression type causes
flb_startto fail (line 290) and handles both success and failure paths in the cleanup (lines 292-294).plugins/out_s3/s3_blob.c (2)
195-302: LGTM - Comprehensive aborted file recovery logic.The function properly handles different scenarios:
- Validates existing upload IDs before resuming
- Distinguishes between resumable and fresh-start cases
- Handles retry limits and cleanup
- Provides detailed logging for each decision path
The conservative approach on validation failures (lines 250-259) is appropriate for data safety.
432-512: Good architecture decision: Lightweight flush processing.The comment block (lines 419-431) and implementation correctly defer heavy operations to timer callbacks, keeping the coroutine-context flush lightweight. This prevents stack overflow issues.
The function properly:
- Parses events and extracts metadata
- Persists to database
- Enqueues for deferred processing
plugins/out_s3/s3_store.h (2)
51-52: Good: Context-free API enables broader usage.The new
flb_chunk_file_openfunction can be called from C++ code or contexts without requiring a Fluent Bit context object. This is a clean addition to the API.
32-32: No breaking change from field rename.The field rename from
file_pathtostream_pathinstruct s3_fileis contained within the s3_store module (s3_store.h/c). All references tos3_file->stream_pathare in s3_store.c, and no external code references the old field name. Thefile_pathreferences found in s3_queue.c belong to the separatestruct upload_queue, which was not affected by this change.Likely an incorrect or invalid review comment.
src/aws/flb_aws_util.c (5)
45-46: LGTM - New macro definitions for S3 key templating.These macros define the placeholders for file path and file name variables in S3 key formats, extending the existing template variable support.
184-184: LGTM - Auth retry mechanism with credential refresh.The implementation correctly:
- Uses a flag to prevent infinite retry loops
- Destroys the old HTTP client before retrying
- Logs the retry attempt for debugging
The retry flow handles the common case of credential expiration during normal operation (e.g., IAM role rotation).
Also applies to: 210-226
851-876: LGTM - Well-structured template processing function.The function signature is clear and the initial format length validation is appropriate.
1058-1075: Verify: Error path cleanup is thorough butbufownership check may be redundant.Line 1066 checks
buf && buf != tmpbefore destroyingbuf. However, examining the code flow, when an error occurs:
bufis only assigned viaflb_sds_create_size(line 919)- If
flb_sds_printfsucceeds,tmppoints tobuf(same or reallocated)- After successful printf, both are destroyed and set to NULL
The condition is safe but may be overly cautious since
bufandtmpshould either both be NULL orbufwould have been destroyed earlier in the success path.
1153-1170: LGTM - Backward-compatible path appending logic.When no file variables are used in the format but a file_path is provided, the code correctly:
- Ensures a separator exists between the key format result and the file path
- Appends the cleaned file path
This maintains backward compatibility with existing configurations.
plugins/out_s3/s3_queue.h (1)
1-85: LGTM - Clean and well-documented queue API header.The header provides:
- Clear function declarations for queue lifecycle management
- Well-documented three-phase recovery architecture
- Consistent naming with
s3_queue_prefix- Appropriate separation between public and internal recovery functions
The API surface aligns with the implementation snippets provided in the context.
plugins/out_s3/s3_store.c (4)
109-133: LGTM - Buffer space validation with warning threshold.The implementation correctly:
- Returns early if no limit is configured
- Checks for overflow before adding new bytes
- Emits a warning at 95% capacity (BUFFER_WARNING_THRESHOLD)
- Uses proper logging with context
247-275: LGTM - Chunk data size calculation handles edge cases.The function safely:
- Validates chunk_name is non-null and non-empty
- Handles path construction overflow
- Returns 0 for files smaller than header + metadata
509-534: LGTM - File deletion with proper cleanup.The function correctly:
- Validates both s3_file and fsf pointers
- Verifies fsf->data matches s3_file before cleanup
- Clears fsf->data to prevent use-after-free
- Updates buffer size accounting
- Frees stream_path before freeing s3_file
536-572: No action needed — metadata length is correctly parsed as big-endian.The code correctly reads the metadata length in big-endian format, which matches the CIO library's specification. This is confirmed by the canonical implementation in
lib/chunkio/include/chunkio/cio_file_st.h:76-79(cio_file_st_get_meta_len()), which uses the identical parsing logic:((uint8_t) map[22] << 8) | (uint8_t) map[23]. The setter (cio_file_st_set_meta_len()) also explicitly encodes in big-endian. The implementation is not an assumption but follows the official CIO format.Likely an incorrect or invalid review comment.
plugins/out_s3/s3.h (4)
31-46: LGTM - Well-defined AWS S3 multipart constants.The constants correctly reflect AWS S3 limits:
- 5 MiB minimum part size
- 5 GiB maximum part size
- 10,000 maximum parts
- 100 MiB default part size
Using
ULLsuffix ensures proper handling for 64-bit values.
48-70: LGTM - Upload queue entry structure is comprehensive.The structure includes all necessary fields for tracking:
- Database identifiers (file_id, part_db_id, part_id)
- File references and offsets
- S3-specific data (s3_key, upload_id)
- Retry metadata (retry_counter, upload_time)
- State flags (needs_upload_creation)
171-174: LGTM - Parquet format support fields.The struct includes:
formatto select JSON or Parquetschema_strfor Parquet schema configurationcached_arrow_schemafor performance (avoids re-parsing in coroutines)
146-153: LGTM - Thread safety with upload_queue_lock properly implemented.The mutex is correctly initialized in the plugin init function (line 895 in s3.c) with error handling and destroyed in the exit function (line 1809 in s3.c). Proper lifecycle management and error checking are in place.
tests/runtime/out_s3_error_handling.c (4)
27-44: LGTM - AWS error response templates for testing.The templates cover key S3 error scenarios:
- Access control errors (AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch)
- Resource errors (NoSuchBucket, NoSuchUpload)
- Throttling (SlowDown)
These match the actual AWS S3 XML error response format.
46-83: LGTM - Test structure follows established patterns.The test correctly:
- Sets up test mode environment variables
- Creates and configures Fluent Bit context
- Pushes test data and waits for processing
- Cleans up environment variables after test
358-420: Testflb_test_retry_layer_auto_retryruns two separate contexts without cleanup between them.The test creates two Fluent Bit contexts sequentially (lines 369-392 and 395-418), which is correct. However, the
FLB_S3_PLUGIN_UNDER_TESTenvironment variable is only unset at line 419, after both sub-tests complete. This is the intended behavior since both sub-tests need the test mode enabled.
541-556: LGTM - Complete test list with consistent naming.All 12 tests are registered with descriptive names that match the test function purposes.
plugins/out_s3/s3_multipart.h (1)
1-101: LGTM - Well-organized multipart upload API.The header provides a clean API surface with:
- Utility functions for part size calculation and MD5 hashing
- Lifecycle management (create, destroy)
- AWS S3 API wrappers (create upload, upload part, complete, abort)
- High-level streaming upload helpers
- Appropriate documentation comments
tests/runtime/out_s3_multipart.c (3)
27-63: LGTM - Basic multipart upload test structure.The test follows the established pattern and properly sets up/tears down the environment.
143-183: Test variableideclared but potentially optimized out by compiler.Line 150 declares
int iwhich is used in the loop at line 174. This is fine, but note that in some tests the variable may appear unused if the loop body is simple.
556-571: LGTM - Complete test list registration.All 12 multipart tests are registered with clear, descriptive names.
plugins/out_s3/s3_queue.c (5)
57-232: Queue entry constructors/destructor look consistentThe three add functions (
s3_queue_add_file,s3_queue_add_part,s3_queue_add_pending_file) ands3_queue_entry_destroyhandle allocations, error paths, and SDS ownership correctly, and they integrate with the queue lock as expected. No changes needed here.
469-613:check_and_complete_multipartDB/ETag handling looks soundThe completion path correctly:
- Fetches the oldest ready file from blob DB.
- Reuses the
s3_keypassed from the part entry (no regeneration).- Reconstructs the ETag list via
flb_blob_db_file_fetch_part_ids.- Feeds the ETags into
s3_multipart_complete.- Destroys all SDS resources (
m_upload.s3_key,m_upload.tag,m_upload.upload_id, and everym_upload.etags[i]) and DB strings on all paths.The logic is consistent with the blob DB API and avoids ownership bugs. No changes needed here.
615-657: Retry handling semantics are correct and consistent with S3 plugin design
handle_upload_failurecorrectly:
- Increments
retry_counterand enforcesctx->ins->retry_limitfor both DB and non-DB entries.- Marks DB-tracked files as aborted (with
flb_blob_db_file_set_aborted_state) when retries are exhausted.- Marks non-DB files inactive via
s3_store_file_inactive.- For retries, schedules
upload_timebackoff and maintainsctx->retry_time/ctx->upload_queue_successonly for non-DB entries.This matches the existing out_s3 retry semantics and cleanly separates DB vs non-DB behavior.
925-993: Resume enqueue path (enqueue_file_parts_for_resume) is well-structuredThe resume helper:
- Recreates a consistent
s3_keyvias3_multipart_upload_create.- Fetches all parts for the file from blob DB and filters out those already uploaded via
flb_blob_db_file_part_check_uploaded.- Enqueues only outstanding parts with
s3_queue_add_part, passing the correct offsets, s3_key, upload_id, and tag.- Frees all temporary arrays and destroys the multipart upload context.
This is a solid basis for crash-resume semantics. No changes required.
996-1055: fstore recovery path looks correct
s3_queue_recover_from_fstore:
- Skips metadata streams and already-locked chunks.
- Honors
chunk->failures >= retry_limitand marks such chunks inactive.- Locks each chunk before enqueuing it with
s3_queue_add_file, and unlocks on enqueue failure.The lifecycle around
s3_store_file_lock/s3_store_file_unlockandflb_fstore_file_inactiveis consistent and safe.include/fluent-bit/flb_blob_db.h (1)
195-216: Header additions align with blob DB implementationThe new SQL definitions (
SQL_GET_ALL_PARTS_FOR_FILE,SQL_GET_NEXT_PENDING_FILE,SQL_GET_PART_UPLOAD_STATUS), corresponding error codes, statement fields, and function prototypes match the implementations insrc/flb_blob_db.c. The public surface looks coherent for multipart recovery and resume logic.src/flb_blob_db.c (4)
215-245: New prepared statements are wired correctlyThe additional calls in
prepare_stmtsfor:
SQL_GET_ALL_PARTS_FOR_FILE→stmt_get_all_parts_for_fileSQL_GET_NEXT_PENDING_FILE→stmt_get_next_pending_fileSQL_GET_PART_UPLOAD_STATUS→stmt_get_part_upload_statusmatch the new SQL macros and error codes from the header, and they are finalized in
flb_blob_db_close. This part of the blob DB initialization/teardown looks consistent.
1440-1572:flb_blob_db_file_fetch_all_partsimplementation looks correct and leak-safeThe new helper:
- Allocates four parallel arrays with
flb_callocand cleans all of them if any allocation fails.- Iterates over
stmt_get_all_parts_for_filerows, dynamically grows arrays withflb_realloc, and carefully frees only newly allocated buffers on reallocation failure while also cleaning the original arrays.- Properly unlocks and resets the statement on all paths.
- Returns 0 (no parts), a positive count, or a negative error code distinctly.
This function is well-structured and safe for the S3 queue resume use case.
1333-1407:flb_blob_db_file_fetch_part_idsSDS handling is correct after type switchAfter switching
remote_id_listtoflb_sds_t *, the function correctly:
- Zeroes the array with
memset(remote_id_list, 0, sizeof(flb_sds_t) * remote_id_list_size);- Populates each slot with
flb_sds_create(tmp)and rolls back allocated entries on error.- Performs another full
memseton the error path to ensure callers never see stray pointers.Call sites that pass
flb_sds_t *(e.g.check_and_complete_multipart) are consistent with this change.
1721-1746:flb_blob_db_reset_zombie_partsis simple and appropriateThe zombie reset helper correctly:
- Validates
contextandcontext->db.- Uses a single
UPDATEwithuploaded = 0 AND in_progress = 1to clear stuck parts.- Executes under the DB lock and propagates
sqlite3_execerrors viacontext->last_error.This is a straightforward and effective way to clean up in-progress flags after crashes.
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
da774d2 to
85f1519
Compare
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Fix all issues with AI agents
In @examples/out_s3/log_sink_s3.conf:
- Line 47: The example config currently hard-codes a development bucket name
"pl-exchange-dev-kafka-sink" on several lines; replace each occurrence with a
clear placeholder (e.g., YOUR_BUCKET_NAME or <your-bucket-name>) so users must
substitute their own bucket, and add a prominent comment at the top of the file
indicating the placeholder must be replaced with a real S3 bucket name before
use; update the entries referenced (lines containing bucket values at
occurrences 47, 83, 106, 130, 157, 184) to the placeholder and include the
top-file comment.
- Line 141: The schema_str entries use unsupported Arrow type names ("int" and
"floatingpoint") and should be updated to supported names; locate each
schema_str occurrence (the JSON value assigned to schema_str) and replace
{"fields":[...,"count","type":{"name":"int","isSigned":true,"bitWidth":64}...,
"latency","type":{"name":"floatingpoint","precision":"DOUBLE"}}]} with
{"fields":[...,"count","type":{"name":"int64"}...,
"latency","type":{"name":"double"}}]}; ensure all three schema_str instances are
changed so the types are int64 and double (drop the unsupported parameter keys
which are ignored).
In @plugins/out_s3/s3_auth.c:
- Around line 112-114: The upstream flag save/restore sequence is undoing the
instance flags that flb_output_upstream_set() adds via
flb_stream_enable_flags(); remove the three lines that call
flb_stream_get_flags(&upstream->base), store upstream_flags, and then
flb_stream_set_flags(&upstream->base, upstream_flags), and replace with a single
direct call to flb_output_upstream_set(upstream, ctx->ins) so the instance
TLS/IPv6/keepalive flags applied by flb_output_upstream_set() are preserved.
In @plugins/out_s3/s3_multipart.c:
- Around line 656-676: The code leaks the SDS in variable query_params when
flb_sds_printf(&query_params, ...) fails; update s3_multipart_complete,
s3_multipart_abort, and s3_multipart_check_upload_exists to use a temporary SDS
(e.g., tmp) for flb_sds_printf or ensure you call flb_sds_destroy(query_params)
before returning on flb_sds_printf failure; specifically, after creating
query_params with flb_sds_create_size(), either build the formatted string into
tmp via flb_sds_printf(&tmp, ...) and then assign/destroy appropriately, or on
flb_sds_printf error call flb_sds_destroy(query_params) (and any other allocated
buffers like body) before returning S3_MULTIPART_ERROR_GENERAL so no SDS is
leaked.
- Around line 875-933: s3_multipart_upload_file_part currently subtracts
offset_end - offset_start without validation which can underflow; before
computing part_size validate that offset_end > offset_start (and both offsets
are non-negative if desired), log an error with the offending offsets (use a
cast like (long long)offset_start/(long long)offset_end or a PRI_OFF_T macro for
formatting), return -1 on invalid ranges, and only then compute part_size,
allocate part_buffer and proceed; update error messages in
s3_multipart_upload_file_part accordingly so bad inputs are rejected early.
In @tests/runtime/out_s3_config.c:
- Around line 255-309: The test flb_test_endpoint_http_vs_https sets
FLB_S3_PLUGIN_UNDER_TEST before the first scenario but unsets it mid-test,
causing the second scenario to run without test mode; fix by setting
FLB_S3_PLUGIN_UNDER_TEST once at the start of flb_test_endpoint_http_vs_https
(using setenv) and remove the intermediate unsetenv call so you only call
unsetenv once at the very end after both scenarios complete (or alternatively
re-call setenv before the second scenario and only unset once at the end); apply
the same change pattern to the other multi-scenario tests that use
setenv/unsetenv around S3 plugin tests.
In @tests/runtime/out_s3_queue.c:
- Around line 182-187: The cleanup uses rmdir(store_dir) which only removes
empty directories so nested files under store_dir/bucket persist; replace this
with a recursive removal call — e.g., invoke the existing
rm_rf_best_effort(store_dir) helper used in tests/internal/storage_dlq.c (or
implement an equivalent recursive deletion function) and call that instead of
rmdir/_rmdir, ensuring store_dir and its nested contents are removed on both
Windows and non-Windows builds.
🧹 Nitpick comments (20)
examples/out_s3/log_sink_s3.conf (2)
23-23: Consider reducing the rate for example configuration.A rate of 50 messages/second across three inputs generates 150 msgs/sec total, which is quite aggressive for a test/example configuration. Users running this example may accumulate significant data volume quickly (potentially gigabytes per hour depending on buffer settings).
Consider reducing to
Rate 10or adding a comment warning users about the data generation volume.💡 Suggested adjustment
- Rate 50 + Rate 10Apply to lines 23, 30, and 37, or add a comment:
+# Note: Rate 50 generates significant data volume. Reduce for lighter testing. Rate 50Also applies to: 30-30, 37-37
141-141: Schema definition is duplicated across Parquet outputs.The
schema_strdefinition is identical across all three Parquet outputs (lines 141, 167, 193). While this is functionally correct, consider noting in comments that the schema must match across all outputs writing to the same data source, or adding a reference comment like "# Schema definition (must match input data structure)".This is informational only; duplication in configuration files is acceptable for clarity.
Also applies to: 167-167, 193-193
tests/runtime/out_s3_error_handling.c (1)
46-83: Tests verify graceful error handling but lack outcome assertions.The test structure is consistent with existing S3 runtime tests. However, these tests primarily verify that the plugin doesn't crash when encountering errors, without asserting on specific outcomes (e.g., verifying logs, retry counts, or proper cleanup).
Consider adding assertions or callback mechanisms to verify expected behavior, such as:
- Confirming error messages are logged
- Verifying retry attempts match
retry_limit- Checking that resources are properly cleaned up
This would strengthen the tests from "smoke tests" to "behavioral tests."
tests/runtime/out_s3_edge_cases.c (1)
70-109: Clarify the test intent for chunk size configuration.Line 93-94 sets
total_file_size=100Kandupload_chunk_size=5M. Sinceupload_chunk_sizeis larger thantotal_file_size, this configuration means files will be uploaded before reaching the chunk size threshold.If the intent is to test "large file chunking" as the function name suggests, consider:
- Swapping the values (larger total_file_size, smaller chunk size)
- Or adding a comment explaining this tests the edge case where chunk size exceeds file size
tests/runtime/out_s3_format.c (2)
27-62: Consider documenting test mode limitations.These runtime tests use
FLB_S3_PLUGIN_UNDER_TESTto mock S3 operations, which means they validate code paths but not actual S3 upload behavior or output correctness. Consider adding a comment explaining this limitation at the top of the file.Additionally, the
sleep(2)call makes tests slower. While acceptable for runtime tests, consider if the sleep duration could be reduced or if there's a more deterministic way to wait for processing completion.
404-530: Consider reducing redundant environment variable manipulation.The test sets and unsets
FLB_S3_PLUGIN_UNDER_TESTmultiple times within the same function (lines 412, 440, 468, 500, 529). Since all test iterations are within the same function, you could set it once at the start and unset once at the end, or extract each combination into a separate test function for better isolation and clarity.plugins/out_s3/s3_stream.h (1)
63-87: Clarify the out_buf parameter documentation.Line 75 states "Output buffer (FILE: marker for temp file)" which is somewhat unclear. Based on the implementation,
out_bufreceives a marker string pointing to a temporary file path. Consider rewording to: "Output buffer (receives FILE: marker for temporary file path)" for clarity.tests/runtime/out_s3_recovery.c (1)
42-106: Consider more robust unique path generation.Line 47 uses
time(NULL)for generating unique database paths, which could cause collisions if tests run within the same second. Consider using a counter or PID in addition to the timestamp for better uniqueness, e.g.,time(NULL) * 1000 + getpid() % 1000.Additionally, line 105 calls
unlink(db_path)without checking the return value. If cleanup fails, it could leave artifacts that affect subsequent test runs. Consider at least logging failures during cleanup.plugins/out_s3/s3_auth.c (3)
154-161: Consider validating mutually exclusive authentication methods.The code uses basic auth if username/password are set, otherwise bearer auth. Consider adding validation during configuration parsing to ensure only one authentication method is configured, preventing user confusion.
166-253: LGTM with a minor suggestion for const correctness.The function correctly handles HTTP client creation, request execution, response validation, and resource cleanup. The dual-path SDS allocation (copy vs. create_len) properly handles both new and existing result_url buffers.
Consider making the
urlparameterconst char *for const correctness, as the function does not modify it.
281-285: Add flb_errno() for consistency with error reporting.For consistency with other allocation failures in this file, consider adding
flb_errno()after the NULL check at line 283.Suggested addition
path = flb_sds_create_size(512); if (!path) { + flb_plg_error(ctx->ins, "Failed to allocate memory for presigned URL path"); flb_errno(); goto error; }include/fluent-bit/flb_blob_db.h (1)
489-495: Verify parallel array pattern for part data retrieval.The function returns four separate parallel arrays (part_db_ids, part_nums, offset_starts, offset_ends) along with a count. While this is functional, parallel arrays can be error-prone if indices get out of sync. Consider whether a struct-based approach would be more robust:
struct blob_part_info { uint64_t db_id; uint64_t part_num; off_t offset_start; off_t offset_end; }; int flb_blob_db_file_fetch_all_parts(struct flb_blob_db *context, uint64_t file_id, struct blob_part_info **parts, int *count);However, if the parallel array pattern is established in the codebase or required for FFI/language binding reasons, the current approach is acceptable.
tests/runtime/out_s3_special_features.c (2)
56-57: Consider using explicit synchronization instead of fixed sleep.The tests use
sleep(2)to allow processing time. Fixed sleep durations can make tests flaky on slow CI systems or unnecessarily slow on fast systems. Consider whether the test framework provides synchronization primitives (e.g., waiting for specific output conditions) that would be more reliable.
216-217: Consider checking unlink() return value.While cleanup failures are often not critical in tests, logging a warning if
unlink(db_path)fails could help diagnose issues on shared test systems.Optional enhancement
- /* Cleanup database file */ - unlink(db_path); + /* Cleanup database file */ + if (unlink(db_path) != 0 && errno != ENOENT) { + flb_warn("Failed to cleanup test database: %s", db_path); + }src/aws/flb_aws_util.c (1)
1133-1138: Consider edge case handling for paths ending with '/'.The file name extraction logic uses
strrchrto find the last slash. Ifvalid_file_pathends with '/' (e.g.,"/var/log/"),file_namewould point to an empty string. Consider whether this edge case needs explicit handling or if it's acceptable behavior.Potential edge case handling
/* Extract just the filename from the path */ file_name = strrchr(valid_file_path, '/'); if (file_name) { file_name++; /* skip the '/' */ + /* Handle trailing slash case */ + if (*file_name == '\0') { + file_name = valid_file_path; /* fallback to full path */ + } } else { file_name = valid_file_path; /* no slash, entire string is filename */ }plugins/out_s3/s3_store.c (1)
44-74: Filename generation uses djb2 hash with timestamp.The
generate_filename_hashfunction combines a djb2 hash of the tag with a timestamp-based hash to create unique filenames. While hash collisions are theoretically possible, the timestamp component (line 58) makes collisions extremely unlikely in practice.Consider documenting the hash algorithm choice and collision handling strategy if uniqueness is critical for correctness.
plugins/out_s3/s3_blob.c (1)
196-304: Robust aborted file recovery with upload ID validation.The
handle_aborted_filesfunction implements sophisticated recovery logic:Strengths:
- Validates existing upload_id before deciding resume vs fresh start (lines 228-262)
- Distinguishes three scenarios: valid upload (resume), invalid upload (fresh), validation error (fresh with warning)
- Enforces retry limits and properly aborts/deletes files exceeding limits (lines 273-283)
- Provides detailed statistics on recovery decisions (lines 296-301)
Potential concern:
Line 232:s3_multipart_check_upload_existsis called for each aborted file. If many files are aborted (e.g., after a long outage), this could result in many S3 API calls during startup.Consider rate-limiting or batching validation calls if recovery of many files is common.
plugins/out_s3/s3_queue.c (1)
345-434: Add explicit bounds check to prevent integer overflow when casting part_id.While the part_size calculation in
s3_multipart.cimplicitly limits the number of parts to S3_AWS_MAX_PARTS (10,000), there is no explicit validation at line 366 whereentry->part_id(uint64_t) is cast to int. Add a bounds check before the cast:if (entry->part_id >= S3_AWS_MAX_PARTS) { flb_plg_error(ctx->ins, "Part ID exceeds AWS limit: %" PRIu64, entry->part_id); return -1; } m_upload.part_number = (int)entry->part_id + 1;This provides defense-in-depth and prevents unsafe behavior if the implicit constraint is ever bypassed.
tests/runtime/out_s3_config.c (1)
26-252: Tests behave as smoke tests only (no validation of actual S3 behavior)Most of these tests only assert that
flb_startsucceeds and the engine doesn’t crash; they don’t validate that keys, endpoints, ACLs, content types, chunk sizing, or invalid configs actually behave as intended.If you want these to catch regressions in the new S3 behavior (especially around key formats, multipart sizing, and Parquet schema requirements), consider adding stronger assertions:
- Inspect plugin logs or in‑memory state via existing test hooks.
- For negative tests, assert that the specific bad parameter is rejected (not just
flb_start == -1).This can be done incrementally; current tests are acceptable as smoke tests.
Also applies to: 368-459, 592-809
tests/runtime/out_s3_multipart.c (1)
430-493: Consider cleaning up the temporary blob DB file inflb_test_upload_id_persistence
flb_test_upload_id_persistencewrites/tmp/test_s3_multipart_persistence.dbbut never removes it. Over multiple runs this can clutter/tmpand mask issues if the DB contents accidentally influence subsequent runs.You could
unlinkthe DB after the second session completes (ignoringENOENT) to keep the test self-contained.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (41)
examples/out_s3/blob_sink_s3.confexamples/out_s3/kafka_sink_s3.confexamples/out_s3/log_sink_s3.confinclude/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/fuzzers/aws_util_fuzzer.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3.ctests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
🚧 Files skipped from review as they are similar to previous changes (10)
- src/flb_signv4.c
- tests/runtime/CMakeLists.txt
- plugins/out_s3/s3_blob.h
- tests/internal/CMakeLists.txt
- plugins/in_blob/blob_file.c
- examples/out_s3/blob_sink_s3.conf
- tests/internal/fuzzers/aws_util_fuzzer.c
- plugins/out_s3/s3_queue.h
- plugins/out_s3/s3_auth.h
- plugins/out_s3/s3_store.h
🧰 Additional context used
🧠 Learnings (9)
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_parquet.hinclude/fluent-bit/flb_aws_util.h
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
src/aws/flb_aws_util.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_s3/s3_store.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
📚 Learning: 2025-09-04T12:35:36.904Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:3275-3282
Timestamp: 2025-09-04T12:35:36.904Z
Learning: The out_s3 plugin intentionally uses a simple numeric comparison for retry_limit (chunk->failures >= ctx->ins->retry_limit) rather than the standard Fluent Bit pattern that checks for FLB_OUT_RETRY_UNLIMITED (-1). The maintainer wants to keep this current behavior for consistency within the plugin.
Applied to files:
plugins/out_s3/s3_multipart.cplugins/out_s3/s3_stream.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
🧬 Code graph analysis (18)
tests/runtime/out_s3_error_handling.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
include/fluent-bit/flb_parquet.h (1)
src/flb_parquet_impl.cpp (8)
flb_parquet_validate_schema(705-721)flb_parquet_validate_schema(705-707)flb_parquet_schema_create(723-748)flb_parquet_schema_create(723-725)flb_parquet_schema_destroy(750-756)flb_parquet_schema_destroy(750-750)flb_msgpack_to_parquet_streaming(758-1074)flb_msgpack_to_parquet_streaming(758-763)
src/aws/flb_aws_util.c (1)
src/flb_sds.c (3)
flb_sds_create_len(58-76)flb_sds_destroy(389-399)flb_sds_cat(120-141)
tests/runtime/out_s3_recovery.c (1)
plugins/out_s3/s3.c (1)
setenv(66-69)
tests/internal/aws_util.c (3)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)src/flb_sds.c (1)
flb_sds_destroy(389-399)tests/internal/fuzzers/aws_util_fuzzer.c (1)
initialization_crutch(27-36)
plugins/out_s3/s3_store.c (4)
src/flb_sds.c (4)
flb_sds_create_size(92-95)flb_sds_printf(336-387)flb_sds_destroy(389-399)flb_sds_create(78-90)src/flb_fstore.c (6)
flb_fstore_file_delete(259-277)flb_fstore_file_create(153-190)flb_fstore_create(479-525)flb_fstore_destroy(527-564)flb_fstore_stream_create(335-402)flb_fstore_file_inactive(238-256)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)
plugins/out_s3/s3_stream.h (1)
plugins/out_s3/s3_stream.c (4)
stream_compress_file(32-147)stream_process_msgpack_file(344-412)stream_json_processor(414-442)stream_log_key_processor(517-546)
tests/runtime/out_s3_special_features.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_multipart.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
tests/internal/parquet.c (3)
src/flb_parquet_impl.cpp (1)
field_name(394-394)src/flb_pack.c (1)
flb_pack_json(530-535)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3_blob.c (4)
plugins/out_s3/s3_multipart.c (2)
s3_multipart_upload_destroy(389-414)s3_multipart_abort(720-785)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(346-387)src/flb_blob_db.c (1)
flb_blob_file_update_remote_id(541-572)plugins/out_s3/s3_queue.c (1)
s3_queue_add_pending_file(185-232)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (5)
cb_s3_upload(1413-1528)s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)mock_s3_call(105-179)
plugins/out_s3/s3_multipart.c (5)
include/fluent-bit/flb_compat.h (1)
usleep(128-132)plugins/out_s3/s3.c (2)
write_seq_index(226-246)s3_plugin_under_test(73-79)src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)src/flb_http_client.c (1)
flb_http_client_destroy(1738-1745)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(346-387)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3_stream.c (5)
src/aws/flb_aws_compress.c (1)
flb_aws_compression_compress(103-120)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_printf(336-387)plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(536-572)tests/internal/processor.c (1)
processor(66-126)
include/fluent-bit/flb_blob_db.h (1)
src/flb_blob_db.c (11)
flb_blob_db_file_fetch_part_ids(1385-1459)flb_blob_db_file_fetch_part_ids(1960-1967)flb_blob_db_file_fetch_part_count(1461-1490)flb_blob_db_file_fetch_part_count(1969-1973)flb_blob_db_file_fetch_all_parts(1492-1624)flb_blob_db_file_get_next_pending(1630-1741)flb_blob_db_file_get_next_pending(1975-1984)flb_blob_db_file_part_check_uploaded(1747-1782)flb_blob_db_file_part_check_uploaded(1986-1991)flb_blob_db_reset_zombie_parts(1785-1809)flb_blob_db_reset_zombie_parts(2000-2003)
tests/runtime/out_s3_format.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (8)
flb_create(143-225)flb_input(266-276)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
🔇 Additional comments (60)
examples/out_s3/log_sink_s3.conf (2)
5-13: SERVICE configuration looks good for testing.The configuration is appropriate for a comprehensive S3 output testing scenario.
137-137: All three compression options (none,gzip,zstd) for Parquet format are properly supported in the S3 plugin implementation. Verification confirms:
- Implementation support: Compression codecs are defined and mapped to parquet compression types in
src/flb_parquet_impl.cppand tested intests/internal/parquet.c- ZSTD availability: ZSTD is a core dependency in Fluent Bit (always enabled, no build conditionals required)
- Test coverage: All compression types including none, gzip, snappy, and zstd are validated in the test suite
The configuration values at lines 137, 165, and 191 are correct and fully supported.
plugins/out_kinesis_firehose/firehose.c (1)
496-501: LGTM - Documentation update aligns with available compression types.The simplified help text correctly lists the compression formats supported by Kinesis Firehose (
gzip,snappy,zstd). Based on learnings, ZSTD is always available without conditional compilation, and removing the reference toarrow(which requiresFLB_HAVE_ARROWguards) appropriately reflects that Arrow/Parquet support is specific to the S3 output plugin.examples/out_s3/kafka_sink_s3.conf (2)
1-11: Well-structured example configuration for Kafka to S3 pipeline.The configuration provides a comprehensive example demonstrating multiple Kafka inputs routed to S3 with different configurations per topic. The comments are helpful for users understanding each option.
184-211: Verify Parquet schema field types match actual log data.The Parquet configuration looks good. A few observations:
The
timestampfield is defined asint64(Line 207), which typically represents epoch milliseconds/microseconds. Ensure this matches the filter on Line 109 that renames@timestamptotimestamp- if the source timestamp is ISO 8601 string format, it may need conversion before Parquet encoding.The
schema_stris quite long on a single line. Consider documenting the schema format requirements in the accompanying documentation.tests/runtime/out_s3_error_handling.c (2)
358-420: Good coverage of auto_retry toggle behavior.The
flb_test_retry_layer_auto_retrytest correctly exercises bothauto_retry_requests=trueandauto_retry_requests=falseconfigurations in sequence. This validates that the configuration option is properly parsed and doesn't cause issues regardless of setting.
541-556: Test list is complete and well-organized.All 12 test functions are properly registered in the TEST_LIST with descriptive names matching the test functions.
tests/runtime/out_s3_edge_cases.c (3)
26-31: Good platform-agnostic path handling.The conditional definition of
TEST_STORE_DIRproperly handles Windows vs Unix path conventions, ensuring tests run correctly across platforms.
448-509: Proper cleanup between rapid configuration cycles.The
flb_test_rapid_config_changestest correctly callsunsetenvat Line 482 between the two configuration cycles and again at Line 508 after completion. This prevents environment variable leakage between test iterations.
511-525: Test list is complete.All 11 edge case test functions are properly registered in TEST_LIST.
tests/runtime/out_s3.c (2)
48-50: Good fix - proper environment cleanup.Adding
unsetenv("FLB_S3_PLUGIN_UNDER_TEST")after test completion ensures environment variables don't leak between test runs. This is particularly important when tests are run in sequence within the same process.
85-87: Consistent cleanup applied across all test functions.The
unsetenvcalls are correctly placed afterflb_destroy()in all six test functions, maintaining consistent ordering where test-specific error env vars (likeTEST_PUT_OBJECT_ERROR) are unset before the general test mode flag.Also applies to: 124-126, 161-163, 197-199, 233-235
plugins/out_s3/CMakeLists.txt (1)
1-10: LGTM - Build configuration updated for new S3 subsystems.The CMakeLists correctly adds the new source files supporting the expanded S3 functionality:
s3_stream.c- streaming processings3_multipart.c- multipart upload orchestrations3_blob.c- blob/state recoverys3_auth.c- presigned URL authorizations3_queue.c- upload queue managementinclude/fluent-bit/flb_aws_util.h (1)
200-207: API change is well-implemented with all callers properly updated.The
flb_get_s3_keysignature successfully addsfile_pathas an optional parameter (can be NULL), maintaining backward compatibility. All callers throughout the codebase have been updated:
- Test suite (tests/internal/aws_util.c): 25+ calls properly pass NULL or file paths
- Production code (plugins/out_s3/):
s3.c,s3_multipart.cpass filename/file_path as appropriate- Fuzzer tests properly handle the NULL case
Documentation clearly specifies the supported template variables and when
file_pathis used.tests/runtime/out_s3_format.c (1)
144-208: LGTM - Invalid schema test is well-designed.The test correctly validates that initialization fails with a malformed JSON schema. The pattern of testing both valid and invalid cases in the same function provides good coverage for schema validation.
tests/runtime/out_s3_queue.c (1)
269-308: LGTM - Good concurrent upload test coverage.The test properly validates concurrent upload scenarios with multiple workers (10) and data chunks (20), giving adequate time for processing.
plugins/out_s3/s3_stream.h (1)
28-42: LGTM - Well-designed callback interface.The callback pattern with opaque user data provides good flexibility for different record processors. The context structure appropriately tracks processing metrics.
include/fluent-bit/flb_parquet.h (3)
31-41: LGTM - Good fail-fast validation design.The schema validation API follows best practices by providing early validation with optional error reporting. This helps catch configuration errors at startup rather than at runtime.
43-70: Excellent API design with helpful context.The opaque type pattern is well-implemented, and the documentation explaining the rationale for pre-parsing schemas (avoiding stack overflow in 37KB coroutine stacks) is particularly valuable. This demonstrates thoughtful consideration of Fluent Bit's runtime constraints.
72-92: Clarify the unused total_file_size parameter.The
total_file_sizeparameter is documented as being "for optimization" (line 84), but the implementation insrc/flb_parquet_impl.cpp(line 762) marks it as__attribute__((unused)). This suggests the parameter is either:
- Reserved for future optimization
- No longer needed and should be removed
- An optimization that wasn't implemented
Consider either implementing the optimization, removing the parameter, or updating the documentation to note it's "reserved for future use" to avoid confusion.
tests/runtime/out_s3_recovery.c (2)
26-33: LGTM - Appropriate platform-specific path handling.The conditional compilation for Windows vs Unix paths is correctly implemented and consistent with other test files in the suite.
36-106: LGTM - Well-designed recovery test.The two-phase test pattern effectively validates the recovery mechanism by simulating an incomplete upload followed by successful recovery. The different timeout values (10m vs 1m) appropriately reflect the test scenarios.
plugins/out_s3/s3_auth.c (4)
36-54: LGTM! Proper cleanup with NULL guards.The function correctly checks each member before freeing and follows the pattern of not freeing the container struct itself.
56-76: LGTM! Robust URL parsing with proper validation.The function correctly validates the parsed components and cleans up resources on error.
123-146: LGTM! Robust port parsing with proper validation.The function correctly validates port string conversion and falls back to scheme-based defaults on parsing failures.
346-387: LGTM! Clean composition of presigned URL requests.The function properly handles the no-endpoint case, builds the full URL, delegates to the request function, and cleans up resources.
include/fluent-bit/flb_blob_db.h (4)
126-127: LGTM! Proper SQL statement for bulk part status updates.The statement correctly updates in_progress status for all parts of a given file.
197-217: LGTM! Well-structured SQL queries for part and file management.The queries are properly parameterized and include appropriate filtering and ordering:
- SQL_GET_ALL_PARTS_FOR_FILE: Clean part retrieval with ordering
- SQL_GET_NEXT_PENDING_FILE: Correctly identifies pending files with proper filters
- SQL_GET_PART_UPLOAD_STATUS: Simple and focused status check
367-370: LGTM! Proper struct extensions for new statements.The new prepared statement handles align with the SQL statements defined earlier and follow consistent naming conventions.
140-144: Acknowledge intentional architectural change; flag inconsistency with Azure version and recovery ordering dependency.The removal of
in_progress=0fromSQL_RESET_FILE_PART_UPLOAD_STATESis intentional: in_progress state is now managed separately byflb_blob_db_reset_zombie_parts()during Phase 0 recovery. However, this creates two concerns:
- Inconsistency: The Azure blob plugin (
plugins/out_azure_blob/azure_blob_db.h) still includesin_progress=0in its equivalent statement, suggesting possible oversight.- Recovery ordering dependency:
flb_blob_db_file_reset_part_upload_states()will not reset in_progress flags unlessflb_blob_db_reset_zombie_parts()is called first during Phase 0. Verify this ordering is enforced in all recovery paths.tests/internal/aws_util.c (4)
47-76: LGTM! Comprehensive test coverage including security edge cases.The test macros cover normal usage (FILE_PATH, FILE_NAME, FILE_BOTH) and important edge cases including parent directory traversal (
../../../etc/passwd), which is security-relevant. The expected outputs correctly show path sanitization behavior.
200-397: LGTM! Consistent API migration across all existing tests.All existing tests correctly updated to pass NULL for the new file_path parameter, maintaining backward compatibility and test coverage.
402-533: LGTM! Thorough test coverage for new file path features.The new tests comprehensively cover:
- Basic file path/name substitution
- Edge cases for path sanitization
- Proper resource cleanup
The edge case tests are particularly valuable for ensuring security (parent traversal) and robustness (dotted prefixes, mixed slashes).
550-557: LGTM! All new tests properly registered.tests/runtime/out_s3_special_features.c (2)
265-297: LGTM! Proper negative test for invalid configuration.The test correctly expects failure and defensively handles the unexpected success case.
382-465: LGTM! Important credential expiration test coverage.The tests cover both retry and strict modes for credential expiration, which is critical for production reliability in cloud environments.
src/aws/flb_aws_util.c (5)
45-46: LGTM! Consistent macro naming for new S3 key variables.
184-226: LGTM! Auth retry mechanism improves credential refresh handling.The immediate retry after credential refresh is a good improvement that reduces latency when credentials expire during normal operation. The
auth_retry_doneflag correctly prevents infinite retry loops within a single request.The approach is sound: after detecting an auth error and refreshing credentials, the code performs one immediate retry before returning, which handles the common case of credential expiration without requiring file-level retry mechanisms.
851-1076: LGTM! Well-structured refactoring of template processing.The extraction of common template processing logic into
process_s3_key_templateimproves code organization and maintainability. The function properly handles all standard S3 key variables, validates inputs, checks length limits, and performs comprehensive error cleanup.
1151-1177: LGTM! Well-documented backward compatibility behavior.The code preserves backward compatibility by appending the file path when no file variables are used in the format. The comment clearly explains the design intent, and the implementation correctly ensures proper path separation.
1105-1116: The path sanitization is sufficient for its intended purpose. The code correctly strips leading dots and slashes from file paths before using them in S3 object keys, and existing tests confirm this behavior (e.g.,../../../etc/passwdbecomeslogs/etc/passwd).Embedded
../sequences are not a security concern in this context because S3 object keys are flat string identifiers, not hierarchical filesystem paths—S3 does not interpret../as directory traversal. The code only performs string concatenation to construct S3 keys, not filesystem operations. If embedded traversal sequences exist (e.g.,logs/./file.log), they remain as literal characters in the S3 key, which is harmless and matches expected behavior per test cases.Likely an incorrect or invalid review comment.
plugins/out_s3/s3_multipart.h (1)
1-101: LGTM! Well-structured multipart upload API.The public API surface is clean and well-documented. The functions are logically grouped (utilities, lifecycle management, AWS operations, high-level helpers) with clear comments explaining their purpose.
plugins/out_s3/s3_queue.c (4)
484-628: LGTM! Robust multipart completion with proper error handling.The completion logic correctly:
- Uses the s3_key from the part entry (avoiding regeneration race conditions)
- Fetches and validates all ETags from the database
- Cleans up resources on all error paths
- Sends delivery notifications before cleanup
1070-1131: LGTM! Proper fstore recovery with failure handling.The fstore recovery logic correctly:
- Skips metadata streams and locked files
- Enforces retry limits and marks failed files inactive
- Manages file locks properly (lock before queue, unlock on error)
57-111: The code already implements complete cleanup on allocation failures consistently.All three queue add functions (
s3_queue_add_file,s3_queue_add_part, ands3_queue_add_pending_file) correctly clean up all previously allocated resources when subsequent allocations fail. Each error path maintains a proper cleanup stack where newly allocated resources are freed before returning.
890-907: Prevents infinite recovery loop by immediately marking parts in-progress.Lines 890-907 contain a critical fix that marks all file parts as
in_progressimmediately after fetching. TheSQL_GET_NEXT_PENDING_FILEquery filters withWHERE p.in_progress = 0, so once marked, the same file cannot be returned by subsequent recovery queries, preventing infinite loop scenarios.Ensure recovery scenarios with multiple pending files have adequate test coverage for this mechanism.
tests/internal/parquet.c (4)
196-310: Excellent test infrastructure with context framework.The test context framework (lines 200-310) significantly reduces boilerplate and improves maintainability. The helper functions (
init_test_context,run_conversion,cleanup_test_context,ctx_pack_json,ctx_validate) provide a clean abstraction for test setup and teardown.
1230-1271: Critical test for schema mismatch crash fix.The test
test_edge_schema_more_fieldsexplicitly validates the fix for crashes when schema has more fields than data (lines 1230-1271). This test ensures missing fields receive default values without causing segmentation faults.The validation checks (lines 1260-1268) properly verify both present and missing fields.
1836-1902: Critical boundary testing for Arrow batch limits.The
test_batch_boundariesfunction tests critical record counts around Arrow's 65536 batch boundary (16-bit limit). This ensures the Parquet encoder correctly handles:
- Records within a single batch (65535)
- Exact batch boundary (65536)
- Multiple batches (65537, 131072)
The validation at key boundary points (lines 1883-1896) ensures data integrity across batch transitions.
2042-2117: Comprehensive test coverage with 44 well-organized test cases.The TEST_LIST demonstrates excellent test coverage across 6 categories:
- 9 functional tests for basic operations and type conversions
- 12 data quality tests for schema mismatches and NULL handling
- 9 scale/boundary tests for large datasets and edge values
- 1 optimized compression test covering all 4 compression types
- 10 negative tests for error conditions
- 3 real-world scenario tests for production use cases
This comprehensive coverage provides confidence in the Parquet encoder implementation.
plugins/out_s3/s3_store.c (4)
35-42: CI detection expanded for multiple environments.The
is_running_in_ci()function now detects multiple CI environments (Travis, GitHub Actions, GitLab CI, Jenkins) instead of just Travis. This is used at line 395 to select memory backend in CI to avoid filesystem issues in containerized environments.Based on learnings: Fluent Bit may have CI-specific constraints that justify this environment detection.
109-133: LGTM! Proactive buffer space management with warnings.The
check_buffer_spacefunction provides good user experience by:
- Warning when buffer reaches 95% capacity (lines 127-130)
- Preventing overflow by rejecting writes when full (lines 120-124)
- Returning 0 when limit is disabled (lines 114-116)
This helps users avoid unexpected data loss from buffer exhaustion.
135-208: Excellent error handling with centralized cleanup.The
cleanup_failed_filehelper (lines 135-152) andcreate_new_buffer_file(lines 154-208) demonstrate proper resource management:
- All allocation failures are handled
- Resources are cleaned up in reverse order of allocation
- The fsf is deleted via fstore API on any failure
- Clear error messages identify failure points
This pattern prevents resource leaks even in complex error scenarios.
536-572: New public API for chunk file access with proper validation.
flb_chunk_file_openprovides a safe way to open and validate chunk files:
- Validates magic bytes (0xC1 0x00) to detect corrupted files
- Extracts and skips metadata length from header
- Returns FILE* positioned at data start for streaming reads
This API enables efficient streaming access to chunk data without loading entire files into memory, which is critical for large Parquet conversions.
plugins/out_s3/s3_blob.c (3)
45-79: LGTM! Clean multipart upload abort helper.The
abort_multipart_uploadhelper properly:
- Creates a temporary multipart context with the required fields
- Fetches a presigned URL for the abort operation
- Cleans up all allocated resources regardless of success/failure
- Returns appropriate error codes
345-421: LGTM! Proper chunk persistence with atomic file creation.The
s3_blob_store_chunkfunction correctly:
- Uses
mkstempfor atomic temp file creation (line 371)- Validates write completeness (line 379)
- Tags chunks with source="chunk" in the database (line 389)
- Registers parts and initiates multipart upload
- Cleans up on all error paths (unlinks file, deletes DB entry)
This enables chunk data (log records) to use the same DB-tracked multipart upload mechanism as blob files, providing consistent recovery semantics.
423-516: Excellent architecture: Lightweight flush callback with deferred processing.The comment block (lines 423-434) explains a critical architectural decision: the flush callback runs in a coroutine with limited stack (37KB), so only lightweight operations (parse, persist metadata) are performed here. Heavy operations (CreateMultipartUpload, API calls) are deferred to the timer callback.
The implementation correctly follows this architecture:
- Minimal work per event: extract info, insert DB metadata, enqueue for later
- No synchronous API calls in the flush callback
- Errors skip the current event but continue processing the batch
- Clear logging of processed count
This design prevents stack overflow and ensures the plugin remains responsive.
plugins/out_s3/s3_stream.c (1)
32-412: Streaming pipeline and resource management look solidThe msgpack streaming, temp-file handling, and compression path all correctly clean up FILE*, msgpack unpacker state, and SDS buffers on both success and error. The
keep_temp_fileflag andFILE:<path>marker generation align with hows3_upload_fileconsumes these files.No changes needed here from my side.
src/flb_blob_db.c (1)
215-255: Blob DB multipart helpers and new prepared statements are coherentThe added prepared statements and helper APIs (fetch_all_parts, get_next_pending, part_check_uploaded, reset_zombie_parts, file_parts_in_progress) use the existing locking pattern correctly, clean up bindings/reset statements on all paths, and free allocated arrays/SDS values on error.
The dynamic array growth in
flb_blob_db_file_fetch_all_partsand the row-scanning logic inflb_blob_db_file_get_next_pendinglook safe and consistent with the rest of the module.Also applies to: 364-368, 1000-1039, 1492-1624, 1630-1741, 1747-1809
plugins/out_s3/s3.h (1)
31-47: S3 state struct and multipart-related declarations are consistentThe new S3 limits, multipart error codes,
upload_queuestruct, and the added fields instruct flb_s3(format,schema_str,cached_arrow_schema,upload_queue_lock) all line up with their usage in the updated implementations (s3.c,s3_stream.c,s3_multipart.c, blob DB helpers).Public prototypes for
cb_s3_upload,s3_format_chunk,s3_upload_file,s3_initiate_multipart_upload, andmock_s3_callmatch the call sites in the provided snippets.Also applies to: 48-88, 90-174, 176-197
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
85f1519 to
1c52b03
Compare
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In @plugins/out_s3/s3_auth.c:
- Around line 56-76: parse_url currently returns after flb_utils_url_split
failure without freeing any partially-initialized allocations; modify parse_url
to always call url_parts_destroy(parts) before returning -1 on any error path
(both when flb_utils_url_split returns -1 and when host/uri are missing).
Specifically, after calling flb_utils_url_split in parse_url, if ret == -1 call
url_parts_destroy(parts) then log the error and return -1; keep the existing
url_parts_destroy(parts) call in the host/uri missing branch so both failure
paths clean up the url_parts structure.
- Around line 120-143: get_port_from_url currently accepts "0" as a valid port;
change the validation so that after strtoul conversion you require port_val to
be between 1 and 65535 (inclusive) before returning it. In other words, update
the condition in get_port_from_url to reject port_val == 0 (treat as conversion
failure) so the function falls back to scheme-based defaults; keep the existing
endptr/errno/ERANGE checks but add the lower-bound check (port_val >= 1).
- Around line 252-341: In build_presigned_url_path change the path format
strings that currently use "%s%s" to include a delimiter between bucket and key
(e.g. "%s/%s") for all relevant flb_sds_printf calls
(multipart_creation_presigned_url, multipart_upload_presigned_url,
multipart_complete_presigned_url, multipart_abort_presigned_url); additionally
ensure you handle a possible leading '/' in encoded_key (strip it or normalize)
so you don't produce a double-slash when formatting.
- Around line 78-118: s3_auth_init_endpoint currently creates a TLS context with
verification disabled (flb_tls_create(..., FLB_FALSE, ...)) and always uses
FLB_IO_TCP in flb_upstream_create_url, which can silently mis-handle https
endpoints; change flb_tls_create to enable verification (FLB_TRUE) and determine
the upstream I/O mode from the parsed scheme in parts (use the TLS/HTTPS IO mode
when parts.scheme indicates "https", otherwise use TCP), then call
flb_upstream_create_url with that IO mode and the TLS context only for HTTPS
endpoints so the authorization_endpoint_url is handled with proper verification
and transport.
🧹 Nitpick comments (12)
plugins/out_s3/s3_auth.h (1)
23-47: Tighten the public API types (const char *+s3_presigned_url_type_t).Using
int url_typeandchar *urlmakes it easy to pass invalid values and suggests mutability where none is needed.Proposed header polish
#include "s3.h" +/* Consider including flb_sds.h directly if s3.h doesn’t guarantee it */ +/* #include <fluent-bit/flb_sds.h> */ @@ int s3_auth_request_presigned_url(struct flb_s3 *ctx, flb_sds_t *result_url, - char *url); + const char *url); @@ int s3_auth_fetch_presigned_url(struct flb_s3 *ctx, flb_sds_t *result_url, - int url_type, + s3_presigned_url_type_t url_type, const char *s3_key, const char *upload_id, int part_number);include/fluent-bit/flb_parquet.h (1)
23-93: Reduce unnecessary public includes; clarify compression constants visibility.
#include <msgpack.h>doesn’t seem needed for the current declarations—dropping it will reduce compile-time coupling for downstream users.- Since the API references
FLB_AWS_COMPRESS_*, either include the public header that defines them (if installed), or adjust the doc to not imply the constants are available from this header alone.Possible include trim
#include <fluent-bit/flb_sds.h> -#include <msgpack.h> #include <stdlib.h>plugins/out_s3/s3_auth.c (1)
163-250: Reduce log noise: make “retrieved successfully” debug-level.Presigned URL fetch can happen per part/upload; logging this at
infocan spam logs in normal operation.tests/runtime/out_s3_special_features.c (2)
26-520: Make blob DB path generation robust for parallel/fast test runs.
/tmp/s3_blob_test_%d.dbwithtime(NULL)can collide when tests run in the same second (or in parallel). Prefer PID + timestamp, ormkstemp()-based naming.Example tweak
- snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%d.db", (int)time(NULL)); + snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%d_%d.db", + (int) getpid(), (int) time(NULL));
26-520: Optional: reduce repetition with a shared setup/teardown helper (and guaranteeunsetenvon failures).This file repeats the same boilerplate in every test; a local helper +
goto cleanuppattern would simplify and make teardown (includingunsetenv) more reliable if aTEST_CHECKaborts mid-test.plugins/out_s3/s3.h (1)
78-78: Verify etags array size alignment with S3_AWS_MAX_PARTS.The
etags[10000]array size matchesS3_AWS_MAX_PARTS(10000), which is correct. However, consider using the macro for consistency:flb_sds_t etags[S3_AWS_MAX_PARTS];plugins/out_s3/s3_store.c (1)
44-74: Hash collision risk in filename generation.The
generate_filename_hashfunction combines a djb2 hash with nanosecond timestamp multiplication. Thehash2calculation using multiplication may produce collisions whentv_nsecis 0 (at second boundaries).Consider using addition or XOR instead:
♻️ Suggested improvement
- hash2 = (unsigned long) hash2 * tm.tm.tv_sec * tm.tm.tv_nsec; + hash2 = (unsigned long) (hash2 ^ tm.tm.tv_sec) ^ (tm.tm.tv_nsec + 1);tests/runtime/out_s3_queue.c (1)
178-217: Consider reducing sleep duration for faster test execution.The concurrent uploads test uses
sleep(3)after pushing 20 chunks with 10 workers. Given the mock environment, this could potentially be reduced to speed up the test suite.include/fluent-bit/flb_blob_db.h (1)
203-213: Consider SQL query optimization.The
SQL_GET_NEXT_PENDING_FILEquery uses bothDISTINCTandGROUP BY f.id. TheDISTINCTis redundant sinceGROUP BYalready ensures unique file IDs. Consider removingDISTINCTfor slightly better performance.♻️ Proposed optimization
-#define SQL_GET_NEXT_PENDING_FILE \ - "SELECT DISTINCT f.id, f.path, f.destination, f.remote_id, f.tag, " \ +#define SQL_GET_NEXT_PENDING_FILE \ + "SELECT f.id, f.path, f.destination, f.remote_id, f.tag, " \ " (SELECT COUNT(*) FROM blob_parts p2 WHERE p2.file_id = f.id) as part_count " \ "FROM blob_files f " \ "JOIN blob_parts p ON f.id = p.file_id " \ "WHERE f.aborted = 0 " \ " AND p.uploaded = 0 " \ " AND p.in_progress = 0 " \ "GROUP BY f.id " \ "ORDER BY f.created ASC " \ "LIMIT 1;"plugins/out_s3/s3_queue.c (1)
606-610: Consider optimizing ETag cleanup loop.The loop iterates through all 10000 possible ETag slots, but could be optimized to only iterate through the actual
part_countsince ETags beyond that index won't be allocated.♻️ Proposed optimization
- for (i = 0; i < 10000; i++) { + for (i = 0; i < part_count && i < 10000; i++) { if (m_upload.etags[i]) { flb_sds_destroy(m_upload.etags[i]); } }tests/runtime/out_s3_multipart.c (1)
452-453: Verify database file cleanup between test sessions.The test uses the same database file path
/tmp/test_s3_multipart_persistence.dbacross two sessions. Consider cleaning up this file after the test completes to avoid interference with subsequent test runs.♻️ Proposed fix
Add cleanup after the second session:
flb_stop(ctx); flb_destroy(ctx); unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); + unlink("/tmp/test_s3_multipart_persistence.db"); }Also applies to: 480-481
plugins/out_s3/s3_multipart.c (1)
103-103: Consider moving extern declaration to header.The
write_seq_indexfunction is referenced froms3.c. For better code organization and to avoid redundant extern declarations, consider adding this tos3.hor a shared internal header.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (38)
include/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/fuzzers/aws_util_fuzzer.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3.ctests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
🚧 Files skipped from review as they are similar to previous changes (5)
- plugins/in_blob/blob_file.c
- src/flb_signv4.c
- tests/runtime/out_s3_edge_cases.c
- tests/runtime/out_s3_config.c
- plugins/out_s3/s3_multipart.h
🧰 Additional context used
🧠 Learnings (11)
📓 Common learnings
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
tests/internal/CMakeLists.txt
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_parquet.h
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_s3/s3_store.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-09-04T12:35:36.904Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:3275-3282
Timestamp: 2025-09-04T12:35:36.904Z
Learning: The out_s3 plugin intentionally uses a simple numeric comparison for retry_limit (chunk->failures >= ctx->ins->retry_limit) rather than the standard Fluent Bit pattern that checks for FLB_OUT_RETRY_UNLIMITED (-1). The maintainer wants to keep this current behavior for consistency within the plugin.
Applied to files:
plugins/out_s3/s3_stream.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
src/aws/flb_aws_util.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
plugins/out_kinesis_firehose/firehose.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_kinesis_firehose/firehose.csrc/flb_parquet_impl.cpp
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
🧬 Code graph analysis (15)
plugins/out_s3/s3_stream.h (1)
plugins/out_s3/s3_stream.c (4)
stream_compress_file(32-147)stream_process_msgpack_file(344-412)stream_json_processor(414-442)stream_log_key_processor(517-546)
tests/internal/aws_util.c (2)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)tests/internal/fuzzers/aws_util_fuzzer.c (1)
initialization_crutch(27-36)
tests/internal/parquet.c (3)
src/flb_parquet_impl.cpp (1)
field_name(403-403)src/flb_pack.c (1)
flb_pack_json(530-535)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
plugins/out_s3/s3_blob.h (1)
plugins/out_s3/s3_blob.c (5)
s3_blob_register_parts(306-338)s3_blob_process_events(436-516)s3_blob_store_chunk(345-421)s3_blob_recover_state(128-147)s3_blob_notify_delivery(81-118)
plugins/out_s3/s3_queue.c (5)
plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(343-384)plugins/out_s3/s3_multipart.c (2)
s3_multipart_upload_file_part(884-955)s3_multipart_complete(634-721)src/flb_blob_db.c (1)
flb_blob_db_file_fetch_all_parts(1492-1624)plugins/out_s3/s3.c (1)
s3_format_chunk(1266-1407)lib/cfl/src/cfl_sds.c (2)
cfl_sds_destroy(127-137)cfl_sds_len(88-91)
tests/internal/fuzzers/aws_util_fuzzer.c (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
plugins/out_s3/s3_store.h (1)
plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(536-572)
plugins/out_s3/s3_auth.h (1)
plugins/out_s3/s3_auth.c (3)
s3_auth_init_endpoint(78-118)s3_auth_request_presigned_url(163-250)s3_auth_fetch_presigned_url(343-384)
plugins/out_s3/s3_multipart.c (3)
plugins/out_s3/s3.c (2)
write_seq_index(226-246)s3_plugin_under_test(73-79)src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(343-384)
plugins/out_s3/s3_queue.h (1)
plugins/out_s3/s3_queue.c (9)
s3_queue_remove(267-275)s3_queue_buffer_chunk(315-337)s3_queue_add_file(57-111)s3_queue_add_part(113-183)s3_queue_add_pending_file(185-232)s3_queue_process_entry(680-739)s3_queue_recover_all(782-827)s3_queue_recover_from_database(837-998)s3_queue_recover_from_fstore(1071-1131)
tests/runtime/out_s3_multipart.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (5)
cb_s3_upload(1413-1528)s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)mock_s3_call(105-179)
tests/runtime/out_s3_format.c (1)
src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
🔇 Additional comments (104)
tests/internal/CMakeLists.txt (2)
136-141: LGTM! Follows the established pattern.The Parquet test file registration mirrors the AVRO encoder pattern (lines 129-134) and correctly uses the
FLB_PARQUET_ENCODERguard.
238-240: Include directories are properly configured globally; library validation is in place.The root CMakeLists.txt already handles this at lines 1348–1349: when
FLB_PARQUET_ENCODERis enabled and both libraries are found,include_directories()adds${ARROW_INCLUDE_DIRS}and${PARQUET_INCLUDE_DIRS}globally. Test targets inherit these includes automatically.Library validation is also already in place: lines 1346–1354 of CMakeLists.txt check both
ARROW_FOUNDandPARQUET_FOUND, and if either is false, the code disablesFLB_PARQUET_ENCODERand logs a warning. By the time the test linking at line 239 executes,FLB_PARQUET_ENCODERis guaranteed to be enabled only if both libraries were successfully discovered.No changes needed.
tests/internal/fuzzers/aws_util_fuzzer.c (1)
73-73: LGTM!The fuzzer correctly adapts to the extended
flb_get_s3_keyAPI by passingNULLfor the newfile_pathparameter, which is appropriate for testing basic key generation.include/fluent-bit/flb_aws_util.h (1)
200-207: LGTM!The API extension is well-documented and the function signature correctly reflects the new file path support. The documentation clearly indicates that
file_pathis optional and explains when it's used.tests/internal/aws_util.c (5)
47-77: LGTM!Excellent test coverage for the new file path feature. The test constants cover both normal cases ($FILE_PATH, $FILE_NAME, both) and important edge cases (relative paths, parent directory traversal, hidden files, mixed dots/slashes). This comprehensive testing helps ensure the implementation handles security-sensitive path manipulations correctly.
200-200: LGTM!All existing test calls correctly updated to pass
NULLfor the newfile_pathparameter. This maintains backward compatibility and ensures existing tests continue to validate the core key generation logic.Also applies to: 215-215, 231-231, 247-247, 263-263, 278-278, 301-301, 317-317, 333-333, 349-349, 356-356, 374-374, 380-380, 396-396
402-448: LGTM!The new test functions for
$FILE_PATHand$FILE_NAMEvariables are well-structured and follow established testing patterns. They properly validate the expected behavior and clean up resources.
450-533: LGTM!Excellent edge case coverage for path normalization and security scenarios. The tests validate that the implementation properly sanitizes paths by stripping leading dots and slashes, which is critical for preventing path traversal attacks and ensuring consistent S3 key generation.
550-557: LGTM!All new test functions are properly registered in the
TEST_LISTwith descriptive names following the established naming convention.src/aws/flb_aws_util.c (6)
45-46: LGTM!The new macro definitions follow the established pattern for S3 key template variables and are used consistently throughout the implementation.
184-184: LGTM!The auth retry mechanism improves reliability by immediately retrying requests after credential refresh. The implementation correctly prevents infinite loops with the
auth_retry_doneflag and properly cleans up the old client before retrying. This handles the common case where credentials expire during normal operation (e.g., IAM role rotation in EKS/ECS).Also applies to: 210-226
851-1076: LGTM!The
process_s3_key_templatefunction is well-structured and consolidates common template processing logic. The implementation includes:
- Proper thread-safe string tokenization
- Comprehensive error handling and resource cleanup
- Multiple validation checks with appropriate warnings
- Correct handling of all template variables ($TAG, $TAG[0-9], $UUID, $INDEX, time formatters)
The error cleanup path (lines 1058-1076) is particularly thorough, checking each pointer before freeing and avoiding double-free issues.
1078-1116: LGTM!The function setup and file path validation logic is well-implemented. The path normalization (lines 1105-1109) correctly strips leading dots and slashes to prevent path traversal attacks and ensure consistent S3 key generation. The early return optimizations improve performance for common cases.
1118-1149: LGTM!The token replacement logic for
$FILE_PATHand$FILE_NAMEis correct. The filename extraction usingstrrchrproperly handles both paths with directories and simple filenames. Error handling includes proper cleanup of allocated resources.
1151-1184: LGTM!The fallback path append logic maintains backward compatibility and correctly handles path separators. The implementation ensures proper S3 key structure by:
- Adding separators only when needed (avoiding empty keys and double slashes)
- Preserving directory structure while removing absolute path prefixes
- Deferring URL encoding to HTTP request construction (as noted in the comment)
The final length validation and comprehensive error handling ensure robustness.
plugins/out_kinesis_firehose/firehose.c (1)
496-501: Keep help text in sync withflb_aws_compression_get_type()supported values.The updated string looks aligned with the standardized compression set (
gzip,snappy,zstd), and dropping Arrow/compile-time wording matches the current repo conventions (ZSTD is always enabled; Arrow/Parquet are the ones typically gated). Based on learnings, ZSTD doesn’t need compile-time caveats.tests/runtime/out_s3.c (1)
17-50: Good test isolation: env var teardown added.Unsetting
FLB_S3_PLUGIN_UNDER_TESTat the end of each test reduces inter-test coupling and flakiness when runtime tests are executed in-process.Also applies to: 52-87, 89-127, 129-163, 165-199, 201-235
plugins/out_s3/CMakeLists.txt (1)
1-8: CMake source list update looks consistent with the new S3 subsystems.plugins/out_s3/s3_auth.c (1)
343-384: The type concern in this review is not valid. The struct definition inplugins/out_s3/s3.hline 118 explicitly declaresauthorization_endpoint_urlasflb_sds_t:flb_sds_t authorization_endpoint_url;Since it is guaranteed to be
flb_sds_t, usingflb_sds_len()on it at line 365 is correct and safe. The suggested change tostrlen()is unnecessary.Likely an incorrect or invalid review comment.
tests/runtime/out_s3_special_features.c (1)
26-520: Tests lack assertions for their described behaviors; add hooks to verify state changes.The S3 plugin does auto-adjust
upload_chunk_size(viaflb_s3_calculate_optimal_part_size()) and logs the result, and the code can support credential retry withfile_delivery_attempt_limit > 1. However, the tests do not verify these behaviors:
auto_adjust_chunk_sizeandchunk_size_equals_file_sizeonly check that startup succeeds; they don't verify the adjusted value (e.g., parse logs or inspect a known metric).- Credential-expiration tests set high attempt limits but provide no mechanism to trigger actual credential expiration in test mode, so the retry logic never exercises.
Consider adding a test hook (e.g., an env var that forces credential refresh or returns the computed
upload_chunk_sizeto a queryable location) so these tests can assert the intended post-condition rather than only confirming the plugin starts.plugins/out_s3/s3_store.h (2)
26-34: LGTM - Well-structured s3_file struct with clear field semantics.The
stream_pathfield name appropriately reflects its purpose (storing the stream directory path). The struct layout is clean with good documentation comments.
51-52: Good API design for context-free chunk file access.The
flb_chunk_file_openfunction is correctly declared to enable usage from C++ code without requiring the full S3 context. This aligns well with the Parquet encoder implementation insrc/flb_parquet_impl.cpp.plugins/out_s3/s3.h (4)
36-46: LGTM - AWS S3 multipart constraints are correctly defined.The defined constants match AWS S3 specifications (5 MiB minimum part size, 5 GiB maximum part size, 10,000 maximum parts).
48-70: Well-structured upload queue entry for multipart orchestration.The
upload_queuestruct captures all necessary metadata for multipart upload tracking including file offsets, S3 keys, retry counters, and theneeds_upload_creationflag for deferred upload initialization.
153-153: Thread safety: upload_queue_lock provides necessary synchronization.The
pthread_mutex_t upload_queue_lockcorrectly protects the upload queue from concurrent access. This aligns with the usage pattern shown incb_s3_upload(s3.c) where the lock is acquired before iterating the queue.
171-174: Parquet schema caching support looks good.The
format,schema_str, andcached_arrow_schemafields provide the necessary infrastructure for Parquet support with schema caching for performance.plugins/out_s3/s3_store.c (7)
35-42: CI environment detection for backend selection.The CI detection covers major CI systems (GitHub Actions, GitLab CI, Travis, Jenkins). Using memory backend in CI avoids filesystem-related test flakiness.
109-133: Buffer space validation with early warning.The
check_buffer_spacefunction correctly validates buffer limits and provides a warning at 95% capacity. The threshold-based warning usingBUFFER_WARNING_THRESHOLDis a good practice for operational visibility.
247-275: Potential integer underflow in chunk size calculation.In
calculate_chunk_data_size, ifst.st_sizeis exactly equal toCIO_FILE_HEADER_SIZE + meta_size, the function returns 0. However, the check on line 270 uses<=which is correct. The return typesize_tis unsigned, so the subtraction is safe given the check.
363-386: Cross-platform time formatting with different separators.The Windows path uses hyphen separators (
%H-%M-%S) while POSIX uses colons (%H:%M:%S). This is intentional since Windows doesn't allow colons in filenames, which is correct behavior.
388-432: Store initialization with CI-aware backend selection.The
s3_store_initfunction correctly:
- Detects CI environment and selects appropriate backend
- Creates the fstore with proper cleanup on failure
- Generates timestamp-based stream names
- Restores any buffered files from previous runs
The cleanup paths properly destroy resources on failure.
509-534: Defensive null checks in s3_store_file_delete.Good defensive programming with multiple null checks before operations. The check
fsf->data != s3_fileon line 519 prevents double-free scenarios if the file was already cleaned up.
536-572: Chunk file opener correctly parses CIO header format.The
flb_chunk_file_openfunction properly:
- Validates magic bytes for chunk file identification
- Parses metadata length from header (big-endian)
- Seeks past metadata to position at data start
- Returns positioned FILE* for caller to read data
The error handling closes the file on any failure path.
tests/runtime/CMakeLists.txt (1)
238-245: Comprehensive test coverage expansion for S3 plugin.The S3 test suite is well-organized into focused test modules covering:
- Configuration validation
- Multipart upload mechanics
- Format conversion (JSON/Parquet)
- Error handling scenarios
- Edge cases
- Special features
- Queue operations
- Recovery paths
This modular approach improves test maintainability and debugging.
tests/runtime/out_s3_queue.c (2)
27-62: Basic queue operations test structure is correct.The test properly sets up the mock environment, configures S3 output, and exercises the queue path. The
sleep(2)allows async processing to complete before shutdown.
257-265: TEST_LIST properly registers all queue tests.The test list correctly maps test names to functions following the Fluent Bit testing convention.
tests/runtime/out_s3_format.c (4)
144-208: Parquet schema validation tests cover both valid and invalid cases.The test correctly:
- Tests valid schema configuration and successful start
- Tests invalid JSON schema expecting
flb_startto return -1The guard
#ifdef FLB_HAVE_PARQUET_ENCODERensures these tests only run when Parquet support is built.
329-365: ZSTD compression test is appropriate.Based on learnings, ZSTD is a core dependency in Fluent Bit and doesn't require conditional compilation guards. This test will work on all platforms.
404-530: Format/compression combination test provides matrix coverage.The test exercises multiple combinations:
- JSON + GZIP
- JSON + ZSTD
- Parquet + GZIP (when encoder available)
- log_key + GZIP
Each sub-test properly resets the environment variable at the end.
532-547: TEST_LIST correctly includes conditional Parquet tests.The
#ifdef FLB_HAVE_PARQUET_ENCODERguards around Parquet test entries ensure the test list is valid regardless of build configuration.plugins/out_s3/s3_stream.h (4)
28-42: Well-designed callback architecture for format extensibility.The
record_processor_fntypedef andstream_processor_contextstruct provide a clean abstraction for format-specific processing. This design allows easy addition of new output formats without modifying the core streaming logic.
44-61: Stream compression API with offset support enables efficient multipart uploads.The
stream_compress_filefunction supporting arbitrary offset ranges allows compressing specific file segments for multipart uploads without loading entire files into memory.
63-87: Unified msgpack processor provides consistent streaming framework.The
stream_process_msgpack_filefunction abstracts the common msgpack iteration pattern, allowing format-specific processors to focus solely on record transformation. TheFILE:marker output convention enables the caller to handle temporary files appropriately.
89-109: Built-in processors for JSON and log_key extraction.The
stream_json_processorandstream_log_key_processorprovide ready-to-use implementations for common use cases. These align with the existing S3 output configuration options.plugins/out_s3/s3_blob.h (1)
1-52: LGTM - Well-structured header file with clear API surface.The header file is well-organized with proper include guards, standard copyright notice, and clear function declarations. The comments provide sufficient context for each function's purpose.
include/fluent-bit/flb_blob_db.h (6)
126-128: LGTM - Useful bulk update for file parts.The new SQL macro enables efficient bulk updates of
in_progressstatus for all parts of a file, which is particularly useful during recovery scenarios.
140-144: LGTM - Improved separation of state reset operations.The removal of
in_progressreset from this macro is appropriate, as it's now handled independently viaSQL_UPDATE_FILE_PARTS_IN_PROGRESS. This provides better granularity for recovery operations.
314-324: LGTM - Consistent error code additions.The new error codes follow the established pattern and are properly integrated into the error code hierarchy, with the
TOPmarker correctly updated.
367-370: LGTM - Statement pointers aligned with new SQL queries.The new prepared statement pointers correctly correspond to the newly defined SQL macros, maintaining consistency with the existing pattern.
477-478: LGTM - Useful addition of file creation timestamp.Adding
time_t *file_createdas an output parameter provides valuable metadata for file age tracking and recovery decisions.
489-514: LGTM - Comprehensive API extensions for blob lifecycle management.The new function declarations provide essential capabilities:
- Bulk part querying (
fetch_all_parts)- Pending file retrieval (
get_next_pending)- Upload status verification (
part_check_uploaded)- Bulk in-progress state management (
file_parts_in_progress)- Recovery support (
reset_zombie_parts)These additions enable richer blob-part lifecycle management and robust recovery mechanisms.
plugins/out_s3/s3_queue.h (2)
32-43: Excellent inline documentation for legacy compatibility.The comment clearly explains the purpose and usage patterns for the legacy file-level queue interface, helping developers understand when to use
file_id == 0vsfile_id > 0.
71-79: Outstanding documentation of recovery architecture.The three-phase recovery architecture is clearly documented, making it easy for developers to understand the recovery process flow:
- Cleanup dirty states
- State transitions
- Queue rebuild
This level of documentation significantly improves maintainability.
tests/runtime/out_s3_error_handling.c (2)
46-83: LGTM - Well-structured test with proper cleanup.The test follows a clear pattern with proper environment setup, resource management, and cleanup. The use of environment variables to inject errors is a good testing strategy.
542-556: LGTM - Comprehensive test registry.The TEST_LIST properly registers all 12 error-handling test scenarios with descriptive names, covering AWS errors, network issues, retry mechanisms, and failure cleanup.
tests/runtime/out_s3_recovery.c (4)
27-33: LGTM - Appropriate platform-specific path handling.The conditional compilation correctly handles Windows vs Unix/Linux temporary directory paths. This ensures tests work across platforms.
36-106: LGTM - Well-designed restart recovery test.The test properly simulates a restart scenario by:
- First run: buffering data without completing upload
- Second run: recovering and processing buffered data
- Cleaning up the database file
This is an effective way to test recovery mechanisms.
389-433: Good stress test for concurrent database access.The test effectively exercises concurrent database access by:
- Configuring 5 workers
- Pushing data 10 times in a loop
- Using a small
total_file_size(100K) to trigger more frequent operationsThis helps identify potential race conditions in the blob database implementation.
436-446: LGTM - Comprehensive recovery test coverage.The TEST_LIST registers 8 recovery scenarios covering:
- Blob database vs fstore-only recovery
- Multipart upload resume
- Part-level tracking
- Failed upload cleanup
- Stale upload handling
- Basic DB operations
- Concurrent access
This provides thorough test coverage for recovery mechanisms.
plugins/out_s3/s3_queue.c (1)
891-907: Excellent defensive programming for infinite loop prevention.The immediate marking of parts as
in_progressprevents the same file from being repeatedly processed in the recovery loop. The clear comment explains the critical nature of this fix.tests/internal/parquet.c (2)
200-310: Well-designed test context framework.The test context structure and helper functions effectively reduce boilerplate across the extensive test suite while ensuring consistent resource management and cleanup.
213-248: Proper schema lifecycle management in wrapper.The wrapper function correctly manages the cached schema lifecycle by ensuring cleanup in all paths, preventing memory leaks across repeated test invocations.
plugins/out_s3/s3_blob.c (1)
371-375: Secure temporary file creation using mkstemp.The use of
mkstempfor creating temporary files is the correct approach, preventing race conditions and improving security compared to alternatives liketmpnam.tests/runtime/out_s3_multipart.c (1)
27-63: Consistent test structure across the suite.The test follows a clear and consistent pattern that's maintained across all 12 tests in this file, making the suite easy to understand and maintain.
plugins/out_s3/s3_stream.c (2)
165-192: Well-designed conditional cleanup with keep_temp_file flag.The
keep_temp_fileflag allows precise control over temporary file cleanup, ensuring files are preserved when needed for downstream processing while being cleaned up on error paths.
344-412: Robust orchestration with comprehensive error handling.The function properly manages resources across all paths:
- Cleanup with file deletion on error
- Preservation of output file on success
- Empty output detection
- Proper marker generation
plugins/out_s3/s3_multipart.c (20)
29-31: LGTM! Required includes for MD5/base64 support.The new includes support Content-MD5 header generation for multipart upload integrity verification.
71-101: LGTM! Well-defined constants and header templates.The buffer sizes are appropriate (MD5 base64 requires 24 characters + null terminator), and the static header structures provide reusable templates for S3 requests.
105-141: LGTM! ETag extraction with proper bounds checking.The function correctly parses the ETag header from HTTP responses with appropriate bounds checks (
i < size). The use ofisspace()is acceptable for HTTP header parsing.
143-195: LGTM! Well-designed part size calculation.The function intelligently calculates optimal part sizes while respecting AWS S3 constraints (max 10,000 parts, min/max part sizes). The rounding to MiB/GiB boundaries is a good optimization for S3 performance.
220-269: LGTM! Proper URI encoding and error handling.The function correctly handles presigned URLs and constructs standard S3 URIs with proper URL encoding of the S3 key. Error paths properly clean up allocated resources.
271-325: LGTM! Robust XML payload construction with integrity checks.The function includes an excellent integrity check (lines 280-288) to ensure all parts have valid ETags before constructing the CompleteMultipartUpload XML payload. This prevents subtle bugs where incomplete uploads would fail at completion time.
327-387: LGTM! Header construction with proper memory management.The function correctly builds S3 request headers. Note that Content-MD5 is excluded for multipart operations (line 341), which aligns with S3 API semantics where MD5 is applied per-part during upload, not during initiation or completion.
389-414: LGTM! Proper resource cleanup.The cleanup function correctly handles NULL pointers and frees all allocated resources in the proper order (contents before container).
420-469: LGTM! Proper initialization with seq_index handling.The function correctly handles sequential index increment and persistence, with proper rollback on failure (line 441). The S3 key generation integrates well with the existing key format system.
471-533: LGTM! CreateMultipartUpload API call with proper error handling.The function correctly initiates multipart uploads and extracts the UploadId from the S3 response. The mock support for testing is a good practice.
535-632: LGTM! UploadPart implementation with Content-MD5 support.The function correctly uploads parts with optional Content-MD5 integrity checking. The ETag extraction and storage (line 618) is critical for the final CompleteMultipartUpload call.
634-721: LGTM! CompleteMultipartUpload with NoSuchUpload handling.The function correctly completes multipart uploads. The special handling of
NoSuchUploaderror (lines 705-711) with a distinct error codeS3_MULTIPART_ERROR_NO_SUCH_UPLOADis essential for recovery scenarios where an upload may have already been completed or aborted.
723-791: LGTM! AbortMultipartUpload implementation.The function correctly aborts multipart uploads using the DELETE method and expects a 204 status code, which aligns with S3 API specifications.
801-882: LGTM! Upload ID validation for recovery.The function efficiently checks upload ID validity using
ListPartswithmax-parts=1to minimize response size. The conservative error handling (lines 874-876) ensures that uncertain states don't cause data corruption during recovery.
884-955: LGTM! File part upload with robust validation.The function includes excellent offset validation (lines 898-908) to prevent underflow and invalid ranges. The read verification (line 938) ensures complete data is uploaded, avoiding partial part uploads.
957-993: LGTM! Helper for multipart upload initialization.This static helper provides a simpler initialization path when the S3 key is already known, avoiding the seq_index logic needed in
s3_multipart_upload_create.
996-1104: LGTM! Robust part upload with retry and backoff.The function implements comprehensive retry logic with exponential backoff (capped at 30s). The overflow protection in backoff calculation (lines 1070-1086) is essential to prevent undefined behavior when
part_attemptis large. Fetching a fresh presigned URL on each retry (line 1025) is correct, as URLs may expire.
1106-1198: LGTM! Complete file upload orchestration.The function correctly orchestrates the entire multipart upload lifecycle with proper error handling and cleanup. The size verification (lines 1161-1167) ensures data integrity, and aborts are called appropriately on failures to clean up incomplete uploads on S3.
36-64: No macro naming conflicts detected.The
flb_open,flb_close,flb_read,flb_lseek,flb_stat, andflb_fstatmacros are defined only in this file and don't conflict with existing definitions elsewhere. The cross-platform I/O wrapper pattern is sound.
197-218: No action required.The
flb_hash_simpleandflb_base64_encodefunctions are core, unconditionally-compiled dependencies in Fluent Bit (defined insrc/flb_hash.candsrc/flb_base64.crespectively). They are always available and used without build guards throughout the codebase, including in multiple plugins and core modules. The code is correct as-is.src/flb_parquet_impl.cpp (8)
20-45: LGTM! Proper C++ and C interop setup.The includes are well-organized with appropriate use of
extern "C"for C library integration. The Arrow, Parquet, MsgPack, and yyjson dependencies are correctly included.
47-53: LGTM! Well-chosen constants for Parquet processing.The batch size (64K) aligns with Arrow standards, and the row group size (512K) is appropriate for S3 storage efficiency. Use of
constexprand anonymous namespace is good modern C++ practice.
67-114: LGTM! Robust string parsing with bounds checking.The template function
parse_string_to_numberincludes explicit bounds checking (lines 79-84) before casting to prevent silent overflow. The use ofstd::optionalfor error handling is excellent modern C++ practice.
116-348: LGTM! Comprehensive type conversion with clamping.The
MsgpackToArrowConverterclass handles type conversions robustly with overflow clamping (e.g., lines 183-189, 192-199) and appropriate logging of clamping events. This prevents data loss while maintaining valid Arrow types.
350-463: LGTM! Flexible JSON schema parsing.The schema parser correctly handles various data types including timestamp with units (lines 426-447). Defaulting unknown types to
utf8(line 450) is a reasonable fallback that prevents data loss.
465-488: LGTM! Default value handling for non-nullable fields.The function provides sensible defaults for all supported types, ensuring non-nullable fields can always be populated when data is missing from the MsgPack input.
497-712: LGTM! Comprehensive Parquet validation utility.The validation function provides thorough checking of Parquet files including metadata, schema, types, and data values. The exception handling (lines 700-711) ensures proper error reporting for Parquet operations.
767-1083: LGTM! Complex streaming implementation with proper resource management.The function implements a robust streaming MsgPack-to-Parquet pipeline with:
- Batched processing (65K records per batch) for memory efficiency
- Proper null/missing value handling (lines 899-916)
- Careful resource cleanup using flags (lines 995-1005) to prevent double-free
- Exception-safe error paths (lines 1065-1080)
The complexity is justified by the need for streaming large datasets efficiently.
src/flb_blob_db.c (10)
215-254: LGTM! New prepared statements for recovery support.The new statements follow the existing pattern with appropriate error codes. These enable the recovery functionality by allowing queries for pending files, part status, and batch updates.
364-367: LGTM! Proper statement cleanup.The new statements are correctly finalized in
flb_blob_db_close, preventing resource leaks.
1000-1039: LGTM! Batch update for file parts.The function efficiently updates all parts of a file in a single statement, which is important for recovery performance when marking entire files as in-progress or resetting them.
1270-1316: LGTM! Addition of file_created timestamp.The signature extension properly extracts the file creation time, which is useful for age-based policies or debugging. The initialization and error-path handling are correct.
Also applies to: 1377-1377
1387-1400: LGTM! Type clarification for remote_id_list.Changing from
char**toflb_sds_t*improves type clarity since the function creates SDS strings. Thememsetsize adjustment (lines 1400, 1447) maintains consistency even thoughflb_sds_tis typedef'd tochar*.Also applies to: 1447-1447
1492-1624: Verify realloc failure handling to prevent memory leaks.The function's dynamic array expansion logic (lines 1540-1579) is complex. In the realloc failure path (lines 1550-1561), ensure that checking
new_db_ids != tmp_db_idscorrectly identifies newly allocated memory vs. the original pointer. Sincerealloccan return NULL on failure (leaving the original pointer valid), the current logic appears correct, but review carefully to prevent leaks.The function provides essential recovery functionality by fetching all parts of a file with their offsets. The caller is properly responsible for freeing the returned arrays.
1747-1782: LGTM! Upload status check.The function correctly queries part upload status. Treating
SQLITE_DONEas an error (lines 1768-1770) is appropriate since the part should exist in the database.
1785-1809: LGTM! Zombie part recovery.The function correctly resets parts that were marked in-progress by a crashed process. Using
sqlite3_execdirectly (line 1799) is appropriate for this one-time recovery operation rather than maintaining another prepared statement.
1926-1927: LGTM! NO-BACKEND stubs updated.The stubs are correctly updated for new and modified function signatures, maintaining consistency with the existing pattern of returning
FLB_BLOB_DB_ERROR_NO_BACKEND_AVAILABLE.Also applies to: 1954-1956, 1962-1964, 1975-2004
1630-1741: No issues identified. The iterator pattern is intentional, properly documented in the function header and inline comments (lines 1627-1629, 1697, 1699), and safely implemented. The only call site in plugins/out_s3/s3_queue.c correctly iterates within awhile(1)loop that breaks onret == 0(SQLITE_DONE) orret < 0(error), ensuring the statement is always reset when appropriate.
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
1c52b03 to
5f6750c
Compare
There was a problem hiding this comment.
Actionable comments posted: 13
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
src/flb_blob_db.c (1)
194-203: Wrong error code returned forSQL_UPDATE_FILE_PART_DELIVERY_ATTEMPT_COUNTprepare failure.
Line 201 returnsFLB_BLOB_DB_ERROR_PREPARING_STATEMENT_UPDATE_FILE_DELIVERY_ATTEMPT_COUNT, but this is preparing the file part statement. This will misreport failures and complicate debugging.Proposed fix
- return FLB_BLOB_DB_ERROR_PREPARING_STATEMENT_UPDATE_FILE_DELIVERY_ATTEMPT_COUNT; + return FLB_BLOB_DB_ERROR_PREPARING_STATEMENT_UPDATE_FILE_PART_DELIVERY_ATTEMPT_COUNT;include/fluent-bit/flb_blob_db.h (1)
126-145: Clarify “part_id” vs “row id” semantics in the API/SQL.
SQL_GET_PART_UPLOAD_STATUSqueries byWHERE id = @id(row PK), but the C API parameter is namedpart_id. That’s easy to misuse (part number vs DB id). Consider renaming the API parameter topart_db_id(and/or adjust SQL to query bypart_idif that’s what callers intend).Also applies to: 215-217
plugins/out_s3/s3.h (1)
28-174: Use Fluent Bit's pthread abstraction layer instead of directly including<pthread.h>.Replace
#include <pthread.h>with#include <fluent-bit/flb_pthread.h>to follow Fluent Bit's established pattern for cross-platform pthread support (as seen inout_stackdriverand other core modules). Fluent Bit provides Windows pthread compatibility through its abstraction layer; direct inclusion bypasses this design.
🤖 Fix all issues with AI agents
In @plugins/out_s3/s3_multipart.c:
- Around line 995-1104: In s3_multipart_upload_file_parts ensure part counts and
retry limits are validated: clamp or check m_upload->part_number against AWS max
(10000) before incrementing/using it and return an error if the next part would
exceed 10000 (refer to m_upload->part_number in s3_multipart_upload_file_parts),
and ensure the per-part retry loop always runs at least once by replacing uses
of ctx->part_delivery_attempt_limit with a local max_attempts =
(ctx->part_delivery_attempt_limit > 0 ? ctx->part_delivery_attempt_limit : 1)
(use max_attempts in the while/for that iterates part_attempt); also update
error messages to report the enforced attempt count.
- Around line 884-955: The code in s3_multipart_upload_file_part allocates
part_size bytes which can OOM for large parts; change it to stream the part from
disk using a fixed-size buffer (e.g., 4–8MB) and loop: allocate a bounded
buffer, seek to offset_start, repeatedly read up to buflen bytes until
offset_end, and for each chunk call an upload routine that supports incremental
chunking (either modify s3_multipart_upload_part to accept repeated chunk calls
for the same part or add a new s3_multipart_upload_part_stream function); ensure
you compute remaining = offset_end - current_pos, read min(remaining, buflen)
each iteration, handle short reads and errors, close fd and free buffer on all
error paths, and remove the large flb_malloc(part_size) usage.
In @src/flb_blob_db.c:
- Around line 1492-1624: In flb_blob_db_file_fetch_all_parts the current realloc
logic can double-free original tmp_* buffers when one or more flb_realloc calls
succeed and others fail; change to an all-or-nothing resize: allocate/resize
into temporary pointers (new_db_ids, new_nums, new_starts, new_ends), check that
all four succeeded before assigning them to
tmp_db_ids/tmp_nums/tmp_starts/tmp_ends, and on failure only free the newly
allocated pointers (not the originals) then perform the existing cleanup and
return FLB_BLOB_DB_ERROR_ALLOCATOR_FAILURE; update the block that currently
frees new_* and tmp_* to follow this safe pattern and keep references to
statement/context->stmt_get_all_parts_for_file unchanged.
- Around line 1630-1741: flb_blob_db_file_get_next_pending currently leaves
context->stmt_get_next_pending_file open after SQLITE_ROW, which with a LIMIT 1
query and a shared prepared statement creates a brittle two-call cadence and
concurrency risk; fix by always clearing bindings and calling
sqlite3_reset(statement) after successfully processing a row (the SQLITE_ROW
branch) so the statement is returned to a clean state for the next caller, and
ensure the reset/clear happens while holding the db lock (or alternatively avoid
sharing stmt_get_next_pending_file across threads and prepare/use/reset a
statement per call).
In @src/flb_parquet_impl.cpp:
- Around line 20-45: This file is missing standard headers required for
portability (used symbols: errno, ERANGE, memcpy, strlen, strerror,
std::strtoll, std::strtof, std::strtod, std::numeric_limits) and uses unlink()
which differs on Windows; add the appropriate headers (<cerrno>, <cstring>,
<cstdlib>, <climits> or <limits> / <limits> for std::numeric_limits, and <cmath>
if needed) at the top of flb_parquet_impl.cpp so those symbols are defined
across toolchains, and replace direct calls to unlink(output_file) with
flb_unlink(output_file) to use the Fluent Bit portability wrapper. Ensure
includes are guarded by C++ headers (e.g., <cerrno>, <cstring>, <cstdlib>,
<limits>) and update all locations referenced in the diff where these symbols
and unlink are used.
- Around line 20-24: CMake currently allows Arrow 10.x but the
parquet::arrow::FileWriter::Open() and reader APIs used by the Parquet encoder
need Arrow 11+, so update the version check in cmake/parquet.cmake to require a
minimum of 11.0.0 (not 10.0.0) and fail configuration if the found Arrow is
older; then update the Parquet encoder documentation/config comments (and any
CMake messages) to state Arrow >= 11.0.0 is required so users know the minimum
supported Arrow version.
In @tests/runtime/out_s3_edge_cases.c:
- Around line 26-31: The test uses POSIX APIs (setenv/unsetenv/sleep) which
break on Windows; replace those calls with cross-platform equivalents by either
using the existing Fluent Bit environment helper wrappers where available or
adding #ifdef FLB_SYSTEM_WINDOWS branches that call _putenv_s/_putenv and Sleep
(ms) on Windows and setenv/unsetenv/sleep on POSIX, and use the common
sleep_ms() helper (as in multipart code) instead of sleep; update all
occurrences (including the setenv/unsetenv/sleep calls referenced around lines
42-68) and keep TEST_STORE_DIR definition unchanged.
- Around line 33-509: Each test currently only ensures no crash; update each
test function (e.g., flb_test_empty_data_upload, flb_test_large_file_chunking,
flb_test_max_concurrent_uploads, flb_test_minimal_timeout,
flb_test_timeout_trigger_priority, flb_test_invalid_s3_key_chars,
flb_test_duplicate_tag_index, flb_test_storage_path_permissions,
flb_test_config_boundary_values, flb_test_compression_boundary,
flb_test_rapid_config_changes) to include one observable assertion after the
sleep: use an existing runtime helper to assert expected behavior — for example,
call the test log checker to assert a relevant substring like "upload",
"multipart upload", "rotated", or "queued" was emitted, or check the local
storage directory (TEST_STORE_DIR) for created files, or assert a retry
counter/DB row exists; add a TEST_CHECK(...) per test that validates the
artifact, then continue to flb_stop/flb_destroy and unsetenv as before.
🧹 Nitpick comments (18)
tests/internal/CMakeLists.txt (1)
238-241: Parquet/Arrow libs are linked to every unit test executable—consider scoping and confirm link-order/include needs.
Right now, enablingFLB_PARQUET_ENCODERadds${ARROW_LIBRARIES} ${PARQUET_LIBRARIES}to all internal test binaries, even those unrelated to Parquet. If the intent is only to satisfyparquet.c, prefer linking only that target (or alternatively, makefluent-bit-staticexport its Parquet deps viaPUBLIC/INTERFACEso downstream targets don’t need to repeat them). Also, if these are static libs, library order may matter (oftenPARQUETbeforeARROW), and ifparquet.cincludes Arrow/Parquet headers directly you may need to add${ARROW_INCLUDE_DIRS}/${PARQUET_INCLUDE_DIRS}viatarget_include_directories.Proposed refinement (scope deps to parquet.c)
- if(FLB_PARQUET_ENCODER) - target_link_libraries(${source_file_we} ${ARROW_LIBRARIES} ${PARQUET_LIBRARIES}) - endif() + if(FLB_PARQUET_ENCODER AND source_file STREQUAL "parquet.c") + # Order can matter for static libs; verify with your supported toolchains. + target_link_libraries(${source_file_we} ${PARQUET_LIBRARIES} ${ARROW_LIBRARIES}) + # If parquet.c includes Arrow/Parquet headers directly: + # target_include_directories(${source_file_we} PRIVATE ${ARROW_INCLUDE_DIRS} ${PARQUET_INCLUDE_DIRS}) + endif()src/aws/flb_aws_util.c (1)
851-1076: Well-structured refactoring with comprehensive error handling.The new
process_s3_key_templatefunction effectively centralizes common S3 key template processing logic. The error handling path properly cleans up all allocated resources.Minor optimization opportunity: multiple
strlen()calls on the same strings (e.g.,formatat lines 877, 882;tagat lines 882, 893) could be cached in local variables for a small performance gain.tests/runtime/out_s3_special_features.c (2)
27-62: Consider reducing test brittleness and adding output verification.The test relies on a hardcoded
sleep(2)to wait for processing, which can be slow and unreliable across different environments. Additionally, the test only verifies thatflb_startreturns 0 but doesn't assert that Snappy compression was actually applied or that data was processed correctly.Consider:
- Using synchronization primitives or callbacks instead of hardcoded sleeps
- Adding assertions to verify compression occurred (e.g., checking test mode output or mock call counts)
Note: This pattern applies to most tests in this file.
177-218: Consider improving database cleanup robustness.The tests generate unique database paths using
time(NULL)and clean them up withunlink(db_path). Two minor suggestions:
- The
unlinkreturn value is not checked. While not critical in tests, logging failures could help debug test environment issues.- In concurrent test execution,
time(NULL)could collide. Consider usinggetpid()in addition to time for better uniqueness.Optional: Enhanced path generation
- snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%d.db", (int)time(NULL)); + snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%d_%d.db", (int)getpid(), (int)time(NULL));Also applies to: 220-262
plugins/out_s3/s3_store.c (1)
35-42: CI detection may have false positives in development environments.The function checks common CI environment variables to select a memory-backed storage backend. However, developers may have these variables set in their local environment (e.g., for testing CI behavior), which would unintentionally trigger memory mode.
Consider adding a dedicated override environment variable (e.g.,
FLB_S3_FORCE_MEMORY_STORE) for explicit control, or logging a clear message when memory mode is activated so developers understand why file-based storage isn't being used.tests/runtime/out_s3_error_handling.c (2)
46-278: Error handling tests verify graceful failure.These tests inject various AWS errors and verify the plugin handles them without crashing. The tests cover important error scenarios including access denial, missing resources, and invalid credentials.
Similar to the special features tests, these would benefit from:
- Assertions beyond
ret == 0to verify specific error handling behavior- Reduced reliance on hardcoded sleep durations
500-539: Test intent not fully validated.The comment at line 534 states "Verify cleanup occurred", but the test doesn't include assertions to actually verify cleanup. Consider adding checks that:
- Temporary files were deleted
- Memory was freed (if measurable)
- Multipart uploads were aborted (if in test mode the plugin exposes this)
tests/runtime/out_s3_format.c (2)
35-35: Consider reducing hardcoded test dependencies.Multiple tests rely on the
FLB_S3_PLUGIN_UNDER_TESTenvironment variable and specific test mocking behavior. While functional, this creates tight coupling between tests and implementation. Additionally, the tests usesleep()calls with hardcoded durations (1-3 seconds), which can cause flakiness in CI environments under load.Consider:
- Using configurable timeouts or polling mechanisms instead of fixed
sleep()calls- Adding test assertions to verify expected outcomes (e.g., file creation, upload attempts) rather than just checking that the engine doesn't crash
- Documenting the expected mock behavior for each test case
Also applies to: 72-72, 112-112, 155-155, 219-219, 273-275, 299-299, 337-337, 375-375, 397-397, 412-412, 437-437, 503-503
256-256: Schema mismatch test may not fail as expected.The test at Line 256 defines a schema with
"nonexistent_field"and pushesJSON_TDdata that likely doesn't contain this field. However, the test expectsflb_start()to succeed (Line 279:TEST_CHECK(ret == 0)), meaning the schema mismatch will only be detected during data processing, not at startup. This test doesn't verify that the mismatch is actually handled—it only checks that the engine starts successfully.Add assertions or log checking to verify that:
- The schema mismatch is detected during flush
- Appropriate error messages are logged
- The data is handled according to the documented behavior (dropped, logged, etc.)
Consider adding a comment explaining the expected behavior for this test case.
Also applies to: 273-274, 282-282
tests/runtime/out_s3_multipart.c (2)
452-452: Hardcoded database path may cause test conflicts.The persistence test uses a hardcoded path
/tmp/test_s3_multipart_persistence.db(Lines 452, 480). If multiple tests run concurrently or if cleanup from previous runs failed, this could cause conflicts or incorrect test results.💡 Use unique test database paths
Generate a unique database path per test run:
char db_path[PATH_MAX]; snprintf(db_path, sizeof(db_path), "/tmp/test_s3_multipart_persistence_%d_%lu.db", getpid(), (unsigned long)time(NULL)); // Use db_path instead of hardcoded path flb_output_set(ctx, out_ffd, "blob_database_file", db_path, NULL); // ... test code ... // Clean up after test unlink(db_path);This prevents test interference and ensures clean test isolation.
Also applies to: 480-480
273-275: Document mock environment variable behavior.Tests use multiple environment variables to control mock behavior:
TEST_COMPLETE_MULTIPART_RETRY(Line 275)TEST_COMPLETE_MULTIPART_FAIL(Line 316)TEST_UPLOAD_PART_FAIL(Lines 357, 397)These variables control critical test behavior but lack documentation explaining what each does and how the mocking mechanism works.
Add a comment block at the top of the file documenting all test environment variables and their effects:
/* * Test Environment Variables: * - FLB_S3_PLUGIN_UNDER_TEST: Enables test mode * - TEST_COMPLETE_MULTIPART_RETRY: First complete attempt fails, second succeeds * - TEST_COMPLETE_MULTIPART_FAIL: All complete attempts fail * - TEST_UPLOAD_PART_FAIL: Part uploads consistently fail */Also applies to: 314-316, 355-357, 396-397
tests/runtime/out_s3_edge_cases.c (2)
33-509: Reduce flakiness: fixedsleep(N)makes CI timing-sensitive.
Prefer waiting on a deterministic condition (log line, file existence/size threshold, queue drained) with a bounded timeout loop.
33-68: Ensure cleanup runs even if aTEST_CHECKfails (env var + ctx teardown).
Today, a failingTEST_CHECKcan short-circuit and leaveFLB_S3_PLUGIN_UNDER_TESTset for subsequent tests. Use agoto cleanup;pattern.Proposed pattern
void flb_test_empty_data_upload(void) { int ret = -1; flb_ctx_t *ctx = NULL; @@ - setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); + setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); ctx = flb_create(); + if (!ctx) { + TEST_CHECK(ctx != NULL); + goto cleanup; + } @@ - ret = flb_start(ctx); - TEST_CHECK(ret == 0); + ret = flb_start(ctx); + TEST_CHECK(ret == 0); @@ - flb_stop(ctx); - flb_destroy(ctx); - unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); +cleanup: + if (ctx) { + flb_stop(ctx); + flb_destroy(ctx); + } + unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); }Also applies to: 70-109, 111-146, 148-182, 184-225, 227-262, 264-302, 304-339, 341-408, 410-446, 448-509
src/flb_blob_db.c (2)
1747-1782: Clarify contract: “part not found” vs “error” inflb_blob_db_file_part_check_uploaded.
SQLITE_DONEcurrently returns-1. If callers use this to check idempotency/recovery, treating “missing row” as a hard error may cause unnecessary abort/retry behavior. Consider returningFLB_BLOB_DB_SUCCESSwith*uploaded = 0when no row is found, and reserve-1for DB failures.
1784-1809: Return code inconsistency inflb_blob_db_reset_zombie_parts.
This function returns-1on failure while most of this module returnsFLB_BLOB_DB_ERROR_*. Recommend returning a consistentFLB_BLOB_DB_ERROR_EXECUTING_STATEMENT_*-style code.src/flb_parquet_impl.cpp (1)
772-1088:AppendNull()/ default appends ignore failures; can silently corrupt output on OOM.
The conversion path intentionally falls back to null/default, but it currently drops the returnedarrow::Status. If an append fails, you likely want to abort the conversion.include/fluent-bit/flb_blob_db.h (1)
203-214:SQL_GET_NEXT_PENDING_FILE: DISTINCT + GROUP BY is redundant; consider simplifying and documenting iteration semantics.
GivenLIMIT 1, callers will treat this as “fetch one pending file”. Simplifying the query and always resetting statements (see src/flb_blob_db.c) will reduce foot-guns.plugins/out_s3/s3.h (1)
171-174: Document ownership/lifetime forschema_strandcached_arrow_schema.
Given the C++ side returns an opaque heap allocation, it’s important to ensure exactly one create/destroy path and that it can’t be used concurrently after destroy (especially with worker threads).
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (38)
include/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/fuzzers/aws_util_fuzzer.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3.ctests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
🚧 Files skipped from review as they are similar to previous changes (8)
- plugins/in_blob/blob_file.c
- plugins/out_kinesis_firehose/firehose.c
- tests/runtime/CMakeLists.txt
- tests/runtime/out_s3.c
- src/flb_signv4.c
- plugins/out_s3/s3_auth.h
- tests/runtime/out_s3_config.c
- plugins/out_s3/s3_multipart.h
🧰 Additional context used
🧠 Learnings (8)
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_parquet.h
📚 Learning: 2025-09-04T12:35:36.904Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:3275-3282
Timestamp: 2025-09-04T12:35:36.904Z
Learning: The out_s3 plugin intentionally uses a simple numeric comparison for retry_limit (chunk->failures >= ctx->ins->retry_limit) rather than the standard Fluent Bit pattern that checks for FLB_OUT_RETRY_UNLIMITED (-1). The maintainer wants to keep this current behavior for consistency within the plugin.
Applied to files:
plugins/out_s3/s3_stream.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_s3/s3_store.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
src/aws/flb_aws_util.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
src/flb_parquet_impl.cpp
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3_auth.c
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
tests/internal/CMakeLists.txt
🧬 Code graph analysis (25)
tests/runtime/out_s3_special_features.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_queue.h (1)
plugins/out_s3/s3_queue.c (9)
s3_queue_remove(267-275)s3_queue_buffer_chunk(315-337)s3_queue_add_file(57-111)s3_queue_add_part(113-183)s3_queue_add_pending_file(185-232)s3_queue_process_entry(680-739)s3_queue_recover_all(782-827)s3_queue_recover_from_database(837-998)s3_queue_recover_from_fstore(1071-1131)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
tests/runtime/out_s3_multipart.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (8)
flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_queue.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_error_handling.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_recovery.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_stream.c (5)
src/aws/flb_aws_compress.c (1)
flb_aws_compression_compress(103-120)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_printf(336-387)plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(536-572)tests/internal/processor.c (1)
processor(66-126)
plugins/out_s3/s3_stream.h (1)
plugins/out_s3/s3_stream.c (4)
stream_compress_file(32-147)stream_process_msgpack_file(344-412)stream_json_processor(414-442)stream_log_key_processor(517-546)
tests/runtime/out_s3_edge_cases.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (8)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_stop(1011-1055)flb_destroy(228-263)
plugins/out_s3/s3_blob.h (1)
plugins/out_s3/s3_blob.c (5)
s3_blob_register_parts(306-338)s3_blob_process_events(436-516)s3_blob_store_chunk(345-421)s3_blob_recover_state(128-147)s3_blob_notify_delivery(81-118)
plugins/out_s3/s3_store.c (4)
src/flb_sds.c (4)
flb_sds_create_size(92-95)flb_sds_printf(336-387)flb_sds_destroy(389-399)flb_sds_create(78-90)src/flb_fstore.c (4)
flb_fstore_file_delete(259-277)flb_fstore_file_create(153-190)flb_fstore_file_meta_set(74-108)flb_fstore_file_append(298-329)include/fluent-bit/flb_mem.h (2)
flb_free(126-128)flb_calloc(84-96)include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)
plugins/out_s3/s3_queue.c (8)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)plugins/out_s3/s3_store.c (5)
s3_store_buffer_put(210-245)s3_store_file_delete(509-534)s3_store_file_unlock(581-586)s3_store_file_inactive(489-507)s3_store_file_lock(574-579)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(359-400)plugins/out_s3/s3_multipart.c (5)
s3_multipart_upload_file_part(884-955)s3_multipart_complete(634-721)s3_multipart_check_upload_exists(801-882)s3_multipart_upload_create(420-469)s3_multipart_upload_destroy(389-414)src/flb_blob_db.c (3)
flb_blob_db_file_part_update_remote_id(1164-1195)flb_blob_file_update_remote_id(541-572)flb_blob_db_file_fetch_all_parts(1492-1624)plugins/out_s3/s3.c (3)
s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)plugins/out_s3/s3_blob.c (1)
s3_blob_notify_delivery(81-118)lib/cfl/src/cfl_sds.c (2)
cfl_sds_destroy(127-137)cfl_sds_len(88-91)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
tests/internal/aws_util.c (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
tests/internal/fuzzers/aws_util_fuzzer.c (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
plugins/out_s3/s3_store.h (1)
plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(536-572)
include/fluent-bit/flb_parquet.h (1)
src/flb_parquet_impl.cpp (8)
flb_parquet_validate_schema(719-735)flb_parquet_validate_schema(719-721)flb_parquet_schema_create(737-762)flb_parquet_schema_create(737-739)flb_parquet_schema_destroy(764-770)flb_parquet_schema_destroy(764-764)flb_msgpack_to_parquet_streaming(772-1088)flb_msgpack_to_parquet_streaming(772-777)
plugins/out_s3/s3_auth.c (5)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_utils.c (1)
flb_utils_url_split(1531-1667)src/flb_upstream.c (1)
flb_upstream_create_url(416-481)src/flb_http_client.c (5)
flb_http_add_header(1013-1045)flb_http_basic_auth(1373-1377)flb_http_bearer_auth(1385-1420)flb_http_client(864-909)flb_http_do(1622-1682)src/flb_sds.c (4)
flb_sds_copy(260-281)flb_sds_create_len(58-76)flb_sds_destroy(389-399)flb_sds_printf(336-387)
tests/runtime/out_s3_format.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/internal/parquet.c (3)
src/flb_parquet_impl.cpp (1)
field_name(408-408)src/flb_pack.c (1)
flb_pack_json(530-535)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
include/fluent-bit/flb_blob_db.h (1)
src/flb_blob_db.c (11)
flb_blob_db_file_fetch_part_ids(1385-1459)flb_blob_db_file_fetch_part_ids(1960-1967)flb_blob_db_file_fetch_all_parts(1492-1624)flb_blob_db_file_get_next_pending(1630-1741)flb_blob_db_file_get_next_pending(1975-1984)flb_blob_db_file_part_check_uploaded(1747-1782)flb_blob_db_file_part_check_uploaded(1986-1991)flb_blob_db_file_parts_in_progress(1004-1039)flb_blob_db_file_parts_in_progress(1993-1998)flb_blob_db_reset_zombie_parts(1785-1809)flb_blob_db_reset_zombie_parts(2000-2003)
plugins/out_s3/s3_multipart.c (2)
plugins/out_s3/s3.c (1)
write_seq_index(226-246)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(359-400)
src/flb_blob_db.c (3)
src/flb_sqldb.c (2)
flb_sqldb_lock(173-178)flb_sqldb_unlock(180-185)include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)lib/cfl/src/cfl_sds.c (2)
cfl_sds_create(113-125)cfl_sds_destroy(127-137)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (4)
s3_format_chunk(1266-1407)s3_upload_file(1089-1194)s3_initiate_multipart_upload(982-1087)mock_s3_call(105-179)
🔇 Additional comments (68)
tests/internal/CMakeLists.txt (1)
136-141: Good: conditionalparquet.cinclusion mirrors the existing AVRO pattern.
Keeps internal unit tests feature-gated behindFLB_PARQUET_ENCODER.plugins/out_s3/s3_store.h (2)
32-32: LGTM: Field rename improves clarity.The rename from
file_pathtostream_pathwith the updated comment accurately reflects that this field holds a stream directory path rather than a file path.
51-52: LGTM: Well-designed context-free API.The new
flb_chunk_file_openfunction is appropriately designed as a context-free helper that can be called from C++. The implementation properly validates chunk file headers and handles errors gracefully.src/aws/flb_aws_util.c (5)
45-46: LGTM: Macros follow established patterns.The new
FILE_PATH_STRINGandFILE_NAME_STRINGmacros are consistent with the existing naming conventions for template variable strings.
184-226: Excellent reliability improvement with immediate auth retry.The addition of immediate retry after credential refresh is a well-designed improvement that handles credential expiration during normal operation (e.g., IAM role rotation). The
auth_retry_doneflag properly prevents infinite retry loops.
1105-1116: Robust file path validation and sanitization.The path sanitization logic properly strips leading dots and slashes, and validates that the result is not empty. The warning message clearly communicates when substitution is skipped.
1151-1177: Backward-compatible fallback logic preserves directory structure.The fallback append logic correctly handles the case when no file variables are used in the format string. It ensures proper path separation and maintains backward compatibility while preserving directory structure.
1078-1184: Well-designed extension maintaining backward compatibility.The updated
flb_get_s3_keyfunction cleanly extends the API with optional file path support while maintaining full backward compatibility. The three-case handling (NULL path, explicit variables, fallback append) is clear and correctly implemented.tests/internal/fuzzers/aws_util_fuzzer.c (1)
73-73: LGTM: Correctly updated fuzzer to match new API.The call to
flb_get_s3_keycorrectly includes the newNULLparameter for the optionalfile_pathargument.plugins/out_s3/CMakeLists.txt (1)
4-8: LGTM: New modules added for extended S3 functionality.The addition of
s3_stream.c,s3_multipart.c,s3_blob.c,s3_auth.c, ands3_queue.caligns with the PR's objective to add Parquet format support and extend S3 output capabilities.include/fluent-bit/flb_aws_util.h (1)
200-207: LGTM: Clear documentation of extended functionality.The updated documentation clearly describes the new template variables (
$FILE_PATH,$FILE_NAME) and notes that thefile_pathparameter is optional. The signature correctly reflects the implementation changes.tests/internal/parquet.c (6)
1-52: Well-structured test file with comprehensive coverage.The test file is well-organized with clear categorization of tests and appropriate helper functions. The extern C declaration for
validate_parquet_fileproperly handles C++ linkage.
57-98: Helper function implementation is correct.The
write_msgpack_to_chunk_filefunction properly handles the chunk format with magic bytes and metadata. Error handling with early returns andfclosecleanup is appropriate.
200-266: Test context framework provides good resource management.The
test_contextstructure and associated functions (init_test_context,run_conversion,cleanup_test_context) provide a clean abstraction for test setup and teardown, reducing boilerplate across tests.
868-903: Long string test correctly handles memory allocation.The test properly checks for allocation failure and frees the allocated memory before cleanup. The pattern
free(long_str)followed bycleanup_test_context(&ctx)ensures no memory leaks.
1922-1925: Division by zero to create special float values is intentional.Using
0.0 / 0.0and1.0 / 0.0to create NaN and infinity values is a valid approach for IEEE 754 compliant systems. Some compilers may warn about this, but it's acceptable in test code.
2042-2117: Comprehensive and well-organized test registry.The
TEST_LISTis well-structured with clear category comments (Functional, Data Quality, Scale & Boundary, Compression, Negative, Real-World). The 44 tests provide thorough coverage of the Parquet encoder functionality.plugins/out_s3/s3_stream.h (1)
20-111: Clean header design with well-documented streaming API.The header provides a well-designed callback-based streaming API. The
record_processor_fntypedef enables flexible record processing, and thestream_processor_contextstruct tracks processing state. Documentation comments clearly explain each function's purpose and parameters.plugins/out_s3/s3_auth.c (7)
29-54: URL parts structure and cleanup are correctly implemented.The
url_partsstructure andurl_parts_destroyfunction properly manage URL component memory. The NULL checks beforeflb_freeare defensive but harmless.
56-77: URL parsing with proper validation and cleanup.The function correctly validates that host and URI components are present after parsing, and properly cleans up on failure.
79-128: Endpoint initialization with proper TLS and error handling.The function correctly initializes the authorization endpoint upstream, creating TLS context only for HTTPS URLs. Error handling properly cleans up resources on failure, including destroying the TLS context if upstream creation fails.
130-153: Robust port parsing with comprehensive validation.The function properly validates the parsed port value including checking for complete string conversion, errno, and valid port range (1-65535). Falls back appropriately to scheme-based defaults.
173-260: Presigned URL request implementation with proper resource management.The function correctly handles HTTP client lifecycle, connection management, and response processing. The logic to either copy into an existing SDS or create a new one (lines 232-239) properly handles both cases. The cleanup section ensures all resources are released.
262-357: URL path construction with proper encoding and error handling.The function correctly URL-encodes the S3 key and upload ID, strips leading slashes, and constructs the appropriate path based on URL type. The error handling properly cleans up all allocated resources. The return of
tmp(line 344) rather thanpathis correct becauseflb_sds_printfmay reallocate.
359-400: Presigned URL fetch with proper optional endpoint handling.The function correctly handles the case where no authorization endpoint is configured (returns success with NULL result). Memory management for the constructed URL is proper with cleanup of both
url_pathandfull_url.plugins/out_s3/s3_blob.h (1)
20-52: Clean header with well-documented blob handling API.The header provides a clear public interface for S3 blob operations. Function declarations are appropriately documented with brief comments explaining their purpose. The signatures align with the implementations shown in the relevant code snippets.
tests/runtime/out_s3_recovery.c (4)
27-33: Platform-agnostic temporary directory handling.The conditional macros for Windows vs. Unix temporary directories ensure the tests work across platforms.
36-106: Well-designed recovery test with proper cleanup.The test correctly simulates a crash-and-restart scenario by running two Fluent Bit instances with the same database path. The environment variable and database file are properly cleaned up.
One minor note:
unsetenvis called at line 75 before the second run and again at line 102 after (redundant but harmless).
388-433: Concurrent access test appropriately stresses the database.The test configures 5 workers and pushes 10 chunks to trigger concurrent database access. This is a good stress test for the blob database implementation.
435-446: Complete and well-organized test registry.The
TEST_LISTincludes all 8 recovery-related tests with descriptive names that clearly indicate what each test validates.tests/runtime/out_s3_queue.c (4)
27-62: Basic queue operations test with proper setup and teardown.The test correctly configures the S3 output plugin and pushes test data to trigger queue operations. Environment variable and context lifecycle are properly managed.
141-176: Timeout handling test correctly triggers time-based uploads.The test uses a short timeout (2s) with a large file size threshold (100M) to ensure the upload is triggered by timeout rather than size. The 3-second sleep appropriately exceeds the timeout.
178-217: Concurrent uploads test with appropriate stress level.The test configures 10 workers and pushes 20 chunks in a tight loop, providing good coverage for concurrent upload scenarios and potential race conditions.
257-266: Complete test registry for queue functionality.The
TEST_LISTcovers the key queue scenarios: basic operations, multiple workers, retry mechanism, timeout handling, concurrent uploads, and shutdown cleanup.plugins/out_s3/s3_queue.h (1)
20-85: Well-designed queue API with comprehensive documentation.The header provides a clear public interface for the S3 queue subsystem. The documentation is particularly valuable:
- The comment for
s3_queue_add_file(lines 32-37) clearly explains the dual-mode behavior (with/without DB tracking).- The
s3_queue_process_entrycomment (lines 63-66) documents the return value semantics.- The three-phase recovery architecture (lines 71-78) is well-documented, explaining the cleanup, state transition, and queue rebuild phases.
This documentation will greatly help future maintainers understand the queue subsystem's design.
tests/runtime/out_s3_special_features.c (3)
265-297: Good negative test with proper error handling.The test correctly expects
flb_startto fail with an invalid compression type and includes conditional cleanup to avoid callingflb_stopwhen start fails. This defensive pattern prevents potential crashes or undefined behavior.
299-336: Clear test with helpful documentation.The test includes a comment explaining the expected auto-adjustment behavior and uses proper conditional cleanup. Good practice.
381-465: Well-documented credential expiration tests.Both tests include clear comments explaining the credential refresh retry logic and the difference between retry mode and strict mode (
file_delivery_attempt_limit=1). This documentation helps maintainers understand test intent.plugins/out_s3/s3_store.c (9)
44-74: Hash generation looks correct with proper cleanup.The function uses the djb2 algorithm combined with timestamp entropy to generate unique filenames. Error handling properly cleans up intermediate allocations.
76-107: Good defensive check for corrupted file state.The added NULL check for
fsf->dataat lines 86-90 prevents potential crashes during recovery scenarios and logs a clear warning. Deleting the corrupted file is a reasonable recovery strategy.
109-133: Buffer space checking is well-implemented.The function properly validates buffer capacity before writes and provides early warning when approaching the limit. The logic correctly handles the case where no limit is set (
store_dir_limit_size == 0).
135-208: Excellent refactoring for centralized cleanup.The new
cleanup_failed_filehelper andcreate_new_buffer_filefunction improve code maintainability by:
- Centralizing error cleanup logic
- Ensuring all error paths properly free resources
- Providing clear error messages at each failure point
The careful use of NULL checks in cleanup and the staged cleanup calls (lines 182, 188, 195) correctly match the allocation state at each error point.
247-275: Robust file size calculation with comprehensive error handling.The function includes multiple safety checks:
- Empty/NULL filename validation
- Buffer overflow protection for path construction
- Graceful handling of stat failures
- Validation of minimum file size
Returning 0 on any error is safe since it represents "no data" rather than causing incorrect buffer tracking.
363-386: Platform-specific time formatting is correct.The function properly handles the different parameter orders between Windows
gmtime_s(struct tm*, const time_t*)and POSIXgmtime_r(const time_t*, struct tm*).The different strftime formats are intentional: Windows uses dashes in the time component (
%H-%M-%S) because colons are invalid in Windows filenames, while POSIX uses the standard ISO 8601 format with colons (%H:%M:%S).
388-432: CI memory backend limits test coverage of filesystem code paths.The initialization selects a memory-backed store in CI environments (line 395-399) to avoid test flakiness. However, this means CI tests don't exercise the filesystem storage code paths, which could hide file-related bugs.
Consider adding at least one test configuration that explicitly forces filesystem mode in CI to ensure both storage backends are tested. This could be a dedicated test job or a subset of tests with an override flag.
434-534: Improved resource cleanup with defensive checks.The updates to exit/delete functions add important safety measures:
- NULL checks prevent crashes on partially-initialized structures
- Verification that
fsf->data == s3_file(line 519) prevents incorrect cleanup if the pointer was already reassigned- Nulling
fsf->dataafter freeing prevents use-after-free bugsThese defensive patterns improve robustness during error recovery and shutdown.
536-572: Well-implemented chunk file opener with proper validation.The function correctly:
- Validates CIO magic bytes to detect corrupted files
- Extracts metadata length from the header (big-endian format at lines 561-562)
- Positions the file pointer at the start of data
- Cleans up on all error paths
Note: Callers must
fclose()the returnedFILE*to avoid resource leaks.tests/runtime/out_s3_error_handling.c (1)
358-498: Good coverage of retry mechanisms.The retry layer tests exercise both automatic retry configuration and manual retry limits at different layers (auto_retry, part_delivery, file_delivery). The
flb_test_retry_layer_auto_retrytest is particularly thorough as it tests both enabled and disabled states.include/fluent-bit/flb_parquet.h (3)
31-41: Well-designed validation API for fail-fast error detection.The validation function allows schema verification at plugin initialization, preventing runtime failures. The optional error message buffer enables detailed diagnostics when validation fails.
43-70: Excellent API design with thorough documentation.The opaque schema type and separate create/destroy functions address a critical architectural constraint: Fluent Bit's coroutines have limited stack space (37KB), which can cause stack overflow during recursive JSON parsing of complex schemas.
The detailed comment at lines 52-54 provides valuable context for future maintainers about why this separation exists. The opaque type properly encapsulates C++ Arrow/Parquet implementation details from the C API.
72-92: Streaming API design is sound.The function signature accepts a pre-parsed schema (addressing the coroutine stack issue) and provides streaming conversion, appropriate for potentially large log data. The
out_file_sizeoutput parameter allows callers to track actual file sizes. Thecompressionparameter typeintis compatible with theFLB_AWS_COMPRESS_*constants (which are integer macros), and the implementation includes proper validation through a switch statement that handles all valid compression types and defaults to uncompressed for invalid values.tests/internal/aws_util.c (3)
191-400: Consistent migration of all existing test calls.All existing tests have been updated to pass
NULLfor the newfile_pathparameter, preserving their original behavior. The migration is complete and consistent across all 14 existing test functions.
402-533: Comprehensive edge case testing for path sanitization.The new tests verify critical security and correctness behaviors:
- Path traversal prevention (line 485):
../../../etc/passwdis sanitized toetc/passwd- Relative path normalization (line 451):
./logs/file.logbecomeslogs/file.log- Multiple dot handling (line 468, 502, 519): Various combinations of dots and slashes are stripped
These tests ensure the S3 key generation cannot be exploited to write outside intended directories or create malformed keys.
535-558: Test registration is complete and correct.All 8 new test functions are properly registered in the TEST_LIST with descriptive names.
plugins/out_s3/s3_blob.c (3)
378-384: LGTM! Proper validation of write completeness.The code correctly validates that the full
data_sizewas written (Line 379), handles errors appropriately, and cleans up resources (closes fd, unlinks file) before returning.
228-261: Comprehensive upload ID validation logic.The validation logic properly handles three distinct scenarios when an upload_id exists:
- Valid upload (Line 235): Resume the upload
- Invalid upload (Line 242): Reset and start fresh
- Network error (Line 253): Conservatively treat as invalid
This defensive approach prevents data corruption and handles edge cases well.
363-371: Non-standard mkstemp template pattern.The code constructs a template string ending with
.msgpackafter theXXXXXXpattern (Line 363). Whilemkstemp()will successfully replace theXXXXXXwith a unique identifier, the resulting filename will have the extension in the middle of the random suffix (e.g.,chunk_ABC123.msgpackDEF456), not at the end as likely intended.🔧 Proposed fix to place extension after mkstemp pattern
- ret = snprintf(temp_path, sizeof(temp_path), "%s/chunk_XXXXXX.msgpack", + ret = snprintf(temp_path, sizeof(temp_path), "%s/chunk_XXXXXX", ctx->buffer_dir); if (ret < 0 || ret >= sizeof(temp_path)) { flb_plg_error(ctx->ins, "chunk_store: temp path too long"); return -1; } /* Create unique temporary file atomically */ fd = mkstemp(temp_path); if (fd == -1) { flb_plg_error(ctx->ins, "chunk_store: failed to create temp file: %s", temp_path); return -1; } + + /* Rename to add .msgpack extension */ + char final_path[PATH_MAX]; + ret = snprintf(final_path, sizeof(final_path), "%s.msgpack", temp_path); + if (ret < 0 || ret >= sizeof(final_path)) { + close(fd); + unlink(temp_path); + return -1; + } + if (rename(temp_path, final_path) != 0) { + close(fd); + unlink(temp_path); + return -1; + } + strncpy(temp_path, final_path, sizeof(temp_path));Alternatively, if the extension placement doesn't matter, add a comment explaining the intentional pattern.
Likely an incorrect or invalid review comment.
plugins/out_s3/s3_queue.c (4)
68-89: Good defensive memory management in queue entry creation.The function properly allocates resources with comprehensive error handling at each step. The cleanup pattern correctly handles partial allocation failures (Lines 78-88), preventing memory leaks.
891-907: Critical fix: Preventing infinite query loop.The comment at Lines 891-894 correctly identifies that marking parts as
in_progressimmediately prevents the same file from being returned by subsequent queries, avoiding an infinite loop. This is a crucial fix for correctness.The implementation properly handles the failure case (Lines 896-907) by cleaning up and skipping the file rather than continuing with an invalid state.
636-654: No action needed - retry limit handling is consistent and correct.The simple numeric comparison at line 636 is intentional and correct. The S3 plugin initializes negative
retry_limitvalues toMAX_UPLOAD_ERRORS(s3.c lines 446-447), ensuring the value is always non-negative before any comparison. This approach is consistent across all retry limit checks in the plugin (s3_queue.c lines 636, 1108; s3.c lines 1463, 1613).The code at s3_blob.c line 225 checks a different field (
file_delivery_attempt_limit) for a separate concern, not the sameretry_limitfield used here.
352-355: Flag is set appropriately early, but in-flight uploads are interrupted rather than gracefully completed.The
ctx->is_exitingflag is set at the start ofcb_s3_exit()(line 1772), before any cleanup, which correctly prevents new uploads from starting. The timer callback skips entirely when the flag is set (line 1427), and the queue processing loop breaks early (line 1489). However, any uploads already in progress that reachupload_part_with_db_tracking()returnFLB_RETRY(line 354) and are subsequently abandoned in the queue—they are not completed or explicitly marked as failed/aborted. This design prioritizes fast, non-blocking shutdown over graceful completion of in-flight uploads. If graceful drain of pending uploads is required, consider processing remaining queue entries with a timeout before destroying the context, or explicitly marking abandoned entries as failed rather than retrying them.plugins/out_s3/s3_stream.c (5)
54-65: Proper file opening with offset handling.The code correctly opens the input file and seeks to the start offset if needed (Lines 61-65), with appropriate error checking via
flb_errno()and descriptive error messages.
80-121: Verify offset arithmetic for potential overflow.The code handles offset ranges with
remaining(Lines 81, 114-120), but the arithmetic operations don't explicitly check for overflow conditions. Whileoff_tis typically large enough, consider validating:
offset_end - offset_startdoesn't overflow (Line 81)current_offset + bytes_readdoesn't exceedoffset_end(Line 116-117)The current implementation appears safe given typical file sizes and the checks at Line 85-90, but consider adding explicit overflow validation if this code will handle extremely large files (>2GB on 32-bit systems).
165-192: Excellent resource cleanup with optional temp file preservation.The
stream_context_destroy()function properly cleans up all resources in the correct order (msgpack objects, buffers, file handles, paths) and includes akeep_temp_fileparameter (Line 165) to control whether the temporary file is deleted. This is good design for both normal operation and debugging.
459-469: Handle both STR and BIN msgpack types for map keys.The code properly handles both
MSGPACK_OBJECT_STR(Lines 459-462) andMSGPACK_OBJECT_BIN(Lines 463-466) types for map keys, which is important for compatibility with different msgpack encoders. This is a good defensive programming practice.
239-243: No issue found. The code correctly checks the return value ofmsgpack_unpacker_init(). The msgpack-c library documents thatmsgpack_unpacker_init()returnsboolwithtrueon success andfalseon failure. The conditionif (!msgpack_unpacker_init(...))correctly captures the failure case (false), and this pattern is consistently used throughout the codebase and within the msgpack library itself.tests/runtime/out_s3_format.c (1)
152-153: Schema validation test expectations are correct.The test correctly expects
flb_start()to return-1when an invalid schema is provided. Schema validation occurs during plugin initialization (flb_start), whereflb_parquet_schema_create()parses and validates the schema string (plugins/out_s3/s3.c:598-607). The parsed schema is cached at startup and reused during data processing, avoiding re-parsing in coroutine contexts. The invalid schema in the test will fail parsing duringflb_start()and correctly return-1.plugins/out_s3/s3_multipart.c (1)
36-64: Windows compiler guards look fine; keep them minimal per current Windows support.
The_WIN32split + MSVC-friendly wrappers seem consistent with the repo’s Windows approach.
| @@ -0,0 +1,1090 @@ | |||
| /* -*- Mode: C++; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- */ | |||
There was a problem hiding this comment.
We don't want to depend on parquet library itself.
We need to implement parquet glib as much as possible.
This is because libparquet and libarrow themselves are implemented in C++ and we need to avoid to link C++ library if there's another way to implement an equivalent functionality in C language.
There was a problem hiding this comment.
@cosmo0920 agree Fluent Bit core should remain C-based. However, Arrow / Parquet libraries (libarrow, libparquet) are only available in C++, and no viable pure C implementation exists. Previous glibc-based approaches were not feasible due to heavy build and runtime dependencies. To minimize impact, C++ usage is strictly isolated to a small Parquet write wrapper and does not affect the core. Using parquet-glib does not eliminate C++ dependencies and introduces GLib, increasing complexity without clear benefits.
let me know your thoughts. if it's not acceptable, I won't waist time here.
cosmo0920
left a comment
cosmo0920
left a comment
There was a problem hiding this comment.
We should avoid to use C++ as much as possible in this project.
We need to use C language in Fluent Bit core.
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
5f6750c to
d8aa149
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In @cmake/parquet.cmake:
- Around line 43-62: The CMake snippet uses ${ARROW_INCLUDE_DIR} but other
installs/exported configs use ARROW_INCLUDE_DIRS (plural), causing inconsistent
include dir resolution; update the block in the Arrow find_package handling (the
branch that sets ARROW_FOUND TRUE and ARROW_LIBRARIES like
arrow_shared/arrow_static/Arrow::arrow_shared) to prefer target-based includes
when a target exists (use the target's INTERFACE_INCLUDE_DIRECTORIES) and
otherwise set ARROW_INCLUDE_DIRS by checking both variables (if
ARROW_INCLUDE_DIRS is defined use it; else if ARROW_INCLUDE_DIR is defined set
ARROW_INCLUDE_DIRS to ARROW_INCLUDE_DIR) so the rest of the build can reliably
use ARROW_INCLUDE_DIRS.
In @plugins/out_s3/s3_blob.c:
- Around line 377-390: The mkstemp template used when building temp_path
("chunk_XXXXXX.msgpack") is invalid because mkstemp requires the XXXXXX to be
the trailing characters; update the template so the six Xs are at the end (e.g.,
"chunk_XXXXXX") before calling mkstemp or switch to mkstemps to preserve the
".msgpack" suffix (e.g., call mkstemps(temp_path, 8) where 8 ==
strlen(".msgpack")). Ensure the snprintf that writes temp_path still validates
size and that the subsequent call uses the matching function (mkstemp for no
suffix, mkstemps with correct suffix length) and handle fd == -1 error as
before.
🧹 Nitpick comments (21)
tests/runtime/out_s3_recovery.c (3)
29-99: Consider adding assertions to verify recovery behavior.The test creates a database, buffers data, restarts, and waits for recovery - but doesn't verify that recovery actually occurred (e.g., checking database state, file existence, or upload attempts). Currently it only confirms the engine starts and stops without crashing.
This is acceptable for initial smoke testing, but consider adding verification logic to confirm recovery behavior in future iterations.
95-95: Redundantunsetenvcall.The environment variable
FLB_S3_PLUGIN_UNDER_TESTwas already unset at line 68. This second call is harmless but unnecessary.Suggested fix
flb_stop(ctx); flb_destroy(ctx); - unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); /* Cleanup database file */
382-426: Concurrent access test uses multiple workers but lacks explicit concurrency verification.The test configures
workers=5and pushes 10 chunks in a loop, but doesn't verify that concurrent database access actually occurred or that data integrity was maintained. Consider adding verification of the final state or using thread-safe assertions.tests/runtime/out_s3_config.c (1)
369-413: Consider adding test output context for loop iterations.If a storage class test fails, it may not be immediately clear which iteration failed. Consider adding debug output or using separate test functions for better failure diagnostics.
tests/runtime/out_s3_error_handling.c (1)
501-539: Consider adding cleanup verification.The comment at line 534 states "Verify cleanup occurred" but there's no actual verification logic. If the S3 plugin is expected to clean up resources (abort multipart uploads, delete temp files), consider adding assertions or file system checks to confirm cleanup.
plugins/out_s3/s3_queue.h (1)
63-69: Clarify return value documentation.The comment states "Returns: 1 on success, -1 on failure (will retry), 0 if invalid (removed)" but reviewing
s3_queue.cshows it usesS3_QUEUE_ENTRY_SUCCESS,S3_QUEUE_ENTRY_RETRY, andS3_QUEUE_ENTRY_INVALIDconstants. Consider documenting the actual constant names for clarity.Suggested documentation improvement
/* * Process a queue entry (called by timer callback) - * Returns: 1 on success, -1 on failure (will retry), 0 if invalid (removed) + * Returns: + * S3_QUEUE_ENTRY_SUCCESS - Upload succeeded, entry removed + * S3_QUEUE_ENTRY_RETRY - Temporary failure, will retry later + * S3_QUEUE_ENTRY_INVALID - Invalid entry, removed from queue */tests/runtime/out_s3_edge_cases.c (4)
71-113: Clarify the test configuration.The test sets
total_file_size=100Kandupload_chunk_size=5M. Since the chunk size (5M) exceeds the file size (100K), this may not effectively test "large file chunking" as intended. Consider whether these values should be reversed or adjusted to better reflect the test's purpose of triggering file rotation through size-based chunking.
116-154: Test does not validate maximum concurrent uploads.The test sets
workers=10but only pushes a single chunk. To effectively test maximum concurrent upload behavior, consider pushing multiple chunks or adjusting the test name to reflect what's actually being validated (e.g., worker configuration acceptance).
329-367: Test validation could be strengthened.The test checks that the plugin doesn't crash when
store_diris configured, but doesn't verify that the directory was actually created or is writable. Consider adding assertions to check directory existence/permissions, or rename the test to reflect what's actually being validated.
491-562: Test name doesn't match behavior.
flb_test_rapid_config_changescreates two completely separate contexts sequentially with a sleep and cleanup between them. This doesn't test "rapid" configuration changes within a running instance, but rather tests that two different configurations can run successfully in sequence. Consider renaming to better reflect the actual test scenario.cmake/parquet.cmake (1)
201-219: Version check only triggers if config.h exists and contains version string.The version validation silently skips if
config.hdoesn't exist or lacksARROW_VERSION_STRING. Consider logging a warning when version cannot be determined, as users might unknowingly use an unsupported Arrow version.💡 Suggested improvement
if(ARROW_FOUND) # Try to detect Arrow version if(EXISTS "${ARROW_INCLUDE_DIRS}/arrow/util/config.h") file(STRINGS "${ARROW_INCLUDE_DIRS}/arrow/util/config.h" ARROW_VERSION_LINE REGEX "^#define ARROW_VERSION_STRING") if(ARROW_VERSION_LINE) string(REGEX REPLACE "^#define ARROW_VERSION_STRING \"([0-9.]+)\".*" "\\1" ARROW_VERSION ${ARROW_VERSION_LINE}) message(STATUS "Arrow version: ${ARROW_VERSION}") # Check minimum version requirement if(ARROW_VERSION VERSION_LESS "11.0.0") message(FATAL_ERROR "Arrow version ${ARROW_VERSION} is too old. ...") endif() + else() + message(WARNING "Could not parse Arrow version from config.h - assuming compatible version") endif() + else() + message(WARNING "Arrow config.h not found - cannot verify version >= 11.0.0") endif() endif()plugins/out_s3/s3_auth.c (1)
230-230: Consider using debug-level logging for successful presigned URL retrieval.The
flb_plg_infolog on every successful presigned URL retrieval could become verbose in production when many uploads are happening. Consider usingflb_plg_debuginstead to reduce log noise.💡 Suggested fix
- flb_plg_info(ctx->ins, "Pre-signed URL retrieved successfully"); + flb_plg_debug(ctx->ins, "Pre-signed URL retrieved successfully");CMakeLists.txt (1)
1091-1134: Minor: RedundantFLB_DEFINITION(FLB_HAVE_LIBYAML)call in Homebrew detection path.When libyaml is found via Homebrew (lines 1104-1105),
FLB_DEFINITION(FLB_HAVE_LIBYAML)is called. Then line 1133 calls it again unconditionally within the outer else block. This is harmless but redundant.💡 Suggested cleanup
Either remove the call at line 1105 since line 1133 will handle it, or restructure to avoid the duplicate. The current code works correctly, so this is purely a code clarity improvement.
cmake/kafka.cmake (1)
53-65: Dead code: else branches for always-enabled flags.Since
FLB_SASL_ENABLEDandFLB_SASL_OAUTHBEARER_ENABLEDare now unconditionally set toON(lines 38, 41), the else branches at lines 56-58 and 63-65 will never execute. Consider removing the conditionals entirely.♻️ Suggested cleanup
# Export compile-time definitions using FLB_DEFINITION macro -if(FLB_SASL_ENABLED) - FLB_DEFINITION(FLB_HAVE_KAFKA_SASL) - message(STATUS "Kafka SASL authentication: ENABLED") -else() - message(STATUS "Kafka SASL authentication: DISABLED") -endif() +FLB_DEFINITION(FLB_HAVE_KAFKA_SASL) +message(STATUS "Kafka SASL authentication: ENABLED (built-in PLAIN/SCRAM)") -if(FLB_SASL_OAUTHBEARER_ENABLED) - FLB_DEFINITION(FLB_HAVE_KAFKA_OAUTHBEARER) - message(STATUS "Kafka OAuth Bearer: ENABLED") -else() - message(STATUS "Kafka OAuth Bearer: DISABLED") -endif() +FLB_DEFINITION(FLB_HAVE_KAFKA_OAUTHBEARER) +message(STATUS "Kafka OAuth Bearer: ENABLED (built-in)")src/aws/flb_aws_util.c (1)
1038-1041: strftime returning 0 may not always indicate an error.When
strftimereturns 0, it could mean either:
- The buffer was too small (actual error)
- The format string produced an empty result (valid case)
The current warning may be misleading for edge cases with minimal format strings.
Consider more precise error detection
ret = strftime(key, S3_KEY_SIZE, s3_key, &gmt); - if (ret == 0) { + if (ret == 0 && s3_key[0] != '\0') { + /* strftime returned 0 with non-empty format - likely buffer overflow */ flb_warn("[s3_key] Object key length is longer than the 1024 character limit."); }tests/runtime/out_s3_multipart.c (1)
429-493: Consider cleanup of persistent database file between test sessions.The test uses a hardcoded path
/tmp/test_s3_multipart_persistence.dbfor both sessions. If a previous test run left the database in an unexpected state, it could affect subsequent test runs. Consider adding cleanup at the start or end of the test.♻️ Proposed cleanup addition
void flb_test_upload_id_persistence(void) { int ret; flb_ctx_t *ctx; int in_ffd; int out_ffd; + const char *db_path = "/tmp/test_s3_multipart_persistence.db"; setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); + /* Clean up any existing database from previous runs */ + unlink(db_path); + /* First session: Create multipart upload */ ctx = flb_create();And at the end:
flb_stop(ctx); flb_destroy(ctx); unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); + + /* Clean up database file */ + unlink(db_path); }plugins/out_s3/s3_stream.c (3)
114-120: current_offset is updated but never used.The variable
current_offsetis incremented in the loop but its value is only used in the conditioncurrent_offset >= offset_end, which is redundant withremaining <= 0since both track the same state.♻️ Consider removing unused variable
- if (remaining > 0) { - remaining -= bytes_read; - current_offset += bytes_read; - if (remaining <= 0 || current_offset >= offset_end) { - break; - } - } + if (remaining > 0) { + remaining -= bytes_read; + if (remaining <= 0) { + break; + } + }And remove lines 80 and 116 that reference
current_offset.
216-223: Potential temp file collision in high-throughput scenarios.The temp path uses
pid_timefor uniqueness, but multiple streams created within the same second by the same process will collide. Consider adding a counter or usingmkstemp()for guaranteed uniqueness.♻️ Proposed fix using atomic counter
+static uint64_t stream_counter = 0; + static int stream_context_setup(struct flb_s3 *s3_ctx, const char *input_path, const char *output_suffix, struct stream_context *ctx) { flb_sds_t tmp; + uint64_t counter_val; stream_context_init(ctx); /* ... */ + counter_val = __sync_fetch_and_add(&stream_counter, 1); tmp = flb_sds_printf(&ctx->temp_path, "%s/stream_%d_%lu_%lu%s", - s3_ctx->buffer_dir, (int)getpid(), (unsigned long)time(NULL), output_suffix); + s3_ctx->buffer_dir, (int)getpid(), + (unsigned long)time(NULL), (unsigned long)counter_val, + output_suffix);
363-368: Return value semantics for empty input should be clarified.The function returns
-1for empty input but also sets*out_buf = NULLand*out_size = 0. This conflates "error" with "no data to process." Consider returning0with null output for empty input, or documenting the expected caller behavior.src/flb_parquet_impl.cpp (1)
772-789: Unused parameter in function signature.The
total_file_sizeparameter is marked__attribute__((unused))but is part of the public API. Consider:
- Removing it if not needed
- Or using it for validation/pre-allocation optimizations
- Or documenting why it's reserved for future use
plugins/out_s3/s3_multipart.c (1)
895-961: Document known memory limitation in part upload.The extensive comment (lines 895-918) clearly documents the memory constraint where entire part must be buffered. This is good documentation, but consider also documenting recommended workaround in user-facing configuration documentation.
Consider adding this memory limitation to plugin configuration documentation with recommended
upload_chunk_sizevalues for different memory constraints.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (47)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/kafka.cmakecmake/parquet.cmakeinclude/fluent-bit/aws/flb_aws_compress.hinclude/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/flb_aws_compress.csrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/fuzzers/aws_util_fuzzer.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3.ctests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
💤 Files with no reviewable changes (2)
- src/aws/compression/CMakeLists.txt
- src/aws/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (12)
- tests/internal/CMakeLists.txt
- tests/internal/parquet.c
- plugins/out_s3/s3_auth.h
- tests/runtime/CMakeLists.txt
- tests/internal/aws_util.c
- src/CMakeLists.txt
- include/fluent-bit/flb_parquet.h
- plugins/out_s3/s3_queue.c
- tests/runtime/out_s3.c
- plugins/out_s3/CMakeLists.txt
- tests/runtime/out_s3_queue.c
- include/fluent-bit/aws/flb_aws_compress.h
🧰 Additional context used
🧠 Learnings (15)
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3_auth.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
.github/workflows/unit-tests.yamlcmake/parquet.cmakesrc/flb_parquet_impl.cpp
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/parquet.cmakesrc/flb_parquet_impl.cppcmake/kafka.cmake
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/kafka.cmake
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.ccmake/kafka.cmake
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.csrc/flb_parquet_impl.cppcmake/kafka.cmake
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtplugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
.github/workflows/unit-tests.yamlCMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
.github/workflows/unit-tests.yamlplugins/out_s3/s3_store.cCMakeLists.txtsrc/flb_parquet_impl.cppplugins/out_s3/s3_multipart.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
CMakeLists.txtsrc/aws/flb_aws_compress.c
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_aws_util.h
📚 Learning: 2025-09-04T12:35:36.904Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:3275-3282
Timestamp: 2025-09-04T12:35:36.904Z
Learning: The out_s3 plugin intentionally uses a simple numeric comparison for retry_limit (chunk->failures >= ctx->ins->retry_limit) rather than the standard Fluent Bit pattern that checks for FLB_OUT_RETRY_UNLIMITED (-1). The maintainer wants to keep this current behavior for consistency within the plugin.
Applied to files:
plugins/out_s3/s3_stream.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
src/aws/flb_aws_util.c
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
🧬 Code graph analysis (21)
plugins/out_s3/s3_auth.c (3)
src/flb_utils.c (1)
flb_utils_url_split(1531-1667)src/flb_upstream.c (1)
flb_upstream_create_url(416-481)src/flb_output.c (1)
flb_output_upstream_set(1578-1669)
tests/runtime/out_s3_recovery.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_special_features.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
src/flb_signv4.c (1)
src/flb_sds.c (2)
flb_sds_destroy(389-399)flb_sds_create_len(58-76)
plugins/out_s3/s3_store.h (1)
plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(536-572)
plugins/out_s3/s3_blob.h (1)
plugins/out_s3/s3_blob.c (5)
s3_blob_register_parts(320-353)s3_blob_process_events(451-531)s3_blob_store_chunk(360-436)s3_blob_recover_state(124-143)s3_blob_notify_delivery(77-114)
plugins/out_s3/s3_stream.c (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
tests/internal/fuzzers/aws_util_fuzzer.c (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
plugins/in_blob/blob_file.c (1)
plugins/in_blob/blob_db.c (1)
blob_db_file_insert(164-196)
src/aws/flb_aws_compress.c (2)
src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_zstd.c (1)
flb_zstd_compress(33-57)
tests/runtime/out_s3_error_handling.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (8)
flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
include/fluent-bit/flb_blob_db.h (1)
src/flb_blob_db.c (13)
flb_blob_db_file_fetch_part_ids(1385-1459)flb_blob_db_file_fetch_part_ids(1979-1986)flb_blob_db_file_fetch_part_count(1461-1490)flb_blob_db_file_fetch_part_count(1988-1992)flb_blob_db_file_fetch_all_parts(1492-1642)flb_blob_db_file_get_next_pending(1648-1760)flb_blob_db_file_get_next_pending(1994-2003)flb_blob_db_file_part_check_uploaded(1766-1801)flb_blob_db_file_part_check_uploaded(2005-2010)flb_blob_db_file_parts_in_progress(1004-1039)flb_blob_db_file_parts_in_progress(2012-2017)flb_blob_db_reset_zombie_parts(1804-1828)flb_blob_db_reset_zombie_parts(2019-2022)
plugins/out_s3/s3_multipart.h (1)
plugins/out_s3/s3_multipart.c (12)
flb_s3_calculate_optimal_part_size(157-209)s3_multipart_get_md5_base64(211-232)s3_multipart_create_headers(341-401)s3_multipart_upload_create(434-483)s3_multipart_upload_destroy(403-428)s3_multipart_create(485-547)s3_multipart_upload_part(549-646)s3_multipart_complete(648-735)s3_multipart_abort(737-805)s3_multipart_check_upload_exists(818-893)s3_multipart_upload_file_part(919-1069)s3_multipart_upload_file(1229-1321)
plugins/out_s3/s3_stream.h (1)
plugins/out_s3/s3_stream.c (4)
stream_compress_file(32-147)stream_process_msgpack_file(344-412)stream_json_processor(414-442)stream_log_key_processor(517-546)
tests/runtime/out_s3_edge_cases.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_stop(1011-1055)flb_destroy(228-263)flb_lib_push(843-870)
src/flb_blob_db.c (3)
src/flb_sqldb.c (2)
flb_sqldb_lock(173-178)flb_sqldb_unlock(180-185)include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)lib/cfl/src/cfl_sds.c (2)
cfl_sds_create(113-125)cfl_sds_destroy(127-137)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (5)
cb_s3_upload(1412-1527)s3_format_chunk(1268-1409)s3_upload_file(1087-1192)s3_initiate_multipart_upload(980-1085)mock_s3_call(105-179)
plugins/out_s3/s3_queue.h (1)
plugins/out_s3/s3_queue.c (9)
s3_queue_remove(267-275)s3_queue_buffer_chunk(315-337)s3_queue_add_file(57-111)s3_queue_add_part(113-183)s3_queue_add_pending_file(185-232)s3_queue_process_entry(680-739)s3_queue_recover_all(782-827)s3_queue_recover_from_database(837-1017)s3_queue_recover_from_fstore(1090-1150)
plugins/out_s3/s3_multipart.c (5)
plugins/out_s3/s3.c (2)
write_seq_index(227-247)s3_plugin_under_test(73-79)src/flb_sds.c (6)
flb_sds_create_len(58-76)flb_sds_create(78-90)flb_sds_create_size(92-95)flb_sds_destroy(389-399)flb_sds_printf(336-387)flb_sds_cat(120-141)src/aws/flb_aws_util.c (2)
flb_aws_xml_get_val(623-655)flb_aws_print_xml_error(586-610)src/flb_http_client.c (1)
flb_http_client_destroy(1738-1745)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(359-400)
🔇 Additional comments (78)
plugins/out_kinesis_firehose/firehose.c (1)
495-501: Documentation is accurate and properly aligned with implementation.The compression module fully supports the three types documented in firehose.c (lines 495-501): gzip, snappy, and zstd. The
flb_aws_compression_get_type()function validates against the compression_options array, which contains only these three compression algorithms. The removal of 'arrow' from the supported list is intentional and well-documented in both the header and implementation files—Arrow is explicitly deprecated and no longer available in the compression module.src/flb_signv4.c (1)
646-677: Implementation correctly handles URI encoding for both normalized and pre-encoded paths.The code properly distinguishes between AWS services requiring URI normalization (normalize_uri=TRUE) and S3's pre-encoded URI requirement (normalize_uri=FALSE). Memory management is sound: temporary buffers are cleaned up after use, error paths properly destroy the canonical request buffer before returning NULL, and the URI buffer is freed after concatenation.
.github/workflows/unit-tests.yaml (2)
56-56: LGTM - Consistent rename from Arrow to Parquet encoder.The matrix option and exclusion rule are consistently updated from
-DFLB_ARROW=Onto-DFLB_PARQUET_ENCODER=On. The clang exclusion is appropriately maintained.Also applies to: 72-75
117-125: Package selection verified: The workflow correctly installslibarrow-devandlibparquet-dev, which provide the C++ API. This is the appropriate choice if the Parquet encoder uses the C++ API directly rather than the GLib bindings (libarrow-glib-dev,libparquet-glib-dev).src/aws/flb_aws_compress.c (3)
31-37: LGTM - Snappy wrapper correctly adapts the signature.The wrapper adapts
flb_snappy_compress'schar*signature to the genericvoid*interface used by other compression functions (gzip, zstd). The casts are appropriate.
85-87: Good optimization - early return for "none" compression.This short-circuits the loop iteration when no compression is requested, which is a common case.
45-56: Documentation clarifies the distinction between compression and format conversion.The updated comments clearly explain that Arrow has been removed and Parquet should now be configured via
format=parquetrather than as a compression option. This is helpful for maintainers.tests/runtime/out_s3_recovery.c (1)
429-439: LGTM - Test list properly registered.All eight test functions are correctly registered in the TEST_LIST with appropriate names and NULL terminator.
tests/runtime/out_s3_config.c (2)
690-789: Good negative testing for invalid configurations.The test properly validates that invalid configurations fail during initialization:
- Invalid compression type
- Invalid format type
- Parquet without schema (guarded by
FLB_HAVE_PARQUET_ENCODER)- Total file size exceeding AWS 5TB limit
Each case correctly checks that
flb_start()returns-1.
792-809: LGTM - Comprehensive test list covering S3 configuration scenarios.The TEST_LIST includes 15 tests covering key format variations, endpoint configurations, storage options, and invalid parameter handling.
tests/runtime/out_s3_special_features.c (4)
27-62: LGTM - Snappy compression test aligns with new compression option.This test validates the new
FLB_AWS_COMPRESS_SNAPPYoption added insrc/aws/flb_aws_compress.c.
265-297: Good defensive coding pattern.The conditional
flb_stop()at lines 292-294 prevents calling stop on a context that failed to start. This pattern should be consistent across tests expecting failures.
339-379: Good boundary condition test.Testing when
upload_chunk_sizeequalstotal_file_sizevalidates that the plugin handles the equality case without unnecessary adjustments.
506-520: LGTM - Test list properly registered.All 12 special feature tests are correctly registered.
tests/runtime/out_s3_error_handling.c (3)
27-43: LGTM - AWS error response templates correctly defined.The XML error templates match AWS S3 error response format and cover key error scenarios: AccessDenied, NoSuchBucket, NoSuchUpload, InvalidAccessKeyId, SignatureDoesNotMatch, and SlowDown.
359-420: Good coverage of auto_retry_requests flag.The test validates both enabled and disabled states for the
auto_retry_requestsconfiguration option.
542-556: LGTM - Comprehensive error handling test suite.The TEST_LIST covers 12 error scenarios including AWS errors, network timeouts, retry layers, and cleanup behavior.
plugins/out_s3/s3_queue.h (3)
32-43: Good documentation for dual-mode API.The comment clearly explains the
file_idparameter semantics:
file_id == 0: Without DB tracking (chunks)file_id > 0: File-level DB tracking (legacy mode)This helps maintainers understand the backward compatibility design.
71-79: Well-documented three-phase recovery architecture.The comment clearly explains the recovery phases:
- Cleanup dirty states (zombie parts, locked chunks)
- State transitions (stale → pending, aborted → pending/delete)
- Queue rebuild (scan storage and enqueue all pending files)
This provides valuable context for maintainers.
20-85: LGTM - Clean header design with appropriate include guard.The header provides a well-organized public interface for the S3 queue subsystem with proper documentation for the key functions.
tests/runtime/out_s3_edge_cases.c (2)
30-68: Test checks no-crash scenario correctly.The test validates that the S3 plugin handles empty flushes without crashing, which is a valid edge case.
157-241: LGTM!The timeout tests are well-structured, especially
flb_test_timeout_trigger_priority, which includes clear documentation explaining that the 1s timeout should trigger before the 100M file size is reached.plugins/out_s3/s3.h (4)
29-46: LGTM!The AWS S3 multipart constants correctly reflect the AWS service limits (5 MiB minimum part size, 5 GiB maximum part size, 10,000 maximum parts, 5 TB maximum object size). The multipart error codes follow a clear convention with negative values for error cases.
48-70: Well-designed upload queue structure.The
upload_queuestruct comprehensively captures all necessary context for tracking multipart upload operations, including retry state, file offsets, and S3-specific identifiers. The structure supports both file-level and part-level granularity.
176-198: LGTM!The format constants and function prototypes provide a clean API surface. The use of output parameters (
out_buf,out_size) ins3_format_chunkfollows established patterns in the codebase, and the mock function is appropriately exposed for testing purposes.
90-174: Arrow schema lifecycle is properly managed.The
cached_arrow_schemapointer is correctly handled:
- Allocated once during initialization via
flb_parquet_schema_create()- Properly freed in
s3_context_destroy()with null-check before callingflb_parquet_schema_destroy()schema_stris immutable after configuration (config-time parameter, no runtime modifications)- Thread-safe since the schema is created once during single-threaded init and accessed read-only in worker threads
No changes needed.
plugins/out_s3/s3_multipart.h (1)
20-104: Well-designed multipart upload API.The header provides a clean, layered API surface:
- Utilities: Part size calculation, MD5, header creation
- Lifecycle: Create/destroy multipart upload contexts
- AWS Operations: Direct wrappers for S3 multipart APIs
- High-level: File-based streaming uploads
The extensive documentation on
s3_multipart_check_upload_exists(lines 72-80) correctly warns about s3_key regeneration during recovery, which is critical for correctness.tests/internal/fuzzers/aws_util_fuzzer.c (1)
73-73: LGTM!The addition of
NULLas the sixth parameter correctly updates the fuzzer to match the expandedflb_get_s3_keyAPI signature. PassingNULLfor thefile_pathparameter is appropriate in the fuzzing context where file-specific key generation is not relevant.plugins/in_blob/blob_file.c (1)
79-89: LGTM!Guarding the database insertion with
if (ctx->database_file)correctly prevents DB operations when the database is not configured for this context. This change is consistent with the earlier existence check (lines 48-53) and maintains proper error handling within the guarded block.include/fluent-bit/flb_aws_util.h (1)
200-207: LGTM! Clear documentation and backward-compatible API extension.The added
file_pathparameter is well-documented as optional (can be NULL), and the implementation inflb_aws_util.chandles NULL gracefully by returning early. The documentation clearly enumerates all supported template variables.plugins/out_s3/s3_stream.h (1)
20-111: LGTM! Well-designed streaming API with clear documentation.The callback-based processing model (
record_processor_fn) provides good extensibility for different output formats. Thestream_processor_contextstruct appropriately tracks progress metrics. Documentation clearly explains parameters including the-1sentinel for EOF inoffset_end.plugins/out_s3/s3_auth.c (3)
36-77: LGTM! Clean URL parsing with proper resource cleanup.The
url_partsstruct and associated functions provide clean URL handling with proper memory management. Theparse_urlfunction correctly validates required components and cleans up on error.
79-128: LGTM! Proper TLS context and upstream initialization with cleanup on error paths.The endpoint initialization correctly handles both HTTP and HTTPS endpoints, creates TLS context only when needed, and properly destroys resources on failure.
262-357: LGTM! URL path construction with proper encoding and validation.The function correctly URL-encodes path components, validates required parameters for each URL type, and has proper cleanup in both success and error paths.
plugins/out_s3/s3_blob.h (1)
20-52: LGTM! Clean API declarations with helpful documentation comments.The header provides a well-organized public interface for S3 blob handling, including lifecycle management (register, store, recover) and notification callbacks. The declarations align with the implementations in
s3_blob.c.CMakeLists.txt (3)
1337-1355: LGTM! Proper Parquet encoder integration with graceful fallback.The configuration correctly enables C++17 (required by Arrow), uses the dedicated detection module, and gracefully disables Parquet support if libraries are not found rather than failing the build.
202-202: LGTM! Parquet encoder option follows established patterns.The new
FLB_PARQUET_ENCODERoption is correctly defined with a descriptive message and defaults to Off, consistent with other optional encoder features.
284-288: Good decision to exclude Parquet from FLB_ALL.The comment clearly explains that
FLB_PARQUET_ENCODERrequires external dependencies (Arrow/Parquet libraries) and must be explicitly enabled. This prevents build failures for users who just want to enable all built-in features.cmake/kafka.cmake (1)
9-51: Good documentation of librdkafka's built-in SASL capabilities.The comments clearly explain that PLAIN, SCRAM, and OAUTHBEARER are built into librdkafka while only GSSAPI (Kerberos) requires the external cyrus-sasl library. The
FLB_SASL_CYRUS_ENABLEDflag correctly gates only the GSSAPI support.plugins/out_s3/s3_store.h (2)
26-34: LGTM! Clean struct definition with consistent naming.The
stream_pathrename aligns with the streaming workflow introduced in this PR. The struct maintains clear ownership semantics with theflb_sds_ttype.
51-52: Good design: Context-free chunk file opener.The
flb_chunk_file_openfunction is appropriately designed as context-free, enabling safe usage from both C and C++ code paths (including the new Parquet encoder). This supports the broader Parquet integration goal.tests/runtime/out_s3_format.c (4)
27-62: LGTM! Well-structured test for basic JSON format.The test properly sets up the test environment, configures the S3 output with appropriate parameters, and cleans up resources including the environment variable.
144-208: Good coverage of Parquet schema validation scenarios.The test appropriately validates both valid and invalid schema configurations, with the expectation that invalid JSON schema causes
flb_startto fail. The conditional compilation withFLB_HAVE_PARQUET_ENCODERis correct.
438-441: Environment variable cleanup inside Test 1 block — intentional or misplaced?The
unsetenv("FLB_S3_PLUGIN_UNDER_TEST")on line 440 occurs at the end of Test 1 (JSON + GZIP), but the environment variable is set again at line 412 (start of the function). This means Tests 2-4 run without the test mode flag set until it's re-enabled implicitly.If this is intentional (each sub-test should independently set the env var), consider adding
setenvcalls at the start of each sub-test for clarity. Otherwise, move theunsetenvto the end of the function.
532-547: Well-organized TEST_LIST with proper conditional compilation.The test registration correctly gates Parquet tests behind
FLB_HAVE_PARQUET_ENCODERand maintains a clean null-terminated list.plugins/out_s3/s3_store.c (5)
35-42: Good CI environment detection coverage.The function checks common CI environment variables (CI, TRAVIS, GITHUB_ACTIONS, GITLAB_CI, JENKINS_URL) for automatic backend selection. This enables proper testing in CI environments.
109-133: Buffer space check logic is correct.The threshold calculation properly warns when buffer usage exceeds 95% (BUFFER_WARNING_THRESHOLD). The floating-point comparison
(1.0 - BUFFER_WARNING_THRESHOLD)equals 0.05, so the warning triggers when less than 5% space remains.
371-383: Time format differs between Windows and POSIX — verify this is intentional.Windows uses
%H-%M-%S(hyphens) while POSIX uses%H:%M:%S(colons). This may be intentional since colons are invalid in Windows filenames, but the stream name is used for directory creation which should support colons on POSIX systems.If intentional, consider adding a brief comment explaining the difference.
536-572: Robust chunk file validation with proper error handling.The
flb_chunk_file_openfunction correctly validates CIO magic bytes, reads the header, extracts metadata length using big-endian byte order, and positions the file pointer past the metadata. Error paths properly close the file handle.
489-507: Good cleanup in s3_store_file_inactive.The function properly frees
stream_pathbefore freeing the struct, preventing memory leaks. The null check at the start is appropriate defensive programming.src/aws/flb_aws_util.c (3)
210-226: Good addition: Immediate retry with refreshed credentials.The auth retry logic properly handles credential expiration during normal operation (e.g., IAM role rotation). The
auth_retry_doneflag prevents infinite retry loops, and the comment clearly explains the rationale.
1106-1116: Path stripping may be too aggressive for relative paths.The while loop strips all leading dots and slashes, which would transform valid relative paths:
../logs/app.log→logs/app.log(parent reference lost)./current/file.log→current/file.log(acceptable)Consider preserving path semantics or documenting this behavior. If users expect
$FILE_PATHto match the original path structure, this could cause unexpected S3 key construction.
851-860: Well-structured template processing function.The refactored
process_s3_key_templateconsolidates common S3 key variable handling ($TAG, $UUID, $INDEX, time formatting) into a reusable function, improving code maintainability.plugins/out_s3/s3_blob.c (3)
228-247: Excellent documentation of known limitation.The comment clearly explains the trade-off with time-based S3 key formats during recovery, and provides guidance to users about using static key formats for reliable recovery. This is good defensive documentation.
116-143: Well-structured recovery state machine.The phased recovery approach (stale → pending, aborted → pending/delete) with proper database locking ensures consistent state transitions. The debug logging aids troubleshooting.
438-451: Good architectural note about coroutine constraints.The comment clearly explains why heavy operations are deferred to timer callbacks, which is important context for future maintainers working in the flush coroutine path.
tests/runtime/out_s3_multipart.c (2)
27-63: LGTM! Basic multipart upload test structure.The test properly configures upload_chunk_size separately from total_file_size, which is the correct pattern for triggering multipart uploads.
556-571: Well-organized TEST_LIST covering multipart lifecycle.The test list provides good coverage of the multipart upload lifecycle: creation, part uploads, completion (with retry scenarios), abort paths, persistence, and part size calculation.
plugins/out_s3/s3_stream.c (3)
32-147: Solid streaming compression implementation.The function correctly handles offset-based reads, supports multiple compression algorithms, and properly cleans up resources including removing the output file on failure. The chunked processing approach is memory-efficient.
414-442: Clean JSON processor implementation.The processor correctly converts msgpack records to JSON, writes with newline delimiters, and tracks bytes written. Memory cleanup is handled properly.
517-546: Good log_key extraction with fallback to JSON.The processor properly handles string, binary, and other value types, with JSON serialization as a fallback for complex types. This provides flexibility for various log formats.
src/flb_parquet_impl.cpp (6)
67-101: Robust overflow prevention in string-to-number parsing.The implementation correctly enforces target type bounds before casting to avoid silent overflow, and properly handles
errnofor range errors. This is a strong defensive programming practice.
116-353: Well-structured type conversion with robust fallback handling.The
MsgpackToArrowConverterclass provides comprehensive type-safe conversions with proper clamping and logging for out-of-range values. The fallback handling for non-nullable fields (lines 908-918 in the caller) ensures data integrity.
502-717: Comprehensive Parquet validation function.The
validate_parquet_filefunction provides thorough validation of Parquet files with proper error handling, type checking, and value comparison. The exception handling covers bothparquet::ParquetExceptionand general exceptions.
719-770: Well-designed schema API with proper lifecycle management.The schema validation and lifecycle functions properly encapsulate Arrow schema objects behind an opaque C API handle, with consistent error reporting and cleanup paths.
999-1087: Excellent resource cleanup discipline in error paths.The streaming conversion function demonstrates robust cleanup with immediate flag updates after each resource release (lines 999-1010, 1069-1085), preventing double-free and ensuring consistent state in both success and error paths.
43-45: No action required—external functions are properly available in the codebase.The declarations for
flb_msgpack_to_json_strandflb_chunk_file_openat lines 43-45 are valid. Both functions are properly defined in the codebase (src/flb_pack.candplugins/out_s3/s3_store.crespectively) with matching signatures, and both are widely used throughout the project, confirming successful linking.Likely an incorrect or invalid review comment.
src/flb_blob_db.c (4)
215-254: Consistent prepared statement additions.The four new prepared statements follow the established pattern with proper error handling and specific error codes for each statement type.
1540-1597: Exemplary all-or-nothing reallocation strategy.The array expansion logic correctly handles partial reallocation failures by:
- Attempting all reallocations first (lines 1547-1558)
- Checking if all succeeded (line 1560)
- Freeing only successfully reallocated pointers that differ from originals (lines 1566-1577)
- Then cleaning up original arrays (lines 1579-1583)
This prevents double-free and memory leaks.
1648-1760: Robust pending file retrieval with proper cleanup.The function correctly handles all allocation failures, resets the prepared statement in all code paths (success, no data, and error), and only sets
context->last_erroron actual errors (line 1715), not on "no more rows" conditions.
1263-1383: Backward-compatible parameter addition.The addition of the
file_createdoutput parameter is implemented consistently with existing patterns, properly initialized, and cleaned up on error paths.include/fluent-bit/flb_blob_db.h (2)
126-216: Well-structured SQL macro additions.The new SQL query definitions follow the established pattern with clear, readable formatting and proper use of placeholders for bind parameters.
314-324: Consistent error code additions.The new error codes maintain the sequential numbering pattern and correctly update the
FLB_BLOB_DB_ERROR_PREPARING_STATEMENT_TOPsentinel (line 324).plugins/out_s3/s3_multipart.c (5)
36-64: Clean cross-platform file I/O abstraction.The platform-specific macros provide a unified API across Windows and POSIX systems, properly mapping to the corresponding system calls with appropriate 64-bit support on Windows.
Based on learnings, Fluent Bit only supports MSVC on Windows, so additional compiler guards are unnecessary.
105-141: Safe ETag extraction with bounds checking.The function properly validates input, handles missing ETag header, and performs bounds checking during string scanning.
143-209: Well-documented part size calculation with AWS constraint handling.The function correctly handles AWS S3's multipart constraints (5 MiB minimum, 5 GiB maximum per part, 10000 parts maximum) with clear documentation of the behavior for small files.
1110-1227: Robust part upload with exponential backoff and retry.The function implements proper retry logic with:
- Exponential backoff capped at 30 seconds
- Overflow prevention in exponent calculation (lines 1196-1210)
- Per-attempt presigned URL fetching with separate retry
- Clear logging of retry attempts
1229-1321: Complete multipart upload orchestration with proper cleanup.The function correctly orchestrates the entire multipart upload lifecycle with:
- Proper error handling at each stage
- Abort on failures to avoid orphaned uploads
- Size validation before completion
- Consistent cleanup via
s3_multipart_upload_destroy- Appropriate return codes (FLB_RETRY vs FLB_OK)
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Add Parquet format support to S3 output plugin with: - Parquet writer integration - Enhanced multipart upload handling - Improved queue and store management - Stream processing updates - Configuration examples Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
Add test coverage for: - AWS utilities - Parquet implementation - S3 output configuration - Edge cases handling - Error handling - Format conversion - Multipart uploads - Queue management - Recovery mechanisms - Special features Signed-off-by: Arbin Cheng <[email protected]> Signed-off-by: Arbin <[email protected]>
kalavt
force-pushed
the
feature/aws-s3-parquet
branch
from
d8aa149 to
dd833c9
Compare
There was a problem hiding this comment.
Actionable comments posted: 20
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🤖 Fix all issues with AI agents
In @include/fluent-bit/aws/flb_aws_compress.h:
- Around line 24-46: The switch in src/flb_parquet_impl.cpp that maps the
integer variable compression to parquet_compression uses hardcoded numeric case
labels (1,2,3) which is fragile; update the switch to use the FLB_AWS_COMPRESS_*
symbolic constants (FLB_AWS_COMPRESS_GZIP, FLB_AWS_COMPRESS_SNAPPY,
FLB_AWS_COMPRESS_ZSTD) for the GZIP/SNAPPY/ZSTD branches and keep the default
mapping to parquet::Compression::UNCOMPRESSED so the code reads the documented
enum names instead of raw numbers.
In @include/fluent-bit/flb_blob_db.h:
- Around line 140-145: The SQL_RESET_FILE_PART_UPLOAD_STATES update currently
resets delivery_attempts and uploaded but misses clearing in_progress, which can
leave parts stranded; modify the SQL statement identified by
SQL_RESET_FILE_PART_UPLOAD_STATES to also set in_progress=0 in the UPDATE of
blob_parts (alongside delivery_attempts=0 and uploaded=0 WHERE file_id=@id) so
parts can be picked up again.
In @plugins/out_s3/s3_multipart.c:
- Around line 818-893: The function s3_multipart_check_upload_exists incorrectly
calls flb_sds_destroy(s3_key) on a const char* it does not own and returns 0 for
many non-200 responses (contradicting the doc that network/API errors should
return -1). Remove all flb_sds_destroy(s3_key) calls in
s3_multipart_check_upload_exists so the function does not free the caller-owned
s3_key, and change the error-path logic so that unexpected ListParts failures
(non-200 and not matching "<Code>NoSuchUpload</Code>") return -1 (keep returning
1 for 200 and 0 for explicit NoSuchUpload). Ensure flb_http_client_destroy(c)
and other owned resources are still cleaned up before returning.
- Around line 105-141: The extract_etag function risks UB and fragile parsing:
change index variables i, start, end to size_t, validate that tmp (from
strstr(response, "ETag:")) lies within the provided size, and replace
isspace(response[i]) with isspace((unsigned char) response[i]) everywhere;
tighten bounds checks when advancing i to avoid overruns, explicitly skip the
literal "ETag:" then any ASCII whitespace and an optional opening quote before
capturing the token, and pass the computed length safely into flb_sds_create_len
(and check tmp != NULL earlier) so extraction works for negative-char platforms
and arbitrary header spacing.
- Around line 285-339: The ETag quotes are being stripped earlier so
build_complete_multipart_xml emits unquoted <ETag> elements; update
extract_etag() to preserve the surrounding double-quote characters instead of
skipping them (remove or adjust the loop that skips '"' chars and any trimming
that drops leading/trailing quotes) so that m_upload->etags[] contains the exact
quoted ETag returned by UploadPart; no changes to build_complete_multipart_xml
are necessary beyond relying on the preserved values, but keep the existing NULL
checks in build_complete_multipart_xml (and in the etag extraction ensure you
still allocate/copy the full quoted string and handle memory failure cases).
- Around line 1217-1309: The error paths call s3_multipart_abort(ctx, m_upload,
NULL) without a presigned URL; update each abort call in
s3_multipart_upload_file to first call s3_auth_fetch_presigned_url(ctx,
&pre_signed_url, S3_PRESIGNED_URL_ABORT_MULTIPART, m_upload->s3_key,
m_upload->upload_id, 0), check its return and log on failure, then call
s3_multipart_abort(ctx, m_upload, pre_signed_url) and ensure pre_signed_url is
flb_sds_destroy'd after use; apply this for the aborts after
s3_multipart_upload_file_parts failure, after the size mismatch check, and after
failed complete presigned-url fetch, reusing the local pre_signed_url variable
and preserving existing cleanup/error returns.
In @plugins/out_s3/s3_store.c:
- Around line 29-33: Remove the local duplicate defines (CIO_FILE_HEADER_SIZE,
CIO_MAGIC_BYTE_1, CIO_MAGIC_BYTE_2, CIO_META_LENGTH_OFFSET) and instead include
the chunkio header that defines the canonical constants; replace usages with
CIO_FILE_HEADER_MIN, CIO_FILE_ID_00, CIO_FILE_ID_01, and CIO_FILE_CONTENT_OFFSET
from chunkio/cio_file_st.h so the code uses the upstream definitions and avoids
drift; keep BUFFER_WARNING_THRESHOLD as-is.
In @plugins/out_s3/s3_stream.c:
- Around line 344-412: The function stream_process_msgpack_file currently treats
empty input_size and zero temp_stat.st_size as errors by returning -1; change
both branches (the input_size == 0 early-exit and the temp_stat.st_size == 0
path) to return 0 (success/no-op) while still setting *out_buf = NULL and
*out_size = 0 and performing the same cleanup (stream_context_destroy as
currently used). Update the two return sites inside stream_process_msgpack_file
so callers won’t treat skipped/empty files as failures or retryable errors.
- Around line 61-66: Validate offset_start and offset_end before
seeking/reading: ensure offset_start >= 0, offset_end >= 0 (if used), and if
offset_end > 0 then offset_end > offset_start; if this condition fails, log an
error via flb_plg_error (same logger used in this file, e.g., ctx->ins) and goto
cleanup instead of proceeding. Also ensure calculations of remaining (offset_end
- offset_start) use unsigned/64-bit types consistent with file offsets to avoid
underflow/overflow and check remaining > 0 before entering the read/compress
loop (the variables referenced are offset_start, offset_end, remaining and the
read loop in s3_stream.c).
- Around line 20-31: The file s3_stream.c is missing standard headers for
PATH_MAX and unlink, causing implicit-declaration/portability issues; add
explicit includes for <limits.h> and <unistd.h> alongside the existing includes
at the top of s3_stream.c so symbols like PATH_MAX and the unlink() call (used
where temporary file paths are built/removed) are properly declared; update the
include block near the top of the file (affecting the code areas that reference
PATH_MAX and unlink) to resolve the warnings/errors.
- Around line 251-314: The loop in process_unpacked_records does not handle
MSGPACK_UNPACK_NOMEM_ERROR and can spin indefinitely; update
process_unpacked_records to check for MSGPACK_UNPACK_NOMEM_ERROR (value from
msgpack_unpacker_next) before the existing SUCCESS/PARSE_ERROR/EXTRA_BYTES
branches, log an error via flb_plg_error including the msgpack error context (as
done in plugins/out_forward/forward.c), and return -1 to abort processing on
NOMEM errors. Ensure the check references process_unpacked_records and
msgpack_unpacker_next/msgpack_unpack_return so it’s clear where to add the
handling.
In @src/flb_blob_db.c:
- Around line 1492-1571: flb_blob_db_file_fetch_all_parts() computes total_count
before locking so concurrent changes can make idx exceed allocated arrays;
modify the fetch loop that uses statement (sqlite3_step) to guard against
overruns by adding idx < total_count to the loop condition (while ((result =
sqlite3_step(statement)) == SQLITE_ROW && idx < total_count) ) and after the
loop detect if result == SQLITE_ROW (meaning more rows remained) then
clear/reset/unlock, free the allocated arrays, set context->last_error to an
appropriate error code and return an error; this prevents out-of-bounds writes
to *part_db_ids, *part_nums, *offset_starts, *offset_ends without requiring
broader lock changes.
In @src/flb_parquet_impl.cpp:
- Around line 338-401: convert_to_timestamp may overflow when multiplying input
values by scale_factor; before multiplying in convert_to_timestamp (for
MSGPACK_OBJECT_POSITIVE_INTEGER, MSGPACK_OBJECT_NEGATIVE_INTEGER, floats and
parsed strings) check for overflow and clamp or return an error. Specifically,
use safe checks against std::numeric_limits<int64_t>::max()/min(): for unsigned
inputs (obj->via.u64) ensure obj->via.u64 <= (uint64_t)(INT64_MAX /
scale_factor) before casting/multiplying; for signed negatives check
obj->via.i64 >= (INT64_MIN / scale_factor); for doubles and parsed string
doubles check the scaled double is within [INT64_MIN, INT64_MAX] before casting;
on overflow return an arrow::Status::Invalid or OutOfRange with a clear message
rather than performing the multiplication. Ensure all switch branches
(POSITIVE_INTEGER, NEGATIVE_INTEGER, FLOAT32/FLOAT64, STR) apply these checks
prior to builder->Append.
- Around line 115-126: The parse_string_to_bool function uses ::tolower on char
values which is undefined for negative chars; update the std::transform call in
parse_string_to_bool to cast each character to unsigned char before passing to
::tolower (e.g., use a lambda or static_cast<unsigned char> in the transform) so
all calls to ::tolower receive unsigned char values and avoid UB while
preserving the current logic and return values.
- Around line 787-803: flb_parquet_validate_schema currently calls
parse_schema_from_json without guarding against a NULL schema_str which causes a
crash; add a NULL-check at the start of flb_parquet_validate_schema (before
calling parse_schema_from_json) that if schema_str is NULL writes a clear
message into error_msg (when error_msg && error_msg_size > 0) using snprintf and
then returns -1; keep the existing behavior for non-NULL inputs and still call
parse_schema_from_json as before.
In @tests/runtime/out_s3_multipart.c:
- Around line 430-493: The test hard-codes
"/tmp/test_s3_multipart_persistence.db", which causes collisions in parallel
runs; change flb_test_upload_id_persistence to create a unique temporary
filepath (e.g., use mkstemp or tmpfile to generate a unique name), use that
generated path in both flb_output_set calls for "blob_database_file" inside the
function, and at the end of the test ensure the temp file is removed (unlink the
path) before returning; keep all changes local to flb_test_upload_id_persistence
and reference the existing occurrences of "blob_database_file" and the function
name when editing.
- Around line 34-63: The test leaves environment and a fixed persistence DB path
(/tmp/test_s3_multipart_persistence.db) around on early failures; change tests
to create a unique per-test persistence path (mkstemp/mkstemp-style or tmpnam
then snprintf into a char persistence_path[]) and set that via setenv before
starting, and replace the hardcoded path with that variable when configuring the
plugin; add a single cleanup: label (or test-framework teardown) that always
runs to unsetenv("FLB_S3_PLUGIN_UNDER_TEST") and unlink(persistence_path) and
jump to it on any early return/after each TEST_CHECK failure so env vars and
temp files are always removed—apply this pattern to all analogous blocks (the
other ranges noted in the comment).
- Around line 27-63: The test flb_test_create_multipart_basic uses sleep(2)
after flb_lib_push which causes flakiness; replace the blind sleep calls with
condition-based polling using the provided wait_for_file() helper from
flb_tests_runtime.h.in (or equivalent) and add explicit assertions that verify
multipart behavior (e.g., that an upload ID was created, parts were tracked,
completion callback was invoked, or the persistence DB contains expected
entries) by polling the mock S3 artifacts or persistence file instead of
sleeping; also remove hard-coded "/tmp/test_s3_multipart_persistence.db" usage
and create a unique temporary directory/file per test (use mkdtemp/tmpfile or
test framework temp helper) so persistence state is isolated between tests
(update code referencing that literal path to use the temp filename).
🟡 Minor comments (11)
plugins/out_kinesis_firehose/firehose.c-499-499 (1)
499-499: Add 'none' to the documented compression values.The documentation lists only 'gzip', 'snappy', 'zstd', but the implementation also supports 'none' as an explicit compression value (see
flb_aws_compression_get_type()in src/aws/flb_aws_compress.c line 85). Update the documentation to include 'none' in the supported values list, or clarify that users can explicitly setcompression=noneto disable compression.tests/runtime/out_s3_recovery.c-101-163 (1)
101-163: Same redundantunsetenvpattern inflb_test_recovery_without_db.Lines 136 and 162 both unset the environment variable. Consider fixing for consistency with the first test.
plugins/out_s3/s3_auth.c-155-171 (1)
155-171: Return values fromflb_http_add_headerand auth functions are not checked.The header setup function ignores return values from
flb_http_add_header,flb_http_basic_auth, andflb_http_bearer_auth. These can fail on allocation errors.🔧 Suggested fix to check return values
static int setup_http_client_headers(struct flb_s3 *ctx, struct flb_http_client *client) { - flb_http_add_header(client, "Accept", 6, "text/plain", 10); - flb_http_add_header(client, "User-Agent", 10, "Fluent-Bit", 10); + int ret; + + ret = flb_http_add_header(client, "Accept", 6, "text/plain", 10); + if (ret == -1) { + return -1; + } + ret = flb_http_add_header(client, "User-Agent", 10, "Fluent-Bit", 10); + if (ret == -1) { + return -1; + } if (ctx->authorization_endpoint_username && ctx->authorization_endpoint_password) { - flb_http_basic_auth(client, ctx->authorization_endpoint_username, - ctx->authorization_endpoint_password); + ret = flb_http_basic_auth(client, ctx->authorization_endpoint_username, + ctx->authorization_endpoint_password); + if (ret == -1) { + return -1; + } } else if (ctx->authorization_endpoint_bearer_token) { - flb_http_bearer_auth(client, ctx->authorization_endpoint_bearer_token); + ret = flb_http_bearer_auth(client, ctx->authorization_endpoint_bearer_token); + if (ret == -1) { + return -1; + } } return 0; }plugins/out_s3/s3_store.c-363-386 (1)
363-386: Windows uses different time format separators than POSIX.The Windows branch uses
%H-%M-%S(hyphens) while POSIX uses%H:%M:%S(colons). This creates inconsistent stream naming across platforms.💡 Consider using a consistent format
If colons are problematic on Windows filesystems, consider using the hyphenated format for both platforms for consistency:
- strftime(tmp, sizeof(tmp) - 1, "%Y-%m-%dT%H:%M:%S", &tm_buf); + strftime(tmp, sizeof(tmp) - 1, "%Y-%m-%dT%H-%M-%S", &tm_buf);tests/runtime/out_s3_recovery.c-29-99 (1)
29-99: Testflb_test_recovery_with_blob_dbhas a redundantunsetenvcall.The environment variable is unset twice at lines 68 and 95. The first call should be removed since the test continues with a second run.
🔧 Suggested fix
/* Shutdown without completing upload */ flb_stop(ctx); flb_destroy(ctx); - unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); /* Second run: Recovery should pick up buffered data */ + setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); ctx = flb_create();Alternatively, add
setenvbefore the second run to ensure consistent test state.plugins/out_s3/s3.h-176-198 (1)
176-198: Small API polish: usevoidin no-arg prototypes.E.g.
int s3_plugin_under_test(void);(and similarly for other no-arg functions) to avoid old-style C prototypes.plugins/out_s3/s3_queue.h-23-84 (1)
23-84: Public header should be self-sufficient about fundamental types.If
plugins/out_s3/s3.hdoesn’t already guaranteestdint.h/sys/types.h/time.h, consider including them here (or documenting the dependency) since this header exposesuint64_t,off_t, andtime_t.src/flb_blob_db.c-1004-1039 (1)
1004-1039: Error code mismatch: batch in-progress update shouldn’t report “single part” error.On failure,
flb_blob_db_file_parts_in_progress()returnsFLB_BLOB_DB_ERROR_FILE_PART_IN_PROGRESS_UPDATE, which is semantically forSQL_UPDATE_FILE_PART_IN_PROGRESS. Consider a dedicated executing-statement error (or at least a different constant) to make debugging clearer.plugins/out_s3/s3_multipart.c-143-209 (1)
143-209:flb_s3_calculate_optimal_part_size()is OK, but the “small file optimization” comment is misleading.It always clamps to
S3_AWS_MIN_PART_SIZE, so callers shouldn’t interpret it as “use file_size as a part size” for small files.plugins/out_s3/s3_multipart.c-434-483 (1)
434-483:seq_indexupdate needs rollback on key generation failure, and lacks synchronization for concurrent uploads.The
seq_indexis incremented and persisted at line 453, but ifflb_get_s3_keyfails at line 464, the function returns without rolling backseq_index. This permanently skips an index number. Additionally,ctx->seq_index++at line 452 has no synchronization protection—timer callbacks can invokes3_initiate_multipart_upload(s3.c:998) ands3_multipart_upload_createconcurrently, and there is no mutex guardingseq_indexaccess (onlyupload_queue_lockexists, which protects the queue, notseq_index).plugins/out_s3/s3_multipart.c-648-735 (1)
648-735: Consider aborting multipart upload on CompleteMultipartUpload failure.The code explicitly handles
NoSuchUpload(lines 708-712), but whenCompleteMultipartUploadfails with other errors (network timeouts, server errors), the function returns an error without callings3_multipart_abort(). This leaves the multipart upload open on S3, risking storage leaks until AWS lifecycle policies clean it up (typically 7 days). Ins3_multipart_upload_file(), the FLB_RETRY response means the same upload ID persists across retries without abort. This differs from earlier error paths (lines 1267, 1275, 1288) which calls3_multipart_abort()on failures. Consider adding abort on failure for definitive errors or implement a retry-with-abort strategy for unknown outcomes.
🧹 Nitpick comments (17)
tests/runtime/out_s3_special_features.c (1)
187-218: Consider using a safer format specifier for time_t.The cast
(int)time(NULL)could truncate on 64-bit systems after 2038. While acceptable for test uniqueness, consider using%ldwith(long)time(NULL)for better portability.Suggested improvement
- snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%d.db", (int)time(NULL)); + snprintf(db_path, sizeof(db_path), "/tmp/s3_blob_test_%ld.db", (long)time(NULL));tests/internal/parquet.c (1)
1922-1924: Consider using standard macros for special float values.The division-by-zero approach works but may generate compiler warnings. Consider using
NANandINFINITYmacros from<math.h>for clearer intent.Alternative using standard macros
+#include <math.h> + - const double nan_val = 0.0 / 0.0; - const double inf_val = 1.0 / 0.0; - const double neg_inf_val = -1.0 / 0.0; + const double nan_val = NAN; + const double inf_val = INFINITY; + const double neg_inf_val = -INFINITY;cmake/kafka.cmake (1)
52-65: Dead code: SASL disabled branch will never execute.Since
FLB_SASL_ENABLEDis hardcoded toONat line 38, the else branch at lines 56-58 is unreachable.♻️ Consider removing the dead branch
# Export compile-time definitions using FLB_DEFINITION macro -if(FLB_SASL_ENABLED) - FLB_DEFINITION(FLB_HAVE_KAFKA_SASL) - message(STATUS "Kafka SASL authentication: ENABLED") -else() - message(STATUS "Kafka SASL authentication: DISABLED") -endif() +FLB_DEFINITION(FLB_HAVE_KAFKA_SASL) +message(STATUS "Kafka SASL authentication: ENABLED (built-in)")tests/runtime/out_s3_format.c (1)
249-288: Schema mismatch test doesn't verify error handling behavior.The test
flb_test_parquet_schema_mismatchpushes data with a schema that doesn't match the actual fields but only checks thatflb_startsucceeds. Consider adding assertions to verify how the plugin handles the mismatch (e.g., via callbacks or logs).tests/runtime/out_s3_config.c (3)
35-35: Consider using a more portable test mode signal.The tests use
setenv()which is not available on all platforms (notably Windows without compatibility layers). While Line 65 in plugins/out_s3/s3.c shows a Windows shim forsetenv, relying on environment variables for test mode signaling can be fragile.💡 Alternative approach
Consider using a dedicated test configuration parameter or a global test flag that can be set programmatically rather than through environment variables, which would be more reliable and portable.
Also applies to: 72-72, 110-110, 150-150, 187-187, 224-224, 262-262, 319-319, 383-383, 429-429, 469-469, 534-534, 599-599, 636-636, 697-697
57-57: Use ofsleep()for synchronization is unreliable in tests.The tests use
sleep(2)to wait for upload processing, which makes tests slow and can still be flaky under load. Consider implementing a polling mechanism or callback to verify upload completion.Does the test framework provide any synchronization primitives or completion callbacks that could replace the fixed sleep delays?
Also applies to: 95-95, 133-135, 172-172, 209-209, 247-247, 405-406, 451-452, 491-492, 517-518, 556-557, 582-583, 620-621
385-410: Test loop may exhaust resources without cleanup between iterations.The test iterates 4 times creating and destroying contexts. While each iteration properly cleans up, rapid context creation could stress system resources in some environments.
♻️ Consider adding a small delay between iterations
flb_stop(ctx); flb_destroy(ctx); + usleep(100000); /* 100ms between iterations */ }src/aws/flb_aws_util.c (2)
1105-1116: File path validation correctly handles edge cases.The validation that strips leading dots and slashes (lines 1107-1109) and checks for empty path (lines 1112-1116) is well-designed. However, the warning message could be more actionable.
📝 Improve warning message clarity
if (*valid_file_path == '\0') { - flb_warn("[s3_key] file_path contains only dots/slashes, skipping $FILE_PATH substitution"); + flb_warn("[s3_key] file_path '%s' contains only dots/slashes after normalization, skipping $FILE_PATH substitution", file_path); /* Return the processed template without file path modifications */ return s3_key; }This helps debugging by showing the original file_path value.
1058-1076: Error cleanup is comprehensive but could use consistent pattern.The error path (lines 1058-1076) properly cleans up all allocated resources. However, it checks
buf != tmpon line 1066, which suggestsbufandtmpmight alias. This pattern is correct given the SDS usage, but it's subtle.Consider adding a comment explaining when
bufandtmpcan alias (whenflb_sds_printfsucceeds without reallocation).if (buf && buf != tmp) { + /* buf and tmp may alias when flb_sds_printf succeeds without reallocation */ flb_sds_destroy(buf); }plugins/out_s3/s3_queue.c (1)
542-566: Memory management for remote_id_list is correct but could be simplified.The code allocates
remote_id_list, fetches ETags, and on error (lines 551-560) properly frees all allocated strings. However, the error path duplicates cleanup logic that's also present at the end of the function (lines 606-610).♻️ Consider using a single cleanup section with goto
The function already uses a
goto cleanuppattern. The error case at lines 551-560 could usegoto cleanupafter setting an error flag, reducing code duplication.However, this would require careful handling since the cleanup section expects different state at lines 597-610. The current approach is safe, so this is a low-priority refactor.
plugins/out_s3/s3_blob.c (1)
380-383: Consider returning early if blob_db is NULL rather than checking it.The check on lines 380-383 returns an error if
ctx->blob_db.dbis NULL. However, this function is called from the flush path, so the check should ideally happen earlier (during initialization) to prevent unnecessary work.If blob event processing is enabled, the blob_db should be required during plugin initialization. This check could then become an assertion rather than a runtime error check.
However, the current approach is safe and allows for better error messages, so this is a low-priority suggestion.
src/flb_parquet_impl.cpp (1)
553-785:validate_parquet_file()is fine as a test helper, but remove dead code.
infileis opened but not used (the reader is opened fromfile_pathagain). This is minor but makes the function harder to follow.plugins/out_s3/s3_multipart.c (5)
29-64:sleep_ms()should guard against overflow / negative input on POSIX.
usleep(milliseconds * 1000)can overflowintmath if this helper is reused elsewhere; it’s also undefined-ish if a negative ever reaches it.Proposed fix
static inline void sleep_ms(int milliseconds) { - usleep(milliseconds * 1000); + if (milliseconds <= 0) { + return; + } + /* prevent overflow: cap to 60s */ + if (milliseconds > 60000) { + milliseconds = 60000; + } + usleep((useconds_t) milliseconds * 1000u); }
211-232: MD5 base64 helper should explicitly NUL-terminate based onolen.You rely on
memset()elsewhere for termination. Make this function robust by setting the terminator when possible.Proposed fix
int s3_multipart_get_md5_base64(char *buf, size_t buf_size, char *md5_str, size_t md5_str_size) { @@ ret = flb_base64_encode((unsigned char*) md5_str, md5_str_size, &olen, md5_bin, sizeof(md5_bin)); if (ret != 0) { return ret; } + if (md5_str_size > 0) { + md5_str[(olen < (md5_str_size - 1)) ? olen : (md5_str_size - 1)] = '\0'; + } return 0; }
341-401: Header building is fine, but avoid divergence with ad-hoc header creation elsewhere.
UploadPartmanually allocates a one-off Content-MD5 header; consider using this helper consistently so future header additions don’t miss some call paths.
737-805: AbortMultipartUpload: treatNoSuchUploadas non-fatal.If an upload already completed or was aborted elsewhere, abort may return an error payload; it’d be good to downgrade that case to success during recovery paths.
1097-1215: Retry/backoff logic is reasonable; please useS3_AWS_MAX_PARTSinstead of10000.Minor maintainability: avoid duplicating the AWS constant.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (47)
.github/workflows/unit-tests.yamlCMakeLists.txtcmake/kafka.cmakecmake/parquet.cmakeinclude/fluent-bit/aws/flb_aws_compress.hinclude/fluent-bit/flb_aws_util.hinclude/fluent-bit/flb_blob_db.hinclude/fluent-bit/flb_parquet.hplugins/in_blob/blob_file.cplugins/out_kinesis_firehose/firehose.cplugins/out_s3/CMakeLists.txtplugins/out_s3/s3.cplugins/out_s3/s3.hplugins/out_s3/s3_auth.cplugins/out_s3/s3_auth.hplugins/out_s3/s3_blob.cplugins/out_s3/s3_blob.hplugins/out_s3/s3_multipart.cplugins/out_s3/s3_multipart.hplugins/out_s3/s3_queue.cplugins/out_s3/s3_queue.hplugins/out_s3/s3_store.cplugins/out_s3/s3_store.hplugins/out_s3/s3_stream.cplugins/out_s3/s3_stream.hsrc/CMakeLists.txtsrc/aws/CMakeLists.txtsrc/aws/compression/CMakeLists.txtsrc/aws/flb_aws_compress.csrc/aws/flb_aws_util.csrc/flb_blob_db.csrc/flb_parquet_impl.cppsrc/flb_signv4.ctests/internal/CMakeLists.txttests/internal/aws_util.ctests/internal/fuzzers/aws_util_fuzzer.ctests/internal/parquet.ctests/runtime/CMakeLists.txttests/runtime/out_s3.ctests/runtime/out_s3_config.ctests/runtime/out_s3_edge_cases.ctests/runtime/out_s3_error_handling.ctests/runtime/out_s3_format.ctests/runtime/out_s3_multipart.ctests/runtime/out_s3_queue.ctests/runtime/out_s3_recovery.ctests/runtime/out_s3_special_features.c
💤 Files with no reviewable changes (2)
- src/aws/CMakeLists.txt
- src/aws/compression/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (10)
- cmake/parquet.cmake
- .github/workflows/unit-tests.yaml
- src/flb_signv4.c
- tests/internal/CMakeLists.txt
- tests/internal/fuzzers/aws_util_fuzzer.c
- include/fluent-bit/flb_parquet.h
- tests/runtime/out_s3_queue.c
- plugins/out_s3/s3_blob.h
- tests/runtime/CMakeLists.txt
- plugins/out_s3/CMakeLists.txt
🧰 Additional context used
🧠 Learnings (18)
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtsrc/flb_parquet_impl.cppinclude/fluent-bit/aws/flb_aws_compress.hcmake/kafka.cmake
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
plugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_kinesis_firehose/firehose.csrc/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtsrc/flb_parquet_impl.cppinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-09-08T11:21:33.975Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.
Applied to files:
plugins/out_s3/s3_store.csrc/CMakeLists.txtCMakeLists.txtsrc/flb_parquet_impl.cppplugins/out_s3/s3_multipart.c
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
include/fluent-bit/flb_aws_util.h
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.
Applied to files:
src/aws/flb_aws_util.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtinclude/fluent-bit/aws/flb_aws_compress.h
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
src/aws/flb_aws_compress.csrc/CMakeLists.txtCMakeLists.txtsrc/flb_parquet_impl.cpp
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
src/CMakeLists.txtCMakeLists.txtsrc/flb_parquet_impl.cppcmake/kafka.cmake
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
src/CMakeLists.txtCMakeLists.txtcmake/kafka.cmake
📚 Learning: 2025-12-22T05:39:02.291Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: lib/ripser-1.2.1/Makefile:1-18
Timestamp: 2025-12-22T05:39:02.291Z
Learning: In fluent/fluent-bit, the Makefile at lib/ripser-1.2.1/Makefile is imported from upstream Ripser and is not used in the actual build process. The project uses CMake for building (lib/ripser-1.2.1/CMakeLists.txt), so changes to the imported Makefile are not necessary.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T07:28:37.083Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10832
File: src/wasm/CMakeLists.txt:112-131
Timestamp: 2025-09-04T07:28:37.083Z
Learning: In fluent-bit CMake files, the user cosmo0920 prefers treating Git as a command rather than a package, emphasizing that Git is not a pkg-config retrievable package but just a command.
Applied to files:
src/CMakeLists.txtsrc/flb_parquet_impl.cpp
📚 Learning: 2025-09-14T09:46:09.531Z
Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.
Applied to files:
src/CMakeLists.txt
📚 Learning: 2025-09-04T12:35:36.904Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:3275-3282
Timestamp: 2025-09-04T12:35:36.904Z
Learning: The out_s3 plugin intentionally uses a simple numeric comparison for retry_limit (chunk->failures >= ctx->ins->retry_limit) rather than the standard Fluent Bit pattern that checks for FLB_OUT_RETRY_UNLIMITED (-1). The maintainer wants to keep this current behavior for consistency within the plugin.
Applied to files:
plugins/out_s3/s3_stream.cplugins/out_s3/s3_multipart.c
📚 Learning: 2025-08-07T10:15:46.187Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10691
File: .github/workflows/unit-tests.yaml:94-101
Timestamp: 2025-08-07T10:15:46.187Z
Learning: Apache Arrow latest packages have changing checksums, making hardcoded checksum verification impractical for CI workflows that download the latest version.
Applied to files:
src/flb_parquet_impl.cpp
📚 Learning: 2025-09-04T12:35:22.872Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10825
File: plugins/out_s3/s3.c:1339-1344
Timestamp: 2025-09-04T12:35:22.872Z
Learning: In the Fluent Bit S3 plugin, the user prefers to maintain current retry_limit behavior without special handling for FLB_OUT_RETRY_UNLIMITED (-1), as there's no documentation indicating -1 should be used for infinite retries and consistency with current logic is preferred.
Applied to files:
plugins/out_s3/s3_multipart.c
📚 Learning: 2025-12-17T18:45:02.283Z
Learnt from: edsiper
Repo: fluent/fluent-bit PR: 11286
File: src/flb_http_client.c:1657-1715
Timestamp: 2025-12-17T18:45:02.283Z
Learning: In the Fluent Bit OAuth2 implementation (src/flb_oauth2.c), the function flb_oauth2_get_access_token returns a pointer to the cached token stored in the OAuth2 context (ctx->access_token), not a new allocation. The OAuth2 context manages the token lifecycle, so callers must not free the returned token pointer.
Applied to files:
plugins/out_s3/s3.h
🧬 Code graph analysis (20)
plugins/out_s3/s3_auth.c (5)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_utils.c (1)
flb_utils_url_split(1531-1667)src/flb_upstream.c (1)
flb_upstream_create_url(416-481)src/flb_http_client.c (5)
flb_http_add_header(1013-1045)flb_http_basic_auth(1373-1377)flb_http_bearer_auth(1385-1420)flb_http_client(864-909)flb_http_do(1622-1682)src/flb_sds.c (5)
flb_sds_copy(260-281)flb_sds_create_len(58-76)flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_printf(336-387)
plugins/out_s3/s3_store.c (4)
src/flb_sds.c (4)
flb_sds_create_size(92-95)flb_sds_printf(336-387)flb_sds_destroy(389-399)flb_sds_create(78-90)src/flb_fstore.c (7)
flb_fstore_file_create(153-190)flb_fstore_file_meta_set(74-108)flb_fstore_file_append(298-329)flb_fstore_create(479-525)flb_fstore_destroy(527-564)flb_fstore_stream_create(335-402)flb_fstore_file_inactive(238-256)include/fluent-bit/flb_mem.h (2)
flb_free(126-128)flb_calloc(84-96)include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)
tests/runtime/out_s3_special_features.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
include/fluent-bit/flb_aws_util.h (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
tests/runtime/out_s3_config.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
src/aws/flb_aws_util.c (3)
src/flb_http_client.c (1)
flb_http_client_destroy(1738-1745)src/flb_sds.c (2)
flb_sds_create_len(58-76)flb_sds_cat(120-141)include/fluent-bit/flb_mem.h (1)
flb_calloc(84-96)
plugins/out_s3/s3_store.h (1)
plugins/out_s3/s3_store.c (1)
flb_chunk_file_open(536-572)
tests/runtime/out_s3_error_handling.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/runtime/out_s3_format.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (9)
flb_create(143-225)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)
tests/internal/parquet.c (3)
src/flb_parquet_impl.cpp (1)
field_name(457-457)src/flb_pack.c (1)
flb_pack_json(530-535)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
tests/runtime/out_s3_edge_cases.c (2)
plugins/out_s3/s3.c (1)
setenv(66-69)src/flb_lib.c (10)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_input_set(305-335)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_stop(1011-1055)flb_destroy(228-263)flb_lib_push(843-870)
plugins/out_s3/s3_auth.h (1)
plugins/out_s3/s3_auth.c (3)
s3_auth_init_endpoint(79-128)s3_auth_request_presigned_url(173-260)s3_auth_fetch_presigned_url(359-400)
src/flb_blob_db.c (2)
src/flb_sqldb.c (2)
flb_sqldb_lock(173-178)flb_sqldb_unlock(180-185)include/fluent-bit/flb_mem.h (1)
flb_calloc(84-96)
src/flb_parquet_impl.cpp (1)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3_multipart.c (6)
include/fluent-bit/flb_compat.h (1)
usleep(128-132)plugins/out_s3/s3.c (1)
write_seq_index(227-247)include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)src/flb_http_client.c (1)
flb_http_client_destroy(1738-1745)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(359-400)
plugins/in_blob/blob_file.c (3)
plugins/in_blob/blob_db.c (1)
blob_db_file_insert(164-196)lib/cfl/src/cfl_sds.c (1)
cfl_sds_destroy(127-137)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)
plugins/out_s3/s3_queue.h (1)
plugins/out_s3/s3_queue.c (9)
s3_queue_remove(267-275)s3_queue_buffer_chunk(315-337)s3_queue_add_file(57-111)s3_queue_add_part(113-183)s3_queue_add_pending_file(185-232)s3_queue_process_entry(680-739)s3_queue_recover_all(782-827)s3_queue_recover_from_database(837-1017)s3_queue_recover_from_fstore(1090-1150)
tests/internal/aws_util.c (1)
src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)
plugins/out_s3/s3.h (1)
plugins/out_s3/s3.c (4)
cb_s3_upload(1412-1527)s3_format_chunk(1268-1409)s3_initiate_multipart_upload(980-1085)mock_s3_call(105-179)
plugins/out_s3/s3_blob.c (7)
plugins/out_s3/s3_multipart.c (4)
s3_multipart_upload_create(434-483)s3_multipart_abort(737-805)s3_multipart_check_upload_exists(818-893)flb_s3_calculate_optimal_part_size(157-209)plugins/out_s3/s3_auth.c (1)
s3_auth_fetch_presigned_url(359-400)src/flb_input_blob.c (2)
flb_input_blob_delivery_notification_destroy(222-231)flb_input_blob_file_get_info(31-112)src/flb_notification.c (2)
flb_notification_enqueue(211-284)flb_notification_cleanup(372-381)src/flb_blob_db.c (1)
flb_blob_file_update_remote_id(541-572)src/aws/flb_aws_util.c (1)
flb_get_s3_key(1083-1184)plugins/out_s3/s3_queue.c (1)
s3_queue_add_pending_file(185-232)
| /* | ||
| * Compression algorithms (true compression) | ||
| * Valid values: none, gzip, snappy, zstd | ||
| */ | ||
| #define FLB_AWS_COMPRESS_NONE 0 | ||
| #define FLB_AWS_COMPRESS_GZIP 1 | ||
| #define FLB_AWS_COMPRESS_ARROW 2 | ||
| #define FLB_AWS_COMPRESS_PARQUET 3 | ||
| #define FLB_AWS_COMPRESS_ZSTD 4 | ||
| #define FLB_AWS_COMPRESS_SNAPPY 2 | ||
| #define FLB_AWS_COMPRESS_ZSTD 3 | ||
|
|
||
| /* | ||
| * File format conversion (NOT compression algorithms) | ||
| * | ||
| * DEPRECATED: FLB_AWS_COMPRESS_ARROW (4) | ||
| * - Arrow is not a proper file format for S3 | ||
| * - This value is kept only for backward compatibility to avoid compilation errors | ||
| * - DO NOT USE in new code | ||
| * | ||
| * Valid file format: PARQUET (5) | ||
| * - Use format=parquet instead of compression=parquet (deprecated usage) | ||
| * - Supported S3 output formats: json (FLB_S3_FORMAT_JSON), parquet (FLB_S3_FORMAT_PARQUET) | ||
| */ | ||
| #define FLB_AWS_COMPRESS_ARROW 4 /* DEPRECATED - Do not use */ | ||
| #define FLB_AWS_COMPRESS_PARQUET 5 /* Use format=parquet instead */ |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search for potential hardcoded compression type values that might break
# Look for direct numeric comparisons with compression variables
echo "=== Direct numeric comparisons with compression variables ==="
rg -n -C3 '\b(compression_type|compress_type|comp_type)\s*==\s*[0-9]' --type=c --type=cpp
# Look for direct numeric assignments to compression variables
echo "=== Direct numeric assignments to compression variables ==="
rg -n -C3 '\b(compression_type|compress_type|comp_type)\s*=\s*[0-9]' --type=c --type=cpp
# Check for FLB_AWS_COMPRESS_ usage in switch statements
echo "=== Switch statements with compression values ==="
rg -n -B5 'switch.*compression' --type=c --type=cpp | head -100
# Search for all FLB_AWS_COMPRESS_ constant usage
echo "=== All FLB_AWS_COMPRESS_ constant usage ==="
rg 'FLB_AWS_COMPRESS_' --type=c --type=cpp | head -50
# Check specifically for hardcoded 2 or 3 being assigned to compression
echo "=== Potential hardcoded compression values (2 or 3) ==="
rg -n '\bcompress.*=\s*[23]' --type=c --type=cppRepository: fluent/fluent-bit
Length of output: 10079
🏁 Script executed:
#!/bin/bash
# Examine switch statements in detail
echo "=== Switch statement in plugins/out_s3/s3.c (around line 1106) ==="
sed -n '1100,1125p' plugins/out_s3/s3.c
echo ""
echo "=== Switch statement in src/flb_parquet_impl.cpp (around line 869) ==="
sed -n '865,900p' src/flb_parquet_impl.cpp
echo ""
echo "=== Check how compression is set/parsed in S3 plugin ==="
rg -n 'flb_aws_compress_get' --type=c --type=cpp -A3
echo ""
echo "=== Check if compression values are ever persisted/serialized ==="
rg -n 'compression.*json\|serialize.*compression' --type=c --type=cpp | head -20Repository: fluent/fluent-bit
Length of output: 2840
🏁 Script executed:
#!/bin/bash
# Check the flb_parquet_impl.cpp file more thoroughly
echo "=== Full context of parquet compression switch (lines 860-885) ==="
sed -n '860,885p' src/flb_parquet_impl.cpp
echo ""
echo "=== Search for any FLB_AWS_COMPRESS constants in flb_parquet_impl.cpp ==="
rg -n 'FLB_AWS_COMPRESS' src/flb_parquet_impl.cpp
echo ""
echo "=== Check the function that calls this switch statement ==="
sed -n '800,875p' src/flb_parquet_impl.cpp | head -80Repository: fluent/fluent-bit
Length of output: 3796
🏁 Script executed:
#!/bin/bash
# Check where the compression parameter comes from in parquet context
echo "=== Check where flb_msgpack_to_parquet_streaming is called ==="
rg -n 'flb_msgpack_to_parquet_streaming' --type=c --type=cpp -B2 -A2
echo ""
echo "=== Check the header file definition ==="
fd 'flb_parquet' --type=h | xargs cat | grep -A5 'flb_msgpack_to_parquet_streaming'
echo ""
echo "=== Verify what values get passed to this function ==="
rg -n 'flb_msgpack_to_parquet_streaming.*FLB_AWS_COMPRESS|flb_msgpack_to_parquet_streaming.*compression' --type=c --type=cpp -B3 -A3Repository: fluent/fluent-bit
Length of output: 1753
🏁 Script executed:
#!/bin/bash
# Fix the search - check what values are passed to flb_msgpack_to_parquet_streaming
echo "=== Check S3 plugin call to flb_msgpack_to_parquet_streaming ==="
sed -n '1340,1360p' plugins/out_s3/s3.c
echo ""
echo "=== Check test call to flb_msgpack_to_parquet_streaming ==="
sed -n '230,245p' tests/internal/parquet.c
echo ""
echo "=== Check header file definition ==="
sed -n '80,95p' include/fluent-bit/flb_parquet.hRepository: fluent/fluent-bit
Length of output: 2070
Replace hardcoded numeric compression values with FLB_AWS_COMPRESS_ symbolic constants in src/flb_parquet_impl.cpp.*
The concern about hardcoded compression enum values is valid. The switch statement at src/flb_parquet_impl.cpp lines 869-875 uses hardcoded numeric case values (1, 2, 3) instead of the documented symbolic constants, creating a maintenance risk. Although the code currently works because the hardcoded values match the new enum numbering, this fragility will cause silent failures if the enum is renumbered again.
Replace:
switch (compression) {
case 1: parquet_compression = parquet::Compression::GZIP; break;
case 2: parquet_compression = parquet::Compression::SNAPPY; break;
case 3: parquet_compression = parquet::Compression::ZSTD; break;
default: parquet_compression = parquet::Compression::UNCOMPRESSED; break;
}With:
switch (compression) {
case FLB_AWS_COMPRESS_GZIP: parquet_compression = parquet::Compression::GZIP; break;
case FLB_AWS_COMPRESS_SNAPPY: parquet_compression = parquet::Compression::SNAPPY; break;
case FLB_AWS_COMPRESS_ZSTD: parquet_compression = parquet::Compression::ZSTD; break;
default: parquet_compression = parquet::Compression::UNCOMPRESSED; break;
}The S3 plugin (plugins/out_s3/s3.c) correctly uses symbolic constants throughout, so the pattern is already established in the codebase.
🤖 Prompt for AI Agents
In @include/fluent-bit/aws/flb_aws_compress.h around lines 24 - 46, The switch
in src/flb_parquet_impl.cpp that maps the integer variable compression to
parquet_compression uses hardcoded numeric case labels (1,2,3) which is fragile;
update the switch to use the FLB_AWS_COMPRESS_* symbolic constants
(FLB_AWS_COMPRESS_GZIP, FLB_AWS_COMPRESS_SNAPPY, FLB_AWS_COMPRESS_ZSTD) for the
GZIP/SNAPPY/ZSTD branches and keep the default mapping to
parquet::Compression::UNCOMPRESSED so the code reads the documented enum names
instead of raw numbers.
| #define SQL_RESET_FILE_PART_UPLOAD_STATES \ | ||
| "UPDATE blob_parts " \ | ||
| " SET delivery_attempts=0, " \ | ||
| " uploaded=0, " \ | ||
| " in_progress=0 " \ | ||
| " uploaded=0 " \ | ||
| " WHERE file_id=@id;" | ||
|
|
There was a problem hiding this comment.
Reset should also clear in_progress to avoid stranding parts.
SQL_RESET_FILE_PART_UPLOAD_STATES doesn’t reset in_progress; a reset file could still have parts stuck with in_progress=1 and never be picked up again.
Proposed fix
#define SQL_RESET_FILE_PART_UPLOAD_STATES \
"UPDATE blob_parts " \
" SET delivery_attempts=0, " \
- " uploaded=0 " \
+ " uploaded=0, " \
+ " in_progress=0 " \
" WHERE file_id=@id;"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| #define SQL_RESET_FILE_PART_UPLOAD_STATES \ | |
| "UPDATE blob_parts " \ | |
| " SET delivery_attempts=0, " \ | |
| " uploaded=0, " \ | |
| " in_progress=0 " \ | |
| " uploaded=0 " \ | |
| " WHERE file_id=@id;" | |
| #define SQL_RESET_FILE_PART_UPLOAD_STATES \ | |
| "UPDATE blob_parts " \ | |
| " SET delivery_attempts=0, " \ | |
| " uploaded=0, " \ | |
| " in_progress=0 " \ | |
| " WHERE file_id=@id;" |
🤖 Prompt for AI Agents
In @include/fluent-bit/flb_blob_db.h around lines 140 - 145, The
SQL_RESET_FILE_PART_UPLOAD_STATES update currently resets delivery_attempts and
uploaded but misses clearing in_progress, which can leave parts stranded; modify
the SQL statement identified by SQL_RESET_FILE_PART_UPLOAD_STATES to also set
in_progress=0 in the UPDATE of blob_parts (alongside delivery_attempts=0 and
uploaded=0 WHERE file_id=@id) so parts can be picked up again.
| static flb_sds_t extract_etag(char *response, size_t size) | ||
| { | ||
| if (str_len <= 0){ | ||
| str_len = strlen(str); | ||
| } | ||
| if (left <= *off+str_len) { | ||
| return FLB_FALSE; | ||
| char *tmp; | ||
| int start; | ||
| int end; | ||
| int i = 0; | ||
| flb_sds_t etag; | ||
|
|
||
| if (response == NULL) { | ||
| return NULL; | ||
| } | ||
| memcpy(buf+*off, str, str_len); | ||
| *off += str_len; | ||
| return FLB_TRUE; | ||
| } | ||
|
|
||
| tmp = strstr(response, "ETag:"); | ||
| if (!tmp) { | ||
| return NULL; | ||
| } | ||
| i = tmp - response; | ||
| i += 5; | ||
|
|
||
| /* the 'tag' or key in the upload_dir is s3_key + \n + upload_id */ | ||
| static flb_sds_t upload_key(struct multipart_upload *m_upload) | ||
| { | ||
| flb_sds_t key; | ||
| flb_sds_t tmp; | ||
| while (i < size && (response[i] == '\"' || isspace(response[i]) != 0)) { | ||
| i++; | ||
| } | ||
| start = i; | ||
|
|
||
| key = flb_sds_create_size(64); | ||
| while (i < size && (response[i] != '\"' && isspace(response[i]) == 0)) { | ||
| i++; | ||
| } | ||
| end = i; | ||
|
|
||
| tmp = flb_sds_printf(&key, "%s\n%s", m_upload->s3_key, m_upload->upload_id); | ||
| if (!tmp) { | ||
| etag = flb_sds_create_len(response + start, end - start); | ||
| if (!etag) { | ||
| flb_errno(); | ||
| flb_sds_destroy(key); | ||
| return NULL; | ||
| } | ||
| key = tmp; | ||
|
|
||
| return key; | ||
| return etag; | ||
| } |
There was a problem hiding this comment.
extract_etag() has UB risk with isspace() and uses fragile parsing.
isspace(response[i]) is UB for negative char values; also the parser assumes a specific header formatting. At minimum, cast to unsigned char and use size_t indices.
Proposed fix
static flb_sds_t extract_etag(char *response, size_t size)
{
char *tmp;
- int start;
- int end;
- int i = 0;
+ size_t start;
+ size_t end;
+ size_t i = 0;
flb_sds_t etag;
@@
- while (i < size && (response[i] == '\"' || isspace(response[i]) != 0)) {
+ while (i < size && (response[i] == '\"' ||
+ isspace((unsigned char) response[i]) != 0)) {
i++;
}
start = i;
- while (i < size && (response[i] != '\"' && isspace(response[i]) == 0)) {
+ while (i < size && (response[i] != '\"' &&
+ isspace((unsigned char) response[i]) == 0)) {
i++;
}
end = i;
- etag = flb_sds_create_len(response + start, end - start);
+ etag = flb_sds_create_len(response + start, (int) (end - start));📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| static flb_sds_t extract_etag(char *response, size_t size) | |
| { | |
| if (str_len <= 0){ | |
| str_len = strlen(str); | |
| } | |
| if (left <= *off+str_len) { | |
| return FLB_FALSE; | |
| char *tmp; | |
| int start; | |
| int end; | |
| int i = 0; | |
| flb_sds_t etag; | |
| if (response == NULL) { | |
| return NULL; | |
| } | |
| memcpy(buf+*off, str, str_len); | |
| *off += str_len; | |
| return FLB_TRUE; | |
| } | |
| tmp = strstr(response, "ETag:"); | |
| if (!tmp) { | |
| return NULL; | |
| } | |
| i = tmp - response; | |
| i += 5; | |
| /* the 'tag' or key in the upload_dir is s3_key + \n + upload_id */ | |
| static flb_sds_t upload_key(struct multipart_upload *m_upload) | |
| { | |
| flb_sds_t key; | |
| flb_sds_t tmp; | |
| while (i < size && (response[i] == '\"' || isspace(response[i]) != 0)) { | |
| i++; | |
| } | |
| start = i; | |
| key = flb_sds_create_size(64); | |
| while (i < size && (response[i] != '\"' && isspace(response[i]) == 0)) { | |
| i++; | |
| } | |
| end = i; | |
| tmp = flb_sds_printf(&key, "%s\n%s", m_upload->s3_key, m_upload->upload_id); | |
| if (!tmp) { | |
| etag = flb_sds_create_len(response + start, end - start); | |
| if (!etag) { | |
| flb_errno(); | |
| flb_sds_destroy(key); | |
| return NULL; | |
| } | |
| key = tmp; | |
| return key; | |
| return etag; | |
| } | |
| static flb_sds_t extract_etag(char *response, size_t size) | |
| { | |
| char *tmp; | |
| size_t start; | |
| size_t end; | |
| size_t i = 0; | |
| flb_sds_t etag; | |
| if (response == NULL) { | |
| return NULL; | |
| } | |
| tmp = strstr(response, "ETag:"); | |
| if (!tmp) { | |
| return NULL; | |
| } | |
| i = tmp - response; | |
| i += 5; | |
| while (i < size && (response[i] == '\"' || | |
| isspace((unsigned char) response[i]) != 0)) { | |
| i++; | |
| } | |
| start = i; | |
| while (i < size && (response[i] != '\"' && | |
| isspace((unsigned char) response[i]) == 0)) { | |
| i++; | |
| } | |
| end = i; | |
| etag = flb_sds_create_len(response + start, (int) (end - start)); | |
| if (!etag) { | |
| flb_errno(); | |
| return NULL; | |
| } | |
| return etag; | |
| } |
🤖 Prompt for AI Agents
In @plugins/out_s3/s3_multipart.c around lines 105 - 141, The extract_etag
function risks UB and fragile parsing: change index variables i, start, end to
size_t, validate that tmp (from strstr(response, "ETag:")) lies within the
provided size, and replace isspace(response[i]) with isspace((unsigned char)
response[i]) everywhere; tighten bounds checks when advancing i to avoid
overruns, explicitly skip the literal "ETag:" then any ASCII whitespace and an
optional opening quote before capturing the token, and pass the computed length
safely into flb_sds_create_len (and check tmp != NULL earlier) so extraction
works for negative-char platforms and arbitrary header spacing.
| static flb_sds_t build_complete_multipart_xml(struct flb_s3 *ctx, | ||
| struct multipart_upload *m_upload) | ||
| { | ||
| struct multipart_upload *m_upload = NULL; | ||
| char *buffered_data = NULL; | ||
| size_t buffer_size = 0; | ||
| int ret; | ||
| flb_sds_t xml; | ||
| flb_sds_t tmp; | ||
| int i; | ||
| int valid_parts = 0; | ||
|
|
||
| ret = s3_store_file_upload_read(ctx, fsf, &buffered_data, &buffer_size); | ||
| if (ret < 0) { | ||
| flb_plg_error(ctx->ins, "Could not read locally buffered data %s", | ||
| fsf->name); | ||
| return NULL; | ||
| /* Enforce integrity: check for NULL etags */ | ||
| for (i = 0; i < m_upload->part_number; i++) { | ||
| if (m_upload->etags[i] == NULL) { | ||
| flb_plg_error(ctx->ins, "Cannot complete multipart upload: part %d (index %d) " | ||
| "has NULL ETag. Total parts: %d", | ||
| i + 1, i, m_upload->part_number); | ||
| return NULL; | ||
| } | ||
| valid_parts++; | ||
| } | ||
|
|
||
| /* always make sure we have a fresh copy of metadata */ | ||
| ret = s3_store_file_meta_get(ctx, fsf); | ||
| if (ret == -1) { | ||
| flb_plg_error(ctx->ins, "Could not read file metadata: %s", | ||
| fsf->name); | ||
| flb_free(buffered_data); | ||
| flb_plg_debug(ctx->ins, "Building CompleteMultipartUpload payload: " | ||
| "%d valid parts", valid_parts); | ||
|
|
||
| if (valid_parts == 0) { | ||
| flb_plg_error(ctx->ins, "No valid ETags found for CompleteMultipartUpload"); | ||
| return NULL; | ||
| } | ||
|
|
||
| m_upload = flb_calloc(1, sizeof(struct multipart_upload)); | ||
| if (!m_upload) { | ||
| xml = flb_sds_create("<CompleteMultipartUpload xmlns=\"" S3_XML_NAMESPACE "\">"); | ||
| if (!xml) { | ||
| flb_errno(); | ||
| flb_free(buffered_data); | ||
| return NULL; | ||
| } | ||
| m_upload->init_time = time(NULL); | ||
| m_upload->upload_state = MULTIPART_UPLOAD_STATE_COMPLETE_IN_PROGRESS; | ||
|
|
||
| ret = upload_data_from_key(m_upload, fsf->meta_buf); | ||
| if (ret < 0) { | ||
| flb_plg_error(ctx->ins, "Could not extract upload data from: %s", | ||
| fsf->name); | ||
| flb_free(buffered_data); | ||
| multipart_upload_destroy(m_upload); | ||
| return NULL; | ||
| for (i = 0; i < m_upload->part_number; i++) { | ||
| tmp = flb_sds_printf(&xml, | ||
| "<Part><ETag>%s</ETag><PartNumber>%d</PartNumber></Part>", | ||
| m_upload->etags[i], i + 1); | ||
| if (!tmp) { | ||
| flb_errno(); | ||
| flb_sds_destroy(xml); | ||
| return NULL; | ||
| } | ||
| xml = tmp; | ||
| } | ||
|
|
||
| parse_etags(m_upload, buffered_data); | ||
| flb_free(buffered_data); | ||
| if (m_upload->part_number == 0) { | ||
| flb_plg_error(ctx->ins, "Could not extract upload data from %s", | ||
| fsf->name); | ||
| multipart_upload_destroy(m_upload); | ||
| tmp = flb_sds_cat(xml, "</CompleteMultipartUpload>", | ||
| strlen("</CompleteMultipartUpload>")); | ||
| if (!tmp) { | ||
| flb_errno(); | ||
| flb_sds_destroy(xml); | ||
| return NULL; | ||
| } | ||
|
|
||
| /* code expects it to be 1 more than the last part read */ | ||
| m_upload->part_number++; | ||
| return tmp; | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's locate the extract_etag function and understand how ETags are extracted
fd -e c -e h plugins/out_s3/ --exec grep -l "extract_etag" {} \;Repository: fluent/fluent-bit
Length of output: 426
🏁 Script executed:
# Search for the extract_etag function definition
rg "extract_etag\s*\(" plugins/out_s3/ -A 15 -B 2Repository: fluent/fluent-bit
Length of output: 2136
🏁 Script executed:
# Check how the s3_multipart.c file is structured and find related functions
head -100 plugins/out_s3/s3_multipart.cRepository: fluent/fluent-bit
Length of output: 2743
🏁 Script executed:
# Get the complete extract_etag function
rg "extract_etag\s*\(" plugins/out_s3/s3_multipart.c -A 40Repository: fluent/fluent-bit
Length of output: 2155
🏁 Script executed:
# Search for S3 API tests or examples of CompleteMultipartUpload
fd . plugins/out_s3/ --exec grep -l "CompleteMultipartUpload" {} \;Repository: fluent/fluent-bit
Length of output: 90
🏁 Script executed:
# Look for test files related to s3
fd -e c -e h test/ --exec grep -l "CompleteMultipartUpload\|complete.*multipart\|etag" {} \; 2>/dev/null | head -20Repository: fluent/fluent-bit
Length of output: 43
🏁 Script executed:
# Search for test files or integration tests related to multipart
fd -e c -e h test/ --exec grep -l "multipart\|s3_multipart" {} \; 2>/dev/nullRepository: fluent/fluent-bit
Length of output: 43
🏁 Script executed:
# Look for any documentation or README about S3 plugin
fd -e md -e txt -e rst . plugins/out_s3/ 2>/dev/null | head -10Repository: fluent/fluent-bit
Length of output: 90
🏁 Script executed:
# Search for any existing XML examples or comments about ETag format
rg "ETag.*Part\|CompleteMultipart" plugins/out_s3/ -B 3 -A 3Repository: fluent/fluent-bit
Length of output: 43
🌐 Web query:
S3 CompleteMultipartUpload API ETag format XML quotes
💡 Result:
Short answer
- In the CompleteMultipartUpload request XML each for a Part should be the exact ETag value returned by UploadPart — including the double quotes. Example: "a54357…". [1][2]
- The CompleteMultipartUpload response returns an for the assembled object; for multipart objects that ETag typically contains the MD5-like hex plus a "-N" part-count suffix (e.g. 66df11…-3). Providers’ docs show that value in the XML element (clients should treat it as a quoted string when comparing to other ETags). [1][3]
References
[1] AWS CompleteMultipartUpload API — request example with quoted part ETags and response ETag.
[2] Example S3-compatible docs showing request values in quotes.
[3] Example CompleteMultipartUploadResult showing multipart ETag with "-N" suffix.
ETag values must include quotes in CompleteMultipartUpload XML.
The extract_etag() function explicitly strips quotes from the HTTP response (see the while loop skipping '\"' characters), but S3's CompleteMultipartUpload API requires the ETag element to contain the exact quoted value returned by UploadPart. The current code outputs <ETag>abc123</ETag> instead of the required <ETag>"abc123"</ETag>, which will cause CompleteMultipartUpload requests to be rejected or fail.
The simplest fix is to preserve quotes when extracting the ETag (modify extract_etag() to not strip the quote characters), rather than adding them later during XML generation.
🤖 Prompt for AI Agents
In @plugins/out_s3/s3_multipart.c around lines 285 - 339, The ETag quotes are
being stripped earlier so build_complete_multipart_xml emits unquoted <ETag>
elements; update extract_etag() to preserve the surrounding double-quote
characters instead of skipping them (remove or adjust the loop that skips '"'
chars and any trimming that drops leading/trailing quotes) so that
m_upload->etags[] contains the exact quoted ETag returned by UploadPart; no
changes to build_complete_multipart_xml are necessary beyond relying on the
preserved values, but keep the existing NULL checks in
build_complete_multipart_xml (and in the etag extraction ensure you still
allocate/copy the full quoted string and handle memory failure cases).
| int s3_multipart_check_upload_exists(struct flb_s3 *ctx, | ||
| const char *s3_key, | ||
| const char *upload_id) | ||
| { | ||
| flb_sds_t uri = NULL; | ||
| flb_sds_t query_params = NULL; | ||
| flb_sds_t tmp; | ||
| struct flb_http_client *c = NULL; | ||
| struct flb_aws_client *s3_client; | ||
| struct flb_aws_header *headers = NULL; | ||
| int num_headers = 0; | ||
| int ret; | ||
| int result; | ||
|
|
||
| uri = flb_sds_create_size(flb_sds_len(m_upload->s3_key) + 8); | ||
| if (!uri) { | ||
| flb_errno(); | ||
| if (!s3_key || !upload_id) { | ||
| flb_plg_error(ctx->ins, "Invalid parameters for upload existence check"); | ||
| return -1; | ||
| } | ||
|
|
||
| if (pre_signed_url != NULL) { | ||
| tmp = flb_sds_copy(uri, pre_signed_url, strlen(pre_signed_url)); | ||
| } | ||
| else { | ||
| tmp = flb_sds_printf(&uri, "/%s%s?uploads=", ctx->bucket, m_upload->s3_key); | ||
| /* Build ListParts query (max-parts=1 for minimal response) */ | ||
| query_params = flb_sds_create_size(128); | ||
| if (!query_params) { | ||
| flb_errno(); | ||
| flb_sds_destroy(s3_key); | ||
| return -1; | ||
| } | ||
|
|
||
| tmp = flb_sds_printf(&query_params, "?uploadId=%s&max-parts=1", upload_id); | ||
| if (!tmp) { | ||
| flb_sds_destroy(uri); | ||
| flb_sds_destroy(query_params); | ||
| flb_sds_destroy(s3_key); | ||
| return -1; | ||
| } | ||
| query_params = tmp; | ||
|
|
||
| uri = build_s3_uri(ctx, NULL, s3_key, query_params); | ||
| flb_sds_destroy(query_params); | ||
|
|
||
| if (!uri) { | ||
| flb_sds_destroy(s3_key); | ||
| return -1; | ||
| } | ||
| uri = tmp; | ||
|
|
||
| s3_client = ctx->s3_client; | ||
| if (s3_plugin_under_test() == FLB_TRUE) { | ||
| c = mock_s3_call("TEST_CREATE_MULTIPART_UPLOAD_ERROR", "CreateMultipartUpload"); | ||
|
|
||
| /* Call ListParts API */ | ||
| c = s3_client->client_vtable->request(s3_client, FLB_HTTP_GET, | ||
| uri, NULL, 0, NULL, 0); | ||
| flb_sds_destroy(uri); | ||
|
|
||
| if (!c) { | ||
| /* Network error or request failed */ | ||
| flb_plg_error(ctx->ins, "ListParts request failed (network/API error)"); | ||
| flb_sds_destroy(s3_key); | ||
| return -1; | ||
| } | ||
| else { | ||
| ret = create_headers(ctx, NULL, &headers, &num_headers, FLB_TRUE); | ||
| if (ret == -1) { | ||
| flb_plg_error(ctx->ins, "Failed to create headers"); | ||
| flb_sds_destroy(uri); | ||
| return -1; | ||
| } | ||
| c = s3_client->client_vtable->request(s3_client, FLB_HTTP_POST, | ||
| uri, NULL, 0, headers, num_headers); | ||
| if (headers) { | ||
| flb_free(headers); | ||
| } | ||
|
|
||
| /* Analyze response */ | ||
| if (c->resp.status == 200) { | ||
| /* Upload exists */ | ||
| flb_plg_debug(ctx->ins, "Upload ID validation: exists"); | ||
| result = 1; | ||
| } | ||
| flb_sds_destroy(uri); | ||
| if (c) { | ||
| flb_plg_debug(ctx->ins, "CreateMultipartUpload http status=%d", | ||
| c->resp.status); | ||
| if (c->resp.status == 200) { | ||
| tmp = flb_aws_xml_get_val(c->resp.payload, c->resp.payload_size, | ||
| "<UploadId>", "</UploadId>"); | ||
| if (!tmp) { | ||
| flb_plg_error(ctx->ins, "Could not find upload ID in " | ||
| "CreateMultipartUpload response"); | ||
| flb_plg_debug(ctx->ins, "Raw CreateMultipartUpload response: %s", | ||
| c->resp.payload); | ||
| flb_http_client_destroy(c); | ||
| return -1; | ||
| } | ||
| m_upload->upload_id = tmp; | ||
| flb_plg_info(ctx->ins, "Successfully initiated multipart upload " | ||
| "for %s, UploadId=%s", m_upload->s3_key, | ||
| m_upload->upload_id); | ||
| flb_http_client_destroy(c); | ||
| return 0; | ||
| } | ||
| flb_aws_print_xml_error(c->resp.payload, c->resp.payload_size, | ||
| "CreateMultipartUpload", ctx->ins); | ||
| if (c->resp.payload != NULL) { | ||
| flb_plg_debug(ctx->ins, "Raw CreateMultipartUpload response: %s", | ||
| c->resp.payload); | ||
| } | ||
| flb_http_client_destroy(c); | ||
| else if (c->resp.payload && | ||
| strstr(c->resp.payload, "<Code>NoSuchUpload</Code>")) { | ||
| /* Upload does not exist */ | ||
| flb_plg_debug(ctx->ins, "Upload ID validation: NoSuchUpload"); | ||
| result = 0; | ||
| } | ||
| else { | ||
| /* Other error - log and treat conservatively */ | ||
| flb_plg_warn(ctx->ins, "ListParts returned status %d, " | ||
| "treating upload_id as potentially invalid", c->resp.status); | ||
| result = 0; | ||
| } | ||
|
|
||
| flb_plg_error(ctx->ins, "CreateMultipartUpload request failed"); | ||
| return -1; | ||
| flb_http_client_destroy(c); | ||
| return result; | ||
| } |
There was a problem hiding this comment.
Blocker: s3_multipart_check_upload_exists() frees s3_key it doesn’t own.
s3_key is const char *; calling flb_sds_destroy(s3_key) is invalid and can crash/double-free. Also, the function doc says “-1 on network/API error” but you return 0 for many non-200 statuses.
Proposed fix
int s3_multipart_check_upload_exists(struct flb_s3 *ctx,
const char *s3_key,
const char *upload_id)
{
@@
query_params = flb_sds_create_size(128);
if (!query_params) {
flb_errno();
- flb_sds_destroy(s3_key);
return -1;
}
@@
if (!tmp) {
flb_sds_destroy(query_params);
- flb_sds_destroy(s3_key);
return -1;
}
@@
if (!uri) {
- flb_sds_destroy(s3_key);
return -1;
}
@@
if (!c) {
/* Network error or request failed */
flb_plg_error(ctx->ins, "ListParts request failed (network/API error)");
- flb_sds_destroy(s3_key);
return -1;
}
@@
else {
- /* Other error - log and treat conservatively */
- flb_plg_warn(ctx->ins, "ListParts returned status %d, "
- "treating upload_id as potentially invalid", c->resp.status);
- result = 0;
+ /* Other error */
+ flb_plg_warn(ctx->ins, "ListParts returned status %d", c->resp.status);
+ result = -1;
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| int s3_multipart_check_upload_exists(struct flb_s3 *ctx, | |
| const char *s3_key, | |
| const char *upload_id) | |
| { | |
| flb_sds_t uri = NULL; | |
| flb_sds_t query_params = NULL; | |
| flb_sds_t tmp; | |
| struct flb_http_client *c = NULL; | |
| struct flb_aws_client *s3_client; | |
| struct flb_aws_header *headers = NULL; | |
| int num_headers = 0; | |
| int ret; | |
| int result; | |
| uri = flb_sds_create_size(flb_sds_len(m_upload->s3_key) + 8); | |
| if (!uri) { | |
| flb_errno(); | |
| if (!s3_key || !upload_id) { | |
| flb_plg_error(ctx->ins, "Invalid parameters for upload existence check"); | |
| return -1; | |
| } | |
| if (pre_signed_url != NULL) { | |
| tmp = flb_sds_copy(uri, pre_signed_url, strlen(pre_signed_url)); | |
| } | |
| else { | |
| tmp = flb_sds_printf(&uri, "/%s%s?uploads=", ctx->bucket, m_upload->s3_key); | |
| /* Build ListParts query (max-parts=1 for minimal response) */ | |
| query_params = flb_sds_create_size(128); | |
| if (!query_params) { | |
| flb_errno(); | |
| flb_sds_destroy(s3_key); | |
| return -1; | |
| } | |
| tmp = flb_sds_printf(&query_params, "?uploadId=%s&max-parts=1", upload_id); | |
| if (!tmp) { | |
| flb_sds_destroy(uri); | |
| flb_sds_destroy(query_params); | |
| flb_sds_destroy(s3_key); | |
| return -1; | |
| } | |
| query_params = tmp; | |
| uri = build_s3_uri(ctx, NULL, s3_key, query_params); | |
| flb_sds_destroy(query_params); | |
| if (!uri) { | |
| flb_sds_destroy(s3_key); | |
| return -1; | |
| } | |
| uri = tmp; | |
| s3_client = ctx->s3_client; | |
| if (s3_plugin_under_test() == FLB_TRUE) { | |
| c = mock_s3_call("TEST_CREATE_MULTIPART_UPLOAD_ERROR", "CreateMultipartUpload"); | |
| /* Call ListParts API */ | |
| c = s3_client->client_vtable->request(s3_client, FLB_HTTP_GET, | |
| uri, NULL, 0, NULL, 0); | |
| flb_sds_destroy(uri); | |
| if (!c) { | |
| /* Network error or request failed */ | |
| flb_plg_error(ctx->ins, "ListParts request failed (network/API error)"); | |
| flb_sds_destroy(s3_key); | |
| return -1; | |
| } | |
| else { | |
| ret = create_headers(ctx, NULL, &headers, &num_headers, FLB_TRUE); | |
| if (ret == -1) { | |
| flb_plg_error(ctx->ins, "Failed to create headers"); | |
| flb_sds_destroy(uri); | |
| return -1; | |
| } | |
| c = s3_client->client_vtable->request(s3_client, FLB_HTTP_POST, | |
| uri, NULL, 0, headers, num_headers); | |
| if (headers) { | |
| flb_free(headers); | |
| } | |
| /* Analyze response */ | |
| if (c->resp.status == 200) { | |
| /* Upload exists */ | |
| flb_plg_debug(ctx->ins, "Upload ID validation: exists"); | |
| result = 1; | |
| } | |
| flb_sds_destroy(uri); | |
| if (c) { | |
| flb_plg_debug(ctx->ins, "CreateMultipartUpload http status=%d", | |
| c->resp.status); | |
| if (c->resp.status == 200) { | |
| tmp = flb_aws_xml_get_val(c->resp.payload, c->resp.payload_size, | |
| "<UploadId>", "</UploadId>"); | |
| if (!tmp) { | |
| flb_plg_error(ctx->ins, "Could not find upload ID in " | |
| "CreateMultipartUpload response"); | |
| flb_plg_debug(ctx->ins, "Raw CreateMultipartUpload response: %s", | |
| c->resp.payload); | |
| flb_http_client_destroy(c); | |
| return -1; | |
| } | |
| m_upload->upload_id = tmp; | |
| flb_plg_info(ctx->ins, "Successfully initiated multipart upload " | |
| "for %s, UploadId=%s", m_upload->s3_key, | |
| m_upload->upload_id); | |
| flb_http_client_destroy(c); | |
| return 0; | |
| } | |
| flb_aws_print_xml_error(c->resp.payload, c->resp.payload_size, | |
| "CreateMultipartUpload", ctx->ins); | |
| if (c->resp.payload != NULL) { | |
| flb_plg_debug(ctx->ins, "Raw CreateMultipartUpload response: %s", | |
| c->resp.payload); | |
| } | |
| flb_http_client_destroy(c); | |
| else if (c->resp.payload && | |
| strstr(c->resp.payload, "<Code>NoSuchUpload</Code>")) { | |
| /* Upload does not exist */ | |
| flb_plg_debug(ctx->ins, "Upload ID validation: NoSuchUpload"); | |
| result = 0; | |
| } | |
| else { | |
| /* Other error - log and treat conservatively */ | |
| flb_plg_warn(ctx->ins, "ListParts returned status %d, " | |
| "treating upload_id as potentially invalid", c->resp.status); | |
| result = 0; | |
| } | |
| flb_plg_error(ctx->ins, "CreateMultipartUpload request failed"); | |
| return -1; | |
| flb_http_client_destroy(c); | |
| return result; | |
| } | |
| int s3_multipart_check_upload_exists(struct flb_s3 *ctx, | |
| const char *s3_key, | |
| const char *upload_id) | |
| { | |
| flb_sds_t uri = NULL; | |
| flb_sds_t query_params = NULL; | |
| flb_sds_t tmp; | |
| struct flb_http_client *c = NULL; | |
| struct flb_aws_client *s3_client; | |
| int result; | |
| if (!s3_key || !upload_id) { | |
| flb_plg_error(ctx->ins, "Invalid parameters for upload existence check"); | |
| return -1; | |
| } | |
| /* Build ListParts query (max-parts=1 for minimal response) */ | |
| query_params = flb_sds_create_size(128); | |
| if (!query_params) { | |
| flb_errno(); | |
| return -1; | |
| } | |
| tmp = flb_sds_printf(&query_params, "?uploadId=%s&max-parts=1", upload_id); | |
| if (!tmp) { | |
| flb_sds_destroy(query_params); | |
| return -1; | |
| } | |
| query_params = tmp; | |
| uri = build_s3_uri(ctx, NULL, s3_key, query_params); | |
| flb_sds_destroy(query_params); | |
| if (!uri) { | |
| return -1; | |
| } | |
| s3_client = ctx->s3_client; | |
| /* Call ListParts API */ | |
| c = s3_client->client_vtable->request(s3_client, FLB_HTTP_GET, | |
| uri, NULL, 0, NULL, 0); | |
| flb_sds_destroy(uri); | |
| if (!c) { | |
| /* Network error or request failed */ | |
| flb_plg_error(ctx->ins, "ListParts request failed (network/API error)"); | |
| return -1; | |
| } | |
| /* Analyze response */ | |
| if (c->resp.status == 200) { | |
| /* Upload exists */ | |
| flb_plg_debug(ctx->ins, "Upload ID validation: exists"); | |
| result = 1; | |
| } | |
| else if (c->resp.payload && | |
| strstr(c->resp.payload, "<Code>NoSuchUpload</Code>")) { | |
| /* Upload does not exist */ | |
| flb_plg_debug(ctx->ins, "Upload ID validation: NoSuchUpload"); | |
| result = 0; | |
| } | |
| else { | |
| /* Other error */ | |
| flb_plg_warn(ctx->ins, "ListParts returned status %d", c->resp.status); | |
| result = -1; | |
| } | |
| flb_http_client_destroy(c); | |
| return result; | |
| } |
🤖 Prompt for AI Agents
In @plugins/out_s3/s3_multipart.c around lines 818 - 893, The function
s3_multipart_check_upload_exists incorrectly calls flb_sds_destroy(s3_key) on a
const char* it does not own and returns 0 for many non-200 responses
(contradicting the doc that network/API errors should return -1). Remove all
flb_sds_destroy(s3_key) calls in s3_multipart_check_upload_exists so the
function does not free the caller-owned s3_key, and change the error-path logic
so that unexpected ListParts failures (non-200 and not matching
"<Code>NoSuchUpload</Code>") return -1 (keep returning 1 for 200 and 0 for
explicit NoSuchUpload). Ensure flb_http_client_destroy(c) and other owned
resources are still cleaned up before returning.
| arrow::Status convert_to_timestamp(const msgpack_object* obj, | ||
| arrow::TimestampBuilder* builder, | ||
| const std::shared_ptr<arrow::DataType>& type) { | ||
| /* Get the timestamp type to determine the time unit */ | ||
| auto ts_type = std::static_pointer_cast<arrow::TimestampType>(type); | ||
| arrow::TimeUnit::type time_unit = ts_type->unit(); | ||
|
|
||
| /* Scale factor: Assume input is in seconds, scale to target unit | ||
| * MILLI (ms): multiply by 1,000 | ||
| * MICRO (us): multiply by 1,000,000 | ||
| * NANO (ns): multiply by 1,000,000,000 | ||
| */ | ||
| int64_t scale_factor = 1; | ||
| switch (time_unit) { | ||
| case arrow::TimeUnit::SECOND: | ||
| scale_factor = 1; | ||
| break; | ||
| case arrow::TimeUnit::MILLI: | ||
| scale_factor = 1000LL; | ||
| break; | ||
| case arrow::TimeUnit::MICRO: | ||
| scale_factor = 1000000LL; | ||
| break; | ||
| case arrow::TimeUnit::NANO: | ||
| scale_factor = 1000000000LL; | ||
| break; | ||
| } | ||
|
|
||
| switch (obj->type) { | ||
| case MSGPACK_OBJECT_POSITIVE_INTEGER: { | ||
| int64_t scaled_value = static_cast<int64_t>(obj->via.u64) * scale_factor; | ||
| return builder->Append(scaled_value); | ||
| } | ||
|
|
||
| case MSGPACK_OBJECT_NEGATIVE_INTEGER: { | ||
| int64_t scaled_value = obj->via.i64 * scale_factor; | ||
| return builder->Append(scaled_value); | ||
| } | ||
|
|
||
| case MSGPACK_OBJECT_FLOAT32: | ||
| case MSGPACK_OBJECT_FLOAT64: { | ||
| /* For floating point, apply scaling to handle fractional seconds */ | ||
| double scaled_value = obj->via.f64 * static_cast<double>(scale_factor); | ||
| return builder->Append(static_cast<int64_t>(scaled_value)); | ||
| } | ||
|
|
||
| case MSGPACK_OBJECT_STR: { | ||
| std::string str(obj->via.str.ptr, obj->via.str.size); | ||
| if (auto val = parse_string_to_number<double>(str)) { | ||
| double scaled_value = (*val) * static_cast<double>(scale_factor); | ||
| return builder->Append(static_cast<int64_t>(scaled_value)); | ||
| } | ||
| return arrow::Status::Invalid("Cannot parse string to timestamp"); | ||
| } | ||
|
|
||
| case MSGPACK_OBJECT_BOOLEAN: { | ||
| int64_t scaled_value = (obj->via.boolean ? 1 : 0) * scale_factor; | ||
| return builder->Append(scaled_value); | ||
| } | ||
|
|
||
| default: | ||
| return arrow::Status::Invalid("Cannot convert msgpack type to timestamp"); | ||
| } | ||
| } |
There was a problem hiding this comment.
Timestamp scaling needs overflow handling (mul can wrap).
u64/i64 * scale_factor can overflow int64_t and silently corrupt timestamps. Consider clamping or returning an error for out-of-range values.
🤖 Prompt for AI Agents
In @src/flb_parquet_impl.cpp around lines 338 - 401, convert_to_timestamp may
overflow when multiplying input values by scale_factor; before multiplying in
convert_to_timestamp (for MSGPACK_OBJECT_POSITIVE_INTEGER,
MSGPACK_OBJECT_NEGATIVE_INTEGER, floats and parsed strings) check for overflow
and clamp or return an error. Specifically, use safe checks against
std::numeric_limits<int64_t>::max()/min(): for unsigned inputs (obj->via.u64)
ensure obj->via.u64 <= (uint64_t)(INT64_MAX / scale_factor) before
casting/multiplying; for signed negatives check obj->via.i64 >= (INT64_MIN /
scale_factor); for doubles and parsed string doubles check the scaled double is
within [INT64_MIN, INT64_MAX] before casting; on overflow return an
arrow::Status::Invalid or OutOfRange with a clear message rather than performing
the multiplication. Ensure all switch branches (POSITIVE_INTEGER,
NEGATIVE_INTEGER, FLOAT32/FLOAT64, STR) apply these checks prior to
builder->Append.
| int flb_parquet_validate_schema(const char *schema_str, | ||
| char *error_msg, | ||
| size_t error_msg_size) | ||
| { | ||
| /* Reuse parse_schema_from_json to avoid code duplication */ | ||
| auto result = parse_schema_from_json(schema_str); | ||
|
|
||
| if (!result.ok()) { | ||
| if (error_msg && error_msg_size > 0) { | ||
| snprintf(error_msg, error_msg_size, "%s", | ||
| result.status().ToString().c_str()); | ||
| } | ||
| return -1; | ||
| } | ||
|
|
||
| return 0; | ||
| } |
There was a problem hiding this comment.
flb_parquet_validate_schema() must NULL-check schema_str.
As-is, a NULL schema_str will hit strlen() in parse_schema_from_json() and crash.
Proposed fix
int flb_parquet_validate_schema(const char *schema_str,
char *error_msg,
size_t error_msg_size)
{
+ if (!schema_str) {
+ if (error_msg && error_msg_size > 0) {
+ snprintf(error_msg, error_msg_size, "NULL schema_str");
+ }
+ return -1;
+ }
/* Reuse parse_schema_from_json to avoid code duplication */
auto result = parse_schema_from_json(schema_str);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| int flb_parquet_validate_schema(const char *schema_str, | |
| char *error_msg, | |
| size_t error_msg_size) | |
| { | |
| /* Reuse parse_schema_from_json to avoid code duplication */ | |
| auto result = parse_schema_from_json(schema_str); | |
| if (!result.ok()) { | |
| if (error_msg && error_msg_size > 0) { | |
| snprintf(error_msg, error_msg_size, "%s", | |
| result.status().ToString().c_str()); | |
| } | |
| return -1; | |
| } | |
| return 0; | |
| } | |
| int flb_parquet_validate_schema(const char *schema_str, | |
| char *error_msg, | |
| size_t error_msg_size) | |
| { | |
| if (!schema_str) { | |
| if (error_msg && error_msg_size > 0) { | |
| snprintf(error_msg, error_msg_size, "NULL schema_str"); | |
| } | |
| return -1; | |
| } | |
| /* Reuse parse_schema_from_json to avoid code duplication */ | |
| auto result = parse_schema_from_json(schema_str); | |
| if (!result.ok()) { | |
| if (error_msg && error_msg_size > 0) { | |
| snprintf(error_msg, error_msg_size, "%s", | |
| result.status().ToString().c_str()); | |
| } | |
| return -1; | |
| } | |
| return 0; | |
| } |
🤖 Prompt for AI Agents
In @src/flb_parquet_impl.cpp around lines 787 - 803, flb_parquet_validate_schema
currently calls parse_schema_from_json without guarding against a NULL
schema_str which causes a crash; add a NULL-check at the start of
flb_parquet_validate_schema (before calling parse_schema_from_json) that if
schema_str is NULL writes a clear message into error_msg (when error_msg &&
error_msg_size > 0) using snprintf and then returns -1; keep the existing
behavior for non-NULL inputs and still call parse_schema_from_json as before.
| void flb_test_create_multipart_basic(void) | ||
| { | ||
| int ret; | ||
| flb_ctx_t *ctx; | ||
| int in_ffd; | ||
| int out_ffd; | ||
|
|
||
| /* mocks calls - signals that we are in test mode */ | ||
| setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); | ||
|
|
||
| ctx = flb_create(); | ||
| flb_service_set(ctx, "flush", "1", "grace", "1", NULL); | ||
|
|
||
| in_ffd = flb_input(ctx, (char *) "lib", NULL); | ||
| TEST_CHECK(in_ffd >= 0); | ||
| flb_input_set(ctx, in_ffd, "tag", "test", NULL); | ||
|
|
||
| out_ffd = flb_output(ctx, (char *) "s3", NULL); | ||
| TEST_CHECK(out_ffd >= 0); | ||
| flb_output_set(ctx, out_ffd, "match", "*", NULL); | ||
| flb_output_set(ctx, out_ffd, "region", "us-west-2", NULL); | ||
| flb_output_set(ctx, out_ffd, "bucket", "test-bucket", NULL); | ||
| flb_output_set(ctx, out_ffd, "total_file_size", "10M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_chunk_size", "5M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_timeout", "1s", NULL); | ||
|
|
||
| ret = flb_start(ctx); | ||
| TEST_CHECK(ret == 0); | ||
|
|
||
| /* Push data to trigger multipart upload creation */ | ||
| flb_lib_push(ctx, in_ffd, (char *) JSON_TD, (int) sizeof(JSON_TD) - 1); | ||
| sleep(2); | ||
|
|
||
| flb_stop(ctx); | ||
| flb_destroy(ctx); | ||
| unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, check if the file exists and get an overview
ls -la tests/runtime/out_s3_multipart.c 2>/dev/null && wc -l tests/runtime/out_s3_multipart.cRepository: fluent/fluent-bit
Length of output: 180
🏁 Script executed:
# Search for wait/poll helpers in runtime tests as suggested
rg -n --type=c 'flb_test_.*wait|wait_for|eventually|poll' tests/runtime | head -20Repository: fluent/fluent-bit
Length of output: 1541
🏁 Script executed:
# Also check for general wait patterns in the runtime test framework
rg -n --type=c '(while|for).*\(.*\)' tests/runtime/*.c | grep -i 'wait\|poll\|check' | head -10Repository: fluent/fluent-bit
Length of output: 893
🏁 Script executed:
# Get a count of how many times sleep() is used in this specific test file
rg -c 'sleep\(' tests/runtime/out_s3_multipart.cRepository: fluent/fluent-bit
Length of output: 63
🏁 Script executed:
# Check what assertions are used in the test file
rg -n 'TEST_CHECK|assert' tests/runtime/out_s3_multipart.c | wc -lRepository: fluent/fluent-bit
Length of output: 63
🏁 Script executed:
# Sample a few test functions to see if they have multipart-specific assertions
rg -A 20 'void flb_test.*multipart' tests/runtime/out_s3_multipart.c | head -60Repository: fluent/fluent-bit
Length of output: 1747
🏁 Script executed:
# Check if there are any hard-coded /tmp paths in the test file
rg -n '/tmp/' tests/runtime/out_s3_multipart.cRepository: fluent/fluent-bit
Length of output: 274
🏁 Script executed:
# Let's examine the full test file structure to understand the test patterns
head -100 tests/runtime/out_s3_multipart.cRepository: fluent/fluent-bit
Length of output: 3459
🏁 Script executed:
# Get the wait_for_file implementation to understand its interface
rg -A 15 'static inline int wait_for_file' tests/runtime/flb_tests_runtime.h.inRepository: fluent/fluent-bit
Length of output: 534
🏁 Script executed:
# Check what the tests are actually asserting - get all TEST_CHECK statements
rg -B 2 'TEST_CHECK' tests/runtime/out_s3_multipart.c | head -50Repository: fluent/fluent-bit
Length of output: 1016
🏁 Script executed:
# Look for any assertions or verification of multipart behavior (like API calls, side effects)
rg -n 'assert|verify|check.*multipart|check.*upload|check.*completion' tests/runtime/out_s3_multipart.c -iRepository: fluent/fluent-bit
Length of output: 216
🏁 Script executed:
# Check the context around sleep() calls to see if they're waiting for conditions
rg -B 3 -A 3 'sleep\(' tests/runtime/out_s3_multipart.c | head -60Repository: fluent/fluent-bit
Length of output: 1545
🏁 Script executed:
# Get the complete implementation of wait_for_file to see if it's applicable to S3 tests
cat tests/runtime/flb_tests_runtime.h.in | head -80Repository: fluent/fluent-bit
Length of output: 1537
Replace sleep() calls with condition-based polling or meaningful assertions on multipart behavior.
The tests use bare sleep(2) and sleep(3) calls directly after pushing data, with no verification of multipart operations. This creates flaky tests on slower CI systems. While the runtime framework provides wait_for_file() for condition-based waiting (see flb_tests_runtime.h.in), the S3 tests need to assert on actual multipart behavior: upload ID generation, part tracking, completion callbacks, or persistent state. Currently, tests pass as long as Fluent Bit doesn't crash, even if multipart logic is broken.
Additionally, fix hard-coded /tmp/test_s3_multipart_persistence.db paths (lines 452, 480) to use temporary directories per test to avoid cross-test contamination.
🤖 Prompt for AI Agents
In @tests/runtime/out_s3_multipart.c around lines 27 - 63, The test
flb_test_create_multipart_basic uses sleep(2) after flb_lib_push which causes
flakiness; replace the blind sleep calls with condition-based polling using the
provided wait_for_file() helper from flb_tests_runtime.h.in (or equivalent) and
add explicit assertions that verify multipart behavior (e.g., that an upload ID
was created, parts were tracked, completion callback was invoked, or the
persistence DB contains expected entries) by polling the mock S3 artifacts or
persistence file instead of sleeping; also remove hard-coded
"/tmp/test_s3_multipart_persistence.db" usage and create a unique temporary
directory/file per test (use mkdtemp/tmpfile or test framework temp helper) so
persistence state is isolated between tests (update code referencing that
literal path to use the temp filename).
| /* mocks calls - signals that we are in test mode */ | ||
| setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); | ||
|
|
||
| ctx = flb_create(); | ||
| flb_service_set(ctx, "flush", "1", "grace", "1", NULL); | ||
|
|
||
| in_ffd = flb_input(ctx, (char *) "lib", NULL); | ||
| TEST_CHECK(in_ffd >= 0); | ||
| flb_input_set(ctx, in_ffd, "tag", "test", NULL); | ||
|
|
||
| out_ffd = flb_output(ctx, (char *) "s3", NULL); | ||
| TEST_CHECK(out_ffd >= 0); | ||
| flb_output_set(ctx, out_ffd, "match", "*", NULL); | ||
| flb_output_set(ctx, out_ffd, "region", "us-west-2", NULL); | ||
| flb_output_set(ctx, out_ffd, "bucket", "test-bucket", NULL); | ||
| flb_output_set(ctx, out_ffd, "total_file_size", "10M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_chunk_size", "5M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_timeout", "1s", NULL); | ||
|
|
||
| ret = flb_start(ctx); | ||
| TEST_CHECK(ret == 0); | ||
|
|
||
| /* Push data to trigger multipart upload creation */ | ||
| flb_lib_push(ctx, in_ffd, (char *) JSON_TD, (int) sizeof(JSON_TD) - 1); | ||
| sleep(2); | ||
|
|
||
| flb_stop(ctx); | ||
| flb_destroy(ctx); | ||
| unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); | ||
| } |
There was a problem hiding this comment.
Prevent cross-test contamination: always cleanup env vars and temp artifacts even on early failure.
If TEST_CHECK() aborts the test early, the env vars may remain set; also the fixed /tmp/test_s3_multipart_persistence.db can collide across parallel runs. Use per-test unique paths (mkstemp-style) and a single cleanup: label to unsetenv()/unlink().
Proposed pattern (apply across all tests)
void flb_test_create_multipart_basic(void)
{
int ret;
flb_ctx_t *ctx;
int in_ffd;
int out_ffd;
+ int started = FLB_FALSE;
- /* mocks calls - signals that we are in test mode */
setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1);
ctx = flb_create();
+ TEST_CHECK(ctx != NULL);
flb_service_set(ctx, "flush", "1", "grace", "1", NULL);
@@
ret = flb_start(ctx);
TEST_CHECK(ret == 0);
+ started = FLB_TRUE;
@@
- flb_stop(ctx);
- flb_destroy(ctx);
- unsetenv("FLB_S3_PLUGIN_UNDER_TEST");
+cleanup:
+ if (ctx) {
+ if (started) {
+ flb_stop(ctx);
+ }
+ flb_destroy(ctx);
+ }
+ unsetenv("FLB_S3_PLUGIN_UNDER_TEST");
}Also applies to: 73-103, 113-141, 152-183, 193-226, 235-305, 314-346, 355-427, 437-493, 503-554
🤖 Prompt for AI Agents
In @tests/runtime/out_s3_multipart.c around lines 34 - 63, The test leaves
environment and a fixed persistence DB path
(/tmp/test_s3_multipart_persistence.db) around on early failures; change tests
to create a unique per-test persistence path (mkstemp/mkstemp-style or tmpnam
then snprintf into a char persistence_path[]) and set that via setenv before
starting, and replace the hardcoded path with that variable when configuring the
plugin; add a single cleanup: label (or test-framework teardown) that always
runs to unsetenv("FLB_S3_PLUGIN_UNDER_TEST") and unlink(persistence_path) and
jump to it on any early return/after each TEST_CHECK failure so env vars and
temp files are always removed—apply this pattern to all analogous blocks (the
other ranges noted in the comment).
| void flb_test_upload_id_persistence(void) | ||
| { | ||
| int ret; | ||
| flb_ctx_t *ctx; | ||
| int in_ffd; | ||
| int out_ffd; | ||
|
|
||
| setenv("FLB_S3_PLUGIN_UNDER_TEST", "true", 1); | ||
|
|
||
| /* First session: Create multipart upload */ | ||
| ctx = flb_create(); | ||
| flb_service_set(ctx, "flush", "1", "grace", "1", NULL); | ||
|
|
||
| in_ffd = flb_input(ctx, (char *) "lib", NULL); | ||
| TEST_CHECK(in_ffd >= 0); | ||
| flb_input_set(ctx, in_ffd, "tag", "test", NULL); | ||
|
|
||
| out_ffd = flb_output(ctx, (char *) "s3", NULL); | ||
| TEST_CHECK(out_ffd >= 0); | ||
| flb_output_set(ctx, out_ffd, "match", "*", NULL); | ||
| flb_output_set(ctx, out_ffd, "region", "us-west-2", NULL); | ||
| flb_output_set(ctx, out_ffd, "bucket", "test-bucket", NULL); | ||
| flb_output_set(ctx, out_ffd, "blob_database_file", "/tmp/test_s3_multipart_persistence.db", NULL); | ||
| flb_output_set(ctx, out_ffd, "total_file_size", "10M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_chunk_size", "5M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_timeout", "1s", NULL); | ||
|
|
||
| ret = flb_start(ctx); | ||
| TEST_CHECK(ret == 0); | ||
|
|
||
| flb_lib_push(ctx, in_ffd, (char *) JSON_TD, (int) sizeof(JSON_TD) - 1); | ||
| sleep(2); | ||
|
|
||
| flb_stop(ctx); | ||
| flb_destroy(ctx); | ||
| unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); | ||
|
|
||
| /* Second session: Verify upload_id recovery from database */ | ||
| ctx = flb_create(); | ||
| flb_service_set(ctx, "flush", "1", "grace", "1", NULL); | ||
|
|
||
| in_ffd = flb_input(ctx, (char *) "lib", NULL); | ||
| TEST_CHECK(in_ffd >= 0); | ||
| flb_input_set(ctx, in_ffd, "tag", "test", NULL); | ||
|
|
||
| out_ffd = flb_output(ctx, (char *) "s3", NULL); | ||
| TEST_CHECK(out_ffd >= 0); | ||
| flb_output_set(ctx, out_ffd, "match", "*", NULL); | ||
| flb_output_set(ctx, out_ffd, "region", "us-west-2", NULL); | ||
| flb_output_set(ctx, out_ffd, "bucket", "test-bucket", NULL); | ||
| flb_output_set(ctx, out_ffd, "blob_database_file", "/tmp/test_s3_multipart_persistence.db", NULL); | ||
| flb_output_set(ctx, out_ffd, "total_file_size", "10M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_chunk_size", "5M", NULL); | ||
| flb_output_set(ctx, out_ffd, "upload_timeout", "1s", NULL); | ||
|
|
||
| ret = flb_start(ctx); | ||
| TEST_CHECK(ret == 0); | ||
|
|
||
| sleep(2); | ||
|
|
||
| flb_stop(ctx); | ||
| flb_destroy(ctx); | ||
| unsetenv("FLB_S3_PLUGIN_UNDER_TEST"); | ||
| } |
There was a problem hiding this comment.
/tmp/test_s3_multipart_persistence.db should be unique per test and removed after.
Hard-coding a shared path will break with parallel test execution and can make results order-dependent.
🤖 Prompt for AI Agents
In @tests/runtime/out_s3_multipart.c around lines 430 - 493, The test hard-codes
"/tmp/test_s3_multipart_persistence.db", which causes collisions in parallel
runs; change flb_test_upload_id_persistence to create a unique temporary
filepath (e.g., use mkstemp or tmpfile to generate a unique name), use that
generated path in both flb_output_set calls for "blob_database_file" inside the
function, and at the end of the test ensure the temp file is removed (unlink the
path) before returning; keep all changes local to flb_test_upload_id_persistence
and reference the existing occurrences of "blob_database_file" and the function
name when editing.
2 participants
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Bug Fixes
Tests
Deprecated
✏️ Tip: You can customize this high-level summary in your review settings.