fix(zstd decompression): add correct usage of zstd library

[jonastheis]

Purpose or design rationale of this PR

In zstd the usage of stateless mode is described.

// Create a reader that caches decompressors.
// For this operation type we supply a nil Reader.
var decoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(0))

// Decompress a buffer. We don't supply a destination buffer,
// so it will be allocated by the decoder.
func Decompress(src []byte) ([]byte, error) {
    return decoder.DecodeAll(src, nil)
}

In our case we were initializing the reader (ie stateful mode) with a buffer and then using the stateless DecodeAll method.

PR title

Your PR title must follow conventional commits (as we are doing squash merge for each PR), so it must start with one of the following types:

• build: Changes that affect the build system or external dependencies (example scopes: yarn, eslint, typescript)
• ci: Changes to our CI configuration files and scripts (example scopes: vercel, github, cypress)
• docs: Documentation-only changes
• feat: A new feature
• fix: A bug fix
• perf: A code change that improves performance
• refactor: A code change that doesn't fix a bug, or add a feature, or improves performance
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
• test: Adding missing tests or correcting existing tests

Breaking change label

Does this PR have the breaking-change label?

• No, this PR is not a breaking change
• Yes

Summary by CodeRabbit

• Refactor
  • Streamlined data decompression logic to remove unnecessary intermediates, reducing complexity and potential sources of failure.
  • Adopted a more direct decoding path while preserving existing error handling and validation.
  • Expected minor improvements in performance and memory usage; no changes to user-facing behavior.
  • Improves maintainability and prepares the codebase for future optimizations.

[coderabbitai]

Walkthrough

Switches zstd decompression in encoding/codecv7_types.go to use a DecodeAll call with a nil destination and a decoder initialized via zstd.NewReader(nil, zstd.WithDecoderConcurrency(1)). Removes bytes.Reader usage and minor variable pre-declarations. Error and empty-payload checks remain.

Changes

Cohort / File(s)	Summary of Changes
Zstd decompression refactor encoding/codecv7_types.go	Remove bytes import and reader creation; initialize zstd decoder with nil reader and concurrency option; use DecodeAll(compressedBytes, nil) with short var declaration; retain error and empty-result validation.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Caller
  participant Decompressor as decompressV7Bytes
  participant Zstd as zstd.Decoder

  Caller->>Decompressor: decompressV7Bytes(compressedBytes)
  note over Decompressor: Initialize decoder<br/>zstd.NewReader(nil, WithDecoderConcurrency(1))
  Decompressor->>Zstd: DecodeAll(compressedBytes, nil)
  Zstd-->>Decompressor: payload or error
  alt error or empty payload
    Decompressor-->>Caller: return error
  else success
    Decompressor-->>Caller: return payload
  end

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• fix(zstd decompression): limit concurrency to 1 to prevent deadlock in zstd library #61 — Also modifies decompressV7Bytes in encoding/codecv7_types.go, adopting zstd.NewReader with WithDecoderConcurrency(1) and related DecodeAll usage.

Suggested reviewers

• Thegaram
• colinlyguo
• yiweichi

Poem

I nibbled bytes, then hopped to DecodeAll,
No crumbs of readers left to fall.
With zstd’s breeze and single thread,
I puffed the payload, lightly spread—
A tidy burrow, errors small. 🐇✨

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 0f1ad36 and ae5775d.

📒 Files selected for processing (1)

• encoding/codecv7_types.go (1 hunks)

🧰 Additional context used

🧠 Learnings (5)

📓 Common learnings

Learnt from: colinlyguo
PR: scroll-tech/da-codec#25
File: encoding/codecv2.go:38-47
Timestamp: 2024-10-16T18:43:44.520Z
Learning: In the `DecodeTxsFromBlob` method in `encoding/codecv2.go`, the `compressedBytes` variable will not be empty, so additional error handling for empty `compressedBytes` is unnecessary.

📚 Learning: 2024-10-18T03:40:09.800Z

Learnt from: colinlyguo
PR: scroll-tech/da-codec#25
File: encoding/codecv1_types.go:105-116
Timestamp: 2024-10-18T03:40:09.800Z
Learning: The code in `encoding/codecv1_types.go`, specifically the `Encode` method in `daBatchV1`, has been updated. Previous comments regarding hardcoded byte offsets may be outdated.

Applied to files:

• encoding/codecv7_types.go

📚 Learning: 2024-10-16T18:43:44.520Z

Learnt from: colinlyguo
PR: scroll-tech/da-codec#25
File: encoding/codecv2.go:38-47
Timestamp: 2024-10-16T18:43:44.520Z
Learning: In the `DecodeTxsFromBlob` method in `encoding/codecv2.go`, the `compressedBytes` variable will not be empty, so additional error handling for empty `compressedBytes` is unnecessary.

Applied to files:

• encoding/codecv7_types.go

📚 Learning: 2024-10-17T07:33:28.436Z

Learnt from: colinlyguo
PR: scroll-tech/da-codec#25
File: encoding/codecv2.go:0-0
Timestamp: 2024-10-17T07:33:28.436Z
Learning: In `encoding/codecv2.go`, the `constructBlobPayload` function should remain as is; prefer to keep it without refactoring or additional optimizations.

Applied to files:

• encoding/codecv7_types.go

📚 Learning: 2024-10-16T18:17:33.741Z

Learnt from: colinlyguo
PR: scroll-tech/da-codec#25
File: encoding/codecv4.go:307-310
Timestamp: 2024-10-16T18:17:33.741Z
Learning: In `encoding/codecv4.go`, including `batchBytes` and `blobBytes` in warning logs is acceptable for debugging purposes, as they do not contain sensitive information.

Applied to files:

• encoding/codecv7_types.go

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: tests

🔇 Additional comments (3)

encoding/codecv7_types.go (3)

481-481: Correct usage of zstd stateless API pattern.

The initialization now follows the documented stateless usage pattern with zstd.NewReader(nil, ...) instead of using a bytes.Reader, which is the correct approach for the DecodeAll method.

---

487-487: Simplified variable declaration aligns with stateless pattern.

The removal of pre-declaring the res variable and using a short variable declaration is cleaner and more idiomatic Go style.

---

479-496: Could you share more about your decompression workload? For example:

• Are you decoding one large payload at a time, or many small payloads concurrently?
• What sizes are your typical blobs, and what throughput/latency targets do you have?

In general, the klauspost zstd decoder will scale decomposition across goroutines, so a good starting point is to set WithDecoderConcurrency(runtime.NumCPU()) (or GOMAXPROCS(0)). From there, you can:

• Benchmark wall-clock throughput and memory usage at 1, 2, 4, … N (where N ≈ NumCPU).
• If you have many simultaneous streams, consider limiting per-stream concurrency (e.g. 1–2) and using a shared pool of decoders so total goroutine count ≈ NumCPU to avoid CPU oversubscription.

Once you let me know your workload characteristics, I can suggest concrete settings and a short benchmarking plan.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/zstd-usage

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.