[BLOG] Transparent Compression - #469
Conversation
|
@xdk-amz You write 'All the language bindings should be implemented and the example files should be linked in the blog before publishing.' That's a publishing dependency. Can you please explain more so we can plan this one out? |
|
@stockholmux valkey-io/valkey-glide#5299 here is the epic tracking that work |
|
bump @stockholmux the feature is preparing for a release in 2.4 We would like to pair the feature release with the blog, so feedback would be appreciated |
|
@xdk-amz We're currently in a cadence with the the release of Valkey 9.1 so publishing capacity is very tight. Let me see what I can do. |
|
@valkey-io/technical-blog-reviewers may we please get review on this blog? |
| featured_image = "/assets/media/featured/random-03.webp" | ||
| +++ | ||
|
|
||
| If you're caching JSON API responses, storing user sessions, or buffering HTML fragments in Valkey, there's a good chance your data is highly compressible while you're still paying full price for every byte. At scale, redundant data adds up fast. ~1KB JSON payloads across millions of keys means gigabytes of memory that could be reclaimed without losing a single field. Even if your overall cache footprint is small, availability zone and cross-region data transfer fees can still drive up your costs. In this blog, we'll demonstrate how to configure the Valkey GLIDE using the Go client for transparent client-side compression and explore performance benchmarks to evaluate the effectiveness and guide recommendations for usage. |
There was a problem hiding this comment.
availability zone and cross-region data transfer fees -> summarize as network costs -> availability zone can be a term limited to particular cloud providers
|
|
||
| If you're caching JSON API responses, storing user sessions, or buffering HTML fragments in Valkey, there's a good chance your data is highly compressible while you're still paying full price for every byte. At scale, redundant data adds up fast. ~1KB JSON payloads across millions of keys means gigabytes of memory that could be reclaimed without losing a single field. Even if your overall cache footprint is small, availability zone and cross-region data transfer fees can still drive up your costs. In this blog, we'll demonstrate how to configure the Valkey GLIDE using the Go client for transparent client-side compression and explore performance benchmarks to evaluate the effectiveness and guide recommendations for usage. | ||
|
|
||
| Transparent compression in [Valkey GLIDE](https://github.com/valkey-io/valkey-glide) offers a seamless solution to this problem. When you write data with a `SET` command, GLIDE compresses it before sending it to the server. When you read it back with `GET`, GLIDE decompresses it automatically. Your application code doesn't change — you just flip a switch in the client configuration: |
There was a problem hiding this comment.
Quantify - Add how much compression may be achieved, can a percentage range
There was a problem hiding this comment.
- you mention Valkey GLIDE in the first paragraph without linking but link it in the second. Linking the first mention is best practice.
- Should you use the GitHub repo for GLIDE? What about docs?
- You say it's a 'a seamless solution this problem' but I'm unclear what it's solving. I'd be more explicit about the problem.
- Revise to avoid using the word 'just'. In technical writing it implies a level of technical competency that can, if the reader doesn't immediately understand, disempower.
There was a problem hiding this comment.
Quantify - Add how much compression may be achieved, can a percentage range
At scale, redundant data adds up fast.
->
At scale, redundant data adds up fast and compression can offer up to 49% lower value sizes.
you mention Valkey GLIDE in the first paragraph without linking but link it in the second. Linking the first mention is best practice.
linked first (didnt remove the second link, should I?)
Should you use the GitHub repo for GLIDE? What about docs?
The repo points to docs and this is a technical blog for developers who are likely familiar with github
You say it's a 'a seamless solution this problem' but I'm unclear what it's solving. I'd be more explicit about the problem.
offers a seamless solution to this problem.
->
offers a seamless solution to reducing Valkey's storage and bandwidth requirements.
I also rewrote the first paragraph to focus more on the problem and set the frame for the feature/solution
Revise to avoid using the word 'just'. In technical writing it implies a level of technical competency that can, if the reader doesn't immediately understand, disempower.
removed
| result, _ := client.Get(ctx, "user:1001:profile") | ||
| ``` | ||
|
|
||
| This single configuration change delivers 28–49% memory savings depending on algorithm choice and data shape, with LZ4 showing near-zero throughput impact and Zstandard (zstd) nearly halving memory usage at a moderate write throughput and latency cost. |
There was a problem hiding this comment.
This is awesome. Move the line on impact up before the code. The readers should understand the benefits before having to go through code blocks
|
|
||
| Transparent compression in [Valkey GLIDE](https://github.com/valkey-io/valkey-glide) offers a seamless solution to this problem. When you write data with a `SET` command, GLIDE compresses it before sending it to the server. When you read it back with `GET`, GLIDE decompresses it automatically. Your application code doesn't change — you just flip a switch in the client configuration: | ||
|
|
||
| ```go |
There was a problem hiding this comment.
IS this only for the Glide go client? call out in prose
There was a problem hiding this comment.
added 'is available for all GLIDE-supported languages' in the first sentence
| result, _ := client.Get(ctx, "user:1001:profile") | ||
| ``` | ||
|
|
||
| This single configuration change delivers 28–49% memory savings depending on algorithm choice and data shape, with LZ4 showing near-zero throughput impact and Zstandard (zstd) nearly halving memory usage at a moderate write throughput and latency cost. |
There was a problem hiding this comment.
LZ4 showing near-zero throughput impact and Zstandard (zstd) nearly halving memory usage at a moderate write throughput and latency cost.
We address throughput impact in LZ4 but not memory, cost or latency impact
Zstd memory is quantified but not throughput or latency impact (give a range).
Standardize between the two algos - call out impact on memory and latency/throughput
There was a problem hiding this comment.
I changed this to just focus on LZ4 since it shows a default configuration (LZ4 is the default) and just say 'effectively zero throughput/latency impact'.
Spelling out the ranges for both tps and latency is quite wordy and does not read well for an introductory hook. 'effectively zero' is true for LZ4 and everyone knows what zero means, vs. 'moderate' for ZSTD requires numbers and creates the verbosity problem, so this is a neat compromise.
| | Session | 480 | 16.9% | 0.0% | | ||
| | Session | 951 | 24.1% | 11.5% | | ||
|
|
||
| A few patterns jump out: |
There was a problem hiding this comment.
Love this section - Can you add here what are the factors of data on which the compression % depends on? Like size, uniqueness, datatypes etc... this will help guide customers on what to expect for their data
There was a problem hiding this comment.
Like size
Value size matters more than data type. Below ~100 bytes, neither algorithm saves meaningful memory — the 5-byte header overhead and Valkey's per-key metadata dominate. Above ~500 bytes, both algorithms deliver substantial savings across all data types.
I cover size here, should I reference more points? IE I can add that above 200 bytes is where compression starts to be effective for structured data (JSON/Session/HTML/etc.) and that above 1kb compression starts to see its maximum value
Uniqueness
I allude to this in the html section regarding repeated tags, patterns, etc. but I can shift this from HTML specific to be headlined as 'Redundant data compresses well' -- and describe what conditions lead to that (repeated tags, attributes, etc.) ie:
Redundant data compresses best Repeated tags, field names, attributes, delimiters, and structural patterns give compression algorithms plenty to work with.
datatypes
Per above, I have an HTML section and one on session data, but I can revise the headlined sections to have one specific to datatypes instead (side note, I will change 'data type' in reference to the compressed value data to be 'content type' to sufficiently distinguish it from valkey 'data types' ie string, set..):
Value size matters more than data type. Below ~100 bytes, neither algorithm saves meaningful memory ...
Redundant data leads to the best results Repeated tags, attributes, ...
Content type differences Session payloads tend to be more random — UUIDs, tokens, timestamps .. HTML contains many repeated tags... JSON's repeated character structure and field names...
zstd consistently beats LZ4 on compression ratio. Across every data type and size, zstd saves more memory. The gap is widest on highly compressible data and narrowest on small or low-redundancy data.
|
|
||
| ## The Core Tradeoff: Memory vs Throughput | ||
|
|
||
| Benchmarks were generated using the Go GLIDE client on Amazon EC2 r7g.2xlarge instances (8 vCPUs, 64 GB RAM, AWS Graviton3) with the client and Valkey 8.0 server running on separate hosts in the same AWS VPC. The test corpus was JSON payloads averaging ~1,884 bytes per value. This value size was chosen to drive compression to its limits by giving the compression algorithms enough data to work with while still being sensibly-sized. A benchmark script swept a matrix of 80 configurations across goroutine counts (1, 2, 4, 8, 10, 25, 100, 1000) and pipeline batch sizes (1, 5, 10, 20, 50). |
There was a problem hiding this comment.
nit - call out para header as methodology
|
|
||
| Compression works identically with `ClusterClient` — just pass the same compression configuration to your cluster config. Compression and decompression happen entirely on the client side, so there is no difference in behavior between standalone and cluster modes. | ||
|
|
||
| ## Gradual Rollout |
There was a problem hiding this comment.
Can we add a best practices or guidance section here based on all the performance characteristic discussed? Its scatter through the doc and a reader scanning through the blog will find it hard to pick up on the nuances across the doc.
|
|
||
|
|
||
|
|
||
| ## Appendix: Sample Data from the Benchmark Corpus |
There was a problem hiding this comment.
This appendix section can be removed or needs to be moved to a samples section/folder within Glide/Valkey that you can direct customers to.
| featured_image = "/assets/media/featured/random-03.webp" | ||
| +++ | ||
|
|
||
| If you're caching JSON API responses, storing user sessions, or buffering HTML fragments in Valkey, there's a good chance your data is highly compressible while you're still paying full price for every byte. At scale, redundant data adds up fast. ~1KB JSON payloads across millions of keys means gigabytes of memory that could be reclaimed without losing a single field. Even if your overall cache footprint is small, availability zone and cross-region data transfer fees can still drive up your costs. In this blog, we'll demonstrate how to configure the Valkey GLIDE using the Go client for transparent client-side compression and explore performance benchmarks to evaluate the effectiveness and guide recommendations for usage. |
There was a problem hiding this comment.
This blog does a lot more -
Suggested - In this blog, we'll demonstrate how client-side compression with Valkey GLIDE Go client solves this problem, deep-dive into how the compression works and explore performance characteristics. This blog provides you with guidance and best practices on configuring client-side caching to help you get started with savings.
Suggest that this comes after line 14.
stockholmux
left a comment
stockholmux
left a comment
There was a problem hiding this comment.
I like the blog post.
Needs some updating and there are some language issues that need to be tightened up.
It's a little long and feels repetitious - can you keep the technical level but condense it?
| featured_image = "/assets/media/featured/random-03.webp" | ||
| +++ | ||
|
|
||
| If you're caching JSON API responses, storing user sessions, or buffering HTML fragments in Valkey, there's a good chance your data is highly compressible while you're still paying full price for every byte. At scale, redundant data adds up fast. ~1KB JSON payloads across millions of keys means gigabytes of memory that could be reclaimed without losing a single field. Even if your overall cache footprint is small, availability zone and cross-region data transfer fees can still drive up your costs. In this blog, we'll demonstrate how to configure the Valkey GLIDE using the Go client for transparent client-side compression and explore performance benchmarks to evaluate the effectiveness and guide recommendations for usage. |
There was a problem hiding this comment.
- 'to configure the Valkey GLIDE' -> 'to configure Valkey GLIDE' (remove definite article)
- "blog" -> "blog post" or "post" (typically refers to a collection of posts, but a single item inside that blog is a "blog post")
- You go from 2nd person singular to ("you") to 'we'/'our' throughout the this post. Typically, for a post like this, stick with 2nd person singular, otherwise the reader doesn't know who 'we' and 'our' refers to.
|
|
||
| If you're caching JSON API responses, storing user sessions, or buffering HTML fragments in Valkey, there's a good chance your data is highly compressible while you're still paying full price for every byte. At scale, redundant data adds up fast. ~1KB JSON payloads across millions of keys means gigabytes of memory that could be reclaimed without losing a single field. Even if your overall cache footprint is small, availability zone and cross-region data transfer fees can still drive up your costs. In this blog, we'll demonstrate how to configure the Valkey GLIDE using the Go client for transparent client-side compression and explore performance benchmarks to evaluate the effectiveness and guide recommendations for usage. | ||
|
|
||
| Transparent compression in [Valkey GLIDE](https://github.com/valkey-io/valkey-glide) offers a seamless solution to this problem. When you write data with a `SET` command, GLIDE compresses it before sending it to the server. When you read it back with `GET`, GLIDE decompresses it automatically. Your application code doesn't change — you just flip a switch in the client configuration: |
There was a problem hiding this comment.
- you mention Valkey GLIDE in the first paragraph without linking but link it in the second. Linking the first mention is best practice.
- Should you use the GitHub repo for GLIDE? What about docs?
- You say it's a 'a seamless solution this problem' but I'm unclear what it's solving. I'd be more explicit about the problem.
- Revise to avoid using the word 'just'. In technical writing it implies a level of technical competency that can, if the reader doesn't immediately understand, disempower.
| cfg := config.NewClientConfiguration(). | ||
| WithAddress(&config.NodeAddress{Host: "localhost", Port: 6379}). | ||
| WithCompressionConfiguration( | ||
| config.NewCompressionConfiguration(), // That's it. |
There was a problem hiding this comment.
I missed the "// That's it" comment when I read the blog.
Also, there is a lot of extraneous code here. Maybe just show the config.NewClientConfiguration block and show a before and after.
There was a problem hiding this comment.
I removed the imports, and the "That's it" comment
I think the client creation, set/get is a good way to build context around the usage for people unfamiliar with using Valkey GLIDE
a before/after would be as many lines and far less information. I can add a comment to the compression line that makes it clear that its the 'new' part and is more descriptive than "That's it" (also feels like "That's it" can be in the same disempowering space as 'just')
// before
cfg := config.NewClientConfiguration().
WithAddress(&config.NodeAddress{Host: "localhost", Port: 6379})
// after
cfg := config.NewClientConfiguration().
WithAddress(&config.NodeAddress{Host: "localhost", Port: 6379}).
WithCompressionConfiguration(
config.NewCompressionConfiguration(),
)
| 3. Is the compressed result actually smaller than the original? If not, send the original instead. | ||
| 4. Prepend a 5-byte header that identifies the data as GLIDE-compressed, and send it to the server. | ||
|
|
||
| On the read path, the process reverses. GLIDE checks for the header, decompresses if present, and returns the original value. If the header isn't there, the value is returned as-is. This allows compression-enabled clients to seamlessly read uncompressed data written by older clients. |
There was a problem hiding this comment.
I would say 'other clients' instead of 'older clients' (totally up to date clients could be co-existing...)
|
|
||
| On the read path, the process reverses. GLIDE checks for the header, decompresses if present, and returns the original value. If the header isn't there, the value is returned as-is. This allows compression-enabled clients to seamlessly read uncompressed data written by older clients. | ||
|
|
||
| GLIDE uses a 5-byte header (`[Magic Prefix: 3 bytes][Version: 1 byte][Backend ID: 1 byte]`) to tag compressed values. You can identify compressed entries when inspecting raw data in Valkey by looking for this header. The backend ID means a zstd-configured client can read LZ4-compressed data and vice versa. All GLIDE language bindings (Python, Node.js, Java, Go, C#) share the same header format, so compressed data written from one language can be read from another. |
There was a problem hiding this comment.
(Python, Node.js, Java, Go, C#) what about Ruby?
There was a problem hiding this comment.
I just removed the listed languages and changed it to 'All supported Glide language..' since Ruby is not currently listed as supported, and neither is C# (but C# is supported for this feature)
|
|
||
| ## The Core Tradeoff: Memory vs Throughput | ||
|
|
||
| Benchmarks were generated using the Go GLIDE client on Amazon EC2 r7g.2xlarge instances (8 vCPUs, 64 GB RAM, AWS Graviton3) with the client and Valkey 8.0 server running on separate hosts in the same AWS VPC. The test corpus was JSON payloads averaging ~1,884 bytes per value. This value size was chosen to drive compression to its limits by giving the compression algorithms enough data to work with while still being sensibly-sized. A benchmark script swept a matrix of 80 configurations across goroutine counts (1, 2, 4, 8, 10, 25, 100, 1000) and pipeline batch sizes (1, 5, 10, 20, 50). |
There was a problem hiding this comment.
Make clear that Graviton is an ARM core... not everyone knows that.
|
|
||
| ## The Core Tradeoff: Memory vs Throughput | ||
|
|
||
| Benchmarks were generated using the Go GLIDE client on Amazon EC2 r7g.2xlarge instances (8 vCPUs, 64 GB RAM, AWS Graviton3) with the client and Valkey 8.0 server running on separate hosts in the same AWS VPC. The test corpus was JSON payloads averaging ~1,884 bytes per value. This value size was chosen to drive compression to its limits by giving the compression algorithms enough data to work with while still being sensibly-sized. A benchmark script swept a matrix of 80 configurations across goroutine counts (1, 2, 4, 8, 10, 25, 100, 1000) and pipeline batch sizes (1, 5, 10, 20, 50). |
There was a problem hiding this comment.
Can we re-test on Valkey 9.1?
There was a problem hiding this comment.
can be done, will take time to reassemble all the pieces
|
|
||
| Benchmarks were generated using the Go GLIDE client on Amazon EC2 r7g.2xlarge instances (8 vCPUs, 64 GB RAM, AWS Graviton3) with the client and Valkey 8.0 server running on separate hosts in the same AWS VPC. The test corpus was JSON payloads averaging ~1,884 bytes per value. This value size was chosen to drive compression to its limits by giving the compression algorithms enough data to work with while still being sensibly-sized. A benchmark script swept a matrix of 80 configurations across goroutine counts (1, 2, 4, 8, 10, 25, 100, 1000) and pipeline batch sizes (1, 5, 10, 20, 50). | ||
|
|
||
| Here's how throughput scales with goroutines for batch sizes 1 and 10 for SET and GET operations: |
There was a problem hiding this comment.
IIRC, goroutines are fairly different from other threading models. What does this look like in other languages that are threaded differently (Python? Node.js?)
There was a problem hiding this comment.
Goroutines in this case are most comparable to logical callers in other languages using an async backend. Their closest comparison is Python asyncio tasks or JavaScript promises/tasks in Node.js.
Threads are basically the same between different languages, they just run the goroutines/tasks/promises based on the scheduler (which is more language-dependent).
|
|
||
| **Start with LZ4** if latency matters. Switch to zstd if you need maximum memory savings and can take a slight hit to latency. The savings you'll see depend heavily on your data type and value size — HTML compresses best, session data compresses least, and anything under 100 bytes isn't worth compressing. Skip compression entirely for already-compressed data (images, video, pre-compressed content). | ||
|
|
||
| **Throughput** can be recovered when using zstd by investing in more compute. Scale your application horizontally or vertically and you can bring zstd throughput up to par with your previously uncompressed workload. While this increases your application's compute costs, the storage savings can completely offset this depending on your throughput vs. storage needs. As an example using on-demand pricing in us-east-1 as of mid-2025: consider a 250GB caching workload running on an AWS r7g.16xlarge ($3.427/hour) served by an application running on a c7g.4xlarge ($0.5781/hour). If zstd can cut your storage requirements by ~40%, you can downgrade your cache instance to an r7g.8xlarge ($1.714/hour) and upgrade your application to a c7g.8xlarge ($1.1562/hour) to make up for the zstd throughput hit for a net savings of $1.1349/hour or a ~28% cost reduction overall. |
There was a problem hiding this comment.
is this pricing still accurate?
There was a problem hiding this comment.
yes and updated year reference to 2026
|
|
||
|
|
||
|
|
||
| ## Appendix: Sample Data from the Benchmark Corpus |
|
@xdk-amz Please see the technical reviewers comments on this PR. If you have any questions, please do let us know. |
Signed-off-by: Dante Knowles <xdk@amazon.com>
📝 WalkthroughWalkthroughAdds a Dante Knowles author profile and a blog post about transparent client-side compression in Valkey GLIDE, including behavior, benchmarks, Go configuration, backend comparisons, and rollout guidance. ChangesAuthor profile
Transparent compression article
Suggested reviewers: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@content/blog/2026-03-16-transparent-compression/index.md`:
- Line 16: Update the LZ4 summaries at
content/blog/2026-03-16-transparent-compression/index.md:16-16 and :173-173 to
use qualified wording such as “low overhead in this benchmark” instead of
claiming effectively zero impact or being effectively free; at :173-173, also
cite the measured 0.87x–1.07x SET throughput range.
- Line 4: Align the post’s front-matter date with the publication path
`2026-03-16`: update the `date` value to the intended March 16, 2026 publication
date, or rename the directory if July 1, 2025 is canonical. Ensure both
represent one consistent publication date.
- Around line 47-58: Revise the compression compatibility note and Unsupported
Commands section to describe command-specific failures: numeric operations may
error, reads and bit operations may inspect compressed bytes, and mutations may
corrupt the stored frame so it cannot be decoded. Remove the recommendation to
replicate commands client-side, and instead recommend uncompressed keys or an
explicitly concurrency-safe design.
- Line 169: The compression rollout guidance must not imply that unconfigured
clients can safely read data rewritten by compression-enabled clients. Update
the paragraph to require compression support for every reader of affected keys,
or document a versioned/dual-read-write rollout with explicit cutover steps, and
remove the unconditional “no migration scripts, no downtime” promise.
- Around line 18-30: Update the “To get started” section to link to complete
standalone and cluster GLIDE examples, or explicitly label the shown Go blocks
as partial snippets. If keeping them as runnable examples, define ctx and
userDataJSON, handle the error returned by glide.NewClient, and close the client
before completion.
- Line 43: Update the compression interoperability statement in the blog content
to remove the claim that zstd- and LZ4-configured clients can read each other’s
data. Retain the cross-language shared-header claim, and replace the backend
wording with the configuration terminology used in the compression
documentation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5bbc087f-bfee-442c-bc88-9d42839a24b3
⛔ Files ignored due to path filters (8)
content/blog/2026-03-16-transparent-compression/images/graph_batch_impact.pngis excluded by!**/*.pngcontent/blog/2026-03-16-transparent-compression/images/graph_latency_heatmap.pngis excluded by!**/*.pngcontent/blog/2026-03-16-transparent-compression/images/graph_latency_rerun_remote_nodivide.pngis excluded by!**/*.pngcontent/blog/2026-03-16-transparent-compression/images/graph_memory_by_type.pngis excluded by!**/*.pngcontent/blog/2026-03-16-transparent-compression/images/graph_ratio_heatmap.pngis excluded by!**/*.pngcontent/blog/2026-03-16-transparent-compression/images/graph_scaling_get.pngis excluded by!**/*.pngcontent/blog/2026-03-16-transparent-compression/images/graph_scaling_set.pngis excluded by!**/*.pngstatic/assets/media/authors/dknowles.jpegis excluded by!**/*.jpeg
📒 Files selected for processing (2)
content/authors/dknowles.mdcontent/blog/2026-03-16-transparent-compression/index.md
| +++ | ||
| title= "Transparent Compression in Valkey GLIDE: Reduce Memory With a Single Line of Code" | ||
| description= "Learn how to enable automatic compression in Valkey GLIDE to reduce memory usage by up to 49% without modifying your application code." | ||
| date= 2025-07-01 01:01:01 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Align the front-matter date with the publication path.
The path targets March 16, 2026, but date is July 1, 2025. This can place the post in the wrong publication order and before the planned 2.4 release. Set one canonical date, or rename the directory to match the intended publication date.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@content/blog/2026-03-16-transparent-compression/index.md` at line 4, Align
the post’s front-matter date with the publication path `2026-03-16`: update the
`date` value to the intended March 16, 2026 publication date, or rename the
directory if July 1, 2025 is canonical. Ensure both represent one consistent
publication date.
|
|
||
| Transparent compression in [Valkey GLIDE](https://github.com/valkey-io/valkey-glide) is available for all GLIDE-supported languages and offers a seamless solution to reducing Valkey's storage and bandwidth requirements for compatible workloads. When you write data with a `SET` command, GLIDE compresses it before sending it to the server. When you read it back with `GET`, GLIDE decompresses it automatically. You can enable the feature with a single flag in your client configuration and no modifications to your application's logic. In this post, you'll learn how to configure [Valkey GLIDE](https://github.com/valkey-io/valkey-glide) using the Go client for transparent client-side compression and deep-dive into how the compression works. The performance benchmarking data and best practices guidance will help you understand if your caching workload is a good fit for compression and how to get started on savings. | ||
|
|
||
| The single configuration change shown below sets up LZ4, the default compression backend, and delivers memory savings of 27.8% with effectively zero throughput/latency impact on our benchmarked 2KB JSON workload. |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Match the LZ4 summary to the measured throughput range.
The benchmark reports 0.87x–1.07x SET throughput for LZ4 at Line 111. That includes a 13% slowdown, so “effectively zero throughput/latency impact” and “effectively free” overstate the result.
content/blog/2026-03-16-transparent-compression/index.md#L16-L16: use qualified wording such as “low overhead in this benchmark.”content/blog/2026-03-16-transparent-compression/index.md#L173-L173: use the same qualified wording and cite the measured range.
📍 Affects 1 file
content/blog/2026-03-16-transparent-compression/index.md#L16-L16(this comment)content/blog/2026-03-16-transparent-compression/index.md#L173-L173
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@content/blog/2026-03-16-transparent-compression/index.md` at line 16, Update
the LZ4 summaries at
content/blog/2026-03-16-transparent-compression/index.md:16-16 and :173-173 to
use qualified wording such as “low overhead in this benchmark” instead of
claiming effectively zero impact or being effectively free; at :173-173, also
cite the measured 0.87x–1.07x SET throughput range.
| ```go | ||
| cfg := config.NewClientConfiguration(). | ||
| WithAddress(&config.NodeAddress{Host: "localhost", Port: 6379}). | ||
| WithCompressionConfiguration( | ||
| config.NewCompressionConfiguration(), | ||
| ) | ||
|
|
||
| client, err := glide.NewClient(cfg) | ||
|
|
||
| // Everything else stays exactly the same | ||
| client.Set(ctx, "user:1001:profile", string(userDataJSON)) | ||
| result, _ := client.Get(ctx, "user:1001:profile") | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="content/blog/2026-03-16-transparent-compression/index.md"
if [ -f "$file" ]; then
echo "== file exists =="
wc -l "$file"
echo "== lines 1-70 =="
sed -n '1,70p' "$file" | nl -ba -v1
echo "== lines 120-190 =="
sed -n '120,190p' "$file" | nl -ba -v120
else
echo "missing: $file"
git ls-files | rg 'transparent-compression|content/blog' | head -50
fiRepository: valkey-io/valkey-io.github.io
Length of output: 303
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="content/blog/2026-03-16-transparent-compression/index.md"
[ -f "$file" ] || { echo "missing"; exit 0; }
echo "== line count =="
wc -l "$file"
echo "== lines 1-80 =="
awk '{printf "%5d\t%s\n", NR, $0}' "$file" | sed -n '1,80p'
echo "== lines 130-185 =="
awk '{printf "%5d\t%s\n", NR, $0}' "$file" | sed -n '130,185p'
echo "== code block boundaries and identifiers =="
python3 - <<'PY'
from pathlib import Path
p=Path("content/blog/2026-03-16-transparent-compression/index.md")
lines=p.read_text().splitlines()
in_block=False
for i,l in enumerate(lines,1):
if l.startswith("```"):
in_block=not in_block
elif in_block:
for w in l.split():
w=w.rstrip(":;,).[]"):"]"):",)
if "%" not in w and "=" not in w and "(" not in w and "-" not in w:
print(i, w)
PYRepository: valkey-io/valkey-io.github.io
Length of output: 11482
🌐 Web query:
Valkey GLIDE Go client NewClient returns error and Close method
💡 Result:
In the Valkey GLIDE Go client, the NewClient function returns a client instance and an error, while the Close method does not return an error [1][2][3]. NewClient Handling: The NewClient function (e.g., glide.NewClient) is used to initialize a connection to a Valkey server [1][2]. It returns a pointer to the client and an error type [1][2]. If an error occurs during initialization—such as invalid configuration or failure to establish the initial connection—the error should be checked before proceeding to use the client [2]. Example usage: client, err:= glide.NewClient(config) if err!= nil { // Handle the error return } defer client.Close Close Method: The Close method is used to terminate the client and free up associated resources [4][3]. Unlike some other Go Redis clients (such as go-redis), the GLIDE Go client's Close method does not return an error [3][5]. It is designed to be called when the client is no longer needed to ensure proper cleanup [3].
Citations:
- 1: https://github.com/valkey-io/valkey-glide/blob/449f61ac/go/glide_client.go
- 2: https://github.com/valkey-io/valkey-glide/blob/main/go/README.md
- 3: https://glide.valkey.io/migration/go/go-redis/connection-management/
- 4: https://github.com/valkey-io/valkey-glide/blob/main/go/base_client.go
- 5: https://glide.valkey.io/migration/go/go-redis/command-comparison-chart/
🌐 Web query:
github.com valkey-io valkey-glide go v2 NewClient Close ClusterClient NewClientConfiguration documentation
💡 Result:
Valkey GLIDE for Go (v2) provides a high-performance client for interacting with Valkey and Redis. The library is available at github.com/valkey-io/valkey-glide/go/v2 [1][2]. Core components for client initialization and management include: NewClient and NewClusterClient These functions are used to initialize connections for standalone and cluster deployments, respectively [3][4][5]. - NewClient(config *config.ClientConfiguration): Creates a connected instance for a standalone Valkey server [5]. - NewClusterClient(config *config.ClusterClientConfiguration): Creates a connected instance for a Valkey Cluster, handling automatic topology discovery based on the provided seed addresses [3][4]. Configuration Client behavior is defined through dedicated configuration objects [6][3]. - config.NewClientConfiguration: Creates a configuration for a standalone client, where you can specify primary and replica node addresses, credentials, and other options [3]. - config.NewClusterClientConfiguration: Creates a configuration for a cluster client, allowing you to specify initial seed nodes, authentication, TLS, and other settings [3]. - Both configuration types offer fluent API methods such as WithAddress, WithCredentials, and WithRequestTimeout to build the connection settings [3]. Closing Connections The client objects (Client and ClusterClient) inherit from a base client, which includes methods to manage the connection lifecycle. To shut down the client and release resources, you call the Close method on the client instance (e.g., client.Close) [4][5]. Documentation For comprehensive guidance, including examples and configuration details, refer to the official Valkey GLIDE documentation site at https://glide.valkey.io/languages/go [2][7]. Detailed code-level documentation is also available within the repository and the Go package documentation [8][4][5].
Citations:
- 1: https://github.com/valkey-io/valkey-glide/tree/main/go
- 2: https://github.com/valkey-io/valkey-glide/blob/main/go/README.md
- 3: https://github.com/valkey-io/valkey-glide/wiki/Golang-wrapper
- 4: https://github.com/valkey-io/valkey-glide/blob/main/go/glide_cluster_client.go
- 5: https://github.com/valkey-io/valkey-glide/blob/449f61ac/go/glide_client.go
- 6: https://glide.valkey.io/how-to/client-initialization/
- 7: https://github.com/valkey-io/valkey-glide/blob/449f61ac/go/README.md
- 8: https://pkg.go.dev/github.com/valkey-io/valkey-glide/go
Point readers to complete GLIDE examples and name these blocks as snippets.
The blocks here cannot run as-is. The first block uses undefined ctx and userDataJSON, leaves NewClient errors unchecked, and omits Close. Point the “To get started” links to standalone and cluster examples, or label these code blocks as partial snippets.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@content/blog/2026-03-16-transparent-compression/index.md` around lines 18 -
30, Update the “To get started” section to link to complete standalone and
cluster GLIDE examples, or explicitly label the shown Go blocks as partial
snippets. If keeping them as runnable examples, define ctx and userDataJSON,
handle the error returned by glide.NewClient, and close the client before
completion.
|
|
||
| On the read path, the process reverses. GLIDE checks for the header, decompresses if present, and returns the original value. If the header isn't there, the value is returned as-is. This allows compression-enabled clients to seamlessly read uncompressed data written by other clients. | ||
|
|
||
| GLIDE uses a 5-byte header (`[Magic Prefix: 3 bytes][Version: 1 byte][Backend ID: 1 byte]`) to tag compressed values. You can identify compressed entries when inspecting raw data in Valkey by looking for this header. The backend ID means a zstd-configured client can read LZ4-compressed data and vice versa. All supported GLIDE language bindings share the same header format, so compressed data written from one language can be read from another. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 3 \
'CompressionConfiguration|NewCompressionConfiguration|WithBackend|LZ4|ZSTD|Magic|Backend ID|header' .Repository: valkey-io/valkey-io.github.io
Length of output: 31911
🌐 Web query:
valkey-glide compression header magic prefix version Backend ID LZ4 ZSTD
💡 Result:
Valkey GLIDE supports transparent data compression to reduce memory usage and network bandwidth [1][2]. When enabled, the client checks if a value meets a configured minimum size threshold (default: 64 bytes) before compressing it [1][2][3]. Mechanism and Header: When a value is compressed, GLIDE prepends a "magic header" to the payload [1]. This short header acts as a marker to identify the data as compressed and specifies which backend was used (ZSTD or LZ4) [1][4]. During retrieval (e.g., via a GET command), the client examines the first bytes of the received value for this magic header [1]. If found, the client decompresses the payload; otherwise, it returns the value unchanged [1]. Backends: GLIDE supports two compression backends [1][4]: - ZSTD: Offers a better compression ratio at the cost of higher CPU usage [1]. It is the default backend [2][5]. - LZ4: Provides higher speed with lower compression density, making it suitable for latency-sensitive workloads [1]. Safety and Configuration: To prevent "decompression bombs" (maliciously crafted data that expands to an excessively large size), GLIDE includes a maxDecompressedSize limit, which defaults to 512 MB [2][6][7]. This configuration is enforced by the Rust core [7][3][5]. The compression configuration is transparent to the server, which stores the compressed payload without modification [1]. While the internal implementation uses protobuf definitions for its configuration [8], the magic header is specifically managed by the GLIDE client logic to ensure seamless, transparent operation for the end user [1][4].
Citations:
- 1: https://glide.valkey.io/concepts/client-features/compression/
- 2: https://glide.valkey.io/how-to/compressing-data/
- 3: https://glide.valkey.io/languages/java/api/glide/api/models/configuration/CompressionConfiguration.html
- 4: Transparent Compression Feature valkey-glide#4759
- 5: https://github.com/valkey-io/valkey-glide/blob/449f61ac/go/config/compression_config.go
- 6: valkey-io/valkey-glide@97611f8
- 7: https://github.com/valkey-io/valkey-glide/blob/449f61ac/glide-core/src/client/types.rs
- 8: https://github.com/valkey-io/valkey-glide/blob/449f61ac/glide-core/src/protobuf/connection_request.proto
🌐 Web query:
site:github.com/valkey-io/valkey-glide "Backend ID" "GLIDE" compression header Magic Prefix
💡 Result:
In the Valkey GLIDE library, the experimental automatic compression feature utilizes a small header prefixed to compressed data to identify the compression format and manage decompression [1][2][3]. According to the implementation details, this compression header is 5 bytes in size [2]. When a value is compressed, the client stores a header followed by the compressed payload [2]. This 5-byte header allows the Rust core to identify the compression backend (e.g., ZSTD or LZ4) and process the payload appropriately when retrieved [2][4][3]. The feature is currently experimental and requires careful configuration, as it is incompatible with server-side string manipulation commands such as APPEND, INCR, and BITFIELD, which do not understand the compressed format [1][3]. The minimum size threshold for compression defaults to 64 bytes, and the library enforces a default maximum decompressed size of 512MB to prevent decompression bombs [2][3].
Citations:
- 1: https://github.com/valkey-io/valkey-glide
- 2: https://github.com/valkey-io/valkey-glide/blob/449f61ac/go/config/compression_config.go
- 3: https://github.com/valkey-io/valkey-glide/blob/449f61ac/python/glide-shared/glide_shared/config.py
- 4: https://github.com/valkey-io/valkey-glide/blob/main/glide-core/src/client/types.rs
🌐 Web query:
site:glide.valkey.io/concepts/client-features/compression All supported GLIDE language bindings share the same header format
💡 Result:
Valkey GLIDE uses a shared Rust core layer that handles functionality such as data compression across its various language bindings [1]. When compression is enabled, this core layer prepends a magic header to compressed data to identify the compression backend and signify that the payload is compressed [1]. Because this compression logic is implemented within the centralized Rust core rather than in the individual language-specific wrappers, the mechanism for handling this header is consistent across the supported language bindings [1].
Citations:
🌐 Web query:
site:glide.valkey.io/how-to/compressing-data "zstd-configured client can read LZ4-compressed data"
💡 Result:
A zstd-configured client cannot natively read or decompress LZ4-compressed data. The Zstandard (zstd) and LZ4 compression algorithms use distinct formats and implementations; they are not interoperable. While some software systems or libraries, such as Valkey GLIDE, may allow a client to be configured to use either ZSTD or LZ4 as a compression backend, this represents a choice of implementation for that specific client, not the ability for the ZSTD algorithm or binary itself to process LZ4-compressed data [1]. If a client or application is configured specifically to use the ZSTD backend, it expects data compressed in the Zstandard format and will not decompress data compressed with LZ4 [1].
Citations:
🌐 Web query:
glide.valkey.io/concepts/client-features/compression "zstd-configured client can read LZ4-compressed data"
💡 Result:
No, a GLIDE client configured for ZSTD cannot read LZ4-compressed data. GLIDE's transparent compression feature uses a specific backend (either ZSTD or LZ4) that must be selected at the client's creation [1][2]. When compression is enabled, the client prepends a magic header to the data to identify the compression format used [3]. Because the client is configured to use a specific backend, it is not "multi-backend-aware"; a client using a ZSTD backend will not decompress data compressed with LZ4, and vice versa [4]. If you need a client to be able to read existing compressed data stored in Valkey, it must be configured with the corresponding compression backend [4]. As noted in the GLIDE documentation, this feature is currently experimental [5][2].
Citations:
- 1: https://glide.valkey.io/how-to/compressing-data/
- 2: https://github.com/valkey-io/valkey-glide/blob/449f61ac/python/glide-shared/glide_shared/config.py
- 3: https://glide.valkey.io/concepts/client-features/compression/
- 4: Transparent Compression Feature valkey-glide#4759
- 5: https://glide.valkey.io/languages/java/api/glide/api/models/configuration/CompressionConfiguration.html
🌐 Web query:
glide.valkey.io/concepts/client-features/compression "All supported GLIDE language bindings share the same header format"
💡 Result:
Yes, all supported Valkey GLIDE language bindings share the same header format for compressed data [1][2][3]. Because GLIDE is implemented using a shared Rust core, all language-specific wrappers (such as Java, Python, Node.js, and Go) utilize the same underlying logic for compression [3][4][5]. When compression is enabled, the GLIDE client automatically prepends a short magic header to compressed payloads before they are sent to the server [1][6]. This header allows the client to identify that data is compressed and determine which backend (ZSTD or LZ4) was used, ensuring that compressed values can be transparently decompressed upon retrieval regardless of which language client is used [1][7][6]. Technical details regarding this header: - Size: The compression header is 5 bytes in length [7][8]. - Function: It serves as a marker to distinguish compressed data from raw bytes and identifies the compression algorithm [1][6]. - Compatibility: Because the header format is standardized within the shared Rust core, it maintains consistency across all supported language bindings [3][5]. - Server-side status: The Valkey server remains unaware of this compression, simply storing the payload (header plus compressed bytes) as raw data [1][9]. Consequently, operations are transparently handled on the client side [1][9].
Citations:
- 1: https://glide.valkey.io/concepts/client-features/compression/
- 2: https://glide.valkey.io/reference/connection-options/
- 3: https://glide.valkey.io/languages/python/api/
- 4: https://deepwiki.com/valkey-io/valkey-glide/2.2-multi-language-ffi
- 5: https://deepwiki.com/valkey-io/valkey-glide/2-architecture
- 6: Transparent Compression Feature valkey-glide#4759
- 7: https://glide.valkey.io/languages/java/api/glide/api/models/configuration/CompressionConfiguration.html
- 8: https://github.com/valkey-io/valkey-glide/blob/449f61ac/go/config/compression_config.go
- 9: https://glide.valkey.io/how-to/compressing-data/
Fix the backend interoperability claim.
All supported GLIDE bindings share the shared Rust core compression header format, but a zstd-configured client cannot read LZ4-compressed data, and an LZ4-configured client cannot read zstd-compressed data. Remove the “vice versa” claim and use the configuration wording from the compression docs instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@content/blog/2026-03-16-transparent-compression/index.md` at line 43, Update
the compression interoperability statement in the blog content to remove the
claim that zstd- and LZ4-configured clients can read each other’s data. Retain
the cross-language shared-header claim, and replace the backend wording with the
configuration terminology used in the compression documentation.
| ### Important Note: | ||
| Compression is NOT compatible with commands that manipulate string data on the server. Since the client handles the compression, the server is unable to mutate the string data and will corrupt the data. | ||
|
|
||
| **Unsupported Commands:** | ||
|
|
||
| * APPEND, GETRANGE, SETRANGE, STRLEN, LCS | ||
|
|
||
| * INCR, INCRBY, INCRBYFLOAT, DECR, DECRBY | ||
|
|
||
| * GETBIT, SETBIT, BITCOUNT, BITPOS, BITFIELD, BITFIELD_RO, BITOP | ||
|
|
||
| You can replicate these commands on the client-side, but it will come at a performance cost. If you rely on these commands, you may not find compression suitable for your workload. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Document command failures precisely and do not recommend unsafe emulation.
These commands do not have one failure mode. Numeric commands can return an error. Read and bit commands can inspect compressed bytes. Mutating commands can make the stored frame undecodable. A client-side GET/decompress/mutate/SET replacement is also not atomic and can lose concurrent updates. Recommend uncompressed keys or an explicitly concurrency-safe design.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@content/blog/2026-03-16-transparent-compression/index.md` around lines 47 -
58, Revise the compression compatibility note and Unsupported Commands section
to describe command-specific failures: numeric operations may error, reads and
bit operations may inspect compressed bytes, and mutations may corrupt the
stored frame so it cannot be decoded. Remove the recommendation to replicate
commands client-side, and instead recommend uncompressed keys or an explicitly
concurrency-safe design.
|
|
||
| ## Gradual Rollout | ||
|
|
||
| One of the most practical aspects of GLIDE's compression design is that you don't need to plan a migration. Compression-enabled clients read uncompressed data transparently. Clients configured with zstd can read LZ4-compressed data and vice versa. This means you can roll out incrementally: deploy your application clients with compression enabled, and as keys expire or get updated through normal application flow, data naturally migrates to compressed format — no migration scripts, no downtime. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Correct the mixed-client rollout guidance.
Only compression-enabled clients are documented as decoding the header. If a compression-enabled client updates a key, an unconfigured client can receive compressed bytes. Require every reader of an affected key to support compression, or document key versioning, dual-read/write, and cutover steps. Do not promise “no downtime” without these conditions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@content/blog/2026-03-16-transparent-compression/index.md` at line 169, The
compression rollout guidance must not imply that unconfigured clients can safely
read data rewritten by compression-enabled clients. Update the paragraph to
require compression support for every reader of affected keys, or document a
versioned/dual-read-write rollout with explicit cutover steps, and remove the
unconditional “no migration scripts, no downtime” promise.
|
Posted a revision based on feedback, still missing a dedicated Best Practices section |
5 participants
Description
Adds a blog that walks through the new Transparent Compression feature from the perspective of a Go client user. The blog contains examples and benchmark data for different client configurations to help inform potential users of the benefits based on their workload.
There is still some TODO work on the blog but I want to get early feedback so I can iterate on the structure and content. I think I have way too many graphs and can probably cut down on some but I love graphs so I have left them in and will pare down from here if requested.
Issues Resolved
Will resolve #314
Check List
--signoffBy submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.