v1.9.0

[romange]

Dragonfly v1.9.0

This release includes a number of important stability and security fixes, as well as some features that have been requested by users for a long time:

1. ACLs : starting from this release, you can now create multiple users in Dragonfly and control their access via acl categories. The following ACL sub-commands are now supported:
   ACL SETUSER/DELUSER/WHOAMI/LIST. Please note that ACLS are not persisted yet through process restarts.
2. Improved sorted-sets. We rolled out our experimental support for sorted sets (aka ZSET). They require 35% less memory and are more CPU efficient, especially for large sets. To try them now with --use_zset_tree.
3. Support for CONFIG GET/SET - now some runtime options can be changed without restarting the server (config get command support #1700).
4. The maxclients flag now prevents the number of client connections from growing indefinitely (introduce maxclients flag #1496).
5. New flags max_multi_bulk_len and max_client_iobuf_len control the limits for large requests. The former controls how many elements can be sent via arrays (for commands like MSET, SADD etc). The latter controls maximum buffer size when accepting string blobs (Error: Multi bulk len is too big 245002 #1771) .
6. HTTP endpoint authorization: now our HTTP endpoint is protected with the same token that is passed with --requirepass=token (Enable authentication for the HTTP interface. #1792)
   Once basic auth is set, one can access the http port via curl -u user:token localhost:6379/

What's Changed

• feat(server): add oom guard by @adiholden in feat(server): add oom guard #1650
• fix(server): update denyoom flag for all commands by @adiholden in fix(server): update denyoom flag for all commands #1651
• test(server): Run all unit tests with --force_epoll by @chakaz in test(server): Run all unit tests with --force_epoll #1672
• chore: Refactor SortedMap by @romange in chore: Refactor SortedMap #1666
• fix(VersionMonitor): request to update on older version by @kostasrim in fix(VersionMonitor): request to update on older version #1674
• feat(server): support config set for some flags by @adiholden in feat(server): support config set for some flags #1624
• opt(server): Execute lua on target shard, if it's 1 by @chakaz in opt(server): Execute lua on target shard, if it's 1 #1639
• introduce --replicaof flag by @talbii in introduce --replicaof flag #1583
• test(server): test transaction locked keys by @adiholden in test(server): test transaction locked keys #1669
• chore: Connection fixes by @dranikpg in chore: Connection fixes #1663
• fix: fix dchecks by @dranikpg in fix: fix dchecks #1681
• chore: refactor a common code into a function by @romange in chore: refactor a common code into a function #1685
• fix(daily-builds): temporarily comment out bullmq tests by @Abhra303 in fix(daily-builds): temporarily comment out bullmq tests #1679
• opt(server): Run Lua on remote thread only if it's a different thread by @chakaz in opt(server): Run Lua on remote thread only if it's a different thread #1684
• chore: get rid of cmdstats_map by @romange in chore: get rid of cmdstats_map #1687
• chore: move RdImpl functions together by @romange in chore: move RdImpl functions together #1694
• fix(ci): use auth mechanism with tls helm values by @Pothulapati in fix(ci): use auth mechanism with tls helm values #1696
• feat(helm) - Enable configuring service type LoadBalancer with a Static IP by @mutiadavid in feat(helm) - Enable configuring service type LoadBalancer with a Static IP #1689
• chore: implemente User and UserRegistry classes by @kostasrim in chore: implemente User and UserRegistry classes #1693
• chore: several improvements around sorted map by @romange in chore: several improvements around sorted map #1699
• security: Fix the password hashing in UserRegistry. by @royjacobson in security: Fix the password hashing in UserRegistry. #1702
• fix: allow integer tokens in search queries by @dranikpg in fix: allow integer tokens in search queries #1701
• feat: Update search index on expiry by @dranikpg in feat: Update search index on expiry #1697
• fix: MONITOR now works for multi transactions by @dranikpg in fix: MONITOR now works for multi transactions #1675
• feat: Pipeline squashing by @dranikpg in feat: Pipeline squashing #1619
• fix: Fix squashing, pytest arg formatting by @dranikpg in fix: Fix squashing, pytest arg formatting #1712
• fix: fix batching during dispatch mode changes by @dranikpg in fix: fix batching during dispatch mode changes #1713
• feat: Support unicode strings in search by @dranikpg in feat: Support unicode strings in search #1698
• feat: json paths in search by @dranikpg in feat: json paths in search #1695
• feat: introduce partial DfImpl implementation for sorted_map by @romange in feat: introduce partial DfImpl implementation for sorted_map #1714
• feat: implement GetRange/GetLexRange for bptree. by @romange in feat: implement GetRange/GetLexRange for bptree. #1715
• chore: add acl categories to all commands by @kostasrim in chore: add acl categories to all commands #1711
• fix: Add small timeout to monitor by @dranikpg in fix: Add small timeout to monitor #1718
• replication: fix misleading log message about full sync by @royjacobson in replication: fix misleading log message about full sync #1717
• chore: clean up save by @dranikpg in chore: clean up save #1709
• feat: implement DeleteRange functions in DfImpl by @romange in feat: implement DeleteRange functions in DfImpl #1716
• tools: Hash defrag script by @dranikpg in tools: Hash defrag script #1723
• fix: Make restore accept ttl in ms by @dranikpg in fix: Make restore accept ttl in ms #1724
• feat(AclFamilly): add acl list command by @kostasrim in feat(AclFamilly): add acl list command #1722
• chore: Add a daily coverage run by @royjacobson in chore: Add a daily coverage run #1437
• chore: Mark slow pytests as 'slow' so they can be excluded. by @royjacobson in chore: Mark slow pytests as 'slow' so they can be excluded. #1720
• fix: Correctly read container limits when running under k8s by @talbii in fix: Correctly read container limits when running under k8s #1692
• fix(AclFamily): commands parsing and error handling by @kostasrim in fix(AclFamily): commands parsing and error handling #1726
• feat(AclFamily): add acl setuser command by @kostasrim in feat(AclFamily): add acl setuser command #1725
• feat: search index persistence by @dranikpg in feat: search index persistence #1721
• fix: add proper printing of sha256 passwords by @kostasrim in fix: add proper printing of sha256 passwords #1728
• chore(gh-workflows): add regression tests to release job by @kostasrim in chore(gh-workflows): add regression tests to release job #1665
• fix(ACL): Fix the iteration over categories loop in AclToString by @royjacobson in fix(ACL): Fix the iteration over categories loop in AclToString #1731
• chore: Split the build & test into 2 stages by @royjacobson in chore: Split the build & test into 2 stages #1733
• fix: Fix inconsistency when deleting dash entries by @romange in fix: Fix inconsistency when deleting dash entries #1734
• feat: Defragmentation for hash values by @dranikpg in feat: Defragmentation for hash values  #1727
• chore: add clear metric for the fragmentation waste by @romange in chore: add clear metric for the fragmentation waste #1735
• fix: fix search test by @dranikpg in fix: fix search test #1739
• chore: integrate sccache by @romange in chore: integrate sccache #1738
• chore: cover zset over bptree implementation in the CI. by @romange in chore: cover zset over bptree implementation in the CI. #1736
• feat(AclFamily): add AUTH for acl members by @kostasrim in feat(AclFamily): add AUTH for acl members #1732
• fix: fix index loading by @dranikpg in fix: fix index loading #1742
• fix: the bug where iobuf should be enlarged during the replication by @romange in fix: the bug where iobuf should be enlarged during the replication #1744
• feat(server): support multi eval in lock ahead mode by @adiholden in feat(server): support multi eval in lock ahead mode #1662
• fix: extend CI running time by @romange in fix: extend CI running time #1749
• fix: fix defrag stats by @dranikpg in fix: fix defrag stats #1740
• fix: fix memcache bugs by @romange in fix: fix memcache bugs #1745
• feat(server): Support limiting the number of open connections. by @royjacobson in feat(server): Support limiting the number of open connections. #1670
• fix: Run defrag on dbs > 0 as well by @dranikpg in fix: Run defrag on dbs > 0 as well #1737
• fix(server): rdb loader catch bad alloc by @adiholden in fix(server): rdb loader catch bad alloc #1748
• docs: add snapshot_cron flag in README (English and zh-CN) by @Niennienzz in docs: add snapshot_cron flag in README (English and zh-CN) #1729
• fix: broken memcached error reporting by @kostasrim in fix: broken memcached error reporting #1741
• feat(search): named return values by @dranikpg in feat(search): named return values #1746
• bug(server): command stats show origin command name by @adiholden in bug(server): command stats show origin command name #1761
• fix(replication): Handle errors instead of crashing. by @royjacobson in fix(replication): Handle errors instead of crashing. #1757
• fix: make Dragonfly compatible with older systems by @romange in fix: make Dragonfly compatible with older systems #1755
• fix(server): service exit on rdb load fail by @adiholden in fix(server): service exit on rdb load fail #1750
• feat: add validation of acl users by @kostasrim in feat: add validation of acl users #1743
• fix(zset): correct the wrong calculation of range.maxex by @theyueli in fix(zset): correct the wrong calculation of range.maxex #1759
• chore: move rdb save related logic into auxillary file by @romange in chore: move rdb save related logic into auxillary file #1769
• fix: Add db id to client list attributes by @romange in fix: Add db id to client list attributes #1770
• fix: failing memcached regression test by @kostasrim in fix: failing memcached regression test #1778
• fix: run container as dfly user by @sorenhansendk in fix: run container as dfly user #1775
• chore: remove locks from acl validation in the hot path by @kostasrim in chore: remove locks from acl validation in the hot path #1765
• feat (facade): simple argument parser by @dranikpg in feat (facade): simple argument parser #1747
• feat: introduce configurable flags for I/O limits by @romange in feat: introduce configurable flags for I/O limits #1776
• fix: use cd instead of working-directory which does not work by @romange in fix: use cd instead of working-directory which does not work #1781
• fix(search): return whole json documents by @dranikpg in fix(search): return whole json documents #1782
• fix(AclFamily): stream acl updates via dispatch queue in connection by @kostasrim in fix(AclFamily): stream acl updates via dispatch queue in connection #1786
• feat(AclFamily): add acl deluser by @kostasrim in feat(AclFamily): add acl deluser #1773
• feat(AclFamily): add acl whoami command by @kostasrim in feat(AclFamily): add acl whoami command #1774
• fix: account for huge pages when displaying RSS usage by @romange in fix: account for huge pages when displaying RSS usage #1785
• fix(server): Dont apply memory limit when loading/replicating by @adiholden in fix(server): Dont apply memory limit when loading/replicating #1760
• fix(AclFamily): setuser signed interleaved categories updates by @kostasrim in fix(AclFamily): setuser signed interleaved categories updates #1790
• chore(AclFamily): disable flaky tests by @kostasrim in chore(AclFamily): disable flaky tests #1791
• feat(search): FT.PROFILE by @dranikpg in feat(search): FT.PROFILE #1787
• chore: dragonfly compiles on FreeBSD by @romange in chore: dragonfly compiles on FreeBSD #1789
• chore: Update double-conversion library by @romange in chore: Update double-conversion library #1793
• feat(search): Query parameters by @dranikpg in feat(search): Query parameters #1768
• Enable authentication for the HTTP interface. by @royjacobson in Enable authentication for the HTTP interface. #1792
• fix(server): Adjust batching behavior to reduce network latency on MULTI blocks by @royjacobson in fix(server): Adjust batching behavior to reduce network latency on MULTI blocks #1777

New Contributors

• @mutiadavid made their first contribution in feat(helm) - Enable configuring service type LoadBalancer with a Static IP #1689
• @Niennienzz made their first contribution in docs: add snapshot_cron flag in README (English and zh-CN) #1729
• @theyueli made their first contribution in fix(zset): correct the wrong calculation of range.maxex #1759
• @sorenhansendk made their first contribution in fix: run container as dfly user #1775

Huge thanks to all the contributors! ❤️

Slava Ukraini 🇺🇦

Full Changelog: v1.8.0...v1.9.0

---

This discussion was created from the release v1.9.0.