v1.15.0

[romange]

Dragonfly v1.15.0

After more than a month, we are releasing a new version with even more efficient transactional algorithms, improved lua compatibility, better memory management and lots of stability fixes. For Mac enthusiasts - the project now builds natively on MacOs 💻

Some prominent changes include:

• Lua redis.log(..) is recognized (feat(lua): add no-op redis.log command #2476)
• Lua can optionally return doubles as longs when replying to RESP2 clients using --lua_resp2_legacy_float flag (fix: optionally return float as integer from lua script #2672)
• Added backlog option (feat(server): Add backlog flag for listen() #2456)

What's Changed

• chore: improve benchmark for bpptree and zsl tree by @romange in chore: improve benchmark for bpptree and zsl tree #2450
• feat(transaction): Independent out of order execution by @dranikpg in feat(transaction): Independent out of order execution #2426
• fix(RestoreStreamer): Fix a few glitches by @chakaz in fix(RestoreStreamer): Fix a few glitches #2452
• feat(cluster): add tx execution in cluster_shard_migration by @BorysTheDev in feat(cluster): add tx execution in cluster_shard_migration #2385
• feat(server): Add backlog flag for listen() by @chakaz in feat(server): Add backlog flag for listen() #2456
• feat: allow reading offloaded strings without loading to the store by @romange in feat: allow reading offloaded strings without loading to the store #2449
• refactor: return OpResult in DbSlice::AddOrFind instead of throwing std::bad_alloc by @kostasrim in refactor: return OpResult in DbSlice::AddOrFind instead of throwing std::bad_alloc #2427
• fix(transaction): Improve ACTIVE flags management by @dranikpg in fix(transaction): Improve ACTIVE flags management #2458
• test(cluster): Enable seeder to work against a Dragonfly cluster by @chakaz in test(cluster): Enable seeder to work against a Dragonfly cluster #2462
• Update README.md to include dragonfly cloud link by @worldsoup in Update README.md to include dragonfly cloud link #2469
• fix: resolves the crash with MGET and duplicate keys by @romange in fix: resolves the crash with MGET and duplicate keys #2471
• fix: fixes for v1.14.0 by @dranikpg in fix: fixes for v1.14.0 #2473
• feat(lua): add no-op redis.log command by @kostasrim in feat(lua): add no-op redis.log command #2476
• fix: reduce test_pipeline_batching_while_migrating flakiness by @dranikpg in fix: reduce test_pipeline_batching_while_migrating flakiness #2475
• chore: switch json object to pmr allocator by @romange in chore: switch json object to pmr allocator #2483
• chore(transaction): Simplify PollExecution() by @dranikpg in chore(transaction): Simplify PollExecution() #2457
• chore: fix clang warnings by @romange in chore: fix clang warnings #2485
• chore(transaction): Launder copied keys in multi transactions by @dranikpg in chore(transaction): Launder copied keys in multi transactions #2478
• chore: lock keys when going through fast-path execution by @romange in chore: lock keys when going through fast-path execution #2491
• fix(server): mget crash on same key get by @adiholden in fix(server): mget crash on same key get #2474
• chore: Introduce LockKey for LockTable by @romange in chore: Introduce LockKey for LockTable #2463
• fix(cluster): fix slot filtration to RestoreStreamer by @BorysTheDev in fix(cluster): fix slot filtration to RestoreStreamer #2477
• feat(lua): add missing no-op redis.replicate_commands() by @kostasrim in feat(lua): add missing no-op redis.replicate_commands() #2472
• chore: remove batch reply statistics by @romange in chore: remove batch reply statistics #2490
• Fix redis->DF replication tests by @romange in Fix redis->DF replication tests #2495
• chore: introduce TaskQueue for executing shard local tasks by @romange in chore: introduce TaskQueue for executing shard local tasks #2498
• fix(cluster): fix incorrect version checking and resource double free by @BorysTheDev in fix(cluster): fix incorrect version checking and resource double free #2499
• fix(server): Fix crash in RENAME by @chakaz in fix(server): Fix crash in RENAME #2503
• test: Disable flaky test test_policy_based_eviction_propagation by @chakaz in test: Disable flaky test test_policy_based_eviction_propagation #2505
• feat(server): Memory tracker by @chakaz in feat(server): Memory tracker #2501
• fix(server): update post updater iterator in tiering by @adiholden in fix(server): update post updater iterator in tiering #2497
• chore(transaction): Use PhasedBarrier for easier synchronization by @dranikpg in chore(transaction): Use PhasedBarrier for easier synchronization #2455
• feat(server): Implement SCRIPT FLUSH command by @highpon in feat(server): Implement SCRIPT FLUSH command #2493
• fix: do not crash with inconsistent watch queue by @romange in fix: do not crash with inconsistent watch queue #2515
• Fix: awakened_keys inconcistency by @romange in Fix: awakened_keys inconcistency #2517
• feat(cluster): add migration finalization by @BorysTheDev in feat(cluster): add migration finalization #2507
• fix: fix BLOCKING/REVERSE_MAPPING flags for some commands by @romange in fix: fix BLOCKING/REVERSE_MAPPING flags for some commands #2516
• fix: handle null length strings in stored commands by @dranikpg in fix: handle null length strings in stored commands #2518
• feat(server): Add MEMORY TRACK ADDRESS command by @chakaz in feat(server): Add MEMORY TRACK ADDRESS command #2513
• feat(server): Implement CONFIG HELP command by @highpon in feat(server): Implement CONFIG HELP command #2510
• fix(transaction): Add special barrier for blocking tx by @dranikpg in fix(transaction): Add special barrier for blocking tx #2512
• fix(zset): ignore cancelled op status in BZPopMinMax by @andydunstall in fix(zset): ignore cancelled op status in BZPopMinMax #2527
• fix: fix script replication by @dranikpg in fix: fix script replication #2531
• chore: remove redis sorted set implementation by @romange in chore: remove redis sorted set implementation #2522
• fix: add emoji when regression tests timeout by @kostasrim in fix: add emoji when regression tests timeout #2543
• fix(cluster): fix fix cluster tests for "redis client socket isn't closed" error #2532 tests redis client close errors by @BorysTheDev in fix(cluster): fix #2532 tests redis client close errors #2542
• bug: DFLY_PASSWORD environment variable deprecation by @mprimeaux in bug: DFLY_PASSWORD environment variable deprecation #2539
• fix: wrongly printing timedout emoji on test failures by @kostasrim in fix: wrongly printing timedout emoji on test failures #2546
• feat: move lint test chart to seperate action and trigger it on reg tests by @kostasrim in feat: move lint test chart to seperate action and trigger it on reg tests #2536
• Update helio by @romange in Update helio #2538
• fix(server): Do not yield in journal if not allowed by @chakaz in fix(server): Do not yield in journal if not allowed #2540
• fix: fix interpreter acquisition with MULTI by @dranikpg in fix: fix interpreter acquisition with MULTI #2549
• fix(tests): increase interpreter test load, lower notice limit, include 2553 by @dranikpg in fix(tests): increase interpreter test load, lower notice limit, include 2553 #2554
• feat: introduce a skeleton for jsonpath parser by @romange in feat: introduce a skeleton for jsonpath parser #2521
• fix(pytest): Disable flaky interpreter test monitoring by @dranikpg in fix(pytest): Disable flaky interpreter test monitoring #2558
• feat(cluster): add slots blocking when we finalize migration by @BorysTheDev in feat(cluster): add slots blocking when we finalize migration #2484
• fix(server): Init tx time for all multi/lua transactions by @chakaz in fix(server): Init tx time for all multi/lua transactions #2562
• chore: enable ContendExpire by @romange in chore: enable ContendExpire #2563
• fix(server): do not print warning on replica stop/ takeover by @adiholden in fix(server): do not print warning on replica stop/ takeover #2560
• chore: parse a simple jsonpath by @romange in chore: parse a simple jsonpath #2566
• feat(pytest): Gen2 seeder, part 1 by @dranikpg in feat(pytest): Gen2 seeder, part 1 #2556
• fix: make sure SCRIPT FLUSH concludes by @romange in fix: make sure SCRIPT FLUSH concludes #2565
• chore: Remove manual memory management from stats by @dranikpg in chore: Remove manual memory management from stats #2550
• fix: asan errors on unit tests by @kostasrim in fix: asan errors on unit tests #2564
• chore(transaction): Simplify armed state by @dranikpg in chore(transaction): Simplify armed state #2508
• test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> by @chakaz in test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> #2561
• chore: Recognize exit status in regression tests by @romange in chore: Recognize exit status in regression tests #2571
• chore(server): replica not accumulate multi commands untill exec by @adiholden in chore(server): replica not accumulate multi commands untill exec #2557
• chore: use helio MPSCIntrusiveQueue queue by @romange in chore: use helio MPSCIntrusiveQueue queue #2573
• chore: provide plumbing for jsonpath error propagation by @romange in chore: provide plumbing for jsonpath error propagation #2567
• test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests by @chakaz in test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests #2572
• fix: do not migrate during connection close by @romange in fix: do not migrate during connection close #2570
• chore: fix fiber types in the codebase by @romange in chore: fix fiber types in the codebase #2574
• feat(release): Also bundle dragonfly-debug builds by @Pothulapati in feat(release): Also bundle dragonfly-debug builds #2424
• feat: jsonpath supports index/wildcard expressions by @romange in feat: jsonpath supports index/wildcard expressions #2578
• fix(tiering tests): introduce wait until tiering entries num EQ/GT by @adiholden in fix(tiering tests): introduce wait until tiering entries num EQ/GT #2559
• chore(pytest): Refactor snapshot test by @dranikpg in chore(pytest): Refactor snapshot test #2583
• fix(test): Apply pytest filter to regression tests as well by @chakaz in fix(test): Apply pytest filter to regression tests as well #2589
• feat: add descent operator to jsonpath parser by @romange in feat: add descent operator to jsonpath parser #2588
• test(cluster): Fix and uncomment counter sanity checks by @chakaz in test(cluster): Fix and uncomment counter sanity checks #2591
• feat(tiering): add background offload step by @adiholden in feat(tiering): add background offload step #2504
• fix(test): Unflake GenericFamilyTest.Time*Keys by @chakaz in fix(test): Unflake GenericFamilyTest.Time*Keys #2592
• chore: add jsonpath evaluate by @romange in chore: add jsonpath evaluate #2586
• fix(test): Use less memory for STRING and HASH memory tests by @chakaz in fix(test): Use less memory for STRING and HASH memory tests #2593
• chore(transaction): Copy poll flags by @dranikpg in chore(transaction): Copy poll flags #2596
• feat: support descent path operator by @romange in feat: support descent path operator #2594
• feat: Add dbfilename to config registry by @romange in feat: Add dbfilename to config registry #2598
• wire json::Path into json_family by @romange in wire json::Path into json_family #2587
• chore: refactor DfsItem to allow mutability semantics by @romange in chore: refactor DfsItem to allow mutability semantics #2602
• chore: cleanup of redis implementation of sets/hashmaps by @romange in chore: cleanup of redis implementation of sets/hashmaps #2605
• chore: remove ImportRObj routine and reduce reliance on object.c by @romange in chore: remove ImportRObj routine and reduce reliance on object.c #2607
• chore: consolidate json code under core/json by @romange in chore: consolidate json code under core/json #2609
• chore(transaction): Untie scheduling from multi status by @dranikpg in chore(transaction): Untie scheduling from multi status #2590
• chore(transaction): Add debug stats for fail printing by @dranikpg in chore(transaction): Add debug stats for fail printing #2600
• Add mutate method to jsonpath by @romange in Add mutate method to jsonpath #2606
• add loadtest benchmarking program by @romange in add loadtest benchmarking program #2603
• chore: get rid of object.c and robj* in cc code by @romange in chore: get rid of object.c and robj* in cc code #2610
• chore: clean up redis code by @romange in chore: clean up redis code #2614
• chore: allow aggregate functions for jsonpath by @romange in chore: allow aggregate functions for jsonpath #2611
• fix: fix cluster test by @dranikpg in fix: fix cluster test #2612
• fix(rdb_load): EnsureRead(min) requesting more bytes than min by @kostasrim in fix(rdb_load): EnsureRead(min) requesting more bytes than min #2604
• fix(server): saving is not a server state by @adiholden in fix(server): saving is not a server state #2613
• fix(server_family): GetMetrics should show commands in lowercase by @lsvmello in fix(server_family): GetMetrics should show commands in lowercase #2601
• feat(cluster): Add finalization for migration process #2448 add new DFLYMIGRATE ACK cmd by @BorysTheDev in feat(cluster): #2448 add new DFLYMIGRATE ACK cmd #2582
• test: Run slow tests in regression runs (again:) by @chakaz in test: Run slow tests in regression runs (again:) #2616
• fix(bug): crash when starting replica while saving by @adiholden in fix(bug): crash when starting replica while saving #2618
• test: Run slow tests in regression runs (again^2) by @chakaz in test: Run slow tests in regression runs (again^2) #2623
• feat(cluster_mgr): Add support for migrate action by @chakaz in feat(cluster_mgr): Add support for migrate action #2626
• feat(pytest): More types for seeder by @dranikpg in feat(pytest): More types for seeder #2577
• fix: loading state error type to be compatible with redis by @kostasrim in fix: loading state error type to be compatible with redis #2629
• fix(cluster_mgr): Disable CPU affinity by @chakaz in fix(cluster_mgr): Disable CPU affinity #2632
• fix: provide more context in case we have duplicate keys by @romange in fix: provide more context in case we have duplicate keys #2619
• chore: doc_accessors now parse with json::Path as well by @romange in chore: doc_accessors now parse with json::Path as well #2615
• fix: crash when using nonexistent global variable in lua by @thweetkomputer in fix: crash when using nonexistent global variable in lua #2627
• feat(regTests): upload only failed test logs on ci and clean up logging by @kostasrim in feat(regTests): upload only failed test logs on ci and clean up logging #2547
• refactor: remove goto statements from replica.cc by @kostasrim in refactor: remove goto statements from replica.cc #2633
• fix(bug): server crash info memory while saving by @adiholden in fix(bug): server crash info memory while saving #2637
• fix(streamer): Do not yield from the Traverse callback. by @chakaz in fix(streamer): Do not yield from the Traverse callback. #2638
• chore: fix build for MacOs by @romange in chore: fix build for MacOs #2635
• fix: avoid crashing during the connection traversal by @romange in fix: avoid crashing during the connection traversal #2651
• a test with flat buffers by @romange in a test with flat buffers #2520
• refactor: create one type for slots set Create one type for slot ranges #2459 by @BorysTheDev in refactor: create one type for slots set #2459 #2645
• fix: data race in save_stages_controller by @kostasrim in fix: data race in save_stages_controller #2647
• fix: fix zmalloc_size on macos by @andydunstall in fix: fix zmalloc_size on macos #2646
• feat(connection): Support pipelining with Memcached by @dranikpg in feat(connection): Support pipelining with Memcached #2648
• chore: JSON.GET now works with our own jsonpath by @romange in chore: JSON.GET now works with our own jsonpath #2653
• chore: fully switch to dual parsing with json::Path by @romange in chore: fully switch to dual parsing with json::Path #2654
• chore: Move CallFromScript log to level 2 by @dranikpg in chore: Move CallFromScript log to level 2 #2656
• chore: switch to self-built flatbuffers by @romange in chore: switch to self-built flatbuffers #2657
• feat: add current_fork_perc in info all command by @kostasrim in feat: add current_fork_perc in info all command #2640
• chore: Del and NUMINCRBY use json::Path by @romange in chore: Del and NUMINCRBY use json::Path #2655
• chore: move jsoncons path code into dedicated files by @romange in chore: move jsoncons path code into dedicated files #2666
• fix: replace localhost with 127.0.0.1 in cluster tests by @dranikpg in fix: replace localhost with 127.0.0.1 in cluster tests #2665
• chore: fuly cover json_family API with json::Path parsing by @romange in chore: fuly cover json_family API with json::Path parsing #2663
• feat(generic_family): implement RANDOMKEY command by @lsvmello in feat(generic_family): implement RANDOMKEY command #2639
• fix(server): client pause work while blocking commands run by @adiholden in fix(server): client pause work while blocking commands run #2584
• feat(cluster_mgr): Add support for remote Dragonfly servers by @chakaz in feat(cluster_mgr): Add support for remote Dragonfly servers #2671
• fix: optionally return float as integer from lua script by @romange in fix: optionally return float as integer from lua script #2672
• refactor(cluster): refactor slot migration process initiating #2652 initiate migration process from CONFIG cmd by @BorysTheDev in refactor(cluster): #2652 initiate migration process from CONFIG cmd #2667
• chore: update helio dependency by @romange in chore: update helio dependency #2674
• fix(snapshot): Fix flag access by @dranikpg in fix(snapshot): Fix flag access #2676
• chore: more fixes for macos by @romange in chore: more fixes for macos #2677
• feat(server): introduce oom_capacity_ratio flag by @adiholden in feat(server): introduce oom_capacity_ratio flag #2678
• chore: add oom stats to /metrics by @romange in chore: add oom stats to /metrics #2680

New Contributors

• @mprimeaux made their first contribution in bug: DFLY_PASSWORD environment variable deprecation #2539
• @thweetkomputer made their first contribution in fix: crash when using nonexistent global variable in lua #2627

Huge thanks to all the contributors! ❤️

🇮🇱 🇺🇦

Full Changelog: v1.14.0...v1.15.0

---

This discussion was created from the release v1.15.0.