v1.4.0

[romange]

Dragonfly v1.4.0

New Dragonfly release! Some prominent changes include:

• Support for several STREAM commands (xinfo, xtrim, read) - Thanks @Abhra303 and @andydunstall !
• Implementation of bzpopmin and zdiff functionalities - thanks to @royjacobson and @kostasrim !
• Resolving high latency issues encountered in specific corner cases: @royjacobson
• Addressing stability issues reported by our community.
  I would like specifically mention the help of one of the discord members that assisted with finding
  the root cause with BLPOP bug. Thank you, glitch!
  In addition, @andydunstall contributed many fixes across the board, especially with BullMQ compatibility.

What's Changed

• fix(docker): Install libxml2 as part of ubuntu Dockerfile by @Pothulapati in fix(docker): Install libxml2 as part of ubuntu Dockerfile #1239
• refactor(cluster): Move CLUSTER commands to their own methods. by @chakaz in refactor(cluster): Move CLUSTER commands to their own methods. #1230
• feat(stream) Add support for XREAD by @andydunstall in feat(stream) Add support for XREAD #1235
• fix: Dragonfly ignoring NOREPLY option for Memcached commands by @talbii in fix: Dragonfly ignoring NOREPLY option for Memcached commands #1233
• fix(replication): Don't open a replication flow in non-shard threads by @royjacobson in fix(replication): Don't open a replication flow in non-shard threads #1234
• fix(stream): avoid using designated initializer by @Abhra303 in fix(stream): avoid using designated initializer #1247
• fix: fix multi test by @dranikpg in fix: fix multi test #1236
• chore: factor out commandid from the server lib by @romange in chore: factor out commandid from the server lib #1253
• Feat bzpopmin by @royjacobson in Feat bzpopmin #1232
• feature(server): Bring back inline scheduling by @royjacobson in feature(server): Bring back inline scheduling #1130
• refactor(cluster): Move management commands from DFLY CLUSTER prefix to DFLYCLUSTER. by @chakaz in refactor(cluster): Move management commands from DFLY CLUSTER prefix to DFLYCLUSTER. #1251
• feat(cluster): cluster config delete now owned slots data by @adiholden in feat(cluster): cluster config delete now owned slots data #1241
• chore: Update helio by @royjacobson in chore: Update helio #1257
• fix: actually zero initialize the struct by @royjacobson in fix: actually zero initialize the struct #1256
• fix: support XREAD ... STREAMS ... keys derivation by @romange in fix: support XREAD ... STREAMS ... keys derivation #1250
• perf: Use InlinedVector in TransactionData for the common case of single commands by @royjacobson in perf: Use InlinedVector in TransactionData for the common case of single commands #1260
• chore: Try to turn on Werror in the CI by @royjacobson in chore: Try to turn on Werror in the CI #1261
• fix: S3 writes are now working with new format as well by @romange in fix: S3 writes are now working with new format as well #1252
• feat(cluster): Implement CLUSTER INFO. by @chakaz in feat(cluster): Implement CLUSTER INFO. #1258
• feat(cluster): Do not allow node IDs to appear more than once. by @chakaz in feat(cluster): Do not allow node IDs to appear more than once. #1271
• fix(cluster): Fix breaking test by @chakaz in fix(cluster): Fix breaking test #1273
• feat: Use journal LSNs for absolute replication offsets by @royjacobson in feat: Use journal LSNs for absolute replication offsets #1242
• fix: remove xread required arguments by @andydunstall in fix: remove xread required arguments #1263
• Update README.md by @worldsoup in Update README.md #1279
• chore: Add clang workflow to CI by @royjacobson in chore: Add clang workflow to CI #1220
• feat(cluster): support dflycluster myid by @adiholden in feat(cluster): support dflycluster myid #1272
• Refactor search AST by @dranikpg in Refactor search AST #1222
• fix: move version check earlier in the flow by @romange in fix: move version check earlier in the flow #1255
• Reply with MOVED error for non-owned slots. by @chakaz in Reply with MOVED error for non-owned slots. #1283
• fix: fix dispatch ordering by @dranikpg in fix: fix dispatch ordering #1254
• feat(cluster): dflycluster commands only under admin port by @adiholden in feat(cluster): dflycluster commands only under admin port #1281
• feat(cluster): Implement CLUSTER SHARDS. by @chakaz in feat(cluster): Implement CLUSTER SHARDS. #1284
• feat(streams): Add support for XREAD BLOCK by @andydunstall in feat(streams): Add support for XREAD BLOCK #1291
• feat(cluster): select not allowed in cluster mode by @adiholden in feat(cluster): select not allowed in cluster mode #1288
• feat(cluster): Implement CLUSTER SLOTS. by @chakaz in feat(cluster): Implement CLUSTER SLOTS. #1289
• bug(memory): read LXC memory.limit_in_bytes too by @hermanbanken in bug(memory): read LXC memory.limit_in_bytes too #1293
• refactor(cluster): Unify cluster-disabled error message. by @chakaz in refactor(cluster): Unify cluster-disabled error message. #1296
• feat(cluster): Implement CLUSTER NODES command. by @chakaz in feat(cluster): Implement CLUSTER NODES command. #1298
• chore: Silence unwanted asyncio clutter by @royjacobson in chore: Silence unwanted asyncio clutter #1299
• Search: Basic shard state by @dranikpg in Search: Basic shard state #1249
• bug(cluster): Fix failing cluster test. by @chakaz in bug(cluster): Fix failing cluster test. #1300
• chore: improve search tests + fixes by @dranikpg in chore: improve search tests + fixes #1303
• fix: fix search test by @dranikpg in fix: fix search test #1306
• feat(cluster): first config set drop all entries not owned by node by @adiholden in feat(cluster): first config set drop all entries not owned by node #1301
• Add bison to build process, and libxml2 to runtime by @chakaz in Add bison to build process, and libxml2 to runtime #1308
• fix: Batch array length before sending its contents by @romange in fix: Batch array length before sending its contents #1287
• Add latest dependencies. by @chakaz in Add latest dependencies. #1310
• feat(server): Add versioning to dragonfly replication by @royjacobson in feat(server): Add versioning to dragonfly replication #1307
• fix: werror in ci and warning that slipped by by @royjacobson in fix: werror in ci and warning that slipped by #1297
• feat: Support ACKs from replica to master by @royjacobson in feat: Support ACKs from replica to master #1243
• refactor(cluster): Use GetEmulatedShardInfo() for CLUSTER INFO. by @chakaz in refactor(cluster): Use GetEmulatedShardInfo() for CLUSTER INFO. #1317
• fix: Clarify logging for snapshots loading. by @royjacobson in fix: Clarify logging for snapshots loading. #1312
• fix: remove redundant list assertions by @andydunstall in fix: remove redundant list assertions #1338
• feat(streams): Add support for XTRIM by @andydunstall in feat(streams): Add support for XTRIM #1337
• Use search indices by @dranikpg in Use search indices #1294
• fix: Skip optional WEIGHT param in ft.create by @dranikpg in fix: Skip optional WEIGHT param in ft.create #1340
• Chore: Clean up interface of Replica::ReadRespReply by @royjacobson in Chore: Clean up interface of Replica::ReadRespReply #1330
• feat(cluster): Allow actions on non-owned keys on replication port. by @chakaz in feat(cluster): Allow actions on non-owned keys on replication port. #1334
• chore: Double regression tests timeout (10m->20m) by @chakaz in chore: Double regression tests timeout (10m->20m) #1350
• feat: Support tags in search by @dranikpg in feat: Support tags in search #1341
• fix(cluster): Fix crash when getting DFLYCLUSTER GETSLOTINFO with no other args. by @chakaz in fix(cluster): Fix crash when getting DFLYCLUSTER GETSLOTINFO with no other args. #1351
• fix(server): Return null when HRANDKEY doesn't find a key. by @chakaz in fix(server): Return null when HRANDKEY doesn't find a key. #1354
• feat(server): debug poplulate add slots option by @adiholden in feat(server): debug poplulate add slots option #1348
• fix: fix search_test after incorrect rebase by @dranikpg in fix: fix search_test after incorrect rebase #1356
• feat(zset family): Implement ZDiff command issue implement ZDIFF #1311 by @kostasrim in feat(zset family): Implement ZDiff command issue #1311 #1333
• feat(cluster): Add total_reads and total_writes to GETSLOTSINFO. by @chakaz in feat(cluster): Add total_reads and total_writes to GETSLOTSINFO. #1355
• fix(cluster): Fix failing test due to automatic merge by @chakaz in fix(cluster): Fix failing test due to automatic merge #1359
• fix(replication): Restart replication upon receiving FLUSHALL during full sync by @chakaz in fix(replication): Restart replication upon receiving FLUSHALL during full sync #1347
• test(cluster): Test that cluster works with a standard cluster client. by @chakaz in test(cluster): Test that cluster works with a standard cluster client. #1336
• chore(server): Move tests to their respective subdir by @chakaz in chore(server): Move tests to their respective subdir #1360
• chore: Add blpop crash diagnostics by @dranikpg in chore: Add blpop crash diagnostics #1358
• fix: fix linsert pivot by @andydunstall in fix: fix linsert pivot #1365
• fix(alpine): Add libxml2 into the build pipeline by @Pothulapati in fix(alpine): Add libxml2 into the build pipeline #1363
• fix(replication): Destroy rdb saver on the same thread it was created by @chakaz in fix(replication): Destroy rdb saver on the same thread it was created #1366
• feat(ci): Test the built docker image by @Pothulapati in feat(ci): Test the built docker image #1240
• CGroup v1 detection by @talbii in CGroup v1 detection #1329
• bug(debug populate): fix debug populate keys by @adiholden in bug(debug populate): fix debug populate keys #1370
• feat(cluster): Use thread-local cluster config by @chakaz in feat(cluster): Use thread-local cluster config #1361
• fix(server): Do more aggressive batching. by @royjacobson in fix(server): Do more aggressive batching. #1375
• fix(xinfo): send integer info in correct type by @Abhra303 in fix(xinfo): send integer info in correct type #1379
• feat: search limit by @dranikpg in feat: search limit #1367
• Fix reply builder access issue by @dranikpg in Fix reply builder access issue #1378
• fix: aggregate xread and xrange replys by @andydunstall in fix: aggregate xread and xrange replys #1383
• fix: aggregate eval replys by @andydunstall in fix: aggregate eval replys #1386
• feat: Basic search optimizations by @dranikpg in feat: Basic search optimizations #1385
• fix: A more gracefull shutdown by @royjacobson in fix: A more gracefull shutdown #1388
• chore: fix some of FreeBsd compile issues. by @romange in chore: fix some of FreeBsd compile issues. #1374
• fix: Allow readonly commands in replica script by @dranikpg in fix: Allow readonly commands in replica script #1392
• feat: track update and delete for search by @dranikpg in feat: track update and delete for search #1325
• Doc: Warn people about benchmarking/deploying without LTO by @royjacobson in Doc: Warn people about benchmarking/deploying without LTO #1396
• feat(xgroup): add support for xgroup createconsumer by @Abhra303 in feat(xgroup): add support for xgroup createconsumer #1382
• Fix crash on shutdown from fix: A more gracefull shutdown #1388 by @royjacobson in Fix crash on shutdown from #1388 #1399
• fix: remove NotifyPending from UnwatchShardCb by @andydunstall in fix: remove NotifyPending from UnwatchShardCb #1402
• fix(lua): Fix build on arm by @chakaz in fix(lua): Fix build on arm #1404
• feat: Basic KNN (core only) by @dranikpg in feat: Basic KNN (core only) #1397
• feat(cluster): Use shared_ptr for thread-local cluster config by @chakaz in feat(cluster): Use shared_ptr for thread-local cluster config #1395
• fix(test): Fix a race condition in test_flushall_in_full_sync by @royjacobson in fix(test): Fix a race condition in test_flushall_in_full_sync #1409
• Put more keys, make it non-failure if the test fails because of bad params by @royjacobson in Put more keys, make it non-failure if the test fails because of bad params #1411
• test(cluster): Add a Cluster Management script by @chakaz in test(cluster): Add a Cluster Management script #1390
• bug(helm) Fix tolerations and affinity mishandling in Helm template by @kbelokon in bug(helm) Fix tolerations and affinity mishandling in Helm template #1373
• fix: remove bad check-fail in the transaction code by @romange in fix: remove bad check-fail in the transaction code #1420
• fix: drain the connection socket on a protocol error by @andydunstall in fix: drain the connection socket on a protocol error #1422
• fix: Don't remove non-concluding tx from queue on ooo runs by @dranikpg in fix: Don't remove non-concluding tx from queue on ooo runs #1427

New Contributors

• @hermanbanken made their first contribution in bug(memory): read LXC memory.limit_in_bytes too #1293
• @kostasrim made their first contribution in feat(zset family): Implement ZDiff command issue #1311 #1333
• @kbelokon made their first contribution in bug(helm) Fix tolerations and affinity mishandling in Helm template #1373

Huge thanks to all the contributors! ❤️

Full Changelog: v1.3.0...v1.4.0

---

This discussion was created from the release v1.4.0.