feat(core): Add defrag support for json objects #6023
Conversation
abhijat
force-pushed
the
abhijat/feat/defrag-json
branch
5 times, most recently
from
a6e61b4 to
dd35f06
Compare
abhijat
force-pushed
the
abhijat/feat/defrag-json
branch
from
dd35f06 to
e717440
Compare
romange
left a comment
romange
left a comment
There was a problem hiding this comment.
Flat json is not really used, do we use jsoncons in prod. I did not understand how you defragment json as it can.contain lots of pointers due to its hierarchical structure
Yes jsoncons should the default encoding based on
Yes, the current implementation works better for flat encoding as it is a single chunk of data, but for If we don't use flat encoding much and use mostly cons, then this PR needs more work. I will look into defragmenting the tree from the root downwards. |
Actually, it doesn't look like this is possible, as far as I can see it might not be possible to go into the object's fields and individually reallocate them. It might be possible to do the following more expensive operations:
|
abhijat
force-pushed
the
abhijat/feat/defrag-json
branch
from
e717440 to
7cf8541
Compare
I tried this approach in the latest commit, will test it a bit more |
Taking an existing JSON object as input, it is serialized and then deserialized, allocating a new object which is a copy of the old one, but now exists in the page reserved for malloc. It can be used to defragment a JSON object. Signed-off-by: Abhijat Malviya <[email protected]>
abhijat
force-pushed
the
abhijat/feat/defrag-json
branch
from
7cf8541 to
bb7650b
Compare
|
@romange does the current approach look better? We basically copy the JSON object, in the process reallocating memory |
src/core/compact_object.cc
Outdated
| } | ||
|
|
||
| bool CompactObj::JsonConsT::DefragIfNeeded(PageUsage* page_usage) { | ||
| if (JsonType* old = json_ptr; page_usage->IsPageForObjectUnderUtilized(old)) { |
There was a problem hiding this comment.
note: the trigger condition is also problematic. the object may be part of fully utilized page but have hundreds of other heap objects. It is possible to solve this using the latest PATCH of @vyavdoshenko
here: dragonflydb/jsoncons@990b0a2
you can pass into compute_memory_size a callback that will return 0 or 1 if a pointer belongs to the under utilized page and if the final result is > 0, than there is at least one pointer that needs to be defragmented.
There was a problem hiding this comment.
the patch is already present in jsoncons.
There was a problem hiding this comment.
I used compute_memory_size but kept state (a boolean) outside to figure out if any pointer is in an underutilized page.
Signed-off-by: Abhijat Malviya <[email protected]>
Signed-off-by: Abhijat Malviya <[email protected]>
abhijat
force-pushed
the
abhijat/feat/defrag-json
branch
from
bb7650b to
98c3fa4
Compare
* core: Add utility to reallocate a JSON object Taking an existing JSON object as input, it is serialized and then deserialized, allocating a new object which is a copy of the old one, but now exists in the page reserved for malloc. It can be used to defragment a JSON object. * core: Add defrag support for json objects * core: Add tests for json defrag Signed-off-by: Abhijat Malviya <[email protected]> (cherry picked from commit 767ed14)
* fix(evicition): Limit accumulation deleted bytes during eviction (#5995) Cache state variable accumulated_deleted_bytes_during_eviction need to have upper bound when accumulating to prevent over eviction. Adjust deleted bytes for each shard between two heartbeats by comparing previous shard used memory and current shard used memory. Fixes #5945 Signed-off-by: mkaruza <[email protected]> (cherry picked from commit 1125eb4) * feat(core): Add defrag support for json objects (#6023) Signed-off-by: mkaruza <[email protected]> Signed-off-by: Abhijat Malviya <[email protected]> Co-authored-by: mkaruza <[email protected]>
Defrag support is added for JSON compact objects for both cons and flat type encoding.