fix: Automatically compact entity_store.jsonl after a large number of repeat topics

[jarhodes314]

Proposed changes

Adds a compaction mechanism to the entity store file. This will trigger after we have written a significant number (currently 100) messages to topics that already exist in the file, preventing unbounded growth.

It also refactors the existing test for the readers/writers into well-defined unit tests.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

• entity_store.jsonl grows indefinitely when retained messages are repeatedly published to the same topic #4172

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES
• I used just check as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

❌ Patch coverage is 95.34161% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/core/tedge_api/src/store/message_log.rs	95.34%	0 Missing and 15 partials ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
916	0	4	916	100	2h39m6.550187s

[albinsuresh]

The code looks fine. Happy to approve once cleared message handling is also fixed. Also as an additional enhancement, how about doing an unconditional compaction on startup?

[jarhodes314]

The behaviour of this is slightly altered. Previously, truncate mode meant we opened the file with truncate(true), then closed it, then opened it again (via MessageLogWriter::new). Now we just open it the one time with truncation. I believe this won't cause any dramatic behaviour changes as we didn't do anything to make the truncation fsync-durable, and it seemed from reading the code that we mostly did it for easy code reuse between the truncate and non-truncate paths.

This has the side-effect of us also failing if we fail to open the file with truncation (previously we ignored the result of opening the truncated file). I don't believe this will cause any major issues - if we can't open a file to truncate, we probably can't open it in append mode. There's a possible edge case where file system attributes might mean we can only append to a file, but in this case I think we're better off failing rather than pretending everything's working?

[albinsuresh]

LGTM