High memory usage

[muety]

I observed my self-hosted HitKeep to consume quite a significant amount of RAM during operation. My database size, as reported by the System status view is 1.9 GB and my OS reports the hitkeep process to use around 2.2 GB, which I think is quite a lot.

I'm not very familiar with DuckDB, but my assumption is that the high memory usage is due to the fact that the entire database is always kept entirely in memory?

Is there any way to make HitKeep run a bit more "lean"?

[PascaleBeier]

There's a lot of knobs we can tune currently (or, all of them). DuckDB can be compared to a "default" configured redis that happily takes all RAM it can if not told otherwise.

I monitor a lot of pressure in the clouds from time to time, but it should absolutely be possible to have 2 GB shared databases on a 512 MB VPS.

I agree that this is a somewhat high priority issue, I'm currently working on replicating memory pressure behaviour more accurately before tuning anything, I'll keep you posted.

I wrote my current findings here: https://hitkeep.com/guides/installation/performance/

[muety]

Thanks for getting back so quickly! Just for my understanding: is it sensible (or even possible) to set HITKEEP_DUCKDB_MEMORY_LIMIT to a value lower than the actual database size? In other words, does DuckDB require at least as much memory to fit the entire database or can it also read pages from disk "on demand"?

[PascaleBeier]

You could set it lower, in your example to 1GiB. Ingest will go above that.

DuckDB reads from disk if needed, but writes will take memory.

It prefers keeping it in memory but that's not strictly required.

From my tests it doesn't seem to be stable setting RAM too low, though.

[PascaleBeier]

I have some decent results with pinning a 5GB tenant to 2GiB for instance and the dashboard still being decent