It makes sense to use something like https://stevehanov.ca/blog/compressing-dictionaries-with-a-dawg as a compact data structure.
Also look at https://github.com/blevesearch/vellum (its vellum file format v1), https://github.com/morfologik/morfologik-stemming (its CFSA2 format) and https://www.grammalecte.net/ (its graphspell format) for inspiration or maybe even full binary compatibility.
The absolutely best compression is not necessary. A balanced approach favoring good performance and code simiplicity may be preferred over shaving off the last 10% from the data size at any cost.
It makes sense to use something like https://stevehanov.ca/blog/compressing-dictionaries-with-a-dawg as a compact data structure.
Also look at https://github.com/blevesearch/vellum (its vellum file format v1), https://github.com/morfologik/morfologik-stemming (its CFSA2 format) and https://www.grammalecte.net/ (its graphspell format) for inspiration or maybe even full binary compatibility.
The absolutely best compression is not necessary. A balanced approach favoring good performance and code simiplicity may be preferred over shaving off the last 10% from the data size at any cost.