Highlights
- Unified chat-template rendering interface across the different tokenizers (fixes a BOS token duplication bug), performance improvements and code improvements on templating path by @hyeongyun0916
- Logger fix by @hyeongyun0916
- Tiered prefix-cache scoring by @Jay-Pd
- UDS-tokenizer service by @delavet and @osswangxining
- Tokenizer from local disk enablement by @pierDipi
- Valkey kvblock backende by @rishi-jat
- General enhancements by: @zhengkezhou1 @Frapschen @my-git9 @samzong @samber @kyanokashi
What's Changed
- Fix examples precalculated hashes by @vMaroon in #141
- feat: make kvcache.Indexer as gRPC Service by @zhengkezhou1 in #109
- Fix: division by zero in metrics logging by @yankay in #145
- feat: Add Valkey and RDMA support for KV-cache indexing by @rishi-jat in #139
- feat: Add UDS-based external tokenizer service by @delavet in #137
- Fix LookupHits metrics not work by @Frapschen in #146
- [feat] Add tolerations for chart by @my-git9 in #149
- [misc] valkey doc repositioning by @vMaroon in #154
- Update README.md to enhance clarity of flowchart labels and descriptions by @samzong in #157
- perf(prefixstore): sync.Map are much faster in read-intensive applications by @samber in #156
- Add support for local tokenizer files by @pierDipi in #142
- Add @delavet as /services/uds_tokenizer owner by @vMaroon in #164
- Add @osswangxining as /services/uds_tokenizer owner by @vMaroon in #168
- Implementation for Tiering in KV-Cache-Manager by @Jay-Pd in #150
- fix: rename LookupHits metric to MaxHitsPerPod to better reflect what's tracked by @kyanokashi in #160
- fix online example chart format by @delavet in #171
- Minor fix for KV Device Tier by @Jay-Pd in #172
- [Fix] Ensure Correct Logger Usage by Replacing
klog.FromContextwithlog.FromContextby @hyeongyun0916 in #169 - refactor(tokenizer): Unify interface for RenderChatTemplate and eliminate object creation overhead by @hyeongyun0916 in #163
- Minor logger fix by @vMaroon in #173
- General refactoring for v0.4.0 by @vMaroon in #174
New Contributors
- @rishi-jat made their first contribution in #139
- @delavet made their first contribution in #137
- @Frapschen made their first contribution in #146
- @samzong made their first contribution in #157
- @samber made their first contribution in #156
- @pierDipi made their first contribution in #142
- @Jay-Pd made their first contribution in #150
- @kyanokashi made their first contribution in #160
- @hyeongyun0916 made their first contribution in #169
Full Changelog: v0.3.2...v0.4.0