huggingface-tokenizers

Run local LLMs like Gemma, Qwen, and LLaMA on Android for offline, private, real-time chat and question answering with LiteRT and ONNX Runtime.

Go, Wasm bindings for HF Tokenizers and Tiktoken

⛔ [DEPRECATED] Adapt Transformer-based language models to new text domains

We use phonetics as a feature to create a joint semantic-phonetic embedding and improve the neural machine translation between Chinese and Japanese. 🥳

This project leverages deep learning transformers to classify YouTube comments into six distinct emotions.

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

tool to split text into semantic chunks

Natural Language Processing Practice — a hands‑on repository spanning the full spectrum of NLP, from classical algorithms to cutting‑edge large language models (LLMs). Built around the Hugging Face LLM Course, it’s enriched with practical notebooks on foundational libraries and advanced fine‑tuning workflows.

Automated glossary generation and QA assistant that extracts technical terms from text corpora using regex and Levenshtein clustering, tokenizes them with custom BPE, and generates definitions and examples using a local Ollama LLM, all accessible through a CLI interface.

Build, analyze, and visualize a custom Byte Pair Encoding (BPE) tokenizer trained on the WikiText-2 dataset with vocabulary analysis, compression evaluation, and an interactive GitHub Pages demo.

HTGM.2 is a Hindi-first BPE tokenizer trained on ~41GB corpus using streaming architecture for scalable Hindi LLMs, Devanagari NLP, and low-memory tokenizer engineering.

Huggingface tokenizers for Haskell

Hugging Face Transformers offer a powerful framework for state-of-the-art NLP, with the Pipeline API for easy inference, Tokenization for efficient preprocessing, and Quantization for optimized deployment.