FTS5 ICU Tokenizer for SQLite

FTS5 ICU Tokenizer for SQLite: This project, rewritten in Zig from C, tokenizes text input before indexing it with SQLite’s FTS5. This enables full-text search on non-space-tokenized languages like Thai. Supported Zig versions Zig 0.16.0 AI / LLM usage disclosure I’ve rewritten much of my previous C code to Zig using various AI models and iterations. The reason for this rewrite is detailed in the README.md but in short, converting UTF-8 to UTF-16 for ICU and then convert it back to UTF-8 for...