Browser-based LLM Chatbot

To help users on the Glitr.io [http://Glitr.io] app, it was important to document the details around the “how it works”. Using Docusaurus, I can then index that data (like used in the Docusaurus search) to be consumed by some script locally and offer suggested links. I framed this as like a chatbot so that it can be intuitive for users who have questions. At first it was only a fancy autocomplete. I then wanted to investigate running a local LLM to offer a more conversational experience (or better suggestions)? I’m using WebLLM on a browser and it seems to work “reasonably” well. I wired it up and set it up so users can select an LLM to download for offline use. The experience varies between devices and models used. There are models that are 19MB… they seem pretty unusable. There are also some ~1GB models which are better but understandably slow. I hope to add more models available and users can choose one that suits them (or none). Without an AI, i hope the “fancy autocomplete” can fill in the gaps. Its great that the ~1GB models can “show me the bubble-sort in rust”, but i want something “dumber” to aim for smaller (im assuming a ~1GB download from the onset would be off-putting for most users). Ideally, something can can take advantage the data ive already indexed. Something that can respond “quickly enough” for a conversational flow on a “modern” phone. Im aiming for a good user-experience more than trying to wedge in an LLM… and a fancy autocomplete might be enough if i create the appropriate coachmarks and docs. It might be better to scrap the LLM in chat approach and have context aware templates instead for a “fancier autocomplete”. Are there any smaller language models out there im overlooking for me to investigate and compare? * Find out more: https://glitr.io/docs/technical/ai [https://glitr.io/docs/technical/ai] * Test it out: https://app.glitr.io/ai [https://app.glitr.io/ai] Note: I hope it goes without saying, this is not ever going to be competative against offerings out there like ChatGPT, Claude, Grok, etc. This project is experiemental and unstable. Its far from finished. Im still trying things out. Use responsibly.