Implementing a sink function for Tantivy tokenizer

Hey folks, I've just started what will likely be a long journey to provide Pāli language support in Tantivy. The first stage of the project is tokenization, working towards a dictionary stemmer and algorithmic stemmer testing. I've started looking at the documentation for the tantivy_tokenizer_api crate and instantiated a WhitespaceTokenizer to get started. Calling token_stream() gives me a TokenStream, which in turn has this method: I'd like to pass in a function or closure that just print...