A third of the tokens in the code you send an LLM are comments. Indentation is 2.5%.
DEV Community
A third of the tokens in the code you send an LLM are comments. Indentation is 2.5%.
I counted all 131 top-level modules of the Python 3.13 standard library with tiktoken. Comments and docstrings were 34.8% of the tokens, indentation 2.5%, blank lines 0.1%. If you trim code before sending it to a model, trim the prose, not the whitespace.
0 comments
No comments yet.