Why I Stopped Trusting Similarity Scores and Built a System That Says "No"

I'll be honest about where this started. I could get a vision model to look at a photo and spit back...