How to Stop an Image Model From Ignoring Your Reference Image

A user uploaded a floor plan, typed "remove the room labels", and got back the same plan — labels...