Deploy weekly, and the previous version keeps running for 17 hours

After a deploy, the service that collects errors from users' browsers (Sentry or similar) starts...