Patterns for making a long-running multimodal data pipeline restartable?

I am working on a long-running pipeline that collects public image-text and video-text records before they are normalized for training. The model code is straightforward, but the data pipeline needs to handle intermittent failures, duplicate records, retries, different response sizes, and occasional changes in the source format. The current design I am considering has: a durable queue for collection jobs bounded retries with explicit failure reasons a raw-response store separate from process...