Patterns for making a long-running multimodal data loader more reliable?

I’m working on a long-running pipeline that collects public image-text and video-text records before they are normalized for training. The model code is fairly straightforward, but the data loader needs to handle intermittent failures, duplicate records, retries, and different response sizes. I’m considering a design with: - a durable queue for collection jobs - bounded retries with failure reasons - a raw-response store separate from processed samples - deterministic dataset manifests - regio...