How to make GPU training survive spot preemption (without babysitting it)

save your model, optimizer and step counter to one directory, write those files atomically, load them on startup, and run on a platform that copies th