Considerations or quirks for direct Vec file writing/reading

I'm working on a program that will involve storing struct instances in data files, and so far have a working process of serialization, writing, reading, and de-serialization directly manipulating the Vec<u8> buffers (I understand serde is a fantastic crate, but I'm avoiding external dependencies until it's unreasonable to do so). It works in a clean, local environment (Linux), however I intend to compile to both Linux and Windows x64 targets. My question is if there are any issues I'm not seein...