Where to validate in a data pipeline?
Ziggit
Where to validate in a data pipeline?
I have a data pipeline that looks like this: cbor encoded float arrives and placed into a tagged union (cbor.Value) (like std.json.Value) that can represent int, float, bool, str. place cbor.Value into a queue pop N cbor.Value from the queue encode N cbor.Value into JSON send JSON to database http endpoint For floats, JSON cannot represent +inf, -inf, nan and the database will reject queries that contain them. Where in my pipeline do I validate they are noninfinite? Do I specialize my cbor d...
0 comments
No comments yet.