≈ the.bay.news

Where to validate in a data pipeline?

Ziggit
Where to validate in a data pipeline?
I have a data pipeline that looks like this: cbor encoded float arrives and placed into a tagged union (cbor.Value) (like std.json.Value) that can represent int, float, bool, str. place cbor.Value into a queue pop N cbor.Value from the queue encode N cbor.Value into JSON send JSON to database http endpoint For floats, JSON cannot represent +inf, -inf, nan and the database will reject queries that contain them. Where in my pipeline do I validate they are noninfinite? Do I specialize my cbor d...

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.