The real problem with ingesting MongoDB into Delta Lake (and how I built a library to fix it)

If you've ever built ETL pipelines pulling data from MongoDB into Delta Lake using Spark, you've probably hit this wall. The pipeline works fine — until it doesn't. A single document with an unexpected shape is enough to break the entire write, leave the table in an inconsistent state, and send your on-call engineer digging through Spark logs at 11pm. I built and maintained more than 10 of these jobs in my last role. After solving the same problem manually across every single one, I decided to b