From Glue to Horizon: Our Real Journey Building an Iceberg Lakehouse on Snowflake

We set out to build an open lakehouse: Iceberg tables on AWS S3, Spark/Glue for pipelines, Snowflake for analytics compute power. What could go wrong? Everything—from query performance implosions to uncloneable dynamic tables. Here's the unfiltered journey, including why we pivoted to Snowflake-managed Iceberg via Horizon Catalog and abandoned automated dynamic tables for explicit, observable incremental processing. TL;DR ❌ Phase 1 failed : Glue-generated Iceberg files (32-64MB) caused 5-10x slo