What Are Table Formats and Why Were They Needed?

This is Part 1 of a 15-part Apache Iceberg Masterclass . This article covers the fundamental question: what problem do table formats solve, and why does the choice between them matter? A data lake without a table format is a collection of files. It has no concept of a transaction, no mechanism to prevent two writers from producing corrupted state, and no efficient way to determine which files belong to the current version of a table. Table formats exist because the gap between "a pile of Parquet