← Back to Articles
·9 min read

Demystifying Table Formats: Hive vs. Iceberg vs. Hudi

data-lakehouseapache-icebergapache-hudihivedata-engineering

In the early days of big data, the Data Lake promised a simple, cost-effective way to store vast amounts of raw data. We could dump files into HDFS or cloud object storage like Amazon S3 or Google Cloud Storage (GCS) and query them later.

However, as data volumes scaled and real-time use cases emerged, data engineers faced a harsh reality: managing raw files is hard. Doing updates, maintaining data consistency during concurrent writes, or changing schema without breaking downstream pipelines became an operational nightmare.

To address these pain points, the industry evolved from managing raw files to using Table Formats. In this article, we’ll demystify the difference between file formats and table formats, analyze how the traditional Hive format works (and why it fails on modern object storage), and explore how next-generation formats like Apache Iceberg and Apache Hudi turn simple data lakes into robust, transactional Data Lakehouses.

File Formats vs. Table Formats: What's the Difference?

A common point of confusion is distinguishing between file formats and table formats. They operate at completely different levels of the data stack.

+-------------------------------------------------------------+
|               DATA LAKEHOUSE TECH / TABLE FORMATS            |
|          (Apache Iceberg, Apache Hudi, Delta Lake)          |
|      - ACID Transactions, Time Travel, Partition Evolution  |
+-------------------------------------------------------------+
                              ||
                              \/
+-------------------------------------------------------------+
|             TRADITIONAL TABLE FORMAT / METASTORE            |
|             (Apache Hive Metastore / Catalog)               |
|      - Directory-to-Table Mapping, Relational Schema Info   |
+-------------------------------------------------------------+
                              ||
                              \/
+-------------------------------------------------------------+
|                      FILE FORMATS LAYER                     |
|           (Apache Parquet, Apache ORC, Apache Avro)         |
|      - Columnar/Row Serialization, Compression on Disk      |
+-------------------------------------------------------------+
                              ||
                              \/
+-------------------------------------------------------------+
|                    PHYSICAL STORAGE LAYER                   |
|       (Google Cloud Storage - GCS, Amazon S3, HDFS)         |
|      - Raw Bytes, Objects, Blocks, and Directory Nodes      |
+-------------------------------------------------------------+

1. File Formats (The "How" of Data Storage)

A File Format determines how data is serialized, structured, and compressed within a single physical file on disk.

  • Row-oriented formats (Avro, JSON, CSV): Store data row-by-row. Excellent for write-heavy, transactional workloads (OLTP) where you need to access entire rows at once.
  • Columnar formats (Parquet, ORC): Store data column-by-column. Designed for read-heavy, analytical queries (OLAP) because they allow query engines to read only the columns requested, drastically reducing I/O and enabling aggressive compression.

A file format is self-contained. It doesn't know about other files in the directory or how they relate to each other to form a cohesive table.

2. Table Formats (The "What" of Data Organization)

A Table Format is a metadata layer that sits on top of physical data files. It defines how a collection of files is grouped together to represent a single, structured table.

Without a table format, a query engine (like Spark, Presto, or Trino) querying SELECT COUNT(*) FROM user_events would have to list all files in a folder and guess which ones belong to the table. A table format answers critical questions:

  • Which files currently belong to the table?
  • What is the schema of the table?
  • How is the table partitioned?
  • How do we safely write to the table without corrupting ongoing reads?

The Traditional Standard: Apache Hive

For over a decade, Apache Hive was the default table format. Designed at Facebook in the HDFS era, Hive defines a table using a simple directory structure.

/user/hive/warehouse/user_events/
├── country=US/
│   ├── part-0000.parquet
│   └── part-0001.parquet
└── country=IN/
    └── part-0000.parquet

In Hive:

  1. State is defined by directories: The files inside the country=US directory are assumed to be the data for the partition where country = 'US'.
  2. Schema and Partition mappings are stored in a Metastore: A relational database (like MySQL or PostgreSQL) stores the mapping between the table name, its schema, and the directory path.

The Limitations of Hive on Modern Cloud Storage

While Hive worked well on Hadoop (HDFS), it has severe architectural limitations when migrated to modern cloud object storage (S3, GCS, ADLS):

  • No ACID Transactions (Partial Writes/Dirty Reads): Hive does not support atomic transactions across multiple files. If a Spark job fails midway through writing a partition, the files written up to the failure point are visible to readers, leading to corrupted or partial query results.
  • Expensive Directory Listings (O(N) Operations): To plan a query, Hive must recursively list all directories and files in object storage. On cloud storage, directory listing (LIST) is a slow, expensive API call. For tables with millions of files, query planning can take minutes just listing files.
  • Object Rename Overhead: To achieve atomicity, Hive writes files to a temporary staging directory and renames them to the final directory upon commit. In object storage, there is no true directory structure; renaming a "folder" requires copying every single file and deleting the old ones, which is extremely slow and doubles storage costs (especially if bucket soft-delete policies are enabled).
  • Schema & Partition Coupling: If you partition a Hive table by country, queries must include WHERE country = ... to prune partition scans. If you decide to change your partition strategy to state, you have to rewrite the entire table.

Modern Table Formats: Enter Iceberg and Hudi

Modern table formats like Apache Iceberg (developed at Netflix) and Apache Hudi (developed at Uber) replace directory-based tracking with file-level tracking via metadata files.

Instead of asking the file system "what files are in this folder?", the query engine reads a manifest/metadata file that lists the exact paths of the files that make up the table at that specific point in time.

/metadata/
├── v1.metadata.json (Points to manifest lists)
└── manifest-list-a.avro (Points to specific data files)
/data/
├── file1.parquet
└── file2.parquet

Key Features and How They Make Data Lakes Better

1. ACID Transactions (Reliable Writes)

Iceberg and Hudi use Optimistic Concurrency Control (OCC). When a writer writes data, it writes new files to storage and then performs an atomic commit by writing a new metadata file.

  • Reader Isolation: Readers only see data associated with the latest committed metadata file. Concurrent writes or failed jobs do not affect ongoing reads.
  • Atomic Commits: If a write fails, the new data files are simply ignored because they never get referenced in the metadata.

2. Schema Evolution

In Hive, renaming or dropping a column can corrupt historical data because the data files themselves dictate the schema. In Iceberg and Hudi, each column is assigned a unique ID in the metadata.

  • If you rename a column, the metadata maps the old name to the same ID.
  • If you add or drop a column, the metadata tracks it. Data files never need to be rewritten to support schema changes.

3. Partition Evolution (Hidden Partitioning)

This is a game-changing feature, particularly in Apache Iceberg.

  • No User Query Changes: You don't have to specify partition columns in your queries. If a table is partitioned by event_timestamp_day (derived from event_timestamp), Iceberg automatically prunes partitions when you query WHERE event_timestamp > ....
  • Evolution without Rewriting: If you decide to change your partitioning from daily (event_timestamp_day) to hourly (event_timestamp_hour), Iceberg lets you update the partition spec. New data will be partitioned hourly, while old data remains daily. Iceberg handles the split layout seamlessly under the hood without requiring a full table rewrite.

4. Time Travel and Rollbacks

Because every commit creates a new metadata snapshot, you can query the state of the table at any point in the past.

-- Iceberg SQL syntax example
SELECT * FROM user_events FOR SYSTEM_TIME AS OF '2026-08-01 18:00:00';

This is invaluable for debugging, auditing, or retraining machine learning models on the exact dataset that was present weeks ago.

5. Streaming Upserts and Incremental Processing (Hudi's Strength)

While Iceberg is fantastic for analytical query performance and clean table evolution, Apache Hudi was specifically designed for low-latency streaming ingestion and upserts (updates/deletes) on data lakes. Hudi offers two table types:

  • Copy on Write (CoW): Every update forces a rewrite of the entire Parquet data file containing the updated rows. This is read-optimized but has write amplification.
  • Merge on Read (MoR): Updates are written to separate row-based log files (Avro), and reads merge the base Parquet files and the log files on the fly. This enables near real-time streaming upserts into the data lake.

Comparison: Hive vs. Iceberg vs. Hudi

Feature Apache Hive Apache Iceberg Apache Hudi
State Tracking Directory-based (Metastore) File-based (JSON/Avro Metadata) File-based (Timeline/Metadata)
Cloud Object Storage Fit Poor (requires renames/listings) Excellent (direct file paths) Excellent (direct file paths)
ACID Transactions No (or highly limited/unstable) Yes (OCC) Yes (OCC / Multi-writer support)
Upsert Support No (requires full overwrite) Yes (Merge-on-read / Copy-on-write) Excellent (Built-in primary key index)
Partition Evolution No (requires table rewrite) Yes (Hidden Partitioning) Limited
Time Travel No Yes Yes
Primary Use Case Legacy batch tables Analytical tables (BI, Warehousing) Near real-time streaming & Upserts

Conclusion: The Rise of the Data Lakehouse

By shifting table state tracking from directory listings to file-level metadata, formats like Apache Iceberg and Apache Hudi have bridged the gap between data lakes and data warehouses.

You no longer have to choose between the cheap, scalable storage of a data lake and the ACID safety and speed of a database. With modern table formats, you can run high-performance BI queries, perform streaming updates, and enforce strict data quality rules directly on top of your open files in cloud storage.

If you are starting a new data platform today, moving away from legacy Hive structures to Iceberg or Hudi is one of the most impactful architectural upgrades you can make.