Lakehouse Architecture: The Ops Nobody Mentions

You migrated your analytics from a managed warehouse to a self-managed Iceberg lakehouse on S3. The migration went smoothly. Initial Trino query times were faster. Your team celebrated. Six months later, the head of analytics files a ticket: “The customer segmentation query that used to take 8 seconds now takes 4 minutes and 12 seconds.”

Nobody ever scheduled compaction jobs. The orders table has piled up 340,000 small files from months of streaming micro-batches. Every query opens each file to read its metadata before scanning a single row of actual data.

A library where it takes longer to walk between shelves than to read the book. The fix takes an afternoon. The performance debt built up silently for months.

Delta Lake and Iceberg bring ACID to object storage. Two questions matter more than feature comparisons: does your team have the operational discipline to maintain them? Does the complexity justify itself over a managed alternative?

How ACID Works on Object Storage

S3, GCS, and Azure Blob were not designed for concurrent transactional writes. Eventually consistent, object-level, no native atomic multi-file operations. Getting ACID on infrastructure built to store objects and log files required some clever engineering. Getting ACID on infrastructure not built for it required clever engineering.

Open table formats solve this with transaction logs. Every write stages new Parquet files, then commits by appending an entry to the log. The log records which files were added and which were logically deleted (tombstoned). Readers check the log to find the current consistent set of files before scanning data. Concurrent writers use optimistic concurrency: if two writers try to commit overlapping changes, one succeeds and the other gets a conflict error and retries.

In practice, readers always see a consistent snapshot, concurrent writes fail cleanly rather than producing corrupt data, and uncommitted writes are never visible. The overhead is 50-200ms per commit operation. Tiny for batch workloads. Compounds for real-time streaming micro-batches writing every 30 seconds. (Fine for batch. Compounds for streaming micro-batches.)

Choosing between Iceberg, Delta Lake, and Hudi depends on your query engine ecosystem, update frequency, and operational maturity. A solid data engineering practice evaluates these against your actual workload, not the last conference talk.

Z-Ordering and Data Layout

ACID on object storage is the foundation. But ACID alone doesn’t make queries fast. Query performance lives and dies by how data is physically organized within files. ACID alone doesn’t make queries fast.

Z-ordering co-locates related data so the query engine can skip entire files that contain no matching values, turning full scans into targeted reads. A query filtering on customer_id against a 500GB unordered table scans all 500GB. The same query against a Z-ordered table scans a small fraction because the engine reads file-level min/max statistics and skips files that can’t contain the target value. Targeted reads instead of full scans.

Z-ordering sorts data within files by the specified columns. Min/max statistics per file let the query engine skip files that cannot contain matching rows. 97% file skipping on a single-column filter is typical.

The trade-off is compute at write time. Z-ordering is applied during compaction and requires rewriting files. For a large table, a full Z-order rewrite can take tens of minutes on a medium Spark cluster. It pays off on high-read, lower-write tables where the write compute cost is spread across thousands of query executions. Re-alphabetizing the library takes a weekend. Worth it if 1,000 people search the shelves every day.

This is one area where platform engineering investment in optimized compaction scheduling pays back directly through reduced query costs.

The Compaction Requirement

Every write creates new files. Every update marks old files as deleted and creates new ones. Every streaming micro-batch produces a file. After weeks of frequent writes, a table piles up tens of thousands of small files, each requiring the query engine to open it and read its metadata before scanning data. Hundreds of pamphlets where there should be one bound volume.

The surprising part: file count is a performance multiplier independent of total data volume. A table with 10GB in 50,000 files queries slower than a table with 1TB in 200 files. A library with 50,000 pamphlets is slower to search than one with 200 encyclopedias. Every team that skips compaction learns this the hard way.

Compaction rewrites many small files into fewer, larger, well-sorted files:

• Schedule compaction on a cadence matched to write frequency (daily for streaming tables, weekly for batch)
• Tune target file size: 256MB per file for high-write tables, 512MB for batch-loaded tables
• Decide whether to apply Z-ordering during compaction (yes for analyst-facing tables, usually no for intermediate pipeline tables)
• Run vacuum to expire old snapshots and recover storage (keep 7 days of snapshots for time travel, delete everything older)

-- Iceberg compaction: rewrite small files into optimal-sized ones
CALL system.rewrite_data_files(
  table => 'analytics.orders',
  options => map(
    'target-file-size-bytes', '268435456',  -- 256MB target
    'min-file-size-bytes', '67108864',       -- Don't touch files >64MB
    'max-concurrent-file-group-rewrites', '5'
  )
);

-- Expire old snapshots to reclaim storage
CALL system.expire_snapshots('analytics.orders', TIMESTAMP '2026-03-14 00:00:00', 100);

-- Remove orphan files left by failed writes
CALL system.remove_orphan_files(table => 'analytics.orders', older_than => TIMESTAMP '2026-03-07');

    

      ×
      ❚❚
      ‹
      ›
      +
      −
    
    


  

```d2 { alt="Lakehouse compaction and vacuum operational schedule showing daily compaction for high-write tables, weekly for medium-write, and vacuum runs to expire old snapshots and recover time-travel storage" }
direction: right

tables: Table Write Frequency {
  style.fill: "#E8EAF6"
  style.stroke: "#1A2980"

  high: High-Write Tables {
    style.fill: "#C5CAE9"
    label: "Streaming ingestion\n100k+ rows/hour\nCompact: daily\nTarget: 256MB files"
  }

  medium: Medium-Write Tables {
    style.fill: "#E0F7F6"
    label: "Batch ETL\n1-10M rows/day\nCompact: weekly\nTarget: 512MB files"
  }

  low: Low-Write Tables {
    style.fill: "#E0F7F6"
    label: "Reference data\nUpdated rarely\nCompact: monthly\nTarget: 512MB files"
  }
}

ops: Maintenance Operations {
  style.fill: "#E8EAF6"
  style.stroke: "#1A2980"

  compact: File Compaction {
    style.fill: "#E8EAF6"
    label: "Merge small files\nApply Z-ordering\nUpdate table stats"
  }

  vacuum: Snapshot Vacuum {
    style.fill: "#E8EAF6"
    label: "Expire snapshots\nolder than 7 days\nDelete orphaned files\nRecover storage"
  }

  stats: Statistics Refresh {
    style.fill: "#E8EAF6"
    label: "Column min/max stats\nRow count per file\nEnables data skipping"
  }
}

tables -> ops: "Schedule per\nwrite frequency"

That customer segmentation query. Eight seconds to four minutes because nobody scheduled compaction. With compaction matched to write patterns, automated partition management, and file size targets enforced in the pipeline, the debt never piles up. The query stays fast because the maintenance runs before anyone notices it needs to. The lakehouse delivers when operational discipline matches architectural ambition.