No database server, no lock manager — just files and a log. Here's how two Spark jobs can write the same table at the exact same moment and never corrupt it.
Two Spark jobs write to the same orders table at 9:00:00 — the exact same second. On a normal folder of files, that’s a recipe for disaster: both overwrite, one clobbers the other, data goes missing. On a Delta table, nothing bad happens. Both writes land safely.
And here’s the surprising part: there’s no database server anywhere. No lock manager, no transaction coordinator, no connection pool. Delta gives you full ACID transactions using nothing but files in cloud storage. How? Let’s follow those two jobs and find out.
First, what “ACID” even means
ACID is the promise that your data operations are safe. Four letters, one line each:
A — Atomicity: a change happens fully, or not at all. Never half.
C — Consistency: the data always follows the table’s rules (like its schema).
I — Isolation: concurrent operations don’t step on each other.
D — Durability: once it’s committed, it survives crashes.
Traditional databases deliver these with a central engine and locks. Delta delivers all four with a much simpler trick.
The whole trick: a numbered log, not a database
A Delta table is just Parquet data files + a transaction log — a folder called _delta_log full of numbered JSON files, sitting in the same S3 or ADLS bucket as the data. (We break down exactly what’s in those files in Delta table internals.)
Everything follows from one rule:
To change the table, you append the next numbered commit file to the log. Commit 5 is ...0005.json, commit 6 is ...0006.json, and so on. The current table is whatever you get by reading the log up to the highest number.
That’s it. No server decides transactions — the log is the transaction system. Now watch how each ACID guarantee falls out of it.
Atomicity: write files first, commit last
When a job writes, it does two things in order: first it writes the new Parquet data files, then it appends one commit file that says “these files are now part of the table.”
The data files are invisible until that commit lands. So if a job crashes after writing files but before committing, those files are orphans no commit references — every reader ignores them, and the table is untouched. You can never see half a write. That single ordering — files first, commit last — is atomicity.
Isolation: how two writers don’t collide
Now the main event — our two jobs at 9:00:00. Call them Job A and Job B. Both read the table at its current version (say version 4), both prepare their new data, and both try to commit the next version: version 5.
There are no locks. Neither job asks permission. They each just try to create the file ...0005.json. And here’s the piece the cheat sheets skip — the actual mechanism that makes it safe:
Creating that commit file is atomic: “create this file only if it doesn’t already exist.” Only one job can win the race to create ...0005.json. The other gets told “it already exists.”
Say Job A wins — its commit becomes version 5. Job B is told version 5 is taken. But Job B doesn’t just fail. It checks: does my change actually clash with what Job A just did?
If they don’t clash — for example both are just appending new rows, or they touched different files — Job B quietly re-bases its change on top of version 5 and commits as version 6. No error, no lost data. This is why hundreds of jobs can append to one table smoothly: appends only add new files, so they never touch each other.
If they do clash — for example two MERGE operations that modify the same rows — Delta can’t safely merge them, so it raises a ConcurrentAppendException. Your job catches it and retries against the new state.
This approach is called optimistic concurrency: assume no conflict, do the work, and only sort it out at commit time — the opposite of a database that locks rows before writing.
Two jobs race to create the next commit file — only one can win, and the other re-bases or retries.
The one detail that makes it all work
Everything hinges on that “create only if it doesn’t exist” being truly atomic. Most cloud storage supports it natively (Azure ADLS and Google Cloud Storage always have; Amazon S3 now does too via conditional writes). Where storage historically couldn’t guarantee it, Delta used a small coordination layer to stand in. But the idea is always the same: make claiming the next version number a race that exactly one writer can win.
Isolation for readers: a stable snapshot
Readers get isolation too, and it’s just as simple. When a query starts, it pins the current version — say version 5 — and reads that snapshot to the end. If writers commit versions 6, 7, 8 while the query runs, the reader keeps seeing version 5. It never sees a half-applied change, and readers never block writers or vice versa.
This is also why “time travel” isn’t a bolt-on feature — it’s the same mechanism. Reading “the table as of version 3” is just pinning version 3 instead of the latest.
A reader pins one version and sees a consistent snapshot, even as new versions are committed around it.
Consistency and Durability: the easy two
Consistency: before a commit is allowed, Delta checks the write against the table’s schema (and protocol). Data that doesn’t fit the table’s rules is rejected, so the table never ends up in an invalid shape.
Durability: every data file and commit file lives in cloud object storage, which is heavily replicated and highly durable. Once a commit lands, it’s safe — even if every cluster disappears the next second.
Why doing it with files (not a database) is a big deal
A traditional database runs ACID through a central engine with locks and connections. That engine is powerful — but it’s also a bottleneck and a single point of failure, and it doesn’t stretch to petabytes and thousands of concurrent jobs easily.
Delta moves the whole transaction mechanism into the log files themselves. There’s no server to run, nothing to scale up, and no single machine everything depends on. The concurrency model scales with your object storage — which is effectively limitless. That’s how a Delta table can serve huge data and heavy concurrent traffic while behaving like a well-mannered transactional database.
The honest caveats
Not every conflict auto-resolves. Compatible writes (like blind appends) re-base and retry themselves, but operations that modify the same rows — overlapping MERGEs — can still throw ConcurrentAppendException, and you handle the retry. Optimistic concurrency detects conflicts; it doesn’t magically prevent all of them.
The atomic-commit guarantee depends on your storage. It relies on “create-if-not-exists” being atomic; on most modern cloud storage that’s built in, but it’s the assumption everything rests on.
VACUUM is the boundary. Deleting old files past the retention window (7 days by default) reclaims storage but breaks time travel — and can even break long-running readers still pinned to an old version. Lower it with care.
The takeaway
Delta Lake gives you ACID with no database engine at all — just Parquet files and a numbered JSON log in object storage. Atomicity comes from writing files first and committing last, so crashes leave only ignorable orphans. Isolation comes from optimistic concurrency: writers race to atomically create the next commit number, only one wins, and the loser re-bases if compatible or retries if it truly conflicts — while readers pin a version and see a stable snapshot. Consistency is schema enforcement at commit time, and durability is cloud storage. Once you see that the log is the transaction system, “how does Delta do ACID without a database?” answers itself: it turned a folder of files into a ledger, and a ledger is all a transaction ever needed.
Frequently Asked Questions
How does Delta Lake provide ACID transactions without a database?
Delta uses a numbered JSON transaction log stored alongside the data files in object storage. To change the table you append the next commit file, and the current table is computed by reading the log. Atomicity, consistency, isolation, and durability all come from how commits are written and read — no separate database engine, lock manager, or server is involved.
What happens when two jobs write to the same Delta table at once?
Both jobs try to create the next commit file, but only one can — creating that file is atomic, so exactly one writer wins that version. The other checks whether its change conflicts; if not, it re-bases onto the new version and commits the next number, and if it does conflict (like overlapping MERGEs) it gets a ConcurrentAppendException and retries.
Does Delta Lake use locks for concurrency?
No. Delta uses optimistic concurrency control: writers never acquire locks, they just do their work and attempt to commit, and conflicts are detected at commit time rather than prevented by locking. This is the opposite of a traditional database, which locks rows before writing.
How does Delta give readers a consistent view during writes?
Through snapshot isolation. A query pins the table version it starts at and reads only that version’s files, even if new versions are committed while it runs. Readers never see a half-applied change and never block writers. Time travel uses the same mechanism — it just pins an older version.
What makes the Delta commit atomic on object storage?
Claiming a version requires creating its commit file only if it doesn’t already exist, and that create operation must be atomic so only one writer can succeed. Most cloud storage supports this natively; where it historically did not, Delta used a small coordination layer to guarantee that exactly one writer wins each version number.
Why is a file-based transaction log better than a database engine for big data?
A database engine is a central component that can become a bottleneck and a single point of failure, and it doesn’t scale effortlessly to petabytes and thousands of concurrent jobs. Delta’s log lives in object storage, so the transaction mechanism scales with storage instead of a server, with nothing central to run or overload.
