November 2026
November 2026
If you don't read the manual, don't expect AI to sort it for you.
That's the lesson. Not the ZFS lesson — the *agentic delivery* lesson. The one that keeps happening, over and over, in every system I've built this year. Plans get made. Tests get run. Both sides check the work. And neither side reads the documentation deep enough to know what they're actually checking.
This is the story of one instance. It cost three days.
Agentic delivery — human and AI working together to get things done — only works if you both have at least some idea what you're doing. That sounds obvious. It isn't. Because "having an idea" and "having read the manual" are not the same thing, and the gap between them is where every trap in this story lives.
Here's what happened. I had a storage pool at 80% capacity. I had a backup that was a perfect mirror. I had an AI agent that could index 1.5 million files, compare them byte-for-byte, and write a deletion script gated on SHA256 verification. I had a plan. I checked the plan. The AI checked the plan. The plan was sound.
The pool: `sata-one`, raidz2, four disks, ~14 TiB usable. 80% full. Not critical, but trending the wrong direction.
The backup: 1,497,854 files across 11.29 TiB on a separate machine. Comparison result: 1,429,980 files matched on name and size. Zero size mismatches. The backup was perfect. Reclaimable total: 8.07 TiB across a handful of directories.
The plan: delete the directories that are fully backed up. Gate it on a SHA256 sample. Run it. Reclaim 8 TiB. Done.
The AI and I went through this together. It indexed. I reviewed. It compared. I verified the match counts. It wrote the deletion script. I reviewed the script. It added the SHA256 gate. I approved. We both checked twice.
The script ran. 647,318 files deleted. I checked the pool.
80%. Still 80%. There is nothing exactly like the panic you feel when you delete that large a section of files and find you freed up ZERO space on your drives
The plan assumed that deleting files frees space. That's true on a simple filesystem. It's not true on ZFS when snapshots exist.
ZFS snapshots hold references to blocks. Delete a file that's been snapshotted, and the blocks don't leave the pool — they move from `usedbydataset` to `usedbysnapshots`. The space isn't free. It's just re-categorized.
Both of us knew this. In principle. The AI knew it the way it knows most things — as a pattern in its training data, "ZFS snapshots can prevent space reclamation." I knew it the way I know most things about ZFS — as something I read once and filed under "edge case, probably won't bite."
Neither of us checked. Neither of us ran `zfs get usedbysnapshots` before writing the deletion script. The plan was built on an assumption that was correct 90% of the time and wrong 100% of the time for this dataset.
And here's where it gets worse. Not only did the plan fail — every tool I reached for to diagnose the failure gave a different wrong answer, making me absolutely sure I was right and something else was broken. Confirmation bias helped do the balance of the damage because I kept seeing the answer I knew should be there — that the data was deleted — so where is my free space?
Tool 1: `zfs list -t snapshot sata-one`
Result: `no datasets available`.
No snapshots. Problem solved, right? Wrong. `zfs list -t snapshot` without `-r` only inspects the root dataset. The snapshots live on child datasets. The command was technically correct and completely misleading.
The actual list, with `-r`:
sata-one/share@auto-2026-09-28_00-00
sata-one/share@auto-2026-09-29_00-00
... (14 daily snapshots)
sata-one/share@auto-2026-10-04_00-00
sata-one/share@manual-2021-backup
Fourteen daily snapshots. Plus one from 2021. All on `sata-one/share`. All holding references to the blocks I just deleted.
Tool 2: Per-snapshot `USED` column
Every snapshot showed `0B`. Fourteen snapshots, zero bytes each. Looks like they cost nothing.
This is the trap. The per-snapshot `USED` field measures space referenced *exclusively* by that snapshot versus its neighbors. Fourteen snapshots capturing identical data each report `0B` because none holds anything the others don't. The cost isn't visible at the per-snapshot level.
The only number that matters is the dataset-level aggregate:
zfs get -p -o name,property,value usedbysnapshots sata-one/share
sata-one/share usedbysnapshots 6489834421632 (5.90 TiB)
5.90 TiB held by snapshots. Every block I "freed" went straight into `usedbysnapshots`. The pool didn't shrink because nothing actually left it.
Tool 3: `df`
Reported the pool at 68%. It was lying. `df` ignores space held by snapshots entirely. It saw 5.5 TiB free and reported 68% because it didn't know about the 5.9 TiB the snapshots were holding.
Three tools. Three wrong answers. The only tool that told the truth was `zfs get usedbysnapshots` — and I didn't run it until day three.
I would love to give you all a clear simple programatic way to clean this up but the problem is on the people and process side not the technology stack.
zfs destroy -nv 'sata-one/share@auto-2026-09-28_00-00%auto-2026-10-04_00-00'
would destroy sata-one/share@auto-2026-09-28_00-00
would destroy sata-one/share@auto-2026-09-29_00-00
... (13 snapshots)
would reclaim 5.90T
`zfs destroy -nv` is free, instant, and completely non-destructive. It would have told me on day one that the snapshots were holding 5.90 TiB and that destroying them would reclaim it.
I destroyed all 15 snapshots. Pool went from 80% to ~39%. 8.5 TiB free.
The trade-off: destroying snapshots removes rollback. I accepted this because all 647,318 deleted files were live-verified present on the backup at identical size. The backup was the rollback. The next day's automatic snapshot re-established the rolling rollback point.
Here's what actually happened, stripped of the technical details:
1. I had a problem (pool at 80%)
2. I made a plan (delete backed-up data)
3. I checked the plan (AI reviewed it, I reviewed it, SHA256 gate added)
4. The plan was wrong (snapshots held the blocks)
5. I reported "solved" twice before it was actually solved
6. The fix was one command I didn't know about until day three
Steps 1-4 are normal. Plans are wrong all the time. Step 5 is the failure — reporting success without verifying the outcome. Step 6 is the lesson — the tool existed, the man page was right there, and neither of us (human or AI) looked.
The AI didn't catch this because it was also reasoning from pattern, not from documentation. "ZFS snapshots can prevent space reclamation" is in its training data. But the specific behavior — `usedbysnapshots` as the aggregate, the per-snapshot `USED` field being misleading, `df` ignoring snapshots entirely, the `-r` flag on `zfs list` — that's documentation. That's reading the manual. And neither of us did.
Before recommending any deletion on a snapshotted dataset, run `zfs get usedbysnapshots`. If it is non-trivial, deleting files reclaims nothing, and the deletion plan is incomplete without a snapshot-destruction step.
And the meta-rule, the one that applies to every agentic delivery session I've run this year:
**Agentic delivery only works if you both have at least some idea what you're doing.** Not "an idea" — some idea. Enough to know which manual to open. Enough to know that "looks right" and "is right" are different things. Enough to verify the outcome, not just the plan.
Plans and tests have to be done in a safe space. Not because the plan is wrong — plans are always a little wrong — but because "checked twice" doesn't mean "checked against the source." It means "checked against what we think we know." And what we think we know is where every trap lives.
`zfs destroy -nv` — free, instant, non-destructive. Run it before you delete anything. It tells you exactly what would be freed and by how much. If the answer is "5.90T", you know the snapshots are the problem. If the answer is "0", you know the data is actually gone and the pool is genuinely full.
I didn't run it on day one. I ran it on day three. The man page was right there the whole time. So was the AI's training data. Neither of us opened either one.
That's the story. Not the ZFS part — the part where two parties, both reasonably competent, both checking each other's work, still shipped a plan that was wrong because neither one read the manual. That's not a ZFS problem. That's an agentic delivery problem. And it's the one I keep having.
*Published at atkatana.com/blog. Part of the series on operational reality behind the infrastructure.*
Built on a home lab, powered by local models, and owned by Andrew Katana.