Table of content
Sometimes shitty things happen and you need to dig into the insides of
ZFS. To this end, there is the zdb command, some of these
options are presented below, and some example of its use will be
detailed in practical examples.
Configuration
Lists the pool configuration with the attached devices (vdev):
1 | |
Dataset
Lists information about the various dataset (clone, snapshot, …), this information includes: name, id number, last transaction number, used space, and objects count:
1 | |
Labels
Labels correspond to the configuration description of the pool but they are stored on each disk and include disk specific information. There are 4 labels per disk, two at the beginning and two at the end. They should normally be identical.
1 | |
Corrupted mirror
A crash of the system has prevented ZFS to complete its write operations on a mirror which has led to a difference in the metadata stored on the two disks. Consequently, the pool is no longer available, because corrupted, event if each disk is online.
The goal now is to identify which disk metadata has been partly written, in order to be able to find a healthy pool by removing the corrupted disk. This approach, somewhat abrupt, works well only in the mirror case. For this, the ZFS labels (metadata) will be read to compare the transaction numbers (fields: txg) and find the order in which records have been written.
1 2 | |
Actually, during the labels display, the ad10 hard drive
is immediately identified as corrupted as only two labels are legible
on the 4 availables (two labels are written at the beginning and two
others at the end).
The rebuild process chosen is to break the mirror and recreate it:
1 2 | |
This is where a mistake was made. Indeed, at the time (ZFS v15) the
pool’s policy was to automatically increase its capacity if the drive
capacity allows it. Unfortunately the two disks, although with the
same reference, were not of the same capacity, the transition to a
single disk caused an increase in capacity of the pool, preventing
later the mirror reconstruction because the disk added was then too
small. It was therefore necessary to make a backup to entirely rebuild
the pool. Since then, the autoexpand attribut was added and is set to
false by default.
Unable to destroy snapshot
1 2 | |
There was a bug (CR-6860996) leading to this type of problem when
receiving an incremantal stream (zfs recv), a temporary clone, with
the “%” character in its name, is created but not deleted
automatically.
So we will look for this clone and destroy it explicitly:
1 2 | |