Yes, hard disks do break from time to time. But thanks to ZFS and RAIDZ-1, there is no reason to panic. Or is there? Interestingly the disk was not reported by ZFS to be broken but there was a message in the WebUI indicating that something is wrong.

So I guess no need to run a SMART long test, let's go ahead and replace the disk. Before I replaced the failed disk, I first started a ZFS replication to an external USB 3.0 hard disk. If you remember from one of my first TrueNAS articles I already did replicate my zPool to an external disk so this time it only took a couple of minutes to sync the latest snapshots. Then I took the bad drive offline via TrueNAS WebUI

and shut down the system. I still had an old hard disk of the same size and replaced the defective one with this one. So far so good. I restarted the system and started the Replace process via the WebUI. Unfortunately, the UI reported after seconds that the hard disk is "Busy". A somewhat strange message, since the hard disk was not yet known to the system. My first suspicion was confirmed immediately. The hard disk still contained old partitions and TrueNAS therefore rejects it. A more informative error message would be helpful here. A simple delete of the partitions and a reboot helped here.

sgdisk -Z /dev/sdb

BTW. there is also a Force tickbox in the WebUI. Probably this would have worked as well. The pool is now resilvering and hopefully all will be fine once it is completed.

root@truenas[~]# zpool status -vx
  pool: zpool0
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jan  1 10:13:31 2023
	626G scanned at 345M/s, 89.1G issued at 49.1M/s, 639G total
	89.2G resilvered, 13.94% done, 03:11:02 to go
config:

	NAME                                      STATE     READ WRITE CKSUM
	zpool0                                    ONLINE       0     0     0
	  mirror-0                                ONLINE       0     0     0
	    5c6c6ba0-5e73-47c1-91dd-3f8ac048f4e2  ONLINE       0     0     0
	    32be529d-c363-4ea6-815d-47dd182517af  ONLINE       0     0     0  (resilvering)

errors: No known data errors