Swap out your old disks for some new ones with greater capacity Last updated: 24/04/2016 - 3 years, 9 months ago An article tagged as: linux, nas, ubuntu, zfs Overview Eventually your ZFS pool will no longer have enough storage for you. When this happens you will need to add some disks or replace your old disks with new larger ones. Since i no longer have any spare SATA ports, I am going to do the latter, replacing all my 2TB disks with 6TB ones. This article will show you how to replace your existing disks with larger capacity ones on a ZFS pool that is using raidz. The following assumptions are made regarding your setup:- Your ZFS pool uses raidz1, raidz2 or raidz3 Your computer/server is running ZFS on Linux Backup your data!! Please make sure you have a full backup of your data before replacing any disks. This article is a useful guide, but we cannot take any responsibility if anything goes wrong. Get current pool information Before you start swapping disks, you should gather as much information about your existing ZFS pools $ zpool status -v pool: DUMPSTER state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016 config: NAME STATE READ WRITE CKSUM DUMPSTER ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdc2 ONLINE 0 0 0 sdb2 ONLINE 0 0 0 sdd2 ONLINE 0 0 0 sdf2 ONLINE 0 0 0 sdg2 ONLINE 0 0 0 sde2 ONLINE 0 0 0 errors: No known data errors We need some more detailed information, so will also do the following:- $ sudo zdb DUMPSTER: version: 5000 name: 'DUMPSTER' state: 0 txg: 10893419 pool_guid: 66180509451502810599 errata: 0 hostid: 8399329 hostname: 'bob' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 66180509451502810599 children[0]: type: 'raidz' id: 0 guid: 4466980275526910809 nparity: 1 metaslab_array: 23 metaslab_shift: 36 ashift: 12 asize: 11989479849984 is_log: 0 children[0]: type: 'disk' id: 0 guid: 2277585469358930581 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9GB903918-part2' phys_path: '/dev/gptid/c1cbdbc7-00c8-11e1-a37e-f46d046fc73d' whole_disk: 0 DTL: 84 children[1]: type: 'disk' id: 1 guid: 12154086360534719056 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705656-part2' phys_path: '/dev/gptid/c215d0a5-00c8-11e1-a37e-f46d046fc73d' whole_disk: 0 DTL: 83 children[2]: type: 'disk' id: 2 guid: 99122293356262304622 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705655-part2' phys_path: '/dev/gptid/c258b10f-00c8-11e1-a37e-f46d046fc73d' whole_disk: 0 DTL: 82 children[3]: type: 'disk' id: 3 guid: 9833596632931584564 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705653-part2' phys_path: '/dev/gptid/c2989f3a-00c8-11e1-a37e-f46d046fc73d' whole_disk: 0 DTL: 81 children[4]: type: 'disk' id: 4 guid: 887129227780467899 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9GB903914-part2' phys_path: '/dev/gptid/c2da76e5-00c8-11e1-a37e-f46d046fc73d' whole_disk: 0 DTL: 80 children[5]: type: 'disk' id: 5 guid: 1256734120672923987 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705654-part2' phys_path: '/dev/gptid/c32365af-00c8-11e1-a37e-f46d046fc73d' whole_disk: 0 DTL: 79 features_for_read: You might want to save the above information about your disks for later reference. The key information here is that my pool DUMPSTER is made up of 6 disks, with a raidz configuration. raidz, also known as raidz1, means that up to one disk can fail without losing any data. Also, it is important to note the guid and path for each disk. Configure expansion We need to tell our pool to expand automatically once all the new disks have been added. We do this with the following:- $ sudo zpool set autoexpand=on DUMPSTER Swap out the disk The method i am going to go through assumes that you don't have any spare SATA ports. So the current disk will be removed, and a new bigger disk will be put in its place. Do the following for starters:- Shutdown the server Unplug and remove one of your current disks Make a note of the serial number on the new disk Put the new disk in its place, and connect it Turn the server back on Now check that status of the pool $ sudo zpool status pool: DUMPSTER state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016 config: NAME STATE READ WRITE CKSUM DUMPSTER DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 2277585469358930581 UNAVAIL 0 0 0 was /dev/sdc2 sdb2 ONLINE 0 0 0 sdd2 ONLINE 0 0 0 sdf2 ONLINE 0 0 0 sdg2 ONLINE 0 0 0 sde2 ONLINE 0 0 0 errors: No known data errors As you can see, the pool has a state of DEGRADED and the disk we removed is shown as UNAVAIL. Helpfully, this also shows the guid of the removed disk, being 2277585469358930581, and that the location of the removed disk was /dev/sdc2. We will need this information shortly. Now we want to list all the disks to find the newly added one. Mine should be listed under /dev/sdc. $ sudo fdisk -l ... Disk /dev/sdc: 6001.2 GB, 6001175126016 bytes 255 heads, 63 sectors/track, 729601 cylinders, total 11721045168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk identifier: 0x00000000 Disk /dev/sdc doesn't contain a valid partition table ... So the new 6TB disk is there but it isn't part of the pool yet. Replace the disk We have physically swapped our disk, but we need to tell our ZFS pool that we have replaced the old disk with a new one. First, we need to find the path of the new disk:- $ ls -la /dev/disk/by-id ... lrwxrwxrwx 1 root root 9 Apr 14 10:31 ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA -> ../../sdc ... This shows me that my new disk, which has a serial number of WX31D95N3KPA, has a path of /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA Create GPT table Before our new disk can become part of the pool, it needs to have a GPT table. This can be created as follows:- $ sudo parted /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA GNU Parted 2.3 Using /dev/sdc Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel GPT (parted) q Information: You may need to update /etc/fstab. Now we are ready to make the new disk part of our pool. Make sure you replace the following with the guid of your old disk, and the path of your new disk. $ sudo zpool replace DUMPSTER 2277585469358930581 /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA The new disk will now be resilvered. This process can take several hours. But you can check its progress:- $ sudo zpool status pool: DUMPSTER state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Thu Apr 14 10:31:36 2016 74.0M scanned out of 10.6T at 5.28M/s, 11h8m to go 11.9M resilvered, 0.00% done config: NAME STATE READ WRITE CKSUM DUMPSTER DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 replacing-0 UNAVAIL 0 0 0 2277585469358930581 UNAVAIL 0 0 0 was /dev/sdc2 ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA ONLINE 0 0 0 (resilvering) sdb2 ONLINE 0 0 0 sdd2 ONLINE 0 0 0 sdf2 ONLINE 0 0 0 sdg2 ONLINE 0 0 0 sde2 ONLINE 0 0 0 errors: No known data errors This shows me that resilvering progress is going to take another 11 hours. This is only an estimate. I would not recommend using the pool whilst it is resilvering. Check that it has worked Once complete, the pool should return to an online state. $ zpool status -v pool: DUMPSTER state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016 config: NAME STATE READ WRITE CKSUM DUMPSTER ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdc2 ONLINE 0 0 0 sdb2 ONLINE 0 0 0 sdd2 ONLINE 0 0 0 sdf2 ONLINE 0 0 0 sdg2 ONLINE 0 0 0 sde2 ONLINE 0 0 0 errors: No known data errors This process can now be repeated to replace as many of the disk as you need to. However, I will not get the bennefit of the larger capacity disks until all the old smaller ones have been replaced.