Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes.

This bring huge amount of changes, I'll enumerate only user-visible changes:

- Delegated Administration

	Allows regular users to perform ZFS operations, like file system
	creation, snapshot creation, etc.

- L2ARC

	Level 2 cache for ZFS - allows to use additional disks for cache.
	Huge performance improvements mostly for random read of mostly
	static content.

- slog

	Allow to use additional disks for ZFS Intent Log to speed up
	operations like fsync(2).

- vfs.zfs.super_owner

	Allows regular users to perform privileged operations on files stored
	on ZFS file systems owned by him. Very careful with this one.

- chflags(2)

	Not all the flags are supported. This still needs work.

- ZFSBoot

	Support to boot off of ZFS pool. Not finished, AFAIK.

	Submitted by:	dfr

- Snapshot properties

- New failure modes

	Before if write requested failed, system paniced. Now one
	can select from one of three failure modes:
	- panic - panic on write error
	- wait - wait for disk to reappear
	- continue - serve read requests if possible, block write requests

- Refquota, refreservation properties

	Just quota and reservation properties, but don't count space consumed
	by children file systems, clones and snapshots.

- Sparse volumes

	ZVOLs that don't reserve space in the pool.

- External attributes

	Compatible with extattr(2).

- NFSv4-ACLs

	Not sure about the status, might not be complete yet.

	Submitted by:	trasz

- Creation-time properties

- Regression tests for zpool(8) command.

Obtained from:	OpenSolaris

7 files changed, 1167 insertions, 0 deletions

diff --git a/tools/regression/zfs/zpool/replace/cache.t b/tools/regression/zfs/zpool/replace/cache.t
new file mode 100644
index 000000000000..23abc3769191
--- /dev/null
+++ b/tools/regression/zfs/zpool/replace/cache.t
@@ -0,0 +1,43 @@
+#!/bin/sh
+# $FreeBSD$
+
+dir=`dirname $0`
+. ${dir}/../../misc.sh
+
+echo "1..6"
+
+disks_create 4
+names_create 1
+
+expect_ok ${ZPOOL} create ${name0} ${disk0} cache ${disk1}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	cache"
+  echo "	    ${disk1}  ONLINE     0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_fl ${ZPOOL} replace ${name0} ${disk1} ${disk2}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	cache"
+  echo "	    ${disk1}  ONLINE     0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+disks_destroy
diff --git a/tools/regression/zfs/zpool/replace/disk.t b/tools/regression/zfs/zpool/replace/disk.t
new file mode 100644
index 000000000000..51029e6d8a68
--- /dev/null
+++ b/tools/regression/zfs/zpool/replace/disk.t
@@ -0,0 +1,48 @@
+#!/bin/sh
+# $FreeBSD$
+
+dir=`dirname $0`
+. ${dir}/../../misc.sh
+
+echo "1..10"
+
+disks_create 4
+names_create 1
+
+expect_ok ${ZPOOL} create ${name0} ${disk0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk1}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME        STATE   READ WRITE CKSUM"
+  echo "	${name0}    ONLINE     0     0     0"
+  echo "	  ${disk1}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} ${disk0} ${disk1} ${disk2}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME        STATE   READ WRITE CKSUM"
+  echo "	${name0}    ONLINE     0     0     0"
+  echo "	  ${disk0}  ONLINE     0     0     0"
+  echo "	  ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	  ${disk2}  ONLINE     0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+disks_destroy
diff --git a/tools/regression/zfs/zpool/replace/log.t b/tools/regression/zfs/zpool/replace/log.t
new file mode 100644
index 000000000000..6eed8bdc9926
--- /dev/null
+++ b/tools/regression/zfs/zpool/replace/log.t
@@ -0,0 +1,152 @@
+#!/bin/sh
+# $FreeBSD$
+
+dir=`dirname $0`
+. ${dir}/../../misc.sh
+
+echo "1..27"
+
+disks_create 4
+names_create 1
+
+expect_ok ${ZPOOL} create ${name0} ${disk0} log mirror ${disk1} ${disk2}
+expect_ok ${ZPOOL} export ${name0}
+dname1=${disk1}
+fdname1=${fdisk1}
+guid1=`get_guid ${fdisk1}`
+disk_destroy 1
+disk_create 1 ${dname1}
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  ${disk0}    ONLINE       0     0     0"
+  echo "	logs          DEGRADED     0     0     0"
+  echo "	  mirror      DEGRADED     0     0     0"
+  echo "	    ${guid1}  UNAVAIL      0     0     0 was ${fdname1}"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+add_msg="# TODO Sun CR 6710376, Lustre bug 16912"
+expect "${exp}" ${ZPOOL} status ${name0}
+add_msg=""
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: resilver completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	logs          ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk3}  ONLINE     0     0     0"
+  echo "	    ${disk2}  ONLINE     0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} ${disk0} log mirror ${disk1} ${disk2}
+expect_ok ${ZPOOL} replace ${name0} ${disk2} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: resilver completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	logs          ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk1}  ONLINE     0     0     0"
+  echo "	    ${disk3}  ONLINE     0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} ${disk0} log mirror ${disk1} ${disk2}
+expect_ok ${ZPOOL} export ${name0}
+dname1=${disk1}
+fdname1=${fdisk1}
+guid1=`get_guid ${fdisk1}`
+disk_destroy 1
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be opened.  Sufficient replicas exist for"
+  echo "      the pool to continue functioning in a degraded state."
+  echo "action: Attach the missing device and online it using 'zpool online'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-2Q"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  ${disk0}     ONLINE       0     0     0"
+  echo "	logs           DEGRADED     0     0     0"
+  echo "	  mirror       DEGRADED     0     0     0"
+  echo "	    ${guid1}   UNAVAIL      0     0     0  was ${fdname1}"
+  echo "	    ${disk2}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${dname1} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: resilver completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	logs          ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk3}  ONLINE     0     0     0"
+  echo "	    ${disk2}  ONLINE     0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 1 ${dname1}
+
+expect_ok ${ZPOOL} create ${name0} ${disk0} log mirror ${disk1} ${disk2}
+expect_ok ${ZPOOL} offline ${name0} ${disk1}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: resilver completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	logs          ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk3}  ONLINE     0     0     0"
+  echo "	    ${disk2}  ONLINE     0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+disks_destroy
diff --git a/tools/regression/zfs/zpool/replace/mirror.t b/tools/regression/zfs/zpool/replace/mirror.t
new file mode 100644
index 000000000000..6e9dafead23f
--- /dev/null
+++ b/tools/regression/zfs/zpool/replace/mirror.t
@@ -0,0 +1,134 @@
+#!/bin/sh
+# $FreeBSD$
+
+dir=`dirname $0`
+. ${dir}/../../misc.sh
+
+echo "1..27"
+
+disks_create 3
+names_create 1
+
+expect_ok ${ZPOOL} create ${name0} mirror ${disk0} ${disk1}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+disk_create 0 ${dname0}
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  mirror      DEGRADED     0     0     0"
+  echo "	    ${guid0}  UNAVAIL      0     0     0 was ${fdname0}"
+  echo "	    ${disk1}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk2}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk1}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} mirror ${disk0} ${disk1}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk2}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk0}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} mirror ${disk0} ${disk1}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  mirror       DEGRADED     0     0     0"
+  echo "	    ${guid0}   REMOVED      0     0     0  was ${fdname0}"
+  echo "	    ${disk1}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${dname0} ${disk2}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk1}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 0 ${dname0}
+
+expect_ok ${ZPOOL} create ${name0} mirror ${disk0} ${disk1}
+expect_ok ${ZPOOL} offline ${name0} ${disk1}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk2}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  mirror      ONLINE     0     0     0"
+  echo "	    ${disk0}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+disks_destroy
diff --git a/tools/regression/zfs/zpool/replace/raidz1.t b/tools/regression/zfs/zpool/replace/raidz1.t
new file mode 100644
index 000000000000..79bcf8e3060c
--- /dev/null
+++ b/tools/regression/zfs/zpool/replace/raidz1.t
@@ -0,0 +1,140 @@
+#!/bin/sh
+# $FreeBSD$
+
+dir=`dirname $0`
+. ${dir}/../../misc.sh
+
+echo "1..27"
+
+disks_create 4
+names_create 1
+
+expect_ok ${ZPOOL} create ${name0} raidz1 ${disk0} ${disk1} ${disk2}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+disk_create 0 ${dname0}
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  raidz1      DEGRADED     0     0     0"
+  echo "	    ${guid0}  UNAVAIL      0     0     0 was ${fdname0}"
+  echo "	    ${disk1}  ONLINE       0     0     0"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz1      ONLINE     0     0     0"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk1}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} raidz1 ${disk0} ${disk1} ${disk2}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz1      ONLINE     0     0     0"
+  echo "	    ${disk0}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} raidz1 ${disk0} ${disk1} ${disk2}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  raidz1       DEGRADED     0     0     0"
+  echo "	    ${guid0}   REMOVED      0     0     0  was ${fdname0}"
+  echo "	    ${disk1}   ONLINE       0     0     0"
+  echo "	    ${disk2}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${dname0} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz1      ONLINE     0     0     0"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk1}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 0 ${dname0}
+
+expect_ok ${ZPOOL} create ${name0} raidz1 ${disk0} ${disk1} ${disk2}
+expect_ok ${ZPOOL} offline ${name0} ${disk1}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk3}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz1      ONLINE     0     0     0"
+  echo "	    ${disk0}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+disks_destroy
diff --git a/tools/regression/zfs/zpool/replace/raidz2.t b/tools/regression/zfs/zpool/replace/raidz2.t
new file mode 100644
index 000000000000..d961e3c32ff3
--- /dev/null
+++ b/tools/regression/zfs/zpool/replace/raidz2.t
@@ -0,0 +1,607 @@
+#!/bin/sh
+# $FreeBSD$
+
+dir=`dirname $0`
+. ${dir}/../../misc.sh
+
+echo "1..115"
+
+disks_create 6
+names_create 1
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+disk_create 0 ${dname0}
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  raidz2      DEGRADED     0     0     0"
+  echo "	    ${guid0}  UNAVAIL      0     0     0  was ${fdname0}"
+  echo "	    ${disk1}  ONLINE       0     0     0"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "	    ${disk3}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk1}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk4}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk0}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  raidz2       DEGRADED     0     0     0"
+  echo "	    ${guid0}   REMOVED      0     0     0  was ${fdname0}"
+  echo "	    ${disk1}   ONLINE       0     0     0"
+  echo "	    ${disk2}   ONLINE       0     0     0"
+  echo "	    ${disk3}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${dname0} ${disk4}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk1}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 0 ${dname0}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} offline ${name0} ${disk1}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk4}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk0}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${zpool_f_flag} ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+disk_create 0 ${dname0}
+dname1=${disk1}
+fdname1=${fdisk1}
+guid1=`get_guid ${fdisk1}`
+disk_destroy 1
+disk_create 1 ${dname1}
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  raidz2      DEGRADED     0     0     0"
+  echo "	    ${guid0}  UNAVAIL      0     0     0  was ${fdname0}"
+  echo "	    ${guid1}  UNAVAIL      0     0     0  was ${fdname1}"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "	    ${disk3}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${disk2} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk0}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+dname1=${disk1}
+fdname1=${fdisk1}
+guid1=`get_guid ${fdisk1}`
+disk_destroy 1
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  raidz2       DEGRADED     0     0     0"
+  echo "	    ${guid0}   REMOVED      0     0     0  was ${fdname0}"
+  echo "	    ${guid1}   REMOVED      0     0     0  was ${fdname1}"
+  echo "	    ${disk2}   ONLINE       0     0     0"
+  echo "	    ${disk3}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${dname0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${dname1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 0 ${dname0}
+disk_create 1 ${dname1}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} offline ${name0} ${disk0}
+add_msg="# TODO Sun CR 6328632, Lustre bug 16878"
+expect_ok ${ZPOOL} offline ${name0} ${disk1}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices has been taken offline by the administrator."
+  echo "      Sufficient replicas exist for the pool to continue functioning in a"
+  echo "      degraded state."
+  echo "action: Online the device using 'zpool online' or replace the device with"
+  echo "      'zpool replace'."
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  raidz2      DEGRADED     0     0     0"
+  echo "	    ${disk0}  OFFLINE      0     0     0"
+  echo "	    ${disk1}  OFFLINE      0     0     0"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "	    ${disk3}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+add_msg=""
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${zpool_f_flag} ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+disk_create 0 ${dname0}
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  raidz2      DEGRADED     0     0     0"
+  echo "	    ${guid0}  UNAVAIL      0     0     0  was ${fdname0}"
+  echo "	    ${disk1}  ONLINE       0     0     0"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "	    ${disk3}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+disk_create 0 ${dname0}
+dname1=${disk1}
+fdname1=${fdisk1}
+guid1=`get_guid ${fdisk1}`
+disk_destroy 1
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  raidz2       DEGRADED     0     0     0"
+  echo "	    ${guid0}   UNAVAIL      0     0     0  was ${fdname0}"
+  echo "	    ${guid1}   REMOVED      0     0     0  was ${fdname1}"
+  echo "	    ${disk2}   ONLINE       0     0     0"
+  echo "	    ${disk3}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${dname1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 1 ${dname1}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} export ${name0}
+dname0=${disk0}
+fdname0=${fdisk0}
+guid0=`get_guid ${fdisk0}`
+disk_destroy 0
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  raidz2       DEGRADED     0     0     0"
+  echo "	    ${guid0}   REMOVED      0     0     0  was ${fdname0}"
+  echo "	    ${disk1}   ONLINE       0     0     0"
+  echo "	    ${disk2}   ONLINE       0     0     0"
+  echo "	    ${disk3}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${dname0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 0 ${dname0}
+
+expect_ok ${ZPOOL} create ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} offline ${name0} ${disk0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices has been taken offline by the administrator."
+  echo "      Sufficient replicas exist for the pool to continue functioning in a"
+  echo "      degraded state."
+  echo "action: Online the device using 'zpool online' or replace the device with"
+  echo "      'zpool replace'."
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  raidz2      DEGRADED     0     0     0"
+  echo "	    ${disk0}  OFFLINE      0     0     0"
+  echo "	    ${disk1}  ONLINE       0     0     0"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "	    ${disk3}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${zpool_f_flag} ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} offline ${name0} ${disk0}
+expect_ok ${ZPOOL} export ${name0}
+dname1=${disk1}
+fdname1=${fdisk1}
+guid1=`get_guid ${fdisk1}`
+disk_destroy 1
+disk_create 1 ${dname1}
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices could not be used because the label is missing or"
+  echo "      invalid.  Sufficient replicas exist for the pool to continue"
+  echo "      functioning in a degraded state."
+  echo "action: Replace the device using 'zpool replace'."
+  echo "   see: http://www.sun.com/msg/ZFS-8000-4J"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE     READ WRITE CKSUM"
+  echo "	${name0}      DEGRADED     0     0     0"
+  echo "	  raidz2      DEGRADED     0     0     0"
+  echo "	    ${disk0}  OFFLINE      0     0     0"
+  echo "	    ${guid1}  UNAVAIL      0     0     0  was ${fdname1}"
+  echo "	    ${disk2}  ONLINE       0     0     0"
+  echo "	    ${disk3}  ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${disk1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+expect_ok ${ZPOOL} create ${zpool_f_flag} ${name0} raidz2 ${disk0} ${disk1} ${disk2} ${disk3}
+expect_ok ${ZPOOL} offline ${name0} ${disk0}
+expect_ok ${ZPOOL} export ${name0}
+dname1=${disk1}
+fdname1=${fdisk1}
+guid1=`get_guid ${fdisk1}`
+disk_destroy 1
+expect_ok ${ZPOOL} import ${import_flags} ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: DEGRADED"
+  echo "status: One or more devices has been taken offline by the administrator."
+  echo "      Sufficient replicas exist for the pool to continue functioning in a"
+  echo "      degraded state."
+  echo "action: Online the device using 'zpool online' or replace the device with"
+  echo "      'zpool replace'."
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME           STATE     READ WRITE CKSUM"
+  echo "	${name0}       DEGRADED     0     0     0"
+  echo "	  raidz2       DEGRADED     0     0     0"
+  echo "	    ${disk0}   OFFLINE      0     0     0"
+  echo "	    ${guid1}   REMOVED      0     0     0  was ${fdname1}"
+  echo "	    ${disk2}   ONLINE       0     0     0"
+  echo "	    ${disk3}   ONLINE       0     0     0"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} replace ${name0} ${disk0} ${disk4}
+expect_ok ${ZPOOL} replace ${name0} ${dname1} ${disk5}
+wait_for_resilver ${name0}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: (scrub|resilver) completed after [0-9]+h[0-9]+m with 0 errors on .*"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  raidz2      ONLINE     0     0     0"
+  echo "	    ${disk4}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk5}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk2}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "	    ${disk3}  ONLINE     0     0     0(  [0-9.]+[A-Z] resilvered)?"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+disk_create 1 ${dname1}
+
+disks_destroy
diff --git a/tools/regression/zfs/zpool/replace/spare.t b/tools/regression/zfs/zpool/replace/spare.t
new file mode 100644
index 000000000000..5899305c6a30
--- /dev/null
+++ b/tools/regression/zfs/zpool/replace/spare.t
@@ -0,0 +1,43 @@
+#!/bin/sh
+# $FreeBSD$
+
+dir=`dirname $0`
+. ${dir}/../../misc.sh
+
+echo "1..6"
+
+disks_create 4
+names_create 1
+
+expect_ok ${ZPOOL} create ${name0} ${disk0} spare ${disk1}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	spares"
+  echo "	    ${disk1}  AVAIL"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_fl ${ZPOOL} replace ${name0} ${disk1} ${disk2}
+exp=`(
+  echo "  pool: ${name0}"
+  echo " state: ONLINE"
+  echo " scrub: none requested"
+  echo "config:"
+  echo "	NAME          STATE   READ WRITE CKSUM"
+  echo "	${name0}      ONLINE     0     0     0"
+  echo "	  ${disk0}    ONLINE     0     0     0"
+  echo "	spares"
+  echo "	    ${disk1}  AVAIL"
+  echo "errors: No known data errors"
+)`
+expect "${exp}" ${ZPOOL} status ${name0}
+expect_ok ${ZPOOL} destroy ${name0}
+expect_fl ${ZPOOL} status -x ${name0}
+
+disks_destroy