Implement zero-copy iSCSI target transmission/read.

Add ICL_NOCOPY flag to icl_pdu_append_data(), specifying that the method
can just reference the data buffer instead of immediately copying it.

Extend the offload KPI with optional PDU queue method, allowing to specify
completion callback, called when all the data referenced by above has been
transferred and won't be accessed any more (the buffers can be freed).

Implement the above functionality in software iSCSI driver using mbufs
with external storage and reference counter.  Note that some NICs (ixl(4))
may keep the mbuf in TX queue for a long time, so CTL has to be ready.

Add optional method to struct ctl_scsiio for buffer reference counting.
Implement it for CTL block backend, allowing to delay free of the struct
ctl_be_block_io and memory it references as needed.  In first reincarnation
of the patch I tried to delay whole I/O as it is done for FibreChannel,
that was cleaner, but due to the above callback delays I had to rewrite
it this way to not leave LUN referenced potentially for hours or more.

All together on sequential read from ZFS ARC this saves about 30% of CPU
time and memory bandwidth by avoiding one of 3 memory copies (the other
two are from ZFS ARC to DMU cache and then from DMU cache to CTL buffers).
On tests with 2x Xeon Silver 4114 this allows to reach full line rate of
100GigE NIC.  Tests with Gold CPUs and two 100GigE NICs are stil TBD,
but expectations to saturate them are pretty high. ;)

Discussed with:	Chelsio
Sponsored by:	iXsystems, Inc.

1 files changed, 6 insertions, 3 deletions

diff --git a/sys/dev/iscsi/icl.h b/sys/dev/iscsi/icl.h
index 9105f0d68a35..8df7a72e0b26 100644
--- a/sys/dev/iscsi/icl.h
+++ b/sys/dev/iscsi/icl.h
@@ -79,9 +79,8 @@ struct icl_pdu {
 	/*
 	 * User (initiator or provider) private fields.
 	 */
-	uint32_t		ip_prv0;
-	uint32_t		ip_prv1;
-	uint32_t		ip_prv2;
+	void			*ip_prv0;
+	void			*ip_prv1;
 };
 
 #define ICL_CONN_STATE_INVALID		0
@@ -93,6 +92,8 @@ struct icl_pdu {
 
 #define	ICL_MAX_DATA_SEGMENT_LENGTH	(128 * 1024)
 
+#define	ICL_NOCOPY			(1 << 30)
+
 struct icl_conn {
 	KOBJ_FIELDS;
 	struct mtx		*ic_lock;
@@ -136,6 +137,8 @@ struct icl_drv_limits {
 	int spare[4];
 };
 
+typedef void (*icl_pdu_cb)(struct icl_pdu *, int error);
+
 struct icl_conn	*icl_new_conn(const char *offload, bool iser, const char *name,
 		    struct mtx *lock);
 int		icl_limits(const char *offload, bool iser,