[WIP] Raidz Expansion: multiple devices expanding

[fuporovvStack]

Motivation and Context

Allow to attach multiple raidz children at once, by applying single reflow process for all newly attached devices in parallel.

This PR allows to attach multiple devices to raidz in same time in parallel. The zpool command is executed by the next way:

$ zpool attach poolname raidz1-0 /path/to/dev0 /path/to/dev1 ... /path/to/devX

Description

The multiple devices raidz expnding will be completed after single reflow process iteration. The number of attached devices should be less, then raidz have initially. The older versions of ZFS with raidz expansion support will not be able to work with raidz expanded with the multiple raidz attached devices.

The multiple devices raidz expansion become possible by adding new vdev label config variable ZPOOL_CONFIG_RAIDZ_EXPAND_VDEVS additionally to ZPOOL_CONFIG_RAIDZ_EXPAND_TXGS. This variable is an array of values of devices attached per raidz expansion iteration. Updated on every reflow process completion, same as ZPOOL_CONFIG_RAIDZ_EXPAND_TXGS array. Both ZPOOL_CONFIG_RAIDZ_EXPAND_TXGS and ZPOOL_CONFIG_RAIDZ_EXPAND_VDEVS should have the same size and used to build raidz txg avl tree to compute raidz width dependent of bp birth txg value.

I did not found any reasons, why it would be impossible from raidz math point of view.
If you have any concerns or suggestions regarding raidz math and multiple devices raidz expanding process, please let me know.

Notes:

• It possible to try multiple devices raidz expansion using zhack utility, zhack was modified appropriately.
• Single device raidz expansion considered as "worked" with modified raidz logic. Passed some number of ztest iterations.

TODO:

• The appropriate ZFS feature flag need to be added.
• userspace:
  • zpool/cmd:
    • zpool attach should support multiple device paths as arguments - clean reimplementation is required
    • validate same device is not attached multiple times
    • zpool attach man page should be updated
  • ztest:
    • ztest issues need to be reproduced clearly and fixed for multiple devices raidz attaching
    • ztest logic should be refactored and retested
  • zfs-tests.sh:
    • additional tests of raidz_test tool should be added
    • multiple devices raidz attach test cases needed to be added
• kernel:
  • vdev_raidz core logic refactoring is required
  • ensure aggregation logic is compatible with multidev expansion
  • not sure that raidz_reconstruct() should not to be modified

How Has This Been Tested?

No zfs-tests.sh tests scripts for now.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.