- Create a ZFS-based Lustre file system

# Create a ZFS-based Lustre file system

Contents

The procedures in this section assume that you have first assembled and configured the physical hardware: servers, storage devices, network interfaces, etc., and installed Intel® Manager for Lustre* software as instructed in the Intel® Enterprise Edition for Lustre* Software Installation Guide.

To create and manage an OpenZFS-based Lustre file system that is highly-available and managed by Intel® Manager for Lustre* software, perform these steps:

1. | Add all of the physical servers that will comprise your ZFS-based Lustre file system. To do this, perform the steps in Add one or more HA servers. Add each server as a Managed Storage Server.

Note: Steps 2 and 3 below are performed automatically by Intel® Manager for Lustre* software and do not need to be performed. They are included here for topic coverage only. Continue with Step 4.

2.

Having added the servers, you now need to ensure that the server hostids are set before creating any zpools. Each server requires a unique hostid to be configured. Setting the hostid on each server will allow ZFS to apply an attribute on each zpool indicating which host currently has the pool imported. This provides some protection in ZFS against multiple simultaneous imports of zpools when the storage is connected to more than one server. To set the hostid, run the genhostid command on each host and reboot. In the document Lustre* Installation and Configuration using Intel® EE for Lustre* Software and OpenZFS, see the section Protecting File System Volumes from Concurrent Access for more information.

3. | As a further protection against “double-importing” ZFS pools, and to prevent conflicts with the Pacemaker resource management software, disable the ZFS Systemd target by entering the following command:

systemctl disable zfs.target

This will stop the operating system from attempting to auto-import ZFS storage pools during system boot. Disabling the ZFS target affect all ZFS storage pools, including any that are not being used for Lustre.

4.

ZFS pool configuration is executed directly on each of the Lustre server HA pairs using the command line tools supplied with the ZFS software. The storage devices for each ZFS pool must be connected to, and accessible from, each Lustre server in the HA pair. For each ZFS pool, it is easiest to nominate a primary server and use that host to create the zpool. Log into each primary host and use this zpool command to create each storage pool. The general syntax for creating a zpool is as follows:

zpool create [-f] -O canmount=off \

-O mountpoint=none \

[ -o <option> ] \

-o cachefile=none \

-o failmode=panic \

<zpool name> <zpool specification>

ZFS is highly configurable and there are a wide range of optional parameters that can be applied to a ZFS pool, to improve performance or to modify functionality. The most important of these are as follows:

•

cachefile=none: ZFS uses the cachefile to identify pools it can automatically import, but this does not make consideration for shared storage cluster environments. To prevent a ZFS pool from being added to the default cachefile and automatically imported at system boot, it is essential that all ZFS pools that are held on shared storage have the cachefile set to the special value “none”. This, in conjunction with setting the hostid provides a lightweight impediment against double-importing the pool.

•

failmode=panic: zpools should have the "failmode" parameter set to "panic". This will cause a host that has lost connectivity to the underlying storage devices, or has experienced too many device failures, to shut down immediately. The node is thus prevented from issuing undesired writes to a zpool after catastrophic failures, including host disconnection from the zpool. When ZFS pools are operating within an HA cluster framework, panicking a node will also trigger a failover event in the cluster, allowing the zpools to be imported to a standby node, and services to continue operation.

• | ashift: used to override ZFS auto-detection of the physical vdev storage sector size. Advanced format drives are known to cause issues by incorrectly reporting the native sector size. Setting the ashift property forces ZFS to use a specific sector size, and can yield improved performance. The value of ashift is an exponent to power of 2. Typically used to set 4KB sector size, (ashift=12).

In addition to zpool properties, there are also properties that can be applied to the ZFS datasets in a pool. ZFS dataset properties are set in the zpool command using a capital letter -O, rather than the lower case letter -o flag used for the cachefile and ashift settings. The -O flag is used to set properties to the root file system data set in the pool, rather than for properties of the pool itself. Two ZFS dataset properties of particular importance are highlighted here:

• | canmount=off: prevent the dataset from being mounted using the standard zfs mount command. Essential for Lustre OSDs, which must be mounted and unmounted using the mount.lustre (or mount -t lustre) command.

• | recordsize: this sets a suggested block size for the file system. The block size cannot exceed this value. The default value, 128KB, is fine for MGT and MDT OSDs. For OSTs, where large block IO is the dominant workload, set the recordsize=1M, which is the maximum currently allowed in ZFS on Linux version 0.6.5. Future versions of ZFS are expected to be able to further increase this value.

• | mountpoint=none: in monitored mode, the management software requires that ZfsDatasets have the following property values to prevent undesired automatic mounting.

For more details, refer to the section ZFS OSDs, in the guide titled Lustre* Installation and Configuration using Intel® EE for Lustre* Software and OpenZFS, as well as the system man pages for zfs(8) and zpool(8).

Following are three examples of typical pool configurations, one each for MGT, MDT and OST respectively:

• | ZFS pool for MGT (typically a simple mirror):

zpool create -O canmount=off \

-O mountpoint=none \

-o cachefile=none \

-o failmode=panic \

mgspool mirror <dev A> <dev B>

• | ZFS pool for MDT (typically a stripe of mirrors to maximize performance):

zpool create -O canmount=off \

-O mountpoint=none \

-o cachefile=none \

-o failmode=panic \

<fsname>-mdt<n>pool \

mirror <dev A> <dev B> [ mirror <dev C> <dev D> ] ...

Note: The naming convention for the pool name is intended to reflect the Lustre file system name name (<fsname>) and the MDT index number (<n> usually 0, unless DNE is used). The number of mirrors used for the MDT depends on the requirements of the installation and can be scaled up accordingly.

• | ZFS pool for OST (typically a RAIDZ2 pool, balancing reliability against optimal capacity):

zpool create -O canmount=off \

-O recordsize=1M \

-O mountpoint=none \

-o failmode=panic \

-o cachefile=none \

[ -o ashift=12 ] \

<fsname>-ost<n>pool \

raidz2 <dev A> <dev B> \

<dev C> <dev D> <dev E> <dev F> ...

Note: See the document Lustre* Installation and Configuration using Intel® EE for Lustre* Software and OpenZFS for descriptions of the ashift and recordsize properties. RAIDZ2 is the preferred vdev configuration for OSTs, and we recommend an arrangement of at least 11 disks (9+2) per RAIDZ2 vdev for best performance. The pool naming convention is based on the Lustre file system name and OST index number, starting at 0 (zero).

The remainder of this procedure is performed at the Intel® Manager for Lustre* software GUI.

5. | For high-availability, configure your servers as primary and fail-over servers for each zpool. Perform the steps in Configure primary and fail-over servers.

6. | If you are using power distribution units (PDUs) for power control, then for each server, perform the steps in Add power distribution units. Then perform the steps in Assign PDU outlets to servers for each server.

7. | If you are using Baseboard Management Controllers (BMCs) for power control, then perform the steps in Assign BMCs to servers for each server.

8. | Perform the steps in Create the new Lustre file system, using the zpools you created in step 4 above as object storage targets (volumes), rather than direct block devices. Each ZFS pool that you created will appear as a target, with the Type identified as a ZfsPool.