- Unexpected file system events

# Unexpected file system events

Contents

This section discusses several unwanted file system events and how Intel® Manager for Lustre* software responds to them.

A server’s connection to a storage target is lost
Immediate file system consequences: | Lustre clients will block if they have requested a file from an unavailable OST. The block will continue until connection to the OST is restored and the OST is again fully online. For OSTs that are still connected to their servers, client access continues unaffected.
Manager software / Peer server response: | No automatic failover. No alerts.
Suggested remedies: | Repair the connection to the target. In the meantime, the superuser may manually fail the target over to the peer server.
A server’s connection to LNet is lost
Immediate file system consequences: | Lustre clients will block waiting for the connection to be re-established. Those portions of the file system that are presented by the affected server are unavailable until then.
Manager software / Peer server response: | No automatic failover. No alerts.
Suggested remedies: | Repair the server’s connection to LNet. In the meantime, the superuser may manually fail the target over to the peer server.
Manager software connection to a server (via the management network, ring0) is lost
Immediate file system consequences: | No direct file system impact; the file system remains operational. However, Intel® Manger for Lustre* software can no longer manage or monitor the server.
Manager software / Peer server response: | Alerts to administer regarding loss of network connection to server.
Suggested remedies: | Re-establish the management network connection to the server.
A Lustre server loses connectivity with the power control device for its peer server (IPMI or PDU)
Immediate file system consequences: | None. The file system continues to operate normally. In the event of a peer server failure, the server that has lost connectivity to power control will be unable to power off the failed server and assume responsibility for its resources.
Manager software / Peer response: | No response to the loss of connectivity if the file system is operating normally. In the event of a server failure, automatic failover of Lustre targets from the failed server may be disabled.
Suggested remedies: | Repair the network link to power control (IPMI or PDU).
The Intel® Manager for Lustre* loses connection with a server’s power control device (IPMI or PDU)
Immediate file system consequences: | The software's ability to shut down the server is lost.
Manager software/peer server response: | Alerts to administer regarding loss of connection to power control device.
Suggested remedies: | Restore the connection between the Manager software server and affected server’s power control device.
A crossover cable between servers is disconnected or the network is down
Immediate file system consequences: | This is the loss of the ring1 network link, but the ring0 link (the management network) provides complete redundancy. The file system is not affected.
Manager software / Peer server response: | No automatic failover. No alerts.
Suggested remedies: | Replace/reconnect the cross-over cable, restore the network.
A primary server’s OS kernel crashes
Immediate file system consequences: | Each server is used as both a primary and secondary server. Temporarily delayed access to served storage as failover occurs.
Manager software / Peer server response: | Peer server performs STONITH, failover occurs.
Suggested remedies: | None needed by Admin. Successful STONITH causes the server to be rebooted.
LBUG, a Lustre crash on a server
Immediate file system consequences: | This will also crash Linux on the affected server. Temporarily delayed access to served storage as failover occurs.
Manager software / Peer server response: | Peer server performs STONITH, failover occurs.
Suggested remedies: | No Admin action needed.
The primary server spontaneously reboots
Immediate file system consequences: | Temporarily delayed access to served storage as failover occurs.
Manager software / Peer server response: | Peer server performs STONITH, failover occurs.
Suggested remedies: | No Admin action needed.
The management network (ring0) and a peer crossover network (ring1) are both down
Immediate file system consequences: | The file system is not directly affected and client operations may continue. Affected peer servers may attempt STONITH.
Manager software / Peer server response: | Peer server performs STONITH and failover occurs. However, each affected server may attempt STONITH on its peer.
Suggested remedies: | This condition is unlikely and unstable. The superuser needs to restore network connections for the management network and the cross-over link between affected servers.