- View job stats

# View job stats

Contents

Job statistics are available from two locations:

• | Clicking the Jobstats button on the top menu bar lists the top ten jobs currently in process. The listed jobs can be sorted by column and average duration can be selected. Column sorts and duration will be persistent when navigating away and back to the page.

•

Clicking on an OST cell on the Read/Write Heat Map chart. The job statistics feature displays the read and write throughput for the top ten jobs for that OST and selected time interval. The window also shows the top Read IOPS and Write IOPS for that OST and time interval. This feature supports the creation of plug-ins to display user account, command line, job size, and job start/finish times. Information is updated every ten seconds.

## To view job statistics

1. | Before viewing job statistics, you will need to run a command to enable this feature. Run this command for each file system. The following command is an example, to run on the management server (MGS): lctl conf_param <test1>.sys.jobid_var=procname_uid where <test1> is the file system name (refer to Using jobstats with other job schedulers for more information.

2.

The variable testfs.mdt.job_cleanup_interval sets the period after which collected statistics are cleared out. If this interval is to short, statistics may get cleared while you're viewing job statistics. Set this interval to a value greater than your collection/viewing period. As an example, you could set this interval to 70 minutes (4200 seconds) using the following command:
lctl conf_param testfs.mdt.job_cleanup_interval=4200

3. | View the Read/Write Heat Map chart on the dashboard window.

4.

Each row on the Read/Write Heat Map corresponds to an OST, with consecutive columns from left-to-right, corresponding to consecutive time intervals. Mouse over a cell to find an OST and time interval of interest, and click on the desired cell.
The Jobs Stats window opens. The top banner reveals the OST and time interval. Each job executing during that interval is displayed as a row, with its average data throughput revealed for that interval. Only the top five read and write jobs are displayed. The window displays the Read Bytes, Write Bytes, Read OPS, and Write IOPS for the top five jobs, listed by Job ID.

5.

To change the duration of the job statistics sampling period, return to the Read/Write Heat Map chart. Click Change Duration and set the time period for the heat map. If you set the time period to one day (as an example), the 24-hour period will be divided into 20 equal, consecutive cells, starting 24 hours previous and ending now. Each Read/Write Heat Map cell now covers 1.2 hours. Clicking on a cell now will reveal a job statistics window that averages 1.2 hours of read/write operations.

6. | To send this Job Stats window to another person, select and copy the URL from browser URL field. Then paste the URL into an email message body and send.

Note: The Job Stats window is static, specific to that time period and OST. To view another time period or OST, return to the Read/Write Heat Map chart and select the desired cell.

## Using job stats with other job schedulers

The job stats code extracts the job identifier from an environment variable set by the scheduler when the job is started. Intel® EE for Lustre* software sets a jobstats environment variable to work with SLURM, however you can set the variable to work with other job schedulers. To enable job stats to work with a desired scheduler, specify the jobid_var to name the environment variable set by the scheduler. For example, SLURM sets the SLURM_JOB_ID environment variable with the unique job ID on each client. To permanently enable jobstats on the testfs file system, run this command on the MGS:

$ lctl conf_param testfs.sys.jobid_var=<environment variable>

- where <environment variable> is one of the following:

Job Scheduler | environment variable
Simple Linux Utility for Resource Management (SLURM) | SLURM_JOB_ID
Sun Grid Engine (SGE) | JOB_ID
Load Sharing Facility (LSF) | LSB_JOBID
Loadleveler | LOADL_JOBID
Portable Batch Scheduler (PBS)/MAUI | PBS_JOBID
Cray Application Level Placement Scheduler (ALPS) | ALPS_APP_ID

To disable job stats, specify jobid_var as disable:

$ lctl conf_param testfs.sys.jobid_var=disable

To track job stats per process name and user ID (for debugging, or if no job scheduler is in use), specify jobid_var as procname_uid:

$ lctl conf_param testfs.sys.jobid_var=procname_uid