Details

    • New Feature
    • Resolution: Fixed
    • Minor
    • Lustre 2.16.0
    • None
    • None
    • 9223372036854775807

    Description

      In DDN-3356, by Andreas Dilger:

      We don't have a tool to do this today, but it would make sense to write a simple tool "lljobstat" to show the top jobs on a server in order to simplify debugging of high load problems, since this is a reasonably frequent request.

      It should be included with the base Lustre RPMs, so it must not have any complex external dependencies that are not included in the base OS distro (el7, el8, sles15, ubuntu22).

      It should read all of the local "..job_stats" files (by default, or --ost or --mdt, or a specific jobstats file if given as an argument) every 10s interval (configurable, either "-i N" or last argument) and prints the top e.g. 5 jobs (configurable "-c N"), one line per job similar to "iostat -x -k -z 10". It should show something useful when run with minimal arguments (eg. just the interval), so that users can use it to easily determine which jobs are driving the most load.

      Since the job_stats has a large number of stats, it is not possible to fit all of them in a single 80-column line, so any operations that have samples = 0 should not be shown. Priority for display should be to show read, write (counts, if non-zero), read_bytes, write_bytes (in MiB/s units, if non-zero), then the top metadata ops by count. It probably makes sense to use abbreviations for the names, like llobdstat so that more can fit onto the line (cx: create, dx: destroy, st: statfs, pu: punch, etc). In the newer llstat and llobdstst it checks if the terminal width is over 80 and shows more fields, but this doesn't have to be in the first version.

      To determine the "top" jobs, it probably makes sense to sum the operations for the same job name across all watched job_stats files, then sort by total count of operations (read+write, but not bytes) and include this as the second item shown ("ops: N") after the job name ("job: name", with escaping/quoting if needed). The timestamp should be shown for each interval.

      Given that the input is YAML, the output could also be YAML, but only if it can be formatted nicely for human readability (one line per job, no excessive quoting). The main users of this will be people, since monitoring tools will likely read and process all of the job_stats output directly.

      Attachments

        1. glljobstat
          16 kB
        2. lljobstat
          8 kB

        Issue Links

          Activity

            [LU-16228] create lljobstats command
            flei Feng Lei added a comment -

            is there a way to "try" loading the libyaml-dev CLoader, but fall back to the regular Loader if it is not installed?

            It can be checked at runtime:

            if hasattr(yaml, "CLoader"):
                yaml_obj = yaml.load(output, Loader=yaml.CLoader)
            else:
                yaml_obj = yaml.safe_load(output)
            
            flei Feng Lei added a comment - is there a way to "try" loading the libyaml-dev CLoader, but fall back to the regular Loader if it is not installed? It can be checked at runtime: if hasattr (yaml, "CLoader" ): yaml_obj = yaml.load(output, Loader=yaml.CLoader) else : yaml_obj = yaml.safe_load(output)

            Here the lines you would need to change:

             

             

            #!/bin/env python3
            '''
            lljobstat command. Read job_stats files, parse and aggregate data of every
            job on multiple OSS/MDS, show top jobs
            '''
            import argparse
            import errno
            import subprocess
            import sys
            import time
            import yaml
            import signal
            import urllib3
            import warnings
            import configparser
            from multiprocessing import Process, Queue, Pool, Manager, active_children, Pipe
            from subprocess import Popen, PIPE, STDOUT
            from pprint import pprint
            from os.path import expanduser
            from pathlib import Path
            try:
                from yaml import CLoader as Loader, CDumper as Dumper
            except ImportError:
                pass
            warnings.filterwarnings(action='ignore',module='.*paramiko.*')
            urllib3.disable_warnings()
            [...]
            
            bolausson Bjoern Olausson added a comment - Here the lines you would need to change:     #!/bin/env python3 ''' lljobstat command. Read job_stats files, parse and aggregate data of every job on multiple OSS/MDS, show top jobs ''' import argparse import errno import subprocess import sys import time import yaml import signal import urllib3 import warnings import configparser from multiprocessing import Process , Queue, Pool, Manager, active_children, Pipe from subprocess import Popen, PIPE, STDOUT from pprint import pprint from os.path import expanduser from pathlib import Path try :     from yaml import CLoader as Loader, CDumper as Dumper except ImportError:     pass warnings.filterwarnings(action= 'ignore' ,module= '.*paramiko.*' ) urllib3.disable_warnings() [...]
            bolausson Bjoern Olausson added a comment - - edited

            Yes this is possible with a try - except construct.

            The CLoader worked perfeclty fine on default EXAScaler 5.2.7 install.
            It was not neccessary to install any aditional packages except:

            python3 -m venv lljobstat
            . ./lljobstat/bin/activate
            python3 -m pip install pyyaml
            python3 -m pip install paramiko
            python3 -m pip install urllib3 

            By the way, I added my enhneced version to the DDNeu GitHub repo:
            https://github.com/DDNeu/global-lustre-jobstats

            Cheers,
            Bjoern

            bolausson Bjoern Olausson added a comment - - edited Yes this is possible with a try - except construct. The CLoader worked perfeclty fine on default EXAScaler 5.2.7 install. It was not neccessary to install any aditional packages except: python3 -m venv lljobstat . ./lljobstat/bin/activate python3 -m pip install pyyaml python3 -m pip install paramiko python3 -m pip install urllib3 By the way, I added my enhneced version to the DDNeu GitHub repo: https://github.com/DDNeu/global-lustre-jobstats Cheers, Bjoern

            Bjoern, is there a way to "try" loading the libyaml-dev CLoader, but fall back to the regular Loader if it is not installed?

            adilger Andreas Dilger added a comment - Bjoern, is there a way to "try" loading the libyaml-dev CLoader, but fall back to the regular Loader if it is not installed?

            See link for solution:

            https://github.com/yaml/pyyaml/issues/108#issuecomment-370459912

            Essentially libyaml-dev is missing on your system. It is required for the CLoader (which replaces the slow Python loader)

            Greetings, Bjoern

            bolausson Bjoern Olausson added a comment - See link for solution: https://github.com/yaml/pyyaml/issues/108#issuecomment-370459912 Essentially libyaml-dev is missing on your system. It is required for the CLoader (which replaces the slow Python loader) Greetings, Bjoern

            bolausson, I pushed the "simple" version of your patch but it is reporting an error:

            This is causing test failures:
            
             lljobstat -n 1 -i 0 -c 1000
              Traceback (most recent call last):
                File "/usr/bin/lljobstat", line 15, in 
                   from yaml import CLoader as Loader, CDumper as Dumper
              ImportError: cannot import name 'CLoader'
            
            adilger Andreas Dilger added a comment - bolausson , I pushed the "simple" version of your patch but it is reporting an error: This is causing test failures: lljobstat -n 1 -i 0 -c 1000 Traceback (most recent call last): File "/usr/bin/lljobstat", line 15, in from yaml import CLoader as Loader, CDumper as Dumper ImportError: cannot import name 'CLoader'
            pjones Peter Jones added a comment -

            Landed for 2.16

            pjones Peter Jones added a comment - Landed for 2.16

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48888/
            Subject: LU-16228 utils: add lljobstat util
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: e2812e877314bc101efdc5a235c7fae8f7424f96

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48888/ Subject: LU-16228 utils: add lljobstat util Project: fs/lustre-release Branch: master Current Patch Set: Commit: e2812e877314bc101efdc5a235c7fae8f7424f96

            It looks like the newly-added sanity.sh test_205e needs to add a version check for interop testing:

            trevis-82vm3: sh: lljobstat: command not found
            

            There is a version check in test_205d already.

            adilger Andreas Dilger added a comment - It looks like the newly-added sanity.sh test_205e needs to add a version check for interop testing: trevis-82vm3: sh: lljobstat: command not found There is a version check in test_205d already.

            "Feng Lei <flei@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48888
            Subject: LU-16228 utils: add lljobstat util
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 777dd8757b0a121daf22f275f7eeb5a2b00ea62f

            gerrit Gerrit Updater added a comment - "Feng Lei <flei@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48888 Subject: LU-16228 utils: add lljobstat util Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 777dd8757b0a121daf22f275f7eeb5a2b00ea62f

            Feng Lei, this looks mostly good. I would say that the comment is large enough that it shouldn't be printed each time, maybe just document the abbreviations in the man page or if "-h" is used. I would suggest "ln" for link (to match the command name).

            The "top_jobs:" should have an underscore so it is a single word, even though I know YAML does not require this, since it makes parsing easier with scripts (eg. "awk '/keyname:/ { print $2 }'".

            adilger Andreas Dilger added a comment - Feng Lei, this looks mostly good. I would say that the comment is large enough that it shouldn't be printed each time, maybe just document the abbreviations in the man page or if " -h " is used. I would suggest " ln " for link (to match the command name). The " top_jobs: " should have an underscore so it is a single word, even though I know YAML does not require this, since it makes parsing easier with scripts (eg. " awk '/keyname:/ { print $2 }' ".

            People

              flei Feng Lei
              flei Feng Lei
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: