A video card is a special circuit board that controls what is displayed on a computer monitor. It is also called a graphics processing unit (GPU), which calculates 3D images and graphics for Linux gaming and other usages. Let us see the top 7 Linux GPU monitoring and diagnostic command-line tools to solve issues.
The following tools work on Linux for GPU monitoring and diagnostic purposes and other operating systems such as FreeBSD. The majority of Linux and FreeBSD users these days use Nvidia, Intel, and AMD GPUs.
Linux GPU Monitoring and Diagnostic Commands Line Tools
We can use the following tools to monitor, diagnostic, and inspect our Linux or *BSD based systems.
Finding information about GPU on Linux
To get the GPU info simply run:
sudo lshw -C display -short
lspci -v | more
Which is output something as follows:
H/W path Device Class Description =============================================================== /0/100/1/0 display TU117M [GeForce GTX 1650 Mobile / Max-Q] /0/100/2 /dev/fb0 display UHD Graphics 630 (Mobile)
1. glmark2 – Stress-testing GPU performance on Linux
glmark2 is an OpenGL 2.0 and ES 2.0 benchmark command-line utility. We can install it as follows:
$ sudo apt install glmark2
Now run it as follows:
$ glmark2
Then it will begin the test as follows and would stress test your GPU on Linux:
Linux glmark2 test screen
=======================================================
glmark2 2014.03+git20150611.fa71af2d
=======================================================
OpenGL Information
GL_VENDOR: NVIDIA Corporation
GL_RENDERER: GeForce GTX 1650 with Max-Q Design/PCIe/SSE2
GL_VERSION: 4.6.0 NVIDIA 450.80.02
=======================================================
[build] use-vbo=false: FPS: 4980 FrameTime: 0.201 ms
[build] use-vbo=true: FPS: 6927 FrameTime: 0.144 ms
[texture] texture-filter=nearest: FPS: 5144 FrameTime: 0.194 ms
[texture] texture-filter=linear: FPS: 4979 FrameTime: 0.201 ms
[texture] texture-filter=mipmap: FPS: 4030 FrameTime: 0.248 ms
[shading] shading=gouraud: FPS: 6358 FrameTime: 0.157 ms
[shading] shading=blinn-phong-inf: FPS: 5810 FrameTime: 0.172 ms
[shading] shading=phong: FPS: 6425 FrameTime: 0.156 ms
[shading] shading=cel: FPS: 5720 FrameTime: 0.175 ms
[bump] bump-render=high-poly: FPS: 4772 FrameTime: 0.210 ms
[bump] bump-render=normals: FPS: 7187 FrameTime: 0.139 ms
[bump] bump-render=height: FPS: 6724 FrameTime: 0.149 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 5278 FrameTime: 0.189 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 3649 FrameTime: 0.274 ms
[pulsar] light=false:quads=5:texture=false: FPS: 5793 FrameTime: 0.173 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 2776 FrameTime: 0.360 ms
[desktop] effect=shadow:windows=4: FPS: 3913 FrameTime: 0.256 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1555 FrameTime: 0.643 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 1703 FrameTime: 0.587 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1800 FrameTime: 0.556 ms
[ideas] speed=duration: FPS: 5480 FrameTime: 0.182 ms
[jellyfish] : FPS: 4283 FrameTime: 0.233 ms
[terrain] : FPS: 746 FrameTime: 1.340 ms
[shadow] : FPS: 4878 FrameTime: 0.205 ms
[refract] : FPS: 1580 FrameTime: 0.633 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 5081 FrameTime: 0.197 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 4556 FrameTime: 0.219 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 5293 FrameTime: 0.189 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 5048 FrameTime: 0.198 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 4602 FrameTime: 0.217 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 4744 FrameTime: 0.211 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 4515 FrameTime: 0.221 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 4948 FrameTime: 0.202 ms
=======================================================
glmark2 Score: 4584
=======================================================
2. glxgears – Simple Linux GPU performance testing tool
It will displays a set of rotating gears and prints out the frame rate at regular intervals. It has become quite popular as basic benchmarking tool for Linux and Unix-like system such as FreeBSD. Install and run it as follows:
$ apt install mesa-utils
$ glxgears
The GPU frame rate is measured and published out on the screen every five seconds. The final result will look as follows:
Running synchronized to the vertical refresh. The framerate should be approximately the same as the monitor refresh rate. 299 frames in 5.0 seconds = 59.416 FPS 299 frames in 5.0 seconds = 59.731 FPS 300 frames in 5.0 seconds = 59.940 FPS 300 frames in 5.0 seconds = 59.968 FPS 300 frames in 5.0 seconds = 59.943 FPS 300 frames in 5.0 seconds = 59.967 FPS 299 frames in 5.0 seconds = 59.742 FPS 300 frames in 5.0 seconds = 59.951 FPS ..... ... ...
3. gpustat – A simple tool to get Nvidia GPU stats on Linux and FreeBSD Unix
It is written in Python and the perfect tool for CLI users, especially ML/AI developers. One can install it as follows using PIP
$ pip install gpustat
$ pip3 install gpustat
Run it as follows:
$ gpustat
$ gpustat -cp
Here we see name of running process and their PIDs running on Nvidia GPU:
nixcraft-wks01 Tue Nov 24 15:46:37 2020 450.80.02 [0] GeForce GTX 1650 with Max-Q Design | 39'C, ?? %, 2 % | 962 / 3911 MB | Xorg/2454(100M) Xorg/3504(325M) gnome-shell/3689(181M) firefox/4614(1M) firefox/5036(1M) firefox/5143(1M)
See help:
$ gpustat -h
4. intel_gpu_top – Displying a top-like summary of Intel GPU usage on Linux
First install the tool, run:
$ sudo apt install intel-gpu-tools
## CentOS/RHEL/Fedora Linux user try the dnf command ##
$ sudo dnf install intel-gpu-tools
Fedora, RHEL and CentOS Linux user can use the podman command as follows to install the same:
$ podman run --rm --priviledged registry.freedesktop.org/drm/igt-gpu-tools/igt:master
The tool gathers data using perf performance counters (PMU) exposed by i915 and other platform drivers like RAPL (power) and Uncore IMC (memory bandwidth). Run it as follows on Linux system:
$ sudo intel_gpu_top
5. nvidia-smi – NVIDIA System Management Interface program
The nvidia-smi provides monitoring and management capabilities for each of NVIDIA’s Tesla, Quadro, GRID and GeForce devices from Fermi and higher architecture families. GeForce Titan series devices are supported for most functions with very limited information provided for the remainder of the Geforce brand. NVSMI is a cross platform tool that supports all standard NVIDIA driver-supported Linux and FreeBSD. Install it as follows once Nvidia driver installed on Ubuntu Linux:
$ apt install nvidia-smi
Open the terminal and then run:
$ nvidia-smi -q -g 0 -d UTILIZATION -l 1
$ sudo nvidia-smi
$ nvidia-smi --help
Here is what we see:
Tue Nov 24 15:57:43 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 165... Off | 00000000:01:00.0 On | N/A |
| N/A 40C P8 3W / N/A | 1011MiB / 3911MiB | 2% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2454 G /usr/lib/xorg/Xorg 100MiB |
| 0 N/A N/A 3504 G /usr/lib/xorg/Xorg 357MiB |
| 0 N/A N/A 3689 G /usr/bin/gnome-shell 179MiB |
| 0 N/A N/A 4614 G /usr/lib/firefox/firefox 1MiB |
| 0 N/A N/A 5036 G /usr/lib/firefox/firefox 1MiB |
| 0 N/A N/A 5143 G /usr/lib/firefox/firefox 1MiB |
| 0 N/A N/A 6406 G ...AAAAAAAA== --shared-files 225MiB |
| 0 N/A N/A 14462 G ...AAAAAAAA== --shared-files 131MiB |
+-----------------------------------------------------------------------------+
6. nvtop – NVIDIA GPU top
Another fancy but very useful tool for NVIDIA GPU. It is a ncurses-based GPU status viewer for NVIDIA GPUs similarly to the htop command or top command. We can install it as follows:
$ apt install nvtop
## RUN the tool ##
$ nvtop
The following commands are available while in nvtop is on screen:
- Up – Select (highlight) the previous process.
- Down – Select (highlight) the next process.
- Left / Right – Scroll in the process row.
- + – Sort increasingly.
- - – Sort decreasingly.
- F1 – Select a signal to send to the highlighted process.
- F2 – Select the field for sorting. The current sort field is highlighted inside the header bar.
- F3, q, Esc – Exit nvtop and return to your shell
7. radeontop – Tool to show AMD GPU utilization on Linux
View your AMD GPU utilization, both for the total activity percent and individual blocks on Linux. Install it as follows:
$ sudo apt install radeontop
$ sudo radeontop
It works with R600 and up GPUs, even Southern Islands should work fine. Works with both the open source AMD drivers and AMD Catalyst cloused-source drivers:
Conclusion
You learned about the various Linux GPU commands and tools for monitoring and diagnostic purposes on Linux and BSD-based systems. Let me know if I missed your favorite tool in the comment section below.
🐧 Get the latest tutorials on Linux, Open Source & DevOps via:
- RSS feed or Weekly email newsletter
- Share on Twitter • Facebook • 0 comments... add one ↓
| Category | List of Unix and Linux commands |
|---|---|
| File Management | cat |
| Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
| Network Utilities | dig • host • ip • nmap |
| OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
| Package Manager | apk • apt |
| Processes Management | bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time |
| Searching | grep • whereis • which |
| User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
| WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
