Whether you’re training deep learning models, running simulations, or just curious about your GPU’s performance, nvidia-smi is your go-to command-line tool. Short for NVIDIA System Management Interface, this utility provides essential real-time information about your NVIDIA GPU’s health, workload, and performance.
In this blog, we’ll explore what nvidia-smi is, how to use it, and walk through a real output from a system using an NVIDIA T1000 8GB GPU.
What is nvidia-smi?
nvidia-smi is a CLI utility bundled with the NVIDIA driver. It enables:
- Real-time GPU monitoring
- Driver and CUDA version discovery
- Process visibility and control
- GPU configuration and performance tuning
You can execute it using:
nvidia-smi
Breakdown by Section
Let’s use real life output from a system using the T1000 8GB Nvidia GPU to review each section in detail.
Driver and CUDA Info
- Driver Version: Installed NVIDIA kernel driver
- CUDA Version: Max supported CUDA runtime version
GPU Status Table
In this case, the GPU is mostly idle, used lightly by background processes.
Running GPU Processes
G: Graphics processC+G: Uses both compute and graphicsseed-version-...: Likely a custom or sandboxed job with a version tag
To investigate it further:
ps -fp <span class="m">4535</span>
ls -l /proc/4535/exePractical Use Cases
Monitor GPU Live
watch -n <span class="m">1</span> nvidia-smiOr reset the GPU:
sudo nvidia-smi --gpu-reset -i <span class="m">0</span>Query Usage via Script
nvidia-smi --query-gpu<span class="o">=</span>utilization.gpu,memory.used --format<span class="o">=</span>csvUseful for logging and dashboards.
Tips for Advanced Users
Enable persistence mode
sudo nvidia-smi -pm <span class="m">1</span>Restrict compute access
sudo nvidia-smi -c EXCLUSIVE_PROCESSView app clocks
nvidia-smi -q -d CLOCKSummary
With tools like nvidia-smi, you gain critical visibility into GPU usage and health. It’s an essential part of any ML or HPC workflow. We have developed the integrated GPU Dashboards in the Rafay Platform to provide the same information in a graphical manner. In additionl, users do not require any form of privileged, root access to visualize this critical data.
