I'm using NVIDIA UNIX x86_64 Kernel Module version (driver) 280.13 under Debian 64 bit Linux with Linux kernel 2.6.32-5-amd64 x86_64. However, I'm getting the following errors in my /var/log/messages file
Feb 13 05:53:39 wks01 kernel: [26652.425207] NVRM: GPU at 0000:01:00.0 has fallen off the bus.
Feb 14 03:59:14 wks01 kernel: [39846.244283] NVRM: GPU at 0000:01:00.0 has fallen off the bus.
Feb 17 04:47:32 wks01 kernel: [35237.485871] NVRM: GPU at 0000:01:00.0 has fallen off the bus.
Feb 18 06:53:19 wks01 kernel: [49298.937949] NVRM: GPU at 0000:01:00.0 has fallen off the bus.
Feb 19 06:14:01 wks01 kernel: [28508.567838] NVRM: GPU at 0000:01:00.0 has fallen off the bus.
This error occurs randomly and my laptop goes in hard freez mode. The hard reboot is the only way to recover from complete freeze of my Dell M6500 Debian Linux based laptop. How do I fix this problem?
This issue is reported all over the places and most recommended solutions are as follows:
Install Latest Kernel Version and NVIDIA Driver
You need to update your kernel and install the latest NVIDIA Unix driver.
Put NVIDIA Driver In Persistence Mode
You need to set your GPU in persistence mode. From the man page:
A flag that indicates whether persistence mode is enabled for the GPU. Value is either "Enabled" or "Disabled". When persistence mode is enabled the NVIDIA driver remains loaded even when no active clients, such as X11 or nvidia-smi, exist. This minimizes the driver load latency associated with running dependent apps, such as CUDA programs. For all CUDA- capable products. Linux only.
Edit /etc/rc.local file and add the following line before exit 0 statement:
/usr/bin/nvidia-smi -pm 1
Save and close the file. The above line ensures that your GPU is set to persistence mode as soon as it boots into the system.
How Do I Set Persistence Mode From Command Line?
Type the following command as root user:
# /usr/bin/nvidia-smi -pm 1
How Do I Verify That Persistence Mode Is Set From My Device?
Type the following command as root user:
# /usr/bin/nvidia-smi -q | grep -i Persistence
Persistence Mode : Enabled
How Do I View All Settings?
Type the following command to display GPU or unit info:
# nvidia-smi -q | less
==============NVSMI LOG============== Timestamp : Tue Feb 21 07:20:20 2012 Driver Version : 280.13 Attached GPUs : 1 GPU 0000:01:00.0 Product Name : Quadro FX 2800M Display Mode : N/A Persistence Mode : Enabled Driver Model Current : N/A Pending : N/A Serial Number : N/A GPU UUID : N/A Inforom Version OEM Object : N/A ECC Object : N/A Power Management Object : N/A PCI Bus : 1 Device : 0 Domain : 0 Device Id : 061D10DE Bus Id : 0000:01:00.0 Fan Speed : N/A Memory Usage Total : 1023 Mb Used : 74 Mb Free : 949 Mb Compute Mode : Default Utilization Gpu : N/A Memory : N/A Ecc Mode Current : N/A Pending : N/A ECC Errors Volatile Single Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Total : N/A Double Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Total : N/A Aggregate Single Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Total : N/A Double Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Total : N/A Temperature Gpu : 48 C Power Readings Power State : N/A Power Management : N/A Power Draw : N/A Power Limit : N/A Clocks Graphics : N/A SM : N/A Memory : N/A
Update 7/July/2012: Nvidia v302.17 Driver
A few user notified me that if you use NVIDIA v302.17 driver this problem get sorted out with Linux kernel 3.xx.xx series. You need to remove (delete or disable) flash player support from all browsers. This will get rid of this problem. NVIDIA internal bug to track this issue is bug ID # 973068.
- man page nvidia-smi
- NVIDIA Accelerated Linux Graphics Driver README and Installation Guide and version 302.17 guide.
- 30 Cool Open Source Software I Discovered in 2013
- 30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X
- Top 30 Nmap Command Examples For Sys/Network Admins
- 25 PHP Security Best Practices For Sys Admins
- 20 Linux System Monitoring Tools Every SysAdmin Should Know
- 20 Linux Server Hardening Security Tips
- Linux: 20 Iptables Examples For New SysAdmins
- Top 20 OpenSSH Server Best Security Practices
- Top 20 Nginx WebServer Best Security Practices
- 20 Examples: Make Sure Unix / Linux Configuration Files Are Free From Syntax Errors
- 15 Greatest Open Source Terminal Applications Of 2012
- My 10 UNIX Command Line Mistakes
- Top 10 Open Source Web-Based Project Management Software
- Top 5 Email Client For Linux, Mac OS X, and Windows Users
- The Novice Guide To Buying A Linux Laptop