Today there was a leap second at 23:59:60 UTC. On one of my systems, this caused a high CPU load starting from around 02h00 GMT+2 (which corresponds with the time of the leap second). ksoftirqd and some java (glassfish) process where using lots of CPU time. This system was running Debian Squeeze with kernel 2.6.32-45. The problem is very easy to fix: just run
# date -s "`date`"
and everything will be fine again. I found this solution on the Linux Kernel Mailing List: http://marc.info/?l=linux-kernel&m=134113389621450&w=2. Apparently a similar problem can happen with Firefox, Thunderbird, Chrome/Chromium, Java, Mysql, Virtualbox and probably other processes.
I was a bit suprised that this problem only happened on this particular machine, because I have several other servers running similar kernel versions.
Very strange. Had the same issue from around yesterday ~21h00 GMT+2 on several java application servers.
What is the cause of the issue?
A bug in the Linux kernel triggered by the leap second on July 1 2012, 23:59:60 UTC. Patches fixing this bug have been posted on the Linux Kernel Mailing List now. See also https://lkml.org/lkml/2012/7/1/203 and https://lkml.org/lkml/2012/7/1/176
Wonderful man, this saved my buttocks. SOLR refused to update with ksoftirqds going mad.
We were hit by this unfortunate bug. Thank you for sharing your experience!
After spending a lot of time locating the issue, I have isolated it to the ksoftirqd. Then your post showed up and everything was immediately very clear. Thank you!
Thanks for sharing this writeup, Frederik – and for linking to this post, Ondrej!
Much appreciated – we’re in public QA on a new release, and the server had been slowed significantly by four ksoftirqd processes, collectively using some 50% of CPU …
I’ve only just noticed this problem, I have an Intel i7 and it was running 8 of these processes all at 40% cpu. Speeded up now thanks to this fix. Thanks.
Hi!
Just wanted to thank you soooo much for this! :) It just made my day :D We were having a load of 200 and now it’s just dropping and dropping!.. It’s amazing! :D Thanks!!!
Thanks man.
Spent 5 hours up in the whee ours looking for the cause. Thanks loads for the solution.
Or as ‘Cow & Chicken’ would say: “Thanks loads Larry” :-)
Cheers,
JW, Netherlands
Thank you!!!!!!
You saved my day!
Note that the command
date -s now
does the same thing, without requiring a separate ‘date’ process to look up the current time (at least assuming the ‘date’ command is the GNU coreutils version)
Thanks for this!
I was seeing this on an Amazon EC2 Ubuntu instance with OpenJDK where Java and ksoftirqd were using 100% CPU (evenly distributed across many of the Java threads which was a little weird-looking). I’m throwing all of those terms in there for search engines, hope you don’t mind :)