Linux,  Work

Leap second causing ksoftirqd and java to use lots of cpu time

Today there was a leap second at 23:59:60 UTC. On one of my systems, this caused a high CPU load starting from around 02h00 GMT+2 (which corresponds with the time of the leap second). ksoftirqd and some java (glassfish) process where using lots of CPU time. This system was running Debian Squeeze with kernel 2.6.32-45. The problem is very easy to fix: just run

# date -s "`date`"

and everything will be fine again. I found this solution on the Linux Kernel Mailing List: http://marc.info/?l=linux-kernel&m=134113389621450&w=2. Apparently a similar problem can happen with Firefox, Thunderbird, Chrome/Chromium, Java, Mysql, Virtualbox and probably other processes.

I was a bit suprised that this problem only happened on this particular machine, because I have several other servers running similar kernel versions.

12 Comments

  • AS

    Very strange. Had the same issue from around yesterday ~21h00 GMT+2 on several java application servers.
    What is the cause of the issue?

  • Marcus

    Wonderful man, this saved my buttocks. SOLR refused to update with ksoftirqds going mad.

  • Ondrej

    After spending a lot of time locating the issue, I have isolated it to the ksoftirqd. Then your post showed up and everything was immediately very clear. Thank you!

  • Aron Roberts

    Thanks for sharing this writeup, Frederik – and for linking to this post, Ondrej!

    Much appreciated – we’re in public QA on a new release, and the server had been slowed significantly by four ksoftirqd processes, collectively using some 50% of CPU …

  • Steve

    I’ve only just noticed this problem, I have an Intel i7 and it was running 8 of these processes all at 40% cpu. Speeded up now thanks to this fix. Thanks.

  • Dario

    Hi!

    Just wanted to thank you soooo much for this! :) It just made my day :D We were having a load of 200 and now it’s just dropping and dropping!.. It’s amazing! :D Thanks!!!

  • JW

    Thanks man.

    Spent 5 hours up in the whee ours looking for the cause. Thanks loads for the solution.

    Or as ‘Cow & Chicken’ would say: “Thanks loads Larry” :-)

    Cheers,

    JW, Netherlands

  • nathan

    Note that the command
    date -s now
    does the same thing, without requiring a separate ‘date’ process to look up the current time (at least assuming the ‘date’ command is the GNU coreutils version)

  • Ed Norris

    Thanks for this!
    I was seeing this on an Amazon EC2 Ubuntu instance with OpenJDK where Java and ksoftirqd were using 100% CPU (evenly distributed across many of the Java threads which was a little weird-looking). I’m throwing all of those terms in there for search engines, hope you don’t mind :)