How to live patch Ubuntu Linux Kernel without rebooting the server

in Categories , , last updated October 23, 2016

Kernel live patching enables runtime correction of critical security issues in running kernel without rebooting. How do I enable or patch my Ubuntu Linux 16.04 LTS server without rebooting the box?

Ubuntu Linux version 16.04 LTS supports live patching for both enterprise and the Ubuntu community members. The Canonical Livepatch Service is an authenticated, encrypted, signed stream of livepatch kernel modules for Ubuntu servers, virtual machines and desktops. Please note that this service is free up to 3 servers running 64-bit Intel/AMD Ubuntu 16.04 LTS.

Before you start

Make sure you are using the following entries in the /etc/apt/sources.list:
$ cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu xenial-security main restricted universe multiverse

Make sure your system is updated using apt command or apt-get command:
$ sudo apt update
$ sudo apt upgrade

If snapd (the snappy software platform daemon) installed on your system:
$ sudo apt install snapd
Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  snap-confine ubuntu-core-launcher
The following NEW packages will be installed:
  snap-confine snapd ubuntu-core-launcher
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 6,262 kB of archives.
After this operation, 32.5 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 snap-confine amd64 1.0.43-0ubuntu1~16.04.1 [28.9 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 ubuntu-core-launcher amd64 1.0.43-0ubuntu1~16.04.1 [2,702 B]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 snapd amd64 2.15.2ubuntu1 [6,231 kB]
Fetched 6,262 kB in 1s (4,850 kB/s)
Selecting previously unselected package snap-confine.
(Reading database ... 244122 files and directories currently installed.)
Preparing to unpack .../snap-confine_1.0.43-0ubuntu1~16.04.1_amd64.deb ...
Unpacking snap-confine (1.0.43-0ubuntu1~16.04.1) ...
Selecting previously unselected package ubuntu-core-launcher.
Preparing to unpack .../ubuntu-core-launcher_1.0.43-0ubuntu1~16.04.1_amd64.deb ...
Unpacking ubuntu-core-launcher (1.0.43-0ubuntu1~16.04.1) ...
Selecting previously unselected package snapd.
Preparing to unpack .../snapd_2.15.2ubuntu1_amd64.deb ...
Unpacking snapd (2.15.2ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up snap-confine (1.0.43-0ubuntu1~16.04.1) ...
Setting up ubuntu-core-launcher (1.0.43-0ubuntu1~16.04.1) ...
Setting up snapd (2.15.2ubuntu1) ...

Step 1: Generate a livepatch key

In order to get started login and generate a key from the following url (a free account is needed):

https://ubuntu.com/livepatch

Sample outputs after login and generated a key for my personal server at home:

Fig.01: Getting started with "Hotfixing Ubuntu Kernels"
Fig.01: Getting started with “Hotfixing Ubuntu Kernels”

Step 2: Enable live patching

Install the canonical-livepatch snap (package):
$ sudo snap install canonical-livepatch
Sample outputs:

Fig.02: Installing live patch
Fig.02: Installing live patch

Make sure /snap/bin in your PATH, run:

echo 'export PATH=$PATH:/snap/bin' >> ~/.bashrc
 
# Load the file
source ~/.bashrc
 
#Verify path
echo "$PATH"

Now, enable the service with your token. The syntax is:
$ sudo canonical-livepatch enable {YOUR-TOKEN-HERE-FROM-STEP-1}
So if token was d3b07384d213edec49eaa6238ad5ff00, enter:
$ sudo canonical-livepatch enable d3b07384d213edec49eaa6238ad5ff00
Sample outputs:

Successfully enabled device. Using machine-token: d3b07384d213edec49eaa6238ad5ff00

Step 3: View status

Type the following command to view kernel’s livepatch status:
$ canonical-livepatch status
Sample outputs:

kernel: 4.4.0-43.63-generic
fully-patched: true
version: ""

My kernel is fully patched. You can pass the --verbose option to see more details:
$ canonical-livepatch status --verbose
Sample outputs:

Fig.03: Canonical enterprise kernel livepatch service in action
Fig.03: Canonical enterprise kernel livepatch service in action

Applied and patched kernel will display status as follows:
$ canonical-livepatch status --verbose
Sample outputs:

client-version: "5"
machine-id: 727********************
machine-token: 034*************************
architecture: x86_64
cpu-model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
last-check: 2016-10-20T17:37:14.088531661-05:00
boot-time: 2016-10-16T12:27:58-05:00
uptime: 102h5m20s
status:
- kernel: 4.4.0-43.63-generic
  running: true
  livepatch:
    state: applied
    version: "13.3"
    fixes: '* CVE-2016-5195 LP: #1633547'

The patch is applied by canonical-livepatchd daemon on Ubuntu server automatically. You can view and confirm running service with the following simple command:
$ ps aux | grep '[c]anonical-livepatchd'
root 28631 0.0 0.0 1390464 23744 ? Ssl Oct19 0:08 /snap/canonical-livepatch/15/canonical-livepatchd

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.

Share this on (or read 11 comments/add one below):

11 comment

  1. Hi,

    Thanks for the valuable post. How come the new kernel would be active without rebooting the server, Curious to know the methodology. Thanks

  2. I am getting schizophrenic behaviour. I followed your recipe.

    nwayno@Willy:~$ echo $PATH
    /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/sbin:/usr/sbin:/sbin:/usr/sbin:/usr/sbin:/snap/bin

    So yup, it’s there.

    If I check the status:

    Machine is not enabled. Please run ‘sudo canonical-livepatch enable’ with the
    token obtained from https://ubuntu.com/livepatch.

    But, when I try to enable it:

    nwayno@Willy:~$ sudo canonical-livepatch enable cfce5706e8fd4934a65ff548ad2d036esudo: canonical-livepatch: command not found
    nwayno@Willy:~$

    Okay, the status PROVES the programme is there….So how can the programme be there, yet when I try to enable it, is says, it’s not?

    Wayno

      1. Sadly, I even tried rebooting (shudder) and that did NOT fix the issue.

        Same error message. I am not a LInux neophyte (pkill-9.com). But this has me stymied.

        Wayno

  3. Something newer: (overkill)

    nwayno@Willy:/usr/bin$ sudo ./snap enable cfce5706e8fd4934a65ff548ad2d036eerror: cannot enable “cfce5706e8fd4934a65ff548ad2d036e”: cannot find snap “cfce5706e8fd4934a65ff548ad2d036e”

    Wayno

  4. Got it. This time I assumed root’s environment:
    I feel like an idiot:

    nwayno@Willy:~$ su –
    Password:
    root@Willy:~# canonical-livepatch enable cfce5706e8fd4934a65ff548ad2d036e
    Successfully enabled device. Using machine-token: 3c3daee897d54f6f8ed80e9729dbb5c0
    root@Willy:~# canonical-livepatch status –verbose
    client-version: “6”
    machine-id: 0536db4c457e7ce19f5bf6e554863acb
    machine-token: 3c3daee897d54f6f8ed80e9729dbb5c0
    architecture: x86_64
    cpu-model: AMD Phenom(tm) 9550 Quad-Core Processor
    last-check: 2016-11-18T19:45:53.534421339-07:00
    boot-time: 2016-11-17T21:07:49-07:00
    uptime: 22h38m28s
    status:
    – kernel: 4.4.0-47.68-generic
    running: true
    livepatch:
    checkState: checked
    patchState: nothing-to-apply
    version: “”
    fixes: “”

    root@Willy:~#

    So, that’s it? When I apply updates now, it will automatically patch the kernel? No more reboots?

    Wayno

  5. I still have no patch deployed. Even for the latest CVE-2016-8655 :(

    status:
    – kernel: 4.4.0-46.67-generic
    running: true
    livepatch:
    checkState: checked
    patchState: nothing-to-apply
    version: “”
    fixes: “”

    canonical-livepatch[49827]: Starting client version 6
    canonical-livepatch[49827]: Checking with livepatch service.
    canonical-livepatch[49827]: No updates available at this time.
    canonical-livepatch[49827]: No payload available.

    Have a question? Post it on our forum!