KVM Thrashing
Tuesday, July 14th, 2009I discovered a thrashing behaviour of the Kernel based virtual machine (KVM) during my research for my thesis on security within virtual os environments. This is how it worked. With the Kernel based virtual machine it is possible to overcommit memory between multiple virtual machines. Memory overcommitment is the fact that you assign more virtual memory than physically available.
Some background knowledge first. KVM is a kernel module. It is included within the vanilla linux kernel since version 2.6.20 (Status). KVM uses Qemu for hardware emulation. Qumranet, the leading company behind KVM has been bought by Red Hat Linux. Red Hat is planning to use KVM as a base technology for virtualization purposes, desktop-virtualization as well as for server-virtualization (RedHat). KVM establishes a full virtualization through Intels Vanderpool or AMDs Pacifica Technology. KVM uses Shadow-Page-Tables to virtualize memory, so KVM can swap pages to disk if the virtual amount of memory does not fit into the physical memory.
Following scenario.
- QEMU PC emulator version 0.9.1 (kvm-72), Copyright (c) 2003-2008 Fabrice Bellard
- 2 GB of physical memory
- Intel Core 2 Duo E8400
- 750 GB Hdd
- 1 Gigabit ethernet controller
- Debian 5.0 Lenny (18.03.2009) minimal
- Kernel Linux version 2.6.26-1-amd64 (Debian 2.6.26-13)
- 2 guests, good and evil, running same distribution
- kvm evil.img -m 2048 -curses -k de
-net nic,macaddr=52:54:00:12:34:56,model=rtl8139
-net tap,ifname=tap1 - kvm good.img -m 1024 -curses -k de
-net nic,macaddr=52:54:00:12:34:55,model=rtl8139
-net tap,ifname=tap0 - Evil without additional services
- Good runs an Apache/2.2.9 (Debian)
1 GB more than physically available has been commited to guests to force KVM to swap memory to disk when Evil is using all memory commited to Evil. I expected the performance of Good to be affacted in a high degree within this scenario. To measure the performance penalty I used the httperf benchmark utilty, version: httperf-0.9.0 compiled Jun 23 2008. The benchmark tried to query a static HTML page of 3704 bytes located on the apache webserver running on Good. I then forced Evil to use all its memory and KVM to swap pages to disk through a simple c-program.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MEMORY_MB 1950
#define MEMORY_MAX (MEMORY_MB ∗ 1048576)
main ( ) {
char ∗ ch = (char∗) malloc(MEMORY_MAX);
memset(ch, 0, MEMORY_MAX);
printf("%s\n", "Memory allocated");
while(1) {
memset(ch, 1, MEMORY_MAX);
}
}
I executed the thrashing and benchmark utility from an other host located within the same subnet.
httperf --server good.local --uri /test.html --num-conn 3000 --num-call 10 --rate 100 --timeout 5
This is a very simple benchmark under normal circumstances, but within this scenario I got 3000 errors of 3000 tries through timeouts on my first attempt. The second time 2269 errors and 2295 errors on my third attempt. So I softened the benchmark even more.
httperf --server good.local --uri /test.html --num-conn 500 --num-call 10 --rate 10 --timeout 5
But again, 97, 184 and 153 errors of 500 connection attempts. To be sure that this is due to memory overcommitment, swapping and thrashing I then quit the thrashing utility - no errors occured, so the webserver was not able to perform due to thrashing behaviour of the VMM and the attack is a DoS.
Why does it happen?
Memory is overcommitted. KVM has to swap, but KVM can’t have as much information about the pages it want to swap as an usual OS. So swapping can result in thrashing behaviour when KVM is choosing pages of highly used workloads.
Well, how risky is this kind of vulnerability?
An attacker first has to get access to a VM, but this access does not have to be highly privileged - so you have to consider it as a local DoS vulnerability. Memory has to be overcommited. The KVM version used here is not the newest one available, but it is the default one available with debian’s default distribution. I will check out newer versions soon. KVM is not the most used VMM for server virtualization, but Red Hat’s attempts to use it as a base for server virtualization show that its use potentially will grow.