Xen
Jump to navigation
Jump to search
I was using a Xen install on a Debian Wheezy 7.3 with a Xen Kernel: 3.2.0-4-amd64.
On more recent architectures i had some issues with SSD devices (+Software RAID) servers: The server wasn't booting anymore on Wheezy + Xen Kernel 3.2.0-4-amd64 complaining about the fact it can't find the md0 of my mdadm array.
I had to install Debian Jessie 8.1 to solve this issue. Maybe cause the Xen kernel shipped with this debian release is more recent (3.16.0-4-amd64).
Then i figured out Xen isn't using Xend Toolstack anymore: xl is now the default toolstack, backward compatible with Xend. You could also choose your own Toolstack among serveral:
http://wiki.xen.org/wiki/Choice_of_Toolstacks
"The other notable difference is that xl, unlike xend, will not perform any host networking configuration for you":
http://wiki.xen.org/wiki/MigrationGuideToXen4.1%2B#Toolstack_upgrade_notes
On each Dom0 i have a public ipv4 and ipv6 on eth0.
I wanted all Domu to have:
- A private IP address to communicate between DomU on eth0. ex: 10.0.1.1, 10.0.1.2, etc. ==> Routed setup.
- A public IPV6 to access to communicate to IPV6 outside and be accessed from the outside. ==> Routed setup.
- A way to reach the outside (this way only) with the Dom0 public ipv4 ==> Dymamic Nat (Masquerading).
- To install needed packaged for hypervisor:
apt-get update && apt-get install -y libxen-4.4 libxenstore3.0 xen-hypervisor-4.4-amd64 xen-linux-system-3.16.0-4-amd64 xen-linux-system-amd64 xen-system-amd64 xen-tools xen-utils-4.4 xen-utils-common xenstore-utils bridge-utils
- To use the new kernel:
dpkg-divert --divert /etc/grub.d/08_OVHKernel --rename /etc/grub.d/06_OVHkernel
dpkg-divert --divert /etc/grub.d/07_linux_xen --rename /etc/grub.d/20_linux_xen
- To create an DomU example:
xen-create-image --hostname kafka1-preprod -ip 10.1.23.1 --dist=wheezy --memory=2048Mb --size=30G --password=mypassword --dir=/var/xen
- We've to enable forwarding and proxy_arp on Dom0:
# Enabling Routing
net.ipv4.ip_forward = 1
net.ipv4.conf.eth0.proxy_arp = 1
- To enable masquerading via the device eth0 add the following rule to iptables:
iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -o eth0 -j MASQUERADE
- The IPs of the DomU must be declared in /etc/xen/VM-NAME.cfg:
vif = [ 'ip=10.0.18.2 2001:41d0:2:7dde::18:2,mac=00:16:3E:BF:90:36' ]
Where the 10.0.18.2 was created when the DomU was deployed and 2001:41d0:2:7dde::18:2 is the "routable" IPv6 of the DomU.
- There is no ipv6 support in xen routed at the moment. To get it, use the following for /etc/xen/scripts/vif-route:
#!/bin/bash
#============================================================================
# ${XEN_SCRIPT_DIR}/vif-route
#
# Script for configuring a vif in routed mode.
# The hotplugging system will call this script if it is specified either in
# the device configuration given to Xend, or the default Xend configuration
# in ${XEN_CONFIG_DIR}/xend-config.sxp. If the script is specified in
# neither of those places, then vif-bridge is the default.
#
# Usage:
# vif-route (add|remove|online|offline)
#
# Environment vars:
# vif vif interface name (required).
# XENBUS_PATH path to this device's details in the XenStore (required).
#
# Read from the store:
# ip list of IP networks for the vif, space-separated (default given in
# this script).
#============================================================================
set -x
dir=$(dirname "$0")
. "$dir/vif-common.sh"
ip6_of()
{
ip -6 addr show "$1" | perl -wane '/scope global/ && /inet6 (([0-9a-f]+:*)+)/ && print $1;'
}
dom0_ip6()
{
local nd=${netdev:-eth0}
local result=$(ip6_of "$nd")
if [ -z "$result" ]
then
""
else
echo "$result"
fi
}
is_ipv6()
{
echo "$1" | grep -q ':' && echo "yes" || echo "no"
}
main_ip=$(dom0_ip)
main_ip6=$(dom0_ip6)
case "$command" in
online)
log info "[vif-route] online request, ip ${ip} with main_ip ${main_ip} and main_ip6 ${main_ip6} for $vif."
ifconfig ${vif} ${main_ip} netmask 255.255.255.255 up
if [ ! -z "${main_ip6}" ]; then
ip -6 addr add ${main_ip6} dev ${vif}
echo 1 >/proc/sys/net/ipv6/conf/${vif}/proxy_ndp
echo 1 >/proc/sys/net/ipv6/conf/${vif}/forwarding
echo 1 >/proc/sys/net/ipv6/conf/all/proxy_ndp
echo 1 >/proc/sys/net/ipv6/conf/all/forwarding
fi
echo 1 >/proc/sys/net/ipv4/conf/${vif}/proxy_arp
echo 1 >/proc/sys/net/ipv4/conf/all/proxy_arp
echo 1 >/proc/sys/net/ipv4/ip_forward
ipcmd='add'
cmdprefix=''
;;
offline)
do_without_error ifdown ${vif}
ipcmd='del'
cmdprefix='do_without_error'
;;
esac
if [ "${ip}" ] ; then
# If we've been given a list of IP addresses, then add routes from dom0 to
# the guest using those addresses.
for addr in ${ip} ; do
result=$(is_ipv6 "${addr}")
if [ "${result}" = no ] ; then
log info "[vif-route] Adding IPv4 address ${addr} with src ${main_ip} for $vif."
result=`${cmdprefix} ip route ${ipcmd} ${addr} dev ${vif} src ${main_ip} 2>&1`
else
log info "[vif-route] Adding IPv6 address ${addr} with src ${main_ip6} for $vif."
result=`${cmdprefix} ip -6 route ${ipcmd} ${addr} dev ${vif} src ${main_ip6} 2>&1`
result=`${cmdprefix} ip -6 neigh ${ipcmd} proxy ${addr} dev ${netdev:-eth0} 2>&1`
fi
done
fi
handle_iptable
log debug "Successful vif-route $command for $vif."
if [ "$command" = "online" ]
then
success
fi
- In the DomU, /etc/network/interfaces should look something like that:
auto eth0
iface eth0 inet static
address 10.0.18.2
netmask 255.255.255.255
post-up /sbin/ip route add 94.23.250.254 dev eth0
post-up /sbin/ip route add default via 94.23.250.254
iface eth0 inet6 static
address 2001:41d0:2:7dde::18:2
netmask 128
post-up /sbin/ip -f inet6 route add 2001:41d0:2:7dde::1 dev eth0
post-up /sbin/ip -f inet6 route add default via 2001:41d0:2:7dde::1
Where 94.23.250.254 is the gateway/router used by dom0 and shown my running this command in dom0:
ip route show | grep default
- DomU /etc/resolv.conf:
nameserver 127.0.0.1
nameserver 2001:41d0:3:163::1
The 2nd one is cdns.ovh.net (OVH is the provider we're using here).
- When the 10.0.0.0/8 machines (ie the VMs) need access to internet, run this in dom0 (ie the physical machine):
- Misc
cat /etc/default/xen
TOOLSTACK=xl
cat /etc/xen/xend-config.sxp
(network-script network-route)
(vif-script vif-route)
(dom0-min-mem 2048)
(enable-dom0-ballooning no)
(total_available_memory 0)
(dom0-cpus 0)
(vncpasswd '')
egrep -v '^$|^#' /etc/xen/xl.conf
vif.default.script="vif-route"
- To attach from Dom0 to a DomU:
xl console domu_name
- To detach from a DomU:
Ctrl + Alt + ]
- Resize xen image:
dd if=/dev/zero bs=1024k count=100000 >> /home/xen/domains/XXX/disk.img
e2fsck -f /home/xen/domains/XXX/disk.img
resize2fs /home/xen/domains/XXX/disk.img
xm create /etc/xen/XXX
- Shrink:
Check fs:
e2fsck -f -y xxx.img
Shrink the filesystem to the minimum size:
resize2fs -M xxx.img
Grow to the desired size (will add 100GB in this example):
dd if=/dev/zero bs=1024k count=100000 >> xxx.img
Grow the filesystem to the final size:
resize2fs xxx.img
Mixed Networking - Routed and Bridged
In our scenario we have:
- Dom0 (Hypervisor)
- eth0 with public IPV4 and internet connectivity.
- xenbr1 bridged on a dummy0 interface.
- DomU's with:
- eth0 on 192.168.0.0/16 for internet connectivity through Dom0. (Masquerade).
- eth1 on 10.1.0.0/16 for private LAN between DomU's and Dom0
Here's an ASCII schema with kafka1-p & kafka2-p as DomU'S:
XXXX XXXX
XX XXXXX X
XXX XXXXXXXX
+------------------>XXX Internet XX
| X X
| XX XX XXXXXX
| XXX XXXXXXXXXXX
| XXXXXX
|
| +--------------------------------------------------------------+----------+
| | +-------------------+ | |
| | | | | kafka1-p |
| | | | +--+ | |
| | | | | <---------------------+ |
| | | | | | +----------+
+--------^---v----------+ Dom0 +------> | +----------+
| | | | <---------------------+ |
| | | +--+ | kafka2-p |
| | | Bridge: 10.1.0.0/16 | |
| +-------------------+ | |
| +-----+----+
| |
| |
| |
+------------------------------------------------------------------------+
- DomU /etc/network/interfaces:
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 152.80.43.5
netmask 255.255.255.0
network 152.80.33.0
broadcast 151.80.43.255
gateway 152.80.43.254
iface eth0 inet6 static
address 2001:41D0:D:2005::
netmask 64
post-up /sbin/ip -family inet6 route add 2001:41D0:D:10ff:ff:ff:ff:ff dev eth0
post-up /sbin/ip -family inet6 route add default via 2001:41D0:D:10ff:ff:ff:ff:ff
pre-down /sbin/ip -family inet6 route del default via 2001:41D0:D:10ff:ff:ff:ff:ff
pre-down /sbin/ip -family inet6 route del 2001:41D0:D:10ff:ff:ff:ff:ff dev eth0
auto dummy0
iface dummy0 inet manual
auto xenbr1
iface xenbr1 inet static
address 10.3.1.1
netmask 255.255.0.0
network 10.3.0.0
broadcast 10.3.1.255
bridge_ports dummy0
bridge_stp off
bridge_maxwait 0
- Dom0 /etc/network/interfaces:
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.0.2
broadcast 192.168.0.255
netmask 255.255.255.0
gateway 192.168.0.1
auto eth1
iface eth1 inet static
address 10.3.1.2
netmask 255.255.0.0
- Xen example DomU network .cfg:
# Networking
vif = [ 'ip=192.168.0.3,mac=00:16:3e:65:c9:31,bridge=eth0','ip=10.3.1.3,mac=00:16:3e:38:a8:e0,bridge=xenbr1' ]
/!\ We need to generate 2 new unique MAC address and declare as described above.
/!\ A Python script is given on this tutorial to generate those MAC address.
- /etc/xl.conf:
[...] vif.default.script="vif-route-ap" [...]
- /etc/xen/xend-config.sxp:
#(network-script network-bridge)
#(vif-script vif-bridge)
(network-script network-route-ap)
(vif-script vif-route-ap)
(dom0-min-mem 2048)
(enable-dom0-ballooning no)
(total_available_memory 0)
(dom0-cpus 0)
(vncpasswd '')
- /etc/xen/scripts/network-route-ap:
#!/bin/bash
dir=$(dirname "$0")
"$dir/network-route" "$@" netdev=eth0
"$dir/network-bridge" "$@" netdev=dummy0
echo 1 >/proc/sys/net/ipv4/ip_forward
- /etc/xen/scripts/vif-route-ap:
#!/bin/bash
# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | awk -F. '{ print $2 }')
if [[ "$IFNUM" == "0" ]] ; then
"$dir/vif-route" "$@"
else
"$dir/vif-bridge" "$@"
fi
chmod +x /etc/xen/scripts/network-route-ap
chmod +x /etc/xen/scripts/vif-route-ap
- You could use this Python Script to generate unique MAC address:
#!/usr/bin/python
# macgen.py script to generate a MAC address for guests on Xen
#
import random
#
def randomMAC():
mac = [ 0x00, 0x16, 0x3e,
random.randint(0x00, 0x7f),
random.randint(0x00, 0xff),
random.randint(0x00, 0xff) ]
return ':'.join(map(lambda x: "%02x" % x, mac))
#
print randomMAC()
$ ./macgen.py 00:16:3e:20:b0:11
Adding specificity for OpenVPN servers
For OpenVPN (+ Quagga) servers we need to have an IPV4 address accessible from the outside.
This implies some modifications:
- /etc/xen/scripts/vif-route-ap
#!/bin/bash
# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | awk -F. '{ print $2 }')
if [[ "$IFNUM" == "0" ]] ; then
"$dir/vif-route" "$@"
elif [[ "$IFNUM" == "1" ]] ; then
"$dir/vif-bridge" "$@"
else
"$dir/vif-route" "$@"
fi
- /etc/xen/scripts/network-route-ap
#!/bin/bash
dir=$(dirname "$0")
"$dir/network-route" "$@" netdev=eth0
"$dir/network-bridge" "$@" netdev=dummy0
"$dir/network-route" "$@" netdev=eth0
echo 1 >/proc/sys/net/ipv4/ip_forward
- DomU's config files - example for ospf-batch-preprod on xen-batch-preprod
[...]
# Networking
vif = [ 'ip=192.168.0.2 2001:41d0:a:459c::254:1,mac=00:16:3E:28:B8:F3,bridge=eth0','ip=10.1.1.254,mac=00:16:3e:79:f3:a3,bridge=xenbr1','ip=5.137.42.96' ]
[...]
- DomU /etc/network/interfaces - example for ospf-batch-preprod
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.0.2
broadcast 192.168.0.255
netmask 255.255.255.0
gateway 192.168.0.1
iface eth0 inet6 static
address 2001:41d0:a:459c::254:1
netmask 128
post-up /sbin/ip -f inet6 route add 2001:41d0:a:459c::0 dev eth0
post-up /sbin/ip -f inet6 route add default via 2001:41d0:a:459c::0
auto eth1
iface eth1 inet static
address 10.1.1.254
netmask 255.255.0.0
auto eth2
iface eth2 inet static
address 5.137.42.96
netmask 255.255.255.255
post-up /sbin/route add -net 37.167.124.157 netmask 255.255.255.255 eth2
post-up /sbin/route add default gw 37.167.124.157