Xen

I was using a Xen install on a Debian Wheezy 7.3 with a Xen Kernel: 3.2.0-4-amd64.

On more recent architectures i had some issues with SSD devices (+Software RAID) servers: The server wasn't booting anymore on Wheezy + Xen Kernel 3.2.0-4-amd64 complaining about the fact it can't find the md0 of my mdadm array.

I had to install Debian Jessie 8.1 to solve this issue. Maybe cause the Xen kernel shipped with this debian release is more recent (3.16.0-4-amd64).

Then i figured out Xen isn't using Xend Toolstack anymore: xl is now the default toolstack, backward compatible with Xend. You could also choose your own Toolstack among serveral:

http://wiki.xen.org/wiki/Choice_of_Toolstacks

"The other notable difference is that xl, unlike xend, will not perform any host networking configuration for you":

http://wiki.xen.org/wiki/MigrationGuideToXen4.1%2B#Toolstack_upgrade_notes

On each Dom0 i have a public ipv4 and ipv6 on eth0.

I wanted all Domu to have:

• A private IP address to communicate between DomU on eth0. ex: 10.0.1.1, 10.0.1.2, etc. ==> Routed setup.
• A public IPV6 to access to communicate to IPV6 outside and be accessed from the outside. ==> Routed setup.
• A way to reach the outside (this way only) with the Dom0 public ipv4 ==> Dymamic Nat (Masquerading).

• To install needed packaged for hypervisor:

apt-get update && apt-get install -y libxen-4.4 libxenstore3.0 xen-hypervisor-4.4-amd64 xen-linux-system-3.16.0-4-amd64 xen-linux-system-amd64 xen-system-amd64 xen-tools xen-utils-4.4 xen-utils-common xenstore-utils bridge-utils

• To use the new kernel:

dpkg-divert --divert /etc/grub.d/08_OVHKernel --rename /etc/grub.d/06_OVHkernel 
dpkg-divert --divert /etc/grub.d/07_linux_xen --rename /etc/grub.d/20_linux_xen

• To create an DomU example:

xen-create-image --hostname kafka1-preprod -ip 10.1.23.1 --dist=wheezy --memory=2048Mb --size=30G --password=mypassword --dir=/var/xen

• We've to enable forwarding and proxy_arp on Dom0:

# Enabling Routing
net.ipv4.ip_forward = 1
net.ipv4.conf.eth0.proxy_arp = 1

• To enable masquerading via the device eth0 add the following rule to iptables:

iptables -t nat -A POSTROUTING -s 10.0.0.0/8  -o eth0 -j MASQUERADE

• The IPs of the DomU must be declared in /etc/xen/VM-NAME.cfg:

vif         = [ 'ip=10.0.18.2 2001:41d0:2:7dde::18:2,mac=00:16:3E:BF:90:36' ]

Where the 10.0.18.2 was created when the DomU was deployed and 2001:41d0:2:7dde::18:2 is the "routable" IPv6 of the DomU.

• There is no ipv6 support in xen routed at the moment. To get it, use the following for /etc/xen/scripts/vif-route:

#!/bin/bash
#============================================================================
# ${XEN_SCRIPT_DIR}/vif-route
#
# Script for configuring a vif in routed mode.
# The hotplugging system will call this script if it is specified either in
# the device configuration given to Xend, or the default Xend configuration
# in ${XEN_CONFIG_DIR}/xend-config.sxp.  If the script is specified in
# neither of those places, then vif-bridge is the default.
#
# Usage:
# vif-route (add|remove|online|offline)
#
# Environment vars:
# vif         vif interface name (required).
# XENBUS_PATH path to this device's details in the XenStore (required).
#
# Read from the store:
# ip      list of IP networks for the vif, space-separated (default given in
#         this script).
#============================================================================

set -x
dir=$(dirname "$0")
. "$dir/vif-common.sh"

ip6_of()
{
	ip -6 addr show "$1" | perl -wane '/scope global/ && /inet6 (([0-9a-f]+:*)+)/ && print $1;'
}

dom0_ip6()
{
  local nd=${netdev:-eth0}
  local result=$(ip6_of "$nd")
  if [ -z "$result" ]
  then
	""
  else
	echo "$result"
  fi
}

is_ipv6()
{
	echo "$1" | grep -q ':' && echo "yes" || echo "no"
}

main_ip=$(dom0_ip)
main_ip6=$(dom0_ip6)

case "$command" in
    online)
 	log info "[vif-route] online request, ip ${ip} with main_ip ${main_ip} and main_ip6 ${main_ip6} for $vif."
        ifconfig ${vif} ${main_ip} netmask 255.255.255.255 up
	if [ ! -z "${main_ip6}" ]; then
		ip -6 addr add ${main_ip6} dev ${vif}
                echo 1 >/proc/sys/net/ipv6/conf/${vif}/proxy_ndp
                echo 1 >/proc/sys/net/ipv6/conf/${vif}/forwarding
                echo 1 >/proc/sys/net/ipv6/conf/all/proxy_ndp
                echo 1 >/proc/sys/net/ipv6/conf/all/forwarding
	fi
        echo 1 >/proc/sys/net/ipv4/conf/${vif}/proxy_arp
        echo 1 >/proc/sys/net/ipv4/conf/all/proxy_arp
        echo 1 >/proc/sys/net/ipv4/ip_forward
        ipcmd='add'
        cmdprefix=''
        ;;
    offline)
        do_without_error ifdown ${vif}
        ipcmd='del'
        cmdprefix='do_without_error'
        ;;
esac

if [ "${ip}" ] ; then
    # If we've been given a list of IP addresses, then add routes from dom0 to
    # the guest using those addresses.
    for addr in ${ip} ; do
	result=$(is_ipv6 "${addr}")
	if [ "${result}" = no ] ; then
		log info "[vif-route] Adding IPv4 address ${addr} with src ${main_ip} for $vif."
	      result=`${cmdprefix} ip route ${ipcmd} ${addr} dev ${vif} src ${main_ip} 2>&1`
	else
		log info "[vif-route] Adding IPv6 address ${addr} with src ${main_ip6} for $vif."
	      result=`${cmdprefix} ip -6 route ${ipcmd} ${addr} dev ${vif} src ${main_ip6} 2>&1`
	      result=`${cmdprefix} ip -6 neigh ${ipcmd} proxy ${addr} dev ${netdev:-eth0} 2>&1`
	fi
    done 
fi

handle_iptable

log debug "Successful vif-route $command for $vif."
if [ "$command" = "online" ]
then
  success
fi

• In the DomU, /etc/network/interfaces should look something like that:

auto eth0
iface eth0 inet static
 address 10.0.18.2
 netmask 255.255.255.255
 post-up /sbin/ip route add 94.23.250.254 dev eth0
 post-up /sbin/ip route add default via 94.23.250.254
 
iface eth0 inet6 static
 address 2001:41d0:2:7dde::18:2
 netmask 128
 post-up /sbin/ip -f inet6 route add 2001:41d0:2:7dde::1 dev eth0
 post-up /sbin/ip -f inet6 route add default via 2001:41d0:2:7dde::1

Where 94.23.250.254 is the gateway/router used by dom0 and shown my running this command in dom0:

ip route show | grep default

• DomU /etc/resolv.conf:

 
nameserver 127.0.0.1
nameserver 2001:41d0:3:163::1

The 2nd one is cdns.ovh.net (OVH is the provider we're using here).

• When the 10.0.0.0/8 machines (ie the VMs) need access to internet, run this in dom0 (ie the physical machine):

• Misc

cat /etc/default/xen

TOOLSTACK=xl

cat /etc/xen/xend-config.sxp

(network-script network-route)
(vif-script     vif-route)
(dom0-min-mem 2048)
(enable-dom0-ballooning no)
(total_available_memory 0) 
(dom0-cpus 0)
(vncpasswd '')

egrep -v '^$|^#' /etc/xen/xl.conf

vif.default.script="vif-route"

• To attach from Dom0 to a DomU:

xl console domu_name

• To detach from a DomU:

Ctrl + Alt + ]

• Resize xen image:

 dd if=/dev/zero bs=1024k count=100000 >> /home/xen/domains/XXX/disk.img
 e2fsck -f /home/xen/domains/XXX/disk.img 
 resize2fs /home/xen/domains/XXX/disk.img
 xm create /etc/xen/XXX

• Shrink:

Check fs:

e2fsck -f -y xxx.img

Shrink the filesystem to the minimum size:

resize2fs -M xxx.img

Grow to the desired size (will add 100GB in this example):

dd if=/dev/zero bs=1024k count=100000 >> xxx.img

Grow the filesystem to the final size:

resize2fs xxx.img

Mixed Networking - Routed and Bridged

In our scenario we have:

• Dom0 (Hypervisor)
  • eth0 with public IPV4 and internet connectivity.
  • xenbr1 bridged on a dummy0 interface.

• DomU's with:
  • eth0 on 192.168.0.0/16 for internet connectivity through Dom0. (Masquerade).
  • eth1 on 10.1.0.0/16 for private LAN between DomU's and Dom0

Here's an ASCII schema with kafka1-p & kafka2-p as DomU'S:

                           XXXX   XXXX                                                  
                        XX    XXXXX  X                                                  
                      XXX             XXXXXXXX                                          
+------------------>XXX      Internet        XX                                         
|                  X                          X                                         
|                  XX XX                 XXXXXX                                         
|                      XXX    XXXXXXXXXXX                                               
|                        XXXXXX                                                         
|                                                                                       
|            +--------------------------------------------------------------+----------+
|            |          +-------------------+                               |          |
|            |          |                   |                               | kafka1-p |
|            |          |                   |      +--+                     |          |
|            |          |                   |      |  <---------------------+          |
|            |          |                   |      |  |                     +----------+
+--------^---v----------+       Dom0        +------>  |                     +----------+
         |              |                   |      |  <---------------------+          |
         |              |                   |      +--+                     | kafka2-p |
         |              |                   |   Bridge: 10.1.0.0/16         |          |
         |              +-------------------+                               |          |
         |                                                                  +-----+----+
         |                                                                        |     
         |                                                                        |     
         |                                                                        |     
         +------------------------------------------------------------------------+     
 

• DomU /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 152.80.43.5
        netmask 255.255.255.0
        network 152.80.33.0
        broadcast 151.80.43.255
        gateway 152.80.43.254

iface eth0 inet6 static
  address 2001:41D0:D:2005::
  netmask 64
  post-up /sbin/ip -family inet6 route add 2001:41D0:D:10ff:ff:ff:ff:ff dev eth0
  post-up /sbin/ip -family inet6 route add default via 2001:41D0:D:10ff:ff:ff:ff:ff
  pre-down /sbin/ip -family inet6 route del default via 2001:41D0:D:10ff:ff:ff:ff:ff
  pre-down /sbin/ip -family inet6 route del 2001:41D0:D:10ff:ff:ff:ff:ff dev eth0

auto dummy0
iface dummy0 inet manual

auto xenbr1
iface xenbr1 inet static
address 10.3.1.1
netmask 255.255.0.0
network 10.3.0.0
broadcast 10.3.1.255
bridge_ports dummy0
bridge_stp off
bridge_maxwait 0

• Dom0 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
  address   192.168.0.2
  broadcast 192.168.0.255
  netmask   255.255.255.0
  gateway   192.168.0.1

auto eth1
iface eth1 inet static
  address 10.3.1.2
  netmask 255.255.0.0

• Xen example DomU network .cfg:

# Networking
vif = [ 'ip=192.168.0.3,mac=00:16:3e:65:c9:31,bridge=eth0','ip=10.3.1.3,mac=00:16:3e:38:a8:e0,bridge=xenbr1' ]

/!\ We need to generate 2 new unique MAC address and declare as described above.

/!\ A Python script is given on this tutorial to generate those MAC address.

• /etc/xl.conf:

[...]
vif.default.script="vif-route-ap"
[...]

• /etc/xen/xend-config.sxp:

#(network-script network-bridge)
#(vif-script     vif-bridge)
(network-script network-route-ap)
(vif-script vif-route-ap)

(dom0-min-mem 2048)
(enable-dom0-ballooning no)
(total_available_memory 0)
(dom0-cpus 0)
(vncpasswd '')

• /etc/xen/scripts/network-route-ap:

#!/bin/bash

dir=$(dirname "$0")
"$dir/network-route"  "$@" netdev=eth0
"$dir/network-bridge" "$@" netdev=dummy0
echo 1 >/proc/sys/net/ipv4/ip_forward

• /etc/xen/scripts/vif-route-ap:

#!/bin/bash

# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | awk -F. '{ print $2 }')
if [[ "$IFNUM" == "0" ]] ; then
 "$dir/vif-route"  "$@"
else
 "$dir/vif-bridge" "$@"
fi

 chmod +x /etc/xen/scripts/network-route-ap
 chmod +x /etc/xen/scripts/vif-route-ap

• You could use this Python Script to generate unique MAC address:

#!/usr/bin/python
# macgen.py script to generate a MAC address for guests on Xen
#
import random
#
def randomMAC():
	mac = [ 0x00, 0x16, 0x3e,
		random.randint(0x00, 0x7f),
		random.randint(0x00, 0xff),
		random.randint(0x00, 0xff) ]
	return ':'.join(map(lambda x: "%02x" % x, mac))
#
print randomMAC()

$ ./macgen.py 
00:16:3e:20:b0:11

Adding specificity for OpenVPN servers

For OpenVPN (+ Quagga) servers we need to have an IPV4 address accessible from the outside.

This implies some modifications:

• /etc/xen/scripts/vif-route-ap

#!/bin/bash

# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | awk -F. '{ print $2 }')
if [[ "$IFNUM" == "0" ]] ; then
 "$dir/vif-route"  "$@"
elif [[ "$IFNUM" == "1" ]] ; then
 "$dir/vif-bridge" "$@"
else
 "$dir/vif-route" "$@"
fi

• /etc/xen/scripts/network-route-ap

#!/bin/bash

dir=$(dirname "$0")
"$dir/network-route"  "$@" netdev=eth0
"$dir/network-bridge" "$@" netdev=dummy0
"$dir/network-route" "$@" netdev=eth0

echo 1 >/proc/sys/net/ipv4/ip_forward

• DomU's config files - example for ospf-batch-preprod on xen-batch-preprod

[...]

#  Networking
vif = [ 'ip=192.168.0.2 2001:41d0:a:459c::254:1,mac=00:16:3E:28:B8:F3,bridge=eth0','ip=10.1.1.254,mac=00:16:3e:79:f3:a3,bridge=xenbr1','ip=5.137.42.96' ]

[...]

• DomU /etc/network/interfaces - example for ospf-batch-preprod

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address   192.168.0.2
	broadcast 192.168.0.255
	netmask   255.255.255.0
	gateway   192.168.0.1

iface eth0 inet6 static
	address 2001:41d0:a:459c::254:1
	netmask 128
	post-up /sbin/ip -f inet6 route add 2001:41d0:a:459c::0 dev eth0
	post-up /sbin/ip -f inet6 route add default via 2001:41d0:a:459c::0

auto eth1
iface eth1 inet static
	address 10.1.1.254
	netmask 255.255.0.0

auto eth2
iface eth2 inet static
	address 5.137.42.96
	netmask 255.255.255.255
	post-up /sbin/route add -net 37.167.124.157 netmask 255.255.255.255 eth2
	post-up /sbin/route add default gw 37.167.124.157