Xen

I was using a Xen install on a Debian Wheezy 7.3 with a Xen Kernel: 3.2.0-4-amd64.

On more recent architechtures i had some issues with SSD (+Software RAID) composed servers: The server wasn't booting anymore on Wheezy + Xen Kernel 3.2.0-4-amd64 complaining about the fact it can't find the md0 of my mdadm array.

I had to install Debian Jessie 8.1 to solve this issue. Maybe cause the Xen kernel shipped with this debian release is more recent (3.16.0-4-amd64).

Then i figured out Xen isn't using Xend Toolstack anymore: xl is now the default toolstack, backward compatible with Xend. You could also choose your own Toolstack among serveral:

http://wiki.xen.org/wiki/Choice_of_Toolstacks

"The other notable difference is that xl, unlike xend, will not perform any host networking configuration for you":

http://wiki.xen.org/wiki/MigrationGuideToXen4.1%2B#Toolstack_upgrade_notes

On each Dom0 i have a public ipv4 and ipv6 on eth0.

I wanted all Domu to have:

• A private IP address to communicate between DomU on eth0. ex: 10.0.1.1, 10.0.1.2, etc. ==> Routed setup.
• A public IPV6 to access to communicate to IPV6 outside and be accessed from the outside. ==> Routed setup.
• A way to reach the outside (this way only) with the Dom0 public ipv4 ==> Dymamic Nat (Masquerading).

The IPs of the DomU must be declared in /etc/xen/VM-NAME.cfg:

vif         = [ 'ip=10.0.18.2 2001:41d0:2:7dde::18:2,mac=00:16:3E:BF:90:36' ]

Where the 10.0.18.2 was created when the DomU was deployed and 2001:41d0:2:7dde::18:2 is the "routable" IPv6 of the DomU.

• There is no ipv6 support in xen routed at the moment. To get it, use the following for /etc/xen/scripts/vif-route:

#!/bin/bash
#============================================================================
# ${XEN_SCRIPT_DIR}/vif-route
#
# Script for configuring a vif in routed mode.
# The hotplugging system will call this script if it is specified either in
# the device configuration given to Xend, or the default Xend configuration
# in ${XEN_CONFIG_DIR}/xend-config.sxp.  If the script is specified in
# neither of those places, then vif-bridge is the default.
#
# Usage:
# vif-route (add|remove|online|offline)
#
# Environment vars:
# vif         vif interface name (required).
# XENBUS_PATH path to this device's details in the XenStore (required).
#
# Read from the store:
# ip      list of IP networks for the vif, space-separated (default given in
#         this script).
#============================================================================

set -x
dir=$(dirname "$0")
. "$dir/vif-common.sh"

ip6_of()
{
	ip -6 addr show "$1" | perl -wane '/scope global/ && /inet6 (([0-9a-f]+:*)+)/ && print $1;'
}

dom0_ip6()
{
  local nd=${netdev:-eth0}
  local result=$(ip6_of "$nd")
  if [ -z "$result" ]
  then
	""
  else
	echo "$result"
  fi
}

is_ipv6()
{
	echo "$1" | grep -q ':' && echo "yes" || echo "no"
}

main_ip=$(dom0_ip)
main_ip6=$(dom0_ip6)

case "$command" in
    online)
 	log info "[vif-route] online request, ip ${ip} with main_ip ${main_ip} and main_ip6 ${main_ip6} for $vif."
        ifconfig ${vif} ${main_ip} netmask 255.255.255.255 up
	if [ ! -z "${main_ip6}" ]; then
		ip -6 addr add ${main_ip6} dev ${vif}
                echo 1 >/proc/sys/net/ipv6/conf/${vif}/proxy_ndp
                echo 1 >/proc/sys/net/ipv6/conf/${vif}/forwarding
                echo 1 >/proc/sys/net/ipv6/conf/all/proxy_ndp
                echo 1 >/proc/sys/net/ipv6/conf/all/forwarding
	fi
        echo 1 >/proc/sys/net/ipv4/conf/${vif}/proxy_arp
        echo 1 >/proc/sys/net/ipv4/conf/all/proxy_arp
        echo 1 >/proc/sys/net/ipv4/ip_forward
        ipcmd='add'
        cmdprefix=''
        ;;
    offline)
        do_without_error ifdown ${vif}
        ipcmd='del'
        cmdprefix='do_without_error'
        ;;
esac

if [ "${ip}" ] ; then
    # If we've been given a list of IP addresses, then add routes from dom0 to
    # the guest using those addresses.
    for addr in ${ip} ; do
	result=$(is_ipv6 "${addr}")
	if [ "${result}" = no ] ; then
		log info "[vif-route] Adding IPv4 address ${addr} with src ${main_ip} for $vif."
	      result=`${cmdprefix} ip route ${ipcmd} ${addr} dev ${vif} src ${main_ip} 2>&1`
	else
		log info "[vif-route] Adding IPv6 address ${addr} with src ${main_ip6} for $vif."
	      result=`${cmdprefix} ip -6 route ${ipcmd} ${addr} dev ${vif} src ${main_ip6} 2>&1`
	      result=`${cmdprefix} ip -6 neigh ${ipcmd} proxy ${addr} dev ${netdev:-eth0} 2>&1`
	fi
    done 
fi

handle_iptable

log debug "Successful vif-route $command for $vif."
if [ "$command" = "online" ]
then
  success
fi

• In the DomU, /etc/network/interfaces should look something like that:

auto eth0
iface eth0 inet static
 address 10.0.18.2
 netmask 255.255.255.255
 post-up /sbin/ip route add 94.23.250.254 dev eth0
 post-up /sbin/ip route add default via 94.23.250.254
 
iface eth0 inet6 static
 address 2001:41d0:2:7dde::18:2
 netmask 128
 post-up /sbin/ip -f inet6 route add 2001:41d0:2:7dde::1 dev eth0
 post-up /sbin/ip -f inet6 route add default via 2001:41d0:2:7dde::1

• When the 10.0.0.0/8 machines (ie the VMs) need access to internet, run this in dom0 (ie the physical machine):

iptables -t nat -A POSTROUTING -s 10.0.0.0/8  -o eth0 -j MASQUERADE

• DomU /etc/resolv.conf:

 
nameserver 127.0.0.1
nameserver 2001:41d0:3:163::1

The 2nd one is cdns.ovh.net (OVH is the provider we're using here).

• Misc

cat /etc/default/xen

TOOLSTACK=xl

cat /etc/xen/xend-config.sxp

(network-script network-route)
(vif-script     vif-route)
(dom0-min-mem 2048)
(enable-dom0-ballooning no)
(total_available_memory 0) 
(dom0-cpus 0)
(vncpasswd '')

egrep -v '^$|^#' /etc/xen/xl.conf

vif.default.script="vif-route"

• To attach from Dom0 to a DomU:

xl console domu_name

• To detach from a DomU:

Ctrl + Alt + ]