How to run Linux under FreeBSD 11.1 using Xen

I am interested in trying out different operating systems, but don't have the hardware to dedicate to playing around.  VirtualBox has worked well on my Windows system, but I haven't invested the time to make it work on FreeBSD.  When I saw that FreeBSD 11 now works as a dom0 system with Xen, I thought I'd give it a try.  I had some frustrations, and almost gave up, but I gave it one more try today and had some success.

Here are my notes on how I got to this point, and perhaps they will help you.

I am running:

FreeBSD tigger.PaceHouse.com 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0: Wed Aug  9 11:55:48 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Step 0: Read the FreeBSD Handbook section on Xen. It is mostly the way to go.

Step 1: Install xen with "pkg install xen".  This installed Xen 4.7.0_2.  Also, you'll need a VNC viewer, which I got via "pkg install tightvnc".

Step 2: Add entries to /boot/loader.conf:

• xen_kernel="/boot/xen"
• xen_cmdline="dom0_mem=8192M dom0_max_vcpus=4 dom0pvh=1 console=com1,vga com1=115200,8n1 guest_loglvl=all loglvl=all

Notes: Tune at least the memory and number of cores.  I don't know if all that console stuff is doing anything, but I left it there from the Handbook.

Step 3: Add settings to /etc/rc.conf:

• xencommons_enable="YES"
• cloned_interfaces="lo1 bridge0 tap0"
• autobridge_interface="bridge0"
• autobridge_bridge0="re1"
• ifconfig_bridge0="addm re1 up SYNCDHCP"

Notes: My xen setup requires bridge0 and tap0; lo1 is used by something else (openvpn?), so I combined two lines.  re1 is my internal network interface, which has a DHCPd server managing it.  I'm not certain that the "autobridge" lines are doing anything, so I added that ifconfig_bridge0 line to force things.

Step 4: Reboot.  Your system will automatically load the xen kernel.  During boot, you won't see most of the normal FreeBSD boot stuff.  Also, if you look in the boot loader (by hitting 6), it will make you think that Xen isn't loaded by default, but it is.

Step 5: Create your domU config file.  Here's one that works for me. The Handbook uses disk settings that Linux doesn't seem to like, so notice that I'm using a file instead of a dev ZFS filesystem. Marcin Cieslak suggested adding the "xen_platform_pci=0" which appears to be critical (thanks!). Without it, the booting process takes a long time and then fails.

builder = "hvm"
name = "xen-mint"
memory = 2048
vcpus = 2
vif = [ 'bridge=bridge0' ]
disk = [ '/pool/Virtual/xen/mint.drv,raw,hda,rw', '/pool/Downloads/linuxmint-18.2-cinnamon-64bit.iso,raw,hdc:cdrom,r' ]
vnc = 1
vnclisten = "0.0.0.0"
serial = "pty"
usb=1
usbdevice = ['tablet']
xen_platform_pci=0

Step 6: Create the disk file (and probably adjust the path to it in the above config file).  I made mine 20 GB.

dd if=/dev/zero of=/pool/Virtual/xen/mint.drv bs=1G count=20

Step 7: Download the installer ISO for the operating system you want to use.  As you can see, I'm trying Linux Mint here.  Change the disk line to where ever you have stored the ISO.

Step 8: Boot the virtual system.

xl create /path/to/config.file

If the command returns with no messages, congratulations!  Your virtual operating system is running.  If there were errors, you'll have to fix them before continuing.  I had a bunch of networking issues, such as bridge0 not existing, and tap0 not existing.

Step 9:  Watch your OS boot and then do the install, using VNC.  If you don't have X Windows working, I think you might be hosed; I couldn't get console stuff (via xl console #) to do anything useful.

vncviewer 0.0.0.0 &

During the boot, vncviewer exits when the size of the virtual screen changes, due to changing video modes or whatever. When that happens, just run the vncviewer command again.  "vncviewer 0.0.0.0:1" works for accessing a second running domU.

After you've completed the install, you'll want to comment out the ISO line from the config file, to "eject" the install media.

Notes

• The "usbdevice = ['tablet']" line should make the mouse follow the cursor well. However, on a FreeBSD 11 domU system, this kept the mouse from moving at all. Without it, the mouse and cursor were far apart, so it is mostly unusable with a GUI. So, at this point, my FreeBSD domU's are just terminals. 
• Parrot Linux won't install for me, as it can't find a disk.
• It appears that audio doesn't work in the domU's.  I haven't spent any time on that yet.
• You have to be root to run the xl commands, so be root or add "sudo" as needed above.
• Mint complains about "Running in software rendering mode".  To fix this in VirtualBox, you have to uninstall the Vbox add-ons, and then add them again.  I don't know what the comparable xen actions are.