DevOps Zone is brought to you in partnership with:

Patrick Debois has been working on closing the gap between development and operations for many years. In 2009 he organized the first devopsdays.org conference and since then the world is stuck with the term 'devops'. Always seeking for opportunities to optimize the global IT instead of local optimizations. Patrick is a DZone MVB and is not an employee of DZone and has posted 39 posts at DZone. You can read more from them at their website. View Full User Profile

Advanced Virtualbox and Vagrant tips for VLAN,PXE and Hostonly networking

11.07.2011
| 7664 views |
  • submit to reddit
I've raved about Vagrant many many times before.

Initially it was concieved for single machines running behind a NAT interface. Later support for hostonly-networking was added, opening the possibility to have multiple host talk together over a private network.

But what if the network setup of production is much more complex than this? In this blogpost I'm going to explain to reproduce the following network setup in a vagrant setup:

The network setup

          HOST
            |
            |
          eth2                          eth2
          (NAT)                         (NAT)
            |                             |
        [ Gateway ]      <---->      [ Mgmt Host]  <----->   [Servers1,2]
               br10              br10          eth3       br0
              (eth0.10,        (eth0.10,                  (eth0
               eth1.10)         eth1.10)                   eth1)
              vboxnet0          vboxnet0     vboxnet1    vboxnet1

This is quite a typical production setup:

gateway

  • it acts as a incoming/outgoing firewall, squid caching proxy , nginx etc...
  • it has two production interfaces eth0,eth1 which are VLAN-ed (ID 10) and bridged together

management host

  • connected to the gateway for in and outside traffic
  • contains all necessary tools to manage the servers, dhcp, tftpboot, etc..

servers

  • all servers get installed via the mgmt host
  • server1 will only get installed via PXE
  • server2 will have no disk attached and will boot from the network everytime

Preparing a basebox

Almost all baseboxes out there assume that the DHCP interface for the NAT interface resides on the eth0 interface. Moving the NAT interface to eth2, requires you to change the /etc/network/interfaces to indicate that dhcp is done on eth2, and eth0,eth1 are not dhcp. Otherwise bootup will be very slow, waiting for a timeout, resulting in no ip addresses.

We've used Veewee to add another postinstall script to change the /etc/network/interfaces file to the state we needed. Resulting in a 'squeeze64'

sed -i -e "s/eth0/eth2/" /etc/network/interfaces

Hostonly networking in Vagrant

While Vagrant has support for hostonly networking, it assumes that it has to assign the interface an IP-address. In our case the interfaces eth0,eth1 do not get an IP-address, only the bridge gets an IP-address.

Also using the hostonly networking support built into vagrant, we found that it changed the order of the interfaces: it seems to dynamically remove all host interfaces and recreate them.

We resorted to :

  • adding our hostonly interfaces to our 'basebox' via the VBoxManage commands
  • exporting it via 'vagrant basebox export' subcommand

And thus creating a 'gateway.box' and 'manager.box' vagrant box.

For the gateway node:

IFTYPE="virtio"
VBoxManage modifyvm squeeze64 --nic1 hostonly
VBoxManage modifyvm squeeze64 --nic2 hostonly
VBoxManage modifyvm squeeze64 --nic3 nat
VBoxManage modifyvm squeeze64 --nic4 none
VBoxManage modifyvm squeeze64 --macaddress1 auto
VBoxManage modifyvm squeeze64 --macaddress2 auto
VBoxManage modifyvm squeeze64 --macaddress3 auto
VBoxManage modifyvm squeeze64 --nictype1 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype2 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype3 $IFTYPE
VBoxManage modifyvm squeeze64 --cableconnected2 off
VBoxManage controlvm squeeze64 setlinkstate2 off
VBoxManage modifyvm squeeze64 --hostonlyadapter1 vboxnet0
VBoxManage modifyvm squeeze64 --hostonlyadapter2 vboxnet0

rm squeeze64.box
bundle exec vagrant basebox export squeeze64
mv  squeeze64.box boxes/manager.box
bundle exec vagrant box remove 'manager'
bundle exec vagrant box add 'manager' 'boxes/manager.box

For the manager node:

IFTYPE="virtio"
VBoxManage modifyvm squeeze64 --nic1 hostonly
VBoxManage modifyvm squeeze64 --nic2 hostonly
VBoxManage modifyvm squeeze64 --nic3 nat
VBoxManage modifyvm squeeze64 --nic4 hostonly
VBoxManage modifyvm squeeze64 --macaddress1 auto
VBoxManage modifyvm squeeze64 --macaddress2 auto
VBoxManage modifyvm squeeze64 --macaddress3 auto
VBoxManage modifyvm squeeze64 --macaddress4 auto
VBoxManage modifyvm squeeze64 --nictype1 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype2 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype3 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype4 $IFTYPE
VBoxManage modifyvm squeeze64 --cableconnected2 off
VBoxManage controlvm squeeze64 setlinkstate2 off
VBoxManage modifyvm squeeze64 --hostonlyadapter1 vboxnet0
VBoxManage modifyvm squeeze64 --hostonlyadapter2 vboxnet0
VBoxManage modifyvm squeeze64 --hostonlyadapter4 vboxnet1

mv squeeze64.box manager.box

bundle exec vagrant box remove 'manager'
bundle exec vagrant basebox export squeeze64
mv  squeeze64.box boxes/manager.box
bundle exec vagrant box add 'manager' 'boxes/manager.box'

Adapting Vagrant to use eth2 for NAT

Not specifying the hostonly network in the Vagrantfile, kept the order and the interface names of the hostonly networks intact. But by default, vagrant assumes it has to put the NAT ssh-port mapping on eth0. To make it point to eth2, you can specify the adaptor by explicitly adding the ssh portforwarding mapping to the Vagrantfile.

  config.vm.forward_port("ssh",22,2222,{:auto => true,:adapter => 2})

VLAN support in Virtualbox

I would have expected VLAN-ing to work out of the box with Virtualbox, but it didn't.

For some reason, both gateway and manager were not able to talk to each other. I would see packets come in from both sides on the interfaces, but it just would not work.

After some digging , I found this blogpost on VLAN Stripping in Virtualbox and the research he did .

It turns out that the Intel Pro Network drivers providers, provided by Virtualbox, strip the VLAN tags. The blog suggested changing the interface types to the older AMD PCNet Fast III. This indeed worked.

But on large bursts of network traffic, I was confronted with a kernel warning NETDEV Watchdog: eth0: transmit timed out .

Luckily there is a third option for networking listed on the Virtualbox network documentation: virtio drivers. At first I thought this was not usable because I was running an a Mac, but the virtio are drivers inside the Guest OS, not on the host.

That is why in the above box modifications we set the nictype to virtio

VBoxManage modifyvm squeeze64 --nictype1 virtio

Bridging in Virtualbox

While recreating the vms, many times, I noticed that bridging sometimes would not work. When I looked in the VirtualBox gui, I noticed that sometimes virtualbox indicated, that the active network connection was on eth0 in the gateway and on eth1 in the manager. To fake the behavior and have them both on the same interfaces, I disconnected the cable on the eth1.

VBoxManage modifyvm squeeze64 --cableconnected2 off
VBoxManage controlvm squeeze64 setlinkstate2 off

Now the bridging network interfaces could always see eachother.

PXE booting in Virtualbox

If you want to use pxe booting in virtualbox, be sure to install the Extension pack. This will include the PXE boot functionality.

$ VBoxManage list extpacks
Extension Packs: 1
Pack no. 0:   Oracle VM VirtualBox Extension Pack
Version:      4.1.4
Revision:     74291
Description:  USB 2.0 Host Controller, VirtualBox RDP, PXE ROM with E1000 support.
VRDE Module:  VBoxVRDP
Usable:       true 
Why unusable: 

More details can be found in the documentation of Virtualbox

Creating a pxe-booting basebox

For server1 and server2, we want them to pxe boot from the manager node. This means that we want to have an empty disk, but with network boot enabled.

VBoxManage modifyvm pxe-server1 --boot4 net

Veewee is not yet capable of doing this, but here are the commands we used to create an empty netboot basebox.

VMPATH="/"$(VBoxManage list systemproperties|grep "^Default"| cut -d '/' -f 2-)"/pxe/"
DISKPATH="/"$VMPATH/"/pxe.vdi"
VBoxManage unregistervm pxe --delete
rm -rf "$VMPATH"
VBoxManage createvm --name pxe --ostype Debian_64 --register
VBoxManage storagectl 'pxe' --name 'IDE Controller' --add ide
VBoxManage storagectl 'pxe' --name 'SATA Controller' --add sata --hostiocache off --sataportcount 1

VBoxManage createhd --filename "$DISKPATH" --size 20000 --format VDI 
VBoxManage storageattach 'pxe' --storagectl 'SATA Controller' --port 0 --device 0 --type hdd --medium "$DISKPATH"

PXE boot install on server1

Now that we have a pxe boot basebox, we can start modifying it to:

  • fix the macaddress, so we can put them in dhcp
  • create the necessary hostonly interfaces

Because the first pxe boot takes a long time, we increased the ssh timeouts in the Vagrantfile

    config.ssh.timeout = 1000000
    config.ssh.max_tries = 50

Again, the virtio driver is used

IF_TYPE=virtio
VBoxManage unregistervm pxe-server1 --delete
VBoxManage clonevm pxe --name pxe-server1 --register

VBoxManage modifyvm pxe-server1 --memory 256 --ostype Debian_64

VBoxManage modifyvm pxe-server1 --nic1 hostonly
VBoxManage modifyvm pxe-server1 --nic2 hostonly
VBoxManage modifyvm pxe-server1 --nic3 nat
VBoxManage modifyvm pxe-server1 --nic4 none
VBoxManage modifyvm pxe-server1 --macaddress1 080027BA2DAE
VBoxManage modifyvm pxe-server1 --macaddress2 080027BE8E74
VBoxManage modifyvm pxe-server1 --macaddress3 auto
VBoxManage modifyvm pxe-server1 --macaddress4 auto
VBoxManage modifyvm pxe-server1 --nictype1 $IF_TYPE
VBoxManage modifyvm pxe-server1 --nictype2 $IF_TYPE
VBoxManage modifyvm pxe-server1 --nictype3 $IF_TYPE
VBoxManage modifyvm pxe-server1 --nictype4 $IF_TYPE
VBoxManage modifyvm pxe-server1 --cableconnected2 off
VBoxManage controlvm pxe-server1 setlinkstate2 off
VBoxManage modifyvm pxe-server1 --cableconnected3 off
VBoxManage controlvm pxe-server1 setlinkstate3 off
VBoxManage modifyvm pxe-server1 --hostonlyadapter1 vboxnet1
VBoxManage modifyvm pxe-server1 --hostonlyadapter2 vboxnet1
VBoxManage modifyvm pxe-server1 --boot4 net
rm boxes/pxe-server1.box
bundle exec vagrant box remove 'server1'
bundle exec vagrant basebox export pxe-server1
mv 'pxe-server1.box' boxes/
bundle exec vagrant box add 'server1' 'boxes/pxe-server1.box'

Fixing the mac address of a bridge

Even though we can control the mac address from the physical interfaces, the macaddress of the bridge is randomly assigned.

The trick is to specify the macaddress on if-up of the bridge interface

auto br0
iface br0 inet dhcp
  bridge_ports eth0
  bridge_maxage 12
  bridge_hello 2
  bridge_fd 6
  bridge_stp on
  post-up ip link set br0 address 08:00:27:ba:2d:ae

PXE boot for server2

The main difference with server1, is that server2 always boots from memory.

Vagrant basebox export, expects a disk to be there. So we have to assign it one, even if we don't need one.

We also found that the internal pxe support is limited, using of ipxe.org, allowed us to boot with many special pxe options.

This lead us to put the ipxe boot code on the first disk.

VBoxManage convertfromraw ../ipxe/ipxe.usb "$DISKPATH"

And replace the disk with it, making it play nice with vagrant export

The full creation script looks like this

VMPATH="/"$(VBoxManage list systemproperties|grep "^Default"| cut -d '/' -f 2-)"/pxe-server2/"
DISKPATH="$VMPATH/pxe-server2.vdi"
VBoxManage storageattach 'pxe-server2' --storagectl 'SATA Controller'  --port 0 --device 0 --medium none
VBoxManage closemedium disk  "$DISKPATH" --delete
VBoxManage unregistervm pxe-server2 --delete
rm -rf "$VMPATH"
VBoxManage clonevm pxe --name pxe-server2 --register
VBoxManage modifyvm pxe-server2 --memory 1024 --ostype Debian_64
VBoxManage modifyvm pxe-server2 --nic1 hostonly
VBoxManage modifyvm pxe-server2 --nic2 hostonly
VBoxManage modifyvm pxe-server2 --nic3 nat
VBoxManage modifyvm pxe-server2 --nic4 none
VBoxManage modifyvm pxe-server2 --macaddress1 080027C5DAB8
VBoxManage modifyvm pxe-server2 --macaddress2 0800274020F8
VBoxManage modifyvm pxe-server2 --macaddress3 auto
VBoxManage modifyvm pxe-server2 --macaddress4 auto

VBoxManage modifyvm pxe-server2 --nictype1 virtio
VBoxManage modifyvm pxe-server2 --nictype2 virtio
VBoxManage modifyvm pxe-server2 --nictype3 virtio
VBoxManage modifyvm pxe-server2 --nictype4 virtio
VBoxManage modifyvm pxe-server2 --cableconnected2 off
VBoxManage controlvm pxe-server2 setlinkstate2 off
VBoxManage modifyvm pxe-server2 --cableconnected3 off
VBoxManage controlvm pxe-server2 setlinkstate3 off
VBoxManage modifyvm pxe-server2 --hostonlyadapter1 vboxnet1
VBoxManage modifyvm pxe-server2 --hostonlyadapter2 vboxnet1
VBoxManage modifyvm pxe-server2 --boot4 net
rm "$DISKPATH"
VBoxManage storageattach 'pxe-server2' --storagectl 'SATA Controller'  --port 0 --device 0 --medium none
VBoxManage closemedium disk  "$DISKPATH" --delete
VBoxManage convertfromraw ../ipxe/ipxe.usb "$DISKPATH"
VBoxManage storageattach 'pxe-server2' --storagectl 'SATA Controller' --type hdd --port 1 --device 0 --medium "$DISKPATH"

rm boxes/pxe-server2.box
bundle exec vagrant box remove 'server2'
bundle exec vagrant basebox export pxe-server2
mv 'pxe-server2.box' boxes/
bundle exec vagrant box add 'server2' 'boxes/pxe-server2.box'

Direct ssh into vagrant VM via Hostonly IF

Instead of doing the portmapping to server1 or server2 via NAT, we found it useful to put an interface of our host into the hostonly network. This allowed for connection to services in server1,server2 for example.

VBoxManage hostonlyif ipconfig vboxnet1 --ip 172.168.0.1

And then adjust for example the ssh properties of server2, as follows:

    config.ssh.host = "172.168.0.22"
    config.ssh.port = 22

Conclusion:

It takes a bit of massaging to get it working but it's possible:

  • use virtio driver where possible
  • use ipxe to boot complex pxe setups
  • use hostonly networking for direct access to services


Source:  http://www.jedi.be/blog/2011/11/04/vagrant-virtualbox-hostonly-pxe-vlans/
Published at DZone with permission of Patrick Debois, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)