We are going to add a physical interface, connected to other hosts to the virtual bridge device that the docker containers connect to, so that the virtual network spans across our physical hosts as shown in the following figure:

We need to explicitly manage the IP address of the containers since all containers will be on the same network. This is not the general use case for docker, so we need to get into a few advanced features.

We need to disable docker from managing the bridge device since we want to manage it by hand, and we will need to use lxc instead of libcontainer to have control over setting our own container IP addresses. Configure this in /etc/sysconfig/docker:

    other_args="-b=none -e=lxc“
    service docker restart

Setup our own bridge device

    sudo yum install bridge-utils
    sudo brctl addbr docker0
    sudo brctl addif docker0 eth0

change eth0 to whatever interface you are using to physically connect the hosts together. To make this permanent, change /etc/sysconfig/network-scripts/ifcfg-eth0 to:

    DEVICE=eth0
    HWADDR=<MAC ADDRESS>
    ONBOOT=yes
    BOOTPROTO=none
    BRIDGE=docker0

and /etc/sysconfig/network-scripts/ifcfg-docker0 (IP configuration is optional, but if needed should be configured on bridge device not ethX device):

    DEVICE=docker0
    TYPE=Bridge
    #IPADDR=1.2.3.4
    #GATEWAY=1.2.3.1
    #NETMASK=255.255.255.0
    ONBOOT=yes
    BOOTPROTO=none
    IPV6INIT=no
    IPV6_AUTOCONF=no
    STP=no

If all went well we should see our ethernet device associated with the bridge using this command

    # brctl show
    bridge name bridge id           STP enabled interfaces
    docker0     8000.000000000000   no  eth0

We need to add some lxc configuration options to our docker commands to setup the static IP addressing

    --lxc-conf="lxc.network.type = veth" 
    --lxc-conf="lxc.network.ipv4 = 172.16.0.1/16" 
    --lxc-conf="lxc.network.link = docker0" 
    --lxc-conf="lxc.network.name = eth0" 
    --lxc-conf="lxc.network.flags = up"

This tells docker/lxc to use the virtual ethernet device with and IP address 172.16.0.1, netmask 255.255.0.0, connected to docker0 bridge, configuring device eth0 within the container, and automatically bringing the container interface up. It is important to have an address scheme defined before setup, because this is needed for generating an /etc/hosts file within the containers. An example is:

172.16.0.1 db
172.16.0.2 master (scheduler)
172.16.0.3 agg01
172.16.1.0 - 172.16.1.255 node001 - node256 (on host1)
172.16.2.0 - 172.16.2.255 node257 - node512 (on host2)
172.16.255.0.X reserved for login shells

In the following steps we assume that Sensys is installed in its default path inside the container i.e. '/opt/sensys' and the name of the docker container image is sensys

1. Launch database (any host)

    % docker run -d --name db -h db --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.0.1/16" \
        --lxc-conf="lxc.network.link = docker0" --lxc-conf="lxc.network.name = eth0" \
        --lxc-conf="lxc.network.flags = up" sensys sudo -u postgres /usr/pgsql-9.3/bin/postmaster \
        -p 5432 -D /var/lib/pgsql/9.3/data

2. Launch scheduler (any host)

    % docker run -d --name master -h master -v /root/hosts:/etc/hosts --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.0.2/16" --lxc-conf="lxc.network.link = docker0" \
        --lxc-conf="lxc.network.name = eth0" --lxc-conf="lxc.network.flags = up" sensys
        /opt/sensys/bin/orcmsched

3. Launch aggregator (any host)

    % docker run -d --name agg01 -h agg01 -v /root/hosts:/etc/hosts --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.0.3/16" --lxc-conf="lxc.network.link = docker0" \
        --lxc-conf="lxc.network.name = eth0" --lxc-conf="lxc.network.flags = up" sensys \
        /opt/sensys/bin/orcmd --omca db_postgresq_uri db:4321 --omca db_postgres_user orcmuser:orcmpassword \
        --omca db_postgres_database sensys-db --omca sensor heartbeat,sigar

4. Launch nodes node001 - node256 (on first host)

    % docker run -d --name node001 -h node001 -v /root/hosts:/etc/hosts --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.1.0/16" --lxc-conf="lxc.network.link = docker0" \
        --lxc-conf="lxc.network.name = eth0" --lxc-conf="lxc.network.flags = up" sensys \
        /opt/sensys/bin/orcmd --omca sensor heartbeat,sigar
    ...
    % docker run -d --name node256 -h node256 -v /root/hosts:/etc/hosts --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.1.255/16" --lxc-conf="lxc.network.link = docker0" \
        --lxc-conf="lxc.network.name = eth0" --lxc-conf="lxc.network.flags = up" sensys \
        /opt/sensys/bin/orcmd --omca sensor heartbeat,sigar

5. Launch nodes node257 - node512 (on second host)

    % docker run -d --name node257 -h node257 -v /root/hosts:/etc/hosts --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.2.0/16" --lxc-conf="lxc.network.link = docker0" \
        --lxc-conf="lxc.network.name = eth0" --lxc-conf="lxc.network.flags = up" sensys \
        /opt/sensys/bin/orcmd --omca sensor heartbeat,sigar
    ...
    % docker run -d --name node512 -h node512 -v /root/hosts:/etc/hosts --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.2.255/16" --lxc-conf="lxc.network.link = docker0" \
        --lxc-conf="lxc.network.name = eth0" --lxc-conf="lxc.network.flags = up" sensys \
        /opt/sensys/bin/orcmd --omca sensor heartbeat,sigar

6. Launch interactive shell (simulating login node)

    % docker run -it --rm -v /root/hosts:/etc/hosts --lxc-conf="lxc.network.type = veth" \
        --lxc-conf="lxc.network.ipv4 = 172.16.255.1/16" --lxc-conf="lxc.network.link = docker0" \
        --lxc-conf="lxc.network.name = eth0" --lxc-conf="lxc.network.flags = up" sensys /bin/bash

Please notice that in the above examples, the -v options allows us to bind mount a directory or file into the container from the host. So if you would like to test a new orcm configuration file, you can add the option:

-v /path/to/orcm-site.xml:/opt/open-rcm/etc/orcm-site.xml

to the above docker commands. The hostnames, entries in the config file, and entries in /etc/hosts in the container all have to match up.

If you want a shared home directory across the virtual cluster, you could add:

-v /home:/home

to the shell and compute nodes and they would all share the common home directory.

1. Stop all running containers

    docker stop $(docker ps -a -q)

2. Remove containers

    docker rm $(docker ps -a -q)