Installation Guide

Building a Hardware Pod

See Supported Hardware for details.

Install Controller - ONOS

You could run Trellis with a single instance of ONOS. But it is recommended to run ONOS as a cluster.

The idea is to have a “build machine”, where you host and build ONOS source code. But to launch it in operation you use other “target machines” (VMs, containers or servers).

Typically we use STC to launch ONOS ins 3 target machines which form a ONOS cluster.

Download ONOS 1.12.2

Trellis currently is released as part of ONOS and therefore it follows ONOS version number. This document is written based on ONOS 1.12.2.

You can find more information about how to setup build environment and fetch ONOS source code from Development Environment Setup.

Prepare Your Target Machine

Please refer to target machines requirements for details (you only have to do this once).

You do not need to follow the directions for the “Mininet Target Machine” if you are using hardware switches.

Create a Cell File

We need to compose a cell file in order to give the information about target machines to the build machine. Please refer to test cells for details. For example:

export ONOS_CELL=menlo
export OC1=10.128.0.216
export OC2=10.128.0.217
export OC3=10.128.0.218
export ONOS_APPS=drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,hostprobingprovider,t3
export ONOS_USER=admin
export ONOS_GROUP=admin
export ONOS_USE_SSH=true
export ONOS_WEB_USER=onos
export ONOS_WEB_PASS=rocks

  • OC1, OC2, OC3 are the IP addresses of the 3 target machines where the ONOS cluster will be deployed.

  • ONOS_APPS are the apps you want to automatically deploy at launch.

    • drivers includes drivers for various devices and pipelines. Always required.

    • gui enables graphic user interface. Highly recommended.

    • openflow is a meta app that loads openflow-base, lldpprovider and hostprovider. Always required.

    • segmentrouting controls forwarding in the fabric. Always required.

    • fpm (Forwarding Plane Manager) exchanges forwarding information with Quagga. Required if connecting to external router.

    • dhcprelay relays DHCP packets between clients and servers. Required if using DHCP to configure IP addresses for hosts

    • routeradvertisement periodically sends IPv6 router advertisement packets on configured interfaces. Required if using IPv6

    • hostprobingprovider probes and verifies locations of dual-homed hosts. Required if using dual-homing feature with paired switches

    • t3 (Trellis Troubleshooting Tool) is very useful for debugging. Highly recommended.

  • ONOS_USER is used to login to the target machines, and the password is not required as you have setup the target machines for password-less access over ssh.

Once that’s done, you can use the cell command to load the cell file.

$ source onos/tools/dev/bash_profile
$ cell menlo

Check Your Target Environment

Check if your environment is OK with stc prerequisites (you only have to do this once).

$ stc prerequisites
2017-02-07 12:10:23  Prerequisites started
2017-02-07 12:10:23  Check-Passwordless-Login-1 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.216 date
2017-02-07 12:10:23  Check-ONOS-Bits started -- onos-check-bits
2017-02-07 12:10:23  Check-Passwordless-Login-3 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.218 date
2017-02-07 12:10:23  Check-Passwordless-Login-2 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.217 date
2017-02-07 12:10:23  Check-Environment started -- test -n /Users/sauravdas/onos -a -n 10.128.0.* -a -n 10.128.0.216
2017-02-07 12:10:23  Check-Environment completed
2017-02-07 12:10:23  Check-Passwordless-Login-1 completed
2017-02-07 12:10:23  Check-Passwordless-Login-2 completed
2017-02-07 12:10:23  Check-Passwordless-Login-3 completed
2017-02-07 12:10:25  Check-ONOS-Bits completed
2017-02-07 12:10:25  Prerequisites completed
0:01 Passed! 6 steps succeeded

Launch STC Command

Finally launch with stc setup once you have built ONOS. Here is what it looks like when I launch

$ stc setup
2016-12-23 12:26:44  Setup started
2016-12-23 12:26:44  Uninstall-3 started -- onos-uninstall 10.128.0.218
2016-12-23 12:26:44  Uninstall-2 started -- onos-uninstall 10.128.0.217
2016-12-23 12:26:44  Uninstall-1 started -- onos-uninstall 10.128.0.216
2016-12-23 12:26:44  Push-Bits-2 started -- onos-push-bits 10.128.0.217
2016-12-23 12:26:44  Push-Bits-3 started -- onos-push-bits 10.128.0.218
2016-12-23 12:26:44  Push-Bits-1 started -- onos-push-bits 10.128.0.216
2016-12-23 12:26:45  Push-Bits-1 completed
2016-12-23 12:26:46  Push-Bits-2 completed
2016-12-23 12:26:46  Push-Bits-3 completed
2016-12-23 12:26:46  Uninstall-3 completed
2016-12-23 12:26:46  Kill-3 started -- onos-kill 10.128.0.218
2016-12-23 12:26:46  Uninstall-2 completed
2016-12-23 12:26:46  Kill-2 started -- onos-kill 10.128.0.217
2016-12-23 12:26:46  Uninstall-1 completed
2016-12-23 12:26:46  Kill-1 started -- onos-kill 10.128.0.216
2016-12-23 12:26:46  Kill-3 completed
2016-12-23 12:26:46  Install-3 started -- onos-install 10.128.0.218
2016-12-23 12:26:46  Kill-1 completed
2016-12-23 12:26:46  Install-1 started -- onos-install 10.128.0.216
2016-12-23 12:26:46  Kill-2 completed
2016-12-23 12:26:46  Install-2 started -- onos-install 10.128.0.217
2016-12-23 12:26:52  Install-2 completed
2016-12-23 12:26:52  Secure-SSH-2 started -- onos-secure-ssh -u onos -p rocks 10.128.0.217
2016-12-23 12:26:52  Install-1 completed
2016-12-23 12:26:52  Secure-SSH-1 started -- onos-secure-ssh -u onos -p rocks 10.128.0.216
2016-12-23 12:26:52  Install-3 completed
2016-12-23 12:26:52  Secure-SSH-3 started -- onos-secure-ssh -u onos -p rocks 10.128.0.218
2016-12-23 12:27:07  Secure-SSH-1 completed
2016-12-23 12:27:07  Wait-for-Start-1 started -- onos-wait-for-start 10.128.0.216
2016-12-23 12:27:09  Secure-SSH-3 completed
2016-12-23 12:27:09  Wait-for-Start-3 started -- onos-wait-for-start 10.128.0.218
2016-12-23 12:27:13  Secure-SSH-2 completed
2016-12-23 12:27:13  Wait-for-Start-2 started -- onos-wait-for-start 10.128.0.217
2016-12-23 12:27:14  Wait-for-Start-1 completed
2016-12-23 12:27:14  Check-Components-1 started -- onos-check-components 10.128.0.216
2016-12-23 12:27:14  Check-Nodes-1 started -- onos-check-nodes 10.128.0.216
2016-12-23 12:27:14  Wait-for-Start-3 completed
2016-12-23 12:27:14  Check-Nodes-3 started -- onos-check-nodes 10.128.0.218
2016-12-23 12:27:14  Check-Components-3 started -- onos-check-components 10.128.0.218
2016-12-23 12:27:16  Wait-for-Start-2 completed
2016-12-23 12:27:16  Check-Nodes-2 started -- onos-check-nodes 10.128.0.217
2016-12-23 12:27:16  Check-Components-2 started -- onos-check-components 10.128.0.217
2016-12-23 12:27:18  Check-Nodes-1 completed
2016-12-23 12:27:18  Check-Nodes-3 completed
2016-12-23 12:27:19  Check-Nodes-2 completed
2016-12-23 12:27:20  Check-Components-1 completed
2016-12-23 12:27:20  Check-Apps-1 started -- onos-check-apps 10.128.0.216 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
2016-12-23 12:27:20  Check-Logs-1 started -- onos-check-logs 10.128.0.216
2016-12-23 12:27:20  Check-Components-3 completed
2016-12-23 12:27:20  Check-Apps-3 started -- onos-check-apps 10.128.0.218 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
2016-12-23 12:27:20  Check-Logs-3 started -- onos-check-logs 10.128.0.218
2016-12-23 12:27:20  Check-Apps-1 completed
2016-12-23 12:27:20  Check-Apps-3 completed
2016-12-23 12:27:20  Check-Logs-1 completed
2016-12-23 12:27:20  Check-Logs-3 completed
2016-12-23 12:27:21  Check-Components-2 completed
2016-12-23 12:27:21  Check-Logs-2 started -- onos-check-logs 10.128.0.217
2016-12-23 12:27:21  Check-Apps-2 started -- onos-check-apps 10.128.0.217 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
2016-12-23 12:27:21  Check-Apps-2 completed
2016-12-23 12:27:21  Check-Logs-2 completed
2016-12-23 12:27:21  Setup completed
0:36 Passed! 31 steps succeeded

Useful Utilities

  • Push network configuration to one particular instance of ONOS (all instances will automatically get the same configuration)

    $ onos-netcfg $OC1 network-cfg.json

  • Remotely log in to CLI of one particular ONOS instance

    $ onos $OC1

  • Remotely access the log of one particular ONOS instance

    $ onos-log $OC1

Install Switch OS - ONL

The switches listed in the Supported Hardware are shipped with Open Networking Install Environment (ONIE) boot loader.

After booting up, we should see the ONIE prompt from console.

Here we assume that the management port on the switch already has Internet access. (via DHCP)

Enter ONIE

Has no ONL or Has ONL 1.x previously installed

The way to install ONL 2.x is the same as ONL 1.x.

However, if you have no ONL installed or have older version of ONL, you might find it tricky to (re)install a newer version.

Here’s the instruction:

  1. Plug in the console cable and reboot the switch

  2. (If your boot loader is grub) When you see the boot menu, select ONIE -> ONIE: Rescue

  3. (If your boot loader is uboot) When you see Hit any key to stop autoboot instead of the boot menu, press any key and then enter run onie_rescue to enter ONIE rescue mode.

Has ONL 2.x previously installed

It would be much more easier to reboot into ONIE if you have a previously installed ONL 2. Simply run:

# onl-onie-boot-mode rescue
# reboot

Available ONIE modes are: install, rescue, uninstall, update, embed, diag, none. This can be helpful when you are in a normal ONL and want to upgrade your system.

Install ONL

At the ONIE prompt, we need to download and install ONL. To be able to download ONL the management interface needs to have an IP. ONIE will try to obtain IP address from DHCP. In the case where DHCP service is not available, please configure it manually:

ip addr add  <IP>/<Netmask> dev <interface>
route add default gw <Gateway>
echo "nameserver 8.8.8.8" > /etc/resolv.conf

To fetch and install the latest compatible ONL, run:

ONIE:/ # wget https://github.com/opencord/OpenNetworkLinux/releases/download/2017-10-19.2200-1211610/ONL-2.0.0_ONL-OS_2017-10-19.2200-1211610_AMD64_INSTALLED_INSTALLER
ONIE:/ # sh ONL-2.0.0_ONL-OS_2017-10-19.2200-1211610_AMD64_INSTALLED_INSTALLER

Checksum: sha256:2db316ea83f5dc761b9b11cc8542f153f092f3b49d82ffc0a36a2c41290f5421

The switch will automatically reboot into ONL after installation. Default login credential of ONL is: root/onl

We might want to configure a fixed IP address for the management interface in ONL.

First edit /etc/network/interfaces and configure ma1, which is the management interface.

auto ma1
iface ma1 inet static
address 10.128.10.128
netmask 255.255.0.0
gateway 10.128.0.1
dns-nameservers 192.168.1.1 8.8.8.8

Install Switch Agent - OF-DPA

Note

Community vs. Premium Version

The OF-DPA image we distribute for free with Trellis is a community version.

This documentation is also written based on the community version.

The community version has most of the features available and therefore it is good for small scale deployments such as lab trials.

However, we highly recommend you to get the premium version from Broadcom if you are aiming for production deployments at scale.

Install OF-DPA

We need to use different OF-DPA images depending on the switch model you are using.

Please find the installer URL corresponding to each switch model in the Vendor Specific Information section.

Copy the image to the switch and start the installation process by running:

$ scp ${OFDPA_DEB} ${SWITCH_IP}:/root
$ dpkg -i --force-overwrite /root/${OFDPA_DEB}

For example, assuming the OF-DPA image is ofdpa_3.0.5.5+accton1.7-1_amd64.deb and the switch management IP is 10.128.0.201, you should run:

$ scp ofdpa_3.0.5.5+accton1.7-1_amd64.deb 10.128.0.201:/root
$ dpkg -i --force-overwrite /root/ofdpa_3.0.5.5+accton1.7-1_amd64.deb

Connect Switch to Controller

Caution

We are going to describe two different ways to start OF-DPA and OpenFlow agent in this section.

Some vendors use ofagentd service while others use old launcher app to start the agent.

Please check the Vendor Specific Information to understand which method you should use before continuing this section.

Use ofagentd service

Launch with ofagentd

The OFDPA software and the Indigo agent are now a single process launched by Linux service. First of all, we need to configure /etc/ofagent/ofagent.conf. Uncomment and edit the following line. Make sure all other lines stay commented.

OPT_ARGS="-t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>"

We can choose to run OF-DPA in debug mode

OPT_ARGS="-d 2 -c 2 -c 4 -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>"

  • To start the ofagent, run service ofagentd start

  • To stop the agent, run service ofagentd stop

  • More ofagentapp options can be found by running ofagentapp --help

Launch in listen mode with ofagentd

An optional -l parameter can be added for switch to listen on specific IP:port.

This allows us to use tools like ovs-ofctl or dpctl to connect and control the switch.

OPT_ARGS="-t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>"

Please refer to Connect to Switch in Listen Mode to learn more about how to access switches in listen mode.

Use launcher app

Launch with launcher

  • To start OF-DPA and OpenFlow agent, run:

    # ./launcher ofagentapp -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>

  • We can choose to run OF-DPA in debug mode

    # ./launcher ofagentapp -d 2 -c 2 -c 4 -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>

  • To stop OF-DPA and OpenFlow agent, run:

    # killall ofagentapp

Launch in listen mode with launcher

An optional -l parameter can be added for switch to listen on specific IP:port.

This allows us to use tools like ovs-ofctl or dpctl to connect and control the switch.

# ./launcher ofagentapp <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>

Please refer to Connect to Switch in Listen Mode to learn more about how to access switches in listen mode.

Vendor Specific Information

Edgecore

  • OF-DPA Image

  • Start OF-DPA and OpenFlow agent

  • Configure Port Speed and Breakout

    • By default all the switch ports are running at maximum speed. The port speed can be modified in /etc/accton/ofdpa.conf

      port_speed_1=1000   # configure front port 1 to run at 1G

    • We can also configure the same file to use break out cables on certain ports.

      port_mode_1=4x10g    # configure front port 1 to break out from 40G to 4x10G

    OF-DPA service needs to be restarted after modifying this configuration.

QCT

  • OF-DPA Image

  • Start OF-DPA and OpenFlow agent

  • Configure Port Speed and Breakout

    The command client_drivshell can be used to configure the port speed.

    Use the parameter ps to list all ports or ps <port number> to list one specific port.

    # client_drivshell ps 1
Calling ofdpaBcmCommand rpc with command = "ps 1 ".
                 ena/    speed/ link auto    STP                  lrn  inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
       xe0(  1)  down   10G  FD   SW  No   Forward         Untag    F    SFI  9412

Returned from ofdpaBcmCommand rpc with rc = 0.

    Then use the parameter port <port number> sp=<port speed> to change the port speed.

    # client_drivshell port xe0 sp=1000
Calling ofdpaBcmCommand rpc with command = "port xe0 sp=1000 ".

Returned from ofdpaBcmCommand rpc with rc = 0.

# client_drivshell ps 1
Calling ofdpaBcmCommand rpc with command = "ps 1 ".
                 ena/    speed/ link auto    STP                  lrn  inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
       xe0(  1)  down    1G  FD   SW  No   Forward         Untag    F   GMII  9412

Returned from ofdpaBcmCommand rpc with rc = 0.

Delta

  • OF-DPA Image

    • Delta AG7648 Checksum: sha256:ddfc13cb98ca47291dce5e6938b1d65f0b99bbe77f0585e36ac0007017397f23

  • Start OF-DPA and OpenFlow agent

  • Special instructions to install ONL

    Make sure /etc/machine.conf looks like the following in ONIE before running ONL installation script:

    onie_platform=x86_64-delta_<platform name>-r0
onie_machine=delta_<platform name>

    After the installation of ONL, if you don’t see ‘/usr/bin’ in your PATH variable, please run the following command:

    # export PATH=$PATH:/usr/bin

Useful Information

OF-DPA Commands

There are some useful OF-DPA commands under /usr/bin/

client_cfg_purge
client_debugcomp
client_drivshell
client_flowtable_dump
client_meter_dump
client_port_table_dump
client_tunnel_dump
client_classcolortable_dump
client_debuglvl
client_event
client_grouptable_dump
client_oam_dump
client_queue_config

ONL Boot Mode

OFDPA offers two boot mode: INSTALLED and SWI.

  • In INSTALLED mode, ONL mounts the root filesystem from /dev/sdb7 which is a flash drive that persists everything even during a power cycle.

  • In SWI mode, ONL will load the root filesystem from a read-only image (*.swi) and put the modified files in an copy-on-write overlay filesystem.

    The change will not be persisted during a power cycle.

    The pros and cons are obvious. INSTALLED mode is more convenient while SWI mode is more error-safe.

The boot mode will be determined by which image we used install ONL. We use the INSTALLED mode in this instruction for simplicity. But we can still change it after the installation. Here’s the instruction:

# mount /mnt/onl/boot -o remount,rw
# sed -i 's/BOOTMODE=INSTALLED/BOOTMODE=SWI/g' /mnt/onl/boot/boot-config
# reboot

Persistence Mechanism

INSTALLED mode already persists everything for you. This section is mainly for SWI mode.

The mechanism to persist files on the switch is different from ONL 1.x to ONL 2.x. Following are the steps required to persist a file (e.g. OFDPA package) in ONL 2.x.

  • Files put in /mnt/onl/data will be persisted automatically.

  • To install a deb package during start-up, put the deb file under /mnt/onl/data/install-debs folder and create a plain text file /mnt/onl/data/install-debs/list with all the filename of the deb (e.g. ofdpa_3.0.5.5+accton1.7-1_amd64.deb)

  • To execute some command during start-up, put the command in /mnt/onl/data/rc.boot

    # set static ip and route (dhcp by default)
echo 'ip addr add 10.128.0.203/16 dev ma1' >> /mnt/onl/data/rc.boot
echo 'ip route add default via 10.128.0.1' >> /mnt/onl/data/rc.boot
# grant executable permission
chmod a+x /mnt/onl/data/rc.boot

See PersistWorkflow.md for more detail.

Build a Customized ONL Image

Sometimes we need to build our own ONL image from source to include some extra files. (The most common case is a modified /etc/network/interfaces).

The instruction of build process has already been well-documented in ONL repo Building.md.

But you might find it a little bit tricky to inject some extra files.

To do that, you need to put the files under $ONL_ROOT/builds/any/rootfs/jessie/common/overlay/ and then run the make command again.

Recovery from a Faulty ONL Install

If, for some reason, the ONL install process fails, you may be brought to the grub rescue prompt upon reboot.

You may or may not find system files, or even basic grub rescue commands (i.e. ‘help’).

There are two options for returning to a (known working) ONIE prompt.

  1. Reinstaller - You should be able to acquire an ISO image for resetting the firmware to factory default from your switch vendor. Once you have the ISO, you can either use PXE or a live USB to boot the switch into the image.

  2. Manually boot to ONIE - If the files are there, you can manually configure GRUB to load the necessary modules to boot back into ONIE.

    # This should be the partition on the ONIE partition on the AS6712.
# It might be hd1 or hd0. Change appropriately.
grub rescue> ls (hd0,gpt2)/
./ ../ lost+found/ onie/ grub/ grubenv

# These instructions will load the kernel and ONIE initrd.
grub rescue> set prefix=(hd0,gpt2)/grub
grub rescue> set root=(hd0,gpt2)
grub rescue> insmod normal
grub rescue> insmod linux
grub rescue> ls /onie
./ ../ vmlinuz-3.2.35-onie initrd.img-3.2.35-onie tools/ grub/ grub.d/ config/

# if the ",115200n8" is omitted, baud rate defaults to 9600.
grub rescue> linux /onie/vmlinuz-3.2.35-onie console=tty0 console=ttyS1,115200n8
grub rescue> initrd /onie/initrd.img-3.2.35-onie

# The system will not boot into ONIE and you can recover or re-install from there.
grub rescue> boot

Once back at the ONIE prompt, you can try to install ONL by following the same steps again.

Connect to Switch in Listen Mode

To connect to the switch in Listen Mode using ovs-ofctl, you can use commands such as:

$ ovs-ofctl -O OpenFlow13 show tcp:<ip>:<port>
$ ovs-ofctl -O OpenFlow13 dump-flows tcp:<ip>:<port>
$ ovs-ofctl -O OpenFlow13 dump-groups tcp:<ip>:<port>
$ ovs-ofctl -O OpenFlow13 add-flow tcp:<ip>:<port> table=60,priority=40000,eth_type=0x0800,ip_dst=55.55.55.55,actions=controller

For more command, please see ovs-ofctl --help or the post OpenvSwitch ovs-ofctl and OF-DPA.

Reference