.. |onos_version| replace:: 1.12.2 Installation Guide ****************** Building a Hardware Pod ======================= See :doc:`Supported Hardware ` for details. Install Controller - ONOS ========================= You could run Trellis with a `single instance of ONOS `_. But it is recommended to run ONOS as a cluster. The idea is to have a "build machine", where you host and build ONOS source code. But to launch it in operation you use other "target machines" (VMs, containers or servers). Typically we use `STC `_ to launch ONOS ins 3 target machines which form a ONOS cluster. Download ONOS |onos_version| --------------------------------- Trellis currently is released as part of ONOS and therefore it follows ONOS version number. This document is written based on ONOS |onos_version|. You can find more information about how to setup build environment and fetch ONOS source code from `Development Environment Setup `_. Prepare Your Target Machine --------------------------- Please refer to `target machines requirements `_ for details (you only have to do this once). You do not need to follow the directions for the "Mininet Target Machine" if you are using hardware switches. Create a Cell File -------------------- We need to compose a cell file in order to give the information about target machines to the build machine. Please refer to `test cells `_ for details. For example: .. code-block:: bash export ONOS_CELL=menlo export OC1=10.128.0.216 export OC2=10.128.0.217 export OC3=10.128.0.218 export ONOS_APPS=drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,hostprobingprovider,t3 export ONOS_USER=admin export ONOS_GROUP=admin export ONOS_USE_SSH=true export ONOS_WEB_USER=onos export ONOS_WEB_PASS=rocks - ``OC1, OC2, OC3`` are the IP addresses of the 3 target machines where the ONOS cluster will be deployed. - ``ONOS_APPS`` are the apps you want to automatically deploy at launch. - ``drivers`` includes drivers for various devices and pipelines. Always required. - ``gui`` enables graphic user interface. Highly recommended. - ``openflow`` is a meta app that loads ``openflow-base``, ``lldpprovider`` and ``hostprovider``. Always required. - ``segmentrouting`` controls forwarding in the fabric. Always required. - ``fpm`` (Forwarding Plane Manager) exchanges forwarding information with Quagga. Required if connecting to external router. - ``dhcprelay`` relays DHCP packets between clients and servers. Required if using DHCP to configure IP addresses for hosts - ``routeradvertisement`` periodically sends IPv6 router advertisement packets on configured interfaces. Required if using IPv6 - ``hostprobingprovider`` probes and verifies locations of dual-homed hosts. Required if using dual-homing feature with paired switches - ``t3`` (Trellis Troubleshooting Tool) is very useful for debugging. Highly recommended. - ``ONOS_USER`` is used to login to the target machines, and the password is not required as you have setup the target machines for password-less access over ssh. Once that's done, you can use the ``cell`` command to load the cell file. .. code-block:: console $ source onos/tools/dev/bash_profile $ cell menlo Check Your Target Environment ----------------------------- Check if your environment is OK with ``stc prerequisites`` (you only have to do this once). .. code-block:: console $ stc prerequisites 2017-02-07 12:10:23 Prerequisites started 2017-02-07 12:10:23 Check-Passwordless-Login-1 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.216 date 2017-02-07 12:10:23 Check-ONOS-Bits started -- onos-check-bits 2017-02-07 12:10:23 Check-Passwordless-Login-3 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.218 date 2017-02-07 12:10:23 Check-Passwordless-Login-2 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.217 date 2017-02-07 12:10:23 Check-Environment started -- test -n /Users/sauravdas/onos -a -n 10.128.0.* -a -n 10.128.0.216 2017-02-07 12:10:23 Check-Environment completed 2017-02-07 12:10:23 Check-Passwordless-Login-1 completed 2017-02-07 12:10:23 Check-Passwordless-Login-2 completed 2017-02-07 12:10:23 Check-Passwordless-Login-3 completed 2017-02-07 12:10:25 Check-ONOS-Bits completed 2017-02-07 12:10:25 Prerequisites completed 0:01 Passed! 6 steps succeeded Launch STC Command ------------------ Finally launch with ``stc setup`` once you have built ONOS. Here is what it looks like when I launch .. code-block:: console $ stc setup 2016-12-23 12:26:44 Setup started 2016-12-23 12:26:44 Uninstall-3 started -- onos-uninstall 10.128.0.218 2016-12-23 12:26:44 Uninstall-2 started -- onos-uninstall 10.128.0.217 2016-12-23 12:26:44 Uninstall-1 started -- onos-uninstall 10.128.0.216 2016-12-23 12:26:44 Push-Bits-2 started -- onos-push-bits 10.128.0.217 2016-12-23 12:26:44 Push-Bits-3 started -- onos-push-bits 10.128.0.218 2016-12-23 12:26:44 Push-Bits-1 started -- onos-push-bits 10.128.0.216 2016-12-23 12:26:45 Push-Bits-1 completed 2016-12-23 12:26:46 Push-Bits-2 completed 2016-12-23 12:26:46 Push-Bits-3 completed 2016-12-23 12:26:46 Uninstall-3 completed 2016-12-23 12:26:46 Kill-3 started -- onos-kill 10.128.0.218 2016-12-23 12:26:46 Uninstall-2 completed 2016-12-23 12:26:46 Kill-2 started -- onos-kill 10.128.0.217 2016-12-23 12:26:46 Uninstall-1 completed 2016-12-23 12:26:46 Kill-1 started -- onos-kill 10.128.0.216 2016-12-23 12:26:46 Kill-3 completed 2016-12-23 12:26:46 Install-3 started -- onos-install 10.128.0.218 2016-12-23 12:26:46 Kill-1 completed 2016-12-23 12:26:46 Install-1 started -- onos-install 10.128.0.216 2016-12-23 12:26:46 Kill-2 completed 2016-12-23 12:26:46 Install-2 started -- onos-install 10.128.0.217 2016-12-23 12:26:52 Install-2 completed 2016-12-23 12:26:52 Secure-SSH-2 started -- onos-secure-ssh -u onos -p rocks 10.128.0.217 2016-12-23 12:26:52 Install-1 completed 2016-12-23 12:26:52 Secure-SSH-1 started -- onos-secure-ssh -u onos -p rocks 10.128.0.216 2016-12-23 12:26:52 Install-3 completed 2016-12-23 12:26:52 Secure-SSH-3 started -- onos-secure-ssh -u onos -p rocks 10.128.0.218 2016-12-23 12:27:07 Secure-SSH-1 completed 2016-12-23 12:27:07 Wait-for-Start-1 started -- onos-wait-for-start 10.128.0.216 2016-12-23 12:27:09 Secure-SSH-3 completed 2016-12-23 12:27:09 Wait-for-Start-3 started -- onos-wait-for-start 10.128.0.218 2016-12-23 12:27:13 Secure-SSH-2 completed 2016-12-23 12:27:13 Wait-for-Start-2 started -- onos-wait-for-start 10.128.0.217 2016-12-23 12:27:14 Wait-for-Start-1 completed 2016-12-23 12:27:14 Check-Components-1 started -- onos-check-components 10.128.0.216 2016-12-23 12:27:14 Check-Nodes-1 started -- onos-check-nodes 10.128.0.216 2016-12-23 12:27:14 Wait-for-Start-3 completed 2016-12-23 12:27:14 Check-Nodes-3 started -- onos-check-nodes 10.128.0.218 2016-12-23 12:27:14 Check-Components-3 started -- onos-check-components 10.128.0.218 2016-12-23 12:27:16 Wait-for-Start-2 completed 2016-12-23 12:27:16 Check-Nodes-2 started -- onos-check-nodes 10.128.0.217 2016-12-23 12:27:16 Check-Components-2 started -- onos-check-components 10.128.0.217 2016-12-23 12:27:18 Check-Nodes-1 completed 2016-12-23 12:27:18 Check-Nodes-3 completed 2016-12-23 12:27:19 Check-Nodes-2 completed 2016-12-23 12:27:20 Check-Components-1 completed 2016-12-23 12:27:20 Check-Apps-1 started -- onos-check-apps 10.128.0.216 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes 2016-12-23 12:27:20 Check-Logs-1 started -- onos-check-logs 10.128.0.216 2016-12-23 12:27:20 Check-Components-3 completed 2016-12-23 12:27:20 Check-Apps-3 started -- onos-check-apps 10.128.0.218 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes 2016-12-23 12:27:20 Check-Logs-3 started -- onos-check-logs 10.128.0.218 2016-12-23 12:27:20 Check-Apps-1 completed 2016-12-23 12:27:20 Check-Apps-3 completed 2016-12-23 12:27:20 Check-Logs-1 completed 2016-12-23 12:27:20 Check-Logs-3 completed 2016-12-23 12:27:21 Check-Components-2 completed 2016-12-23 12:27:21 Check-Logs-2 started -- onos-check-logs 10.128.0.217 2016-12-23 12:27:21 Check-Apps-2 started -- onos-check-apps 10.128.0.217 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes 2016-12-23 12:27:21 Check-Apps-2 completed 2016-12-23 12:27:21 Check-Logs-2 completed 2016-12-23 12:27:21 Setup completed 0:36 Passed! 31 steps succeeded Useful Utilities ---------------- - Push network configuration to one particular instance of ONOS (all instances will automatically get the same configuration) .. code-block:: console $ onos-netcfg $OC1 network-cfg.json - Remotely log in to CLI of one particular ONOS instance .. code-block:: console $ onos $OC1 - Remotely access the log of one particular ONOS instance .. code-block:: console $ onos-log $OC1 Install Switch OS - ONL ======================= The switches listed in the :doc:`Supported Hardware ` are shipped with Open Networking Install Environment (ONIE) boot loader. After booting up, we should see the ONIE prompt from console. Here we assume that the management port on the switch already has Internet access. (via DHCP) Enter ONIE ---------- Has no ONL or Has ONL 1.x previously installed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The way to install ONL 2.x is the same as ONL 1.x. However, if you have no ONL installed or have older version of ONL, you might find it tricky to (re)install a newer version. Here's the instruction: 1. Plug in the console cable and reboot the switch 2. (If your boot loader is grub) When you see the boot menu, select **ONIE -> ONIE: Rescue** 3. (If your boot loader is uboot) When you see **Hit any key to stop autoboot** instead of the boot menu, press any key and then enter ``run onie_rescue`` to enter ONIE rescue mode. Has ONL 2.x previously installed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It would be much more easier to reboot into ONIE if you have a previously installed ONL 2. Simply run: .. code-block:: console # onl-onie-boot-mode rescue # reboot Available ONIE modes are: *install, rescue, uninstall, update, embed, diag, none*. This can be helpful when you are in a normal ONL and want to upgrade your system. Install ONL ----------- At the ONIE prompt, we need to download and install ONL. To be able to download ONL the management interface needs to have an IP. ONIE will try to obtain IP address from DHCP. In the case where DHCP service is not available, please configure it manually: .. code-block:: console ip addr add / dev route add default gw echo "nameserver 8.8.8.8" > /etc/resolv.conf To fetch and install the latest compatible ONL, run: .. code-block:: console ONIE:/ # wget https://github.com/opencord/OpenNetworkLinux/releases/download/2017-10-19.2200-1211610/ONL-2.0.0_ONL-OS_2017-10-19.2200-1211610_AMD64_INSTALLED_INSTALLER ONIE:/ # sh ONL-2.0.0_ONL-OS_2017-10-19.2200-1211610_AMD64_INSTALLED_INSTALLER **Checksum**: *sha256:2db316ea83f5dc761b9b11cc8542f153f092f3b49d82ffc0a36a2c41290f5421* The switch will automatically reboot into ONL after installation. Default login credential of ONL is: ``root/onl`` We might want to configure a fixed IP address for the management interface in ONL. First edit ``/etc/network/interfaces`` and configure ``ma1``, which is the management interface. .. code-block:: text auto ma1 iface ma1 inet static address 10.128.10.128 netmask 255.255.0.0 gateway 10.128.0.1 dns-nameservers 192.168.1.1 8.8.8.8 Install Switch Agent - OF-DPA ============================= .. note:: **Community vs. Premium Version** The OF-DPA image we distribute for free with Trellis is a community version. This documentation is also written based on the community version. The **community version** has most of the features available and therefore it is **good for small scale deployments** such as lab trials. However, we highly recommend you to get the **premium version** from Broadcom if you are aiming for **production deployments at scale**. Install OF-DPA -------------- We need to use different OF-DPA images depending on the switch model you are using. Please find the installer URL corresponding to each switch model in the `Vendor Specific Information`_ section. Copy the image to the switch and start the installation process by running: .. code-block:: console $ scp ${OFDPA_DEB} ${SWITCH_IP}:/root $ dpkg -i --force-overwrite /root/${OFDPA_DEB} For example, assuming the OF-DPA image is ``ofdpa_3.0.5.5+accton1.7-1_amd64.deb`` and the switch management IP is ``10.128.0.201``, you should run: .. code-block:: console $ scp ofdpa_3.0.5.5+accton1.7-1_amd64.deb 10.128.0.201:/root $ dpkg -i --force-overwrite /root/ofdpa_3.0.5.5+accton1.7-1_amd64.deb Connect Switch to Controller ============================ .. caution:: We are going to describe two different ways to start OF-DPA and OpenFlow agent in this section. Some vendors use ``ofagentd`` service while others use old ``launcher`` app to start the agent. Please check the `Vendor Specific Information`_ to understand which method you should use before continuing this section. Use ofagentd service -------------------- Launch with ofagentd ^^^^^^^^^^^^^^^^^^^^ The OFDPA software and the Indigo agent are now a single process launched by Linux service. First of all, we need to configure **/etc/ofagent/ofagent.conf**. Uncomment and edit the following line. Make sure all other lines stay commented. .. code-block:: bash OPT_ARGS="-t -t -t -i " We can choose to run OF-DPA in **debug mode** .. code-block:: bash OPT_ARGS="-d 2 -c 2 -c 4 -t -t -t -i " - To **start** the ofagent, run ``service ofagentd start`` - To **stop** the agent, run ``service ofagentd stop`` - More ``ofagentapp`` options can be found by running ``ofagentapp --help`` Launch in listen mode with ofagentd ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**. This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch. .. code-block:: bash OPT_ARGS="-t -t -t -i -l :" Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode. Use launcher app ---------------- Launch with launcher ^^^^^^^^^^^^^^^^^^^^ - To **start** OF-DPA and OpenFlow agent, run: .. code-block:: console # ./launcher ofagentapp -t -t -t -i - We can choose to run OF-DPA in **debug mode** .. code-block:: console # ./launcher ofagentapp -d 2 -c 2 -c 4 -t -t -t -i - To **stop** OF-DPA and OpenFlow agent, run: .. code-block:: console # killall ofagentapp Launch in listen mode with launcher ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**. This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch. .. code-block:: console # ./launcher ofagentapp -t -t -i -l : Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode. Vendor Specific Information =========================== Edgecore -------- - **OF-DPA Image** - `EdgeCore 5712-54X / 5812-54X / 6712-32X / 7712-32X `_ **Checksum**: *sha256:db228b6e79fb15f77497b59689235606b60abc157e72fc3356071bcc8dc4c01f* - **Start OF-DPA and OpenFlow agent** - Please refer to `Use ofagentd service`_ - **Configure Port Speed and Breakout** - By default all the switch ports are running at maximum speed. The port speed can be modified in **/etc/accton/ofdpa.conf** .. code-block:: text port_speed_1=1000 # configure front port 1 to run at 1G - We can also configure the same file to use break out cables on certain ports. .. code-block:: text port_mode_1=4x10g # configure front port 1 to break out from 40G to 4x10G OF-DPA service needs to be restarted after modifying this configuration. QCT --- - OF-DPA Image - `QuantaMesh T3048-LY8 `_ **Checksum**: *sha256:f8201530b1452145c1a0956ea1d3c0402c3568d090553d0d7b3c91a79137da9e* - `QuantaMesh T7032-IX1 / IX1B `_ **Checksum**: *sha256:278b8ffed8a8fc705a1b60d16f8e70377e78342a27a11568a1d80b1efd706a46* - **Start OF-DPA and OpenFlow agent** - Please refer to `Use launcher app`_ - Configure Port Speed and Breakout The command ``client_drivshell`` can be used to configure the port speed. Use the parameter ``ps`` to list all ports or ``ps `` to list one specific port. .. code-block:: console # client_drivshell ps 1 Calling ofdpaBcmCommand rpc with command = "ps 1 ". ena/ speed/ link auto STP lrn inter max loop port link duplex scan neg? state pause discrd ops face frame back xe0( 1) down 10G FD SW No Forward Untag F SFI 9412 Returned from ofdpaBcmCommand rpc with rc = 0. Then use the parameter ``port sp=`` to change the port speed. .. code-block:: console # client_drivshell port xe0 sp=1000 Calling ofdpaBcmCommand rpc with command = "port xe0 sp=1000 ". Returned from ofdpaBcmCommand rpc with rc = 0. # client_drivshell ps 1 Calling ofdpaBcmCommand rpc with command = "ps 1 ". ena/ speed/ link auto STP lrn inter max loop port link duplex scan neg? state pause discrd ops face frame back xe0( 1) down 1G FD SW No Forward Untag F GMII 9412 Returned from ofdpaBcmCommand rpc with rc = 0. Delta ----- - OF-DPA Image - `Delta AG7648 `_ **Checksum**: *sha256:ddfc13cb98ca47291dce5e6938b1d65f0b99bbe77f0585e36ac0007017397f23* - **Start OF-DPA and OpenFlow agent** - Please refer to `Use launcher app`_ - Special instructions to install ONL Make sure ``/etc/machine.conf`` looks like the following in ONIE before running ONL installation script: .. code-block:: text onie_platform=x86_64-delta_-r0 onie_machine=delta_ After the installation of ONL, if you don't see '/usr/bin' in your PATH variable, please run the following command: .. code-block:: console # export PATH=$PATH:/usr/bin Useful Information ================== OF-DPA Commands --------------- There are some useful OF-DPA commands under ``/usr/bin/`` .. code-block:: text client_cfg_purge client_debugcomp client_drivshell client_flowtable_dump client_meter_dump client_port_table_dump client_tunnel_dump client_classcolortable_dump client_debuglvl client_event client_grouptable_dump client_oam_dump client_queue_config ONL Boot Mode ------------- OFDPA offers two boot mode: **INSTALLED** and **SWI**. - In **INSTALLED** mode, ONL mounts the root filesystem from ``/dev/sdb7`` which is a flash drive that persists everything even during a power cycle. - In **SWI** mode, ONL will load the root filesystem from a read-only image (``*.swi``) and put the modified files in an copy-on-write overlay filesystem. The change will not be persisted during a power cycle. The pros and cons are obvious. INSTALLED mode is more convenient while SWI mode is more error-safe. The boot mode will be determined by which image we used install ONL. We use the **INSTALLED** mode in this instruction for simplicity. But we can still change it after the installation. Here's the instruction: .. code-block:: console # mount /mnt/onl/boot -o remount,rw # sed -i 's/BOOTMODE=INSTALLED/BOOTMODE=SWI/g' /mnt/onl/boot/boot-config # reboot Persistence Mechanism --------------------- INSTALLED mode already persists everything for you. This section is mainly for SWI mode. The mechanism to persist files on the switch is different from ONL 1.x to ONL 2.x. Following are the steps required to persist a file (e.g. OFDPA package) in ONL 2.x. - Files put in **/mnt/onl/data** will be persisted automatically. - To install a deb package during start-up, put the deb file under **/mnt/onl/data/install-debs** folder and create a plain text file **/mnt/onl/data/install-debs/list** with all the filename of the deb (e.g. ``ofdpa_3.0.5.5+accton1.7-1_amd64.deb``) - To execute some command during start-up, put the command in **/mnt/onl/data/rc.boot** .. code-block:: bash # set static ip and route (dhcp by default) echo 'ip addr add 10.128.0.203/16 dev ma1' >> /mnt/onl/data/rc.boot echo 'ip route add default via 10.128.0.1' >> /mnt/onl/data/rc.boot # grant executable permission chmod a+x /mnt/onl/data/rc.boot See `PersistWorkflow.md `_ for more detail. Build a Customized ONL Image ---------------------------- Sometimes we need to build our own ONL image from source to include some extra files. (The most common case is a modified /etc/network/interfaces). The instruction of build process has already been well-documented in ONL repo `Building.md `_. But you might find it a little bit tricky to inject some extra files. To do that, you need to put the files under **$ONL_ROOT/builds/any/rootfs/jessie/common/overlay/** and then run the make command again. Recovery from a Faulty ONL Install ---------------------------------- If, for some reason, the ONL install process fails, you may be brought to the grub rescue prompt upon reboot. You may or may not find system files, or even basic grub rescue commands (i.e. 'help'). There are two options for returning to a (known working) ONIE prompt. 1. **Reinstaller** - You should be able to acquire an ISO image for resetting the firmware to factory default from your switch vendor. Once you have the ISO, you can either use PXE or a live USB to boot the switch into the image. 2. **Manually boot to ONIE** - If the files are there, you can manually configure GRUB to load the necessary modules to boot back into ONIE. .. code-block:: console # This should be the partition on the ONIE partition on the AS6712. # It might be hd1 or hd0. Change appropriately. grub rescue> ls (hd0,gpt2)/ ./ ../ lost+found/ onie/ grub/ grubenv # These instructions will load the kernel and ONIE initrd. grub rescue> set prefix=(hd0,gpt2)/grub grub rescue> set root=(hd0,gpt2) grub rescue> insmod normal grub rescue> insmod linux grub rescue> ls /onie ./ ../ vmlinuz-3.2.35-onie initrd.img-3.2.35-onie tools/ grub/ grub.d/ config/ # if the ",115200n8" is omitted, baud rate defaults to 9600. grub rescue> linux /onie/vmlinuz-3.2.35-onie console=tty0 console=ttyS1,115200n8 grub rescue> initrd /onie/initrd.img-3.2.35-onie # The system will not boot into ONIE and you can recover or re-install from there. grub rescue> boot Once back at the ONIE prompt, you can try to install ONL by following the same steps again. Connect to Switch in Listen Mode -------------------------------- To connect to the switch in Listen Mode using **ovs-ofctl**, you can use commands such as: .. code-block:: console $ ovs-ofctl -O OpenFlow13 show tcp:: $ ovs-ofctl -O OpenFlow13 dump-flows tcp:: $ ovs-ofctl -O OpenFlow13 dump-groups tcp:: $ ovs-ofctl -O OpenFlow13 add-flow tcp:: table=60,priority=40000,eth_type=0x0800,ip_dst=55.55.55.55,actions=controller For more command, please see ``ovs-ofctl --help`` or the post `OpenvSwitch ovs-ofctl and OF-DPA `_. Reference --------- - `ONL/OF-DPA installation guide by Edgecore Networks `_ - `ONL/OF-DPA cheat sheet by Phil Huang `_