Troubleshooting Guide¶
Overview¶
The Trellis Troubleshooting Tools (T3) is an ONOS application intended to help you troubleshoot traffic-related problems in your network by tracing installed flows and groups.
T3 works both on simulated OVS-based topologies but also on real HW, basing it’s internal decisions on the driver assigned to the device.
T3 also currently supports dual links between leaf and spine and properly handles the difference between deferred and immediate rules. Dual Homed hosts are also supported in T3.
To activate this tool on your ONOS environment, you need to run:
onos> app activate t3
T3 supports tracing of following packet types:
L2 Unicast
L2 Broadcast
L3 Unicast (both IPv4 and IPv6)
L3 Multicast (both IPv4 and IPv6)
LLDP and BDDP
ARP
DHCP
Network Information Base (NIB)¶
T3 analyzes network states that are stored in the NIB. A NIB is a collection of snapshots captured from various ONOS stores such as flows, groups, links…etc. The NIB, acting as a cache, allows T3 troubleshooting logic to avoid massive interaction with live ONOS subsystems, enabling near-runtime or postmortem analysis in a production environment.
There are two methods of loading the network snapshots into the NIB: (1) from a live running system or (2) from a onos-diagnostics file dump. Users can choose one load command before troubleshooting commands, or can run troubleshooting commands alone (then the NIB is automatically loaded via (1)).
t3-load-file¶
T3 offers this command to fill the NIB with files that represent snapshots of the network states in a certain time.
t3-load-file Help¶
onos> t3-load-file --help
DESCRIPTION
onos:t3-load-file
Load the NIB with onos-diagnostics dump files
SYNTAX
onos:t3-load-file [options] rootDir
ARGUMENTS
rootDir
Specify the location of the directory where the previous onos-diagnostics command has extracted its dump files (e.g. /tmp/onos-diags/127.0.0.1)
OPTIONS
--help
Display this help message
-j, --json
Output JSON
t3-load-file Example¶
User is expected to run the tool onos-diagnostics
1 2
to generate target files first.
user$ onos-diagnostics -xj [IP_ONOS_INSTANCE_1 IP_ONOS_INSTANCE_2 ...]
Collecting diagnostics on [IP_ONOS_INSTANCE_1]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done.
Collecting diagnostics on [IP_ONOS_INSTANCE_2]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done.
Then, execute this command with the path of the location where those JSON files are extracted. User can check the creation time and a summary of the NIB contents.
onos> t3-load-file /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]
Load target files in: /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]/
FlowNib created 03-06-2020 01:30 from FILE
GroupNib created 03-06-2020 01:30 from FILE
LinkNib created 03-06-2020 01:30 from FILE
HostNib created 03-06-2020 01:30 from FILE
DeviceNib created 03-06-2020 01:30 from FILE
DriverNib created 03-06-2020 01:30 from FILE
MastershipNib created 03-06-2020 01:30 from FILE
EdgePortNib created 03-06-2020 01:30 from FILE
RouteNib created 03-06-2020 01:30 from FILE
NetworkConfigNib created 03-06-2020 01:30 from FILE
MulticastRouteNib created 03-06-2020 01:30 from FILE
Alternatively, it is possible to use onos-diagnostics-k8s
3 2
in Kubernetes enabled environments.
t3-load-snapshot¶
T3 offers this command to fill the NIB with snapshots of the network states at the execution time.
t3-load-snapshot Help¶
onos> t3-load-snapshot --help
DESCRIPTION
onos:t3-load-snapshot
Load the NIB with the network states stored in the ONOS instance where the T3 is running
SYNTAX
onos:t3-load-snapshot [options]
OPTIONS
--help
Display this help message
-j, --json
Output JSON
t3-load-snapshot Example¶
User can check the creation time and a summary of the NIB contents. SNAPSHOT indicates that the source of the current NIB is the ONOS instance where the T3 is running.
onos> t3-load-snapshot
Load current network states from ONOS stores
FlowNib created 03-06-2020 01:40 from SNAPSHOT
GroupNib created 03-06-2020 01:40 from SNAPSHOT
LinkNib created 03-06-2020 01:40 from SNAPSHOT
HostNib created 03-06-2020 01:40 from SNAPSHOT
DeviceNib created 03-06-2020 01:40 from SNAPSHOT
DriverNib created 03-06-2020 01:40 from SNAPSHOT
MastershipNib created 03-06-2020 01:40 from SNAPSHOT
EdgePortNib created 03-06-2020 01:40 from SNAPSHOT
RouteNib created 03-06-2020 01:40 from SNAPSHOT
NetworkConfigNib created 03-06-2020 01:40 from SNAPSHOT
MulticastRouteNib created 03-06-2020 01:40 from SNAPSHOT
If one of T3 commands below is used before loading the NIB from SNAPSHOT or FILE (so the NIB is empty), the command internally executes t3-load-snapshot to proceed with auto-filled NIB.
onos> t3-troubleshoot-simple 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None
*** NIB is invalid. Snapshots for the NIB have been auto-filled: ***
Load current network states from ONOS stores
FlowNib created 03-06-2020 01:44 from SNAPSHOT
GroupNib created 03-06-2020 01:44 from SNAPSHOT
LinkNib created 03-06-2020 01:44 from SNAPSHOT
...
Tracing between: 00:AA:00:00:00:01/None and 00:AA:00:00:00:02/None
Tracing Packet: DefaultTrafficSelector{criteria=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32]}
Paths
[of:0000000000000204/3, of:0000000000000204/4]
Result:
Reached required destination Host of:0000000000000204/4
t3-troubleshoot¶
As of right now the tools provides a CLI command (troubleshoot).
As arguments to the CLI command you have to pass arguments describing the packet you want to trace through the network.
t3-troubleshoot Help¶
onos> t3-troubleshoot --help
DESCRIPTION
onos:troubleshoot
troubleshoots flows and groups between source and destination
SYNTAX
onos:troubleshoot [options]
OPTIONS
-sm, --srcMac
Source MAC
-j, --json
Output JSON
--help
Display this help message
-v, --verbose
Outputs complete path
-ipp, --ipProto
IP Proto
-s, --srcIp
Source IP
-dm, --dstMac
Destination MAC
-ml, --mplsLabel
Mpls label of incoming packet
-vv, --veryverbose
Outputs flows and groups for every device
-udps, --udpSrc
UDP Source
-mb, --mplsBos
MPLS BOS
-et, --ethType
ETH Type
(defaults to ipv4)
-sp, --srcPort
Source Port
-dtp, --dstTcpPort
destination TCP Port
-udpd, --udpDst
UDP Destination
-stp, --srcTcpPort
Source TCP Port
-d, --dstIp
Destination IP
-vid, --vlanId
Vlan of incoming packet
(defaults to None)
Default Verbosity Level¶
Let’s look at an example.
onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
with this command we are asking the t3 tool to trace a packet with
Source IP of
10.0.2.1/32
Input port
of:0000000000000204/3
Source MAC
00:AA:00:00:00:01
ethType
IPv4
Source TCP Port
12345
Destination IP
10.0.3.1/32
Destination MAC
00:AA:00:00:00:01
Destination TCP Port
80
andVLAN ID
None
This command has a verbosity level of 0 and will output just the paths that the packet might take inside the network. In the example deployment taken from trellis.py 4, the output will be:
Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Paths
[of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
[of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
Result:
Reached required destination Host
Reached required destination Host
Middle Verbosity Level¶
There are also two other verbosity levels.
onos> t3-troubleshoot -v -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
prints flows selectors and groupId that are met along the path
Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
flowId=a20000bd57cb49, selector=[]
flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
flowId=a2000038b487e7, selector=[]
Groups
groupId=GroupId{id=0x70000010}
groupId=GroupId{id=0x9200000d}
groupId=GroupId{id=0x9000000c}
groupId=GroupId{id=0xffe0002}
groupId=GroupId{id=0x9200000f}
groupId=GroupId{id=0x9000000e}
groupId=GroupId{id=0xffe0001}
Output through of:0000000000000204/2
Device of:0000000000000227
Input from of:0000000000000227/1
Flows
flowId=a200005577bcd6, selector=[]
flowId=900000498457b2, selector=[IN_PORT:1, VLAN_VID:None]
flowId=900000f87746c9, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
flowId=a200005bfd6ed7, selector=[]
flowId=9000009f3d554f, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
flowId=a20000e601fc35, selector=[]
Groups
groupId=GroupId{id=0x70000017}
groupId=GroupId{id=0x20000016}
groupId=GroupId{id=0xffe0002}
Output through of:0000000000000227/2
Device of:0000000000000205
Input from of:0000000000000205/2
Flows
flowId=a200008d542817, selector=[]
flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
flowId=a2000042d8e6b5, selector=[]
Groups
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
Output through of:0000000000000205/3
Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
flowId=a20000bd57cb49, selector=[]
flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
flowId=a2000038b487e7, selector=[]
Groups
groupId=GroupId{id=0x70000010}
groupId=GroupId{id=0x9200000d}
groupId=GroupId{id=0x9000000c}
groupId=GroupId{id=0xffe0002}
groupId=GroupId{id=0x9200000f}
groupId=GroupId{id=0x9000000e}
groupId=GroupId{id=0xffe0001}
Output through of:0000000000000204/1
Device of:0000000000000226
Input from of:0000000000000226/1
Flows
flowId=a20000e9b27d24, selector=[]
flowId=9000004bd76105, selector=[IN_PORT:1, VLAN_VID:None]
flowId=90000069f274d1, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
flowId=a200000ec7529e, selector=[]
flowId=90000050f9a350, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
flowId=a20000b15cada6, selector=[]
Groups
groupId=GroupId{id=0x7000000b}
groupId=GroupId{id=0x20000009}
groupId=GroupId{id=0xffe0002}
Output through of:0000000000000226/2
Device of:0000000000000205
Input from of:0000000000000205/1
Flows
flowId=a200008d542817, selector=[]
flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
flowId=a2000042d8e6b5, selector=[]
Groups
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
Output through of:0000000000000205/3
Result:
Reached required destination Host
Reached required destination Host
High Verbosity Level¶
The most verbose output you can get is
onos> t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
This verbosity level prints all the flows and groups that are met along the path:
Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
Output through of:0000000000000204/2
Device of:0000000000000227
Input from of:0000000000000227/1
Flows
ADDED, bytes=702318, packets=8511, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=343130, packets=4234, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=754, packets=7, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
ADDED, bytes=17566, packets=57, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
ADDED, bytes=510, packets=5, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x70000017], transition=TABLE:60]
ADDED, bytes=17566, packets=57, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x70000017, state=ADDED, type=SELECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=2
id=0x70000017, bucket=1, bytes=754, packets=7, actions=[GROUP:0x20000016]
id=0x20000016, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x20000016, bucket=1, bytes=754, packets=7, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:27, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=754, packets=7, actions=[VLAN_POP, OUTPUT:2]
Output through of:0000000000000227/2
Device of:0000000000000205
Input from of:0000000000000205/2
Flows
ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
Output through of:0000000000000205/3
Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
Output through of:0000000000000204/1
Device of:0000000000000226
Input from of:0000000000000226/1
Flows
ADDED, bytes=701910, packets=8507, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=358984, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=16608, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
ADDED, bytes=17158, packets=53, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
ADDED, bytes=16608, packets=48, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x7000000b], transition=TABLE:60]
ADDED, bytes=17158, packets=53, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x7000000b, state=ADDED, type=SELECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=2
id=0x7000000b, bucket=1, bytes=16608, packets=48, actions=[GROUP:0x20000009]
id=0x20000009, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x20000009, bucket=1, bytes=16608, packets=48, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:26, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=16608, packets=48, actions=[VLAN_POP, OUTPUT:2]
Output through of:0000000000000226/2
Device of:0000000000000205
Input from of:0000000000000205/1
Flows
ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
Output through of:0000000000000205/3
Result:
Reached required destination Host
Reached required destination Host
Successful Example¶
In this example we trace a packet that should be bridged between two hosts in the same subnet
onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:AA:00:00:00:02 -vid None
Tracing packet: [IN_PORT:3, ETH_DST:00:AA:00:00:00:02, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Paths
[of:0000000000000204/3, of:0000000000000204/4]
Result:
Reached required destination Host
Failed Example¶
Obviously failures can happen along one of the paths that the packet can take.
In our sample we removed a flow in of:0000000000000226
thus disabling that
possible path for our initial packet.
The output of our trace would then look something like:
onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Paths
[of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
[of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1]
Result:
Reached required destination Host
No output out of device of:0000000000000226 Packet is dropped
Notice that on the second path that was previously going through
of:0000000000000226
now the packet is dropped due to the rule we removed.
Example of Various Packet Types¶
These sample commands refer to the Mininet environment found in trellis.py 4
L2 Unicast IPv4
t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.2.2 -dm 00:AA:00:00:00:02
L2 Unicast IPv6
t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::302 -dm 00:BB:00:00:00:04 -et ipv6
L3 Unicast IPv4
t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.1 -dm 00:00:00:00:02:04
L3 Unicast IPv6
t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::202 -dm 00:00:00:00:02:05 -et ipv6
ARP
t3-troubleshoot -sp of:0000000000000204/3 -et ARP
DHCP
t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.253 -dm 00:00:00:00:02:04 -ipp 17 -udps 67 -udpd 67
LLDP
t3-troubleshoot -sp of:0000000000000204/3 -et lldp
BDDP
t3-troubleshoot -sp of:0000000000000204/3 -et bddp
LLDP from CONTROLLER
T3 is also capable of tracing packets coming in from the CONTROLLER.
We use the same input port with the possibility to give CONTROLLER as logical port number.
This is a mere convenience to carry the information inside T3 and do the proper flooding to all the ports of the given device.
This case is not treated like all the rest and the input port here is a logical entity that has no match on any physical element on the device.
t3-troubleshoot -sp of:0000000000000204/CONTROLLER -et lldp
Multicast
T3 is also capable of tracing multicast packets.
t3-troubleshoot -vv -sp of:0000000000000205/9 -et ipv4 -d 224.0.0.1 -dm 01:00:5e:00:00:01
t3-troubleshoot-simple¶
T3 also offers a simplified version of the command to be able to ping between two given hosts without having to specify all the information.
This commands bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast. If that is not present the user needs to input this information manually recurring to the the previously shown command. Same verbosity levels and output as the troubleshoot command are possible.
t3-troubleshoot-simple Help¶
onos> t3-troubleshoot-simple --help
DESCRIPTION
onos:troubleshoot-simple
Given two hosts troubleshoots flows and groups between them, in case of segment routing
SYNTAX
onos:troubleshoot-simple [options] one two
ARGUMENTS
one
One host ID
two
Another host ID
OPTIONS
-vv, --veryverbose
Outputs flows and groups for every device
-j, --json
Output JSON
--help
Display this help message
-et, --ethType
ETH Type
(defaults to ipv4)
-v, --verbose
Outputs complete path
t3-troubleshoot-simple Example¶
Examples are given for the same trellis.py 4 script.
L2 Unicast
Hosts are under the same leaf, only interface configuration is needed.
t3-troubleshoot-simple -v -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None
L3 Multicast
Hosts are under different leafs, also segment routing configuration config is needed for the leaf’s routerMac.
t3-troubleshoot-simple -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:03/None
t3-troubleshoot-pingall¶
T3 also offers a ping-all command to be able to test ping between any given hosts of an EthType. This commands bases itself on top of the simplified Host to Host command. It has two verbosity levels, one just with Host and Destination with the result message of the Trace and one that shows also the packet and the paths.
t3-troubleshoot-pingall Help¶
onos> t3-troubleshoot-pingall --help
DESCRIPTION
onos:t3-troubleshoot-pingall
Traces a ping between all hosts in the system of a given ETH type
SYNTAX
onos:t3-troubleshoot-pingall [options]
OPTIONS
-j, --json
Output JSON
--help
Display this help message
-et, --ethType
ETH Type
(defaults to ipv4)
-v, --verbose
Outputs trace for each host to host combination
t3-troubleshoot-pingall Examples¶
IPv4 hosts
t3-troubleshoot-pingall
IPv6 hosts
t3-troubleshoot-pingall -et IPV6
t3-troubleshoot-mcast¶
T3 also offers a mcast command queries the system for all the multicast routes given a certain vlanId. It prints all the traces given a certain group, source and source connect point. It has two “standard” verbosity levels.
t3-troubleshoot-mcast Help¶
onos> t3-troubleshoot-mcast --help
DESCRIPTION
onos:t3-troubleshoot-mcast
Traces all the mcast routes present in the system
SYNTAX
onos:t3-troubleshoot-mcast [options]
OPTIONS
-j, --json
Output JSON
--help
Display this help message
-vv, --veryverbose
Outputs details of every trace
-vid, --vlanId
Vlan of incoming packet
(defaults to None)
-v, --verbose
Outputs trace for each host to host combination
REST API¶
T3 now also offers a way to execute the troubleshoot through REST API endpoints. You can find the description and test the API at the ONOS docs endpoint here:
<ONOS_IP>:8181/onos/v1/docs/#/t3
The rest API offered are a non verbose and a verbose version of the troubleshoot simple command.
It bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast.
The use need to specify just the Source Host, the destination host and the ETH type.
curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>'
curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>/verbose
Last Resort¶
In the case where you can’t figure out what is going wrong even with help of
T3, you can seek help on Trellis developer mailing list
trellis-dev@opennetworking.org
.
There are a few things we would like you to attach:
Issue description
Environment description, such as ONOS version, switch model and OF-DPA version
Steps of reproduction, as detail as possible
Diagnostics.
We have built a tool onos-diagnostics
1 2
to help you easily collect and package ONOS diagnostics.
Reference¶
- 1(1,2)
https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics
- 2(1,2,3,4)
https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools
- 3(1,2)
https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics-k8s
- 4(1,2,3)
https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py