Exploring the Broadcom Tomahawk 5 Chip and its Impact on ASICs in Data Center Environments

Feb 28, 2023

In Q4 last year, I attended Networking Field Day 29 and explored Broadcom Tomahawk 5 new chip for their data center switches. Since I am starting a career as a Cloud Engineer, I decided to briefly explore how switches leverage VxLAN in a data center environment and its relationship with Application-specific integrated circuits (ASICs).

The chip is available for their switches with 64 ports × 800GbE, 128 ports × 400GbE, or 256 ports × 200GbE ports.

It also has a network connectivity of up to 51.2 Tb/s. This is basically the maximum data rate that can be achieved on it. At first, I got a bit confused about how this relates to the speed on the switch port.

How does the Bandwith Relate to the Switch’s ASIC

So, Application-specific integrated circuits (ASICs) are often configured to perform a specific function like support VLANs, by allowing them to be created, configured, and managed, via the Generic VLAN Registration Protocol (GVRP) defined in the IEEE 802.1Q. Functions like assigning a VLAN to a particular port or ports on the switch, and configuring routing between VLANs are all examples of this.

VxLAN marvels me because it is a protocol that wasn’t present in some old ASICs from Broadcom and it’s now popular in the Data Center space. The VxLAN RFC 7348 was released in 2014. Now, I can’t confirm the exact date Broadcom started rolling out the VxLAN protocol from their Tomahawk 3.4Tb/s.

Source: Broadcom Tomahawk 5: World’s Fastest Switch Chip at 51.2Tbps (Tech Field Day Youtube)

Basically, these protocols are programmed on a pipeline in the ASIC. Think logic gates, like the AND gate. It is the basis of the operation of an ASIC and can be used to create complex functions. Folks in the electrical engineering space will get it.

I also realized that when a switch has been configured to perform specific functions pretty well, one might need to wait for a few years for a new one.

Source: ASICs for Network Engineers (NANOG Youtube)

As we can see in the first image, it’s been taking 2 years each for Broadcom to release new chips.

While looking at an ASIC presentation from Pete Lumbis, I saw that an ASIC may be limited to doing a VxLAN decap or a Layer 3 lookup, not both. This could easily be in a multi-tenant network.

Multi-Tenancy and VxLAN

A simple diagram explaining multi-tenancy and VxLAN

We simply have two Tenant hosts in two regions, interconnected and owned by one customer. With a collection of Broadcom switches, per region. However, these two Tenants are logically seen as one single network, all thanks to the VxLAN capability.

VxLAN Decap and L3 Lookup

A VxLAN decap is the process of removing the VxLAN header from a packet before sending it to the next switch for processing.

A layer 3 lookup is the process of using the destination IP address in the packet header to determine the route the packet should take.

The VxLAN decap is typically done before the layer 3 lookups, as the VxLAN header contains information that needs to be removed before the packet can be routed. By removing the VxLAN header from a packet before sending it to the next switch for processing, the switch can determine which layer 2 protocols are associated with the packet and route it accordingly. This allows for the extension of layer 2 protocols over a layer 3 network. Layer 2 protocols like Address Resolution Protocol (ARP).

Unlike other ASICs which have to perform recirculation, the Tomahawk 5 performs single-pass VxLAN routing and bridging.

Lags

The ASIC isn’t programmable yet. I realized that this is currently possible using an open-source language called P4. I look forward to what Broadcom has in store ahead. It’ll be exciting.

Conclusion

The Broadcom Tomahawk 5 is a powerful chip for data center switches. It’s great for supporting VxLAN and other protocols, and its recirculation feature allows for high bandwidth utilization. To get the most out of this chip, it’s important to understand how it works to best match the requirements of your data center environment.

Networking Characteristics

Discussion about this post