VXLAN EVPN – A modern Open Scalable Fabric

Introduction;

NB:The intended audience for this blog is network professionals with a general need to understand how to deploy VXLAN networks in their organizations to unleash the full potential of modern networking. While interested network administrators will reap the most benefits from this content, the information included within this blog may be of use to every IT professional interested in networking technologies. Elements in this blog explore how VXLAN and EVPN solve network challenges that have daunted the industry for years, as well as how to deploy constructs that are typically seen in traditional networks with this new technology.

Virtual Extensible Local Area Network (VXLAN)

Introduction
IT is evolving toward a cloud consumption model. This transition affects the way applications are being architected and implemented, driving an evolution in data center infrastructure design to meet these changing requirements. As the foundation of the
modern data center, the network must also take part in this evolution while also meeting the increasing demands of server virtualization and new microservices-based architectures. This demands a new paradigm that must deliver on the following areas:
• Flex​i​bil​ity to allow workload mobility across any floor tile in any site
• Re​siliency to maintain service levels even in failure conditions (better fault
isolation)
• Multi-ten​ancy capabilities and better workload segmentation
• Per​for​mance to provide for adequate bandwidth and predictable latency, independent of scale for demanding workloads
• Scal​a​bil​ity from small environments to cloud scale while maintaining the above
characteristics
As a result, modern data center networks are evolving from traditional hierarchical designs to horizontally-oriented spine-leaf architectures with hosts and services distributed throughout the network. These networks are capable of supporting the increasingly common east-west traffic flows experienced in modern applications. In addition,
there are clustering technologies and virtualization techniques that require Layer 2
adjacency.
Evolving user demands and application requirements suggest a different approach that
is simple, and more agile. Ease of provisioning and speed are now critical performance
metrics for data center network infrastructure that supports physical, virtual, and
cloud environments – without compromising scalability or security. These are the main
drivers for the industry to look at Software Defined Network (SDN) solutions.
Why a New Approach 13
Cisco Application Centric Infrastructure (ACI) is an innovative data center architecture
that simplifies, optimizes and accelerates the entire application lifecycle through a
common policy management framework. ACI provides a turnkey solution to build and
operate an automated cloud infrastructure. An alternative option is a VXLAN Fabric
with BGP EVPN control plane that provides a scalable, flexible and manageable solution
to support growing demands of cloud environments.
This chapter introduces the concepts of VXLAN EVPN and the problem it has been designed to solve.
Why a New Approach 14

Why VXLAN Overlay
Network overlays are a technique used in state-of-the-art data centers to create a flexible infrastructure over an inherently static network by virtualizing the network. Before going into the details of how overlays work, the challenges they face, and the solutions to overlay problems, it’s worth spending some time to understand why traditional
networks are so static.
When networks were first developed, there was no such thing as an application moving
from one place to another while it was in use. As a result, the original architects of
TCP/IP used the IP address as both the identity of a device and its location on the network. This was a perfectly reasonable thing to do as computers and their applications
did not move, or at least they did not move very fast or very often.
Today in the modern data center, applications are often deployed on virtual machines
(VMs) or containers. The virtualized application workload can be stretched across multiple locations. The application endpoints (VMs, containers) can also be mobile among
different hosts. Their identities (IP addresses) no longer indicate their location. Due to
the tight coupling of an endpoint’s location with its identity in the traditional network
model, the endpoint may need to change its IP address to indicate the new location
when it moves. This breaks the seamless mobility model required by the virtualized applications. Therefore, the network needs to evolve from the static model to a flexible
one in order to continuously support communications among application endpoints regardless of where they are. One approach is to separate the identity of an endpoint
from its physical location on the network so the locations can be changed at will without breaking the communications to the endpoint. This is where overlays come into the
picture.
An overlay takes the original message sent by an application and encapsulates it with
the location it needs to be delivered to before sending it through the network. Once
the message arrives at its final destination, it is decapsulated and delivered as desired.
The identities of the devices (applications) communicating are in the original message,
and the locations are in the encapsulation, thus separating the location from the iden-
Why a New Approach 15
tity. This encapsulation and decapsulation is done on a per-packet basis and therefore
must be done very quickly and efficiently.
Today, according to market research, approximately 60-70% of all application workloads are virtualized, however, more than 80% of the servers in use today are not running a hypervisor. Of course, every data center is unique and the mix of servers running
virtualized workloads vs. non-virtualized workloads covers the entire spectrum. Any
network solution for the data center must address this mix.
Cisco, in partnership with other leading vendors, proposed the Virtual Extensible LAN
(VXLAN) standard to the IETF as a solution to the data center network challenges posed
by traditional VLAN technology. The VXLAN standard provides for the elastic workload
placement and higher scalability of Layer 2 segmentation that is required by today’s application demands.
VXLAN is designed to provide the same Ethernet Layer 2 network services as VLANs do
today, but with greater extensibility and flexibility. Implementing VXLAN technologies
in the network will provide the following benefits to every workload in the data center:
• Flexible placement of any workload in any rack throughout and between data
centers
• Decoupling between physical and virtual networks
• Large Layer 2 network to provide workload mobility
• Centralized Management, provisioning, and automation, from a controller
• Scale, performance, agility and streamlined operations
• Better utilization of available network paths in the underlying infrastructure
Why a New Approach 16

Why a Control Plane
When implementing an overlay, there are three major tasks that have to be accomplished. Firstly, there must be a mechanism to forward packets through the network.
Traditional networking mechanisms are effective for this.
Secondly, there must be a control plane where the location of a device or application
can be looked up and the result used to encapsulate the packet so that it may be forwarded to its destination.
Thirdly, there must be a way to update the control plane such that it is always accurate. Having the wrong information in the control plane could result in packets being
sent to the wrong location and likely dropped.
The first task, forwarding the packet, is something that networking equipment has always delivered. Performance, cost, reliability, and supportability are fundamental considerations for the network which must equally apply to both the physical and overlay
networks respectively.
The second task, control plane lookup and encapsulation, is really an issue of performance and capacity. If these functions were performed in software, they would consume valuable CPU resources and add latency when compared to hardware solutions.
The third component of an overlay is the means by which modifications to the control
plane are updated across all network elements. This updating is a real challenge and a
concern for any data center administrator due to the potential for application impact
from packet loss if the control plane malfunctions.

VXLAN Control Plane
Why a New Approach 17

VXLAN Control Plane
VXLAN as an overlay technology does not provide many of the mechanisms for scale
and fault tolerance that other networking technologies have developed and are now
taking for granted. In a VXLAN network, each switch builds a database with the locally
connected hosts. A mechanism is required so that other switches learn about those
hosts. In a traditional network, there is no mechanism to distribute this information.
The only control plane previously available was a data plane-driven model called flood
and learn. For a host to be reachable, its information has to be flooded across the network. Ethernet networks have operated with this deficiency for decades.
While the demand for scalable networks increases, the effects of flood and learn need
to be mitigated. For a VXLAN overlay, a control plane is required that is capable of distributing the Layer 2 and Layer 3 host reachability information across the network.
Early implementations of VXLAN lacked the ability to carry Layer 2 network reachability
information, therefore, Ethernet VPN (EVPN) extensions were added to Multi-Protocol
BGP (MP-BGP) to carry this information.
MP-BGP EVPN
MP-BGP EVPN for VXLAN provides a distributed control plane solution that significantly improves the ability to build and interconnect SDN overlay networks. MP-BGP
EVPN control plane for VXLAN offers the following key benefits:
• Control plane learning for end host Layer 2 and Layer 3 reachability information.
• Ability to build a more robust and scalable VXLAN overlay network
• Supports multi-tenancy
• Provides integrated routing and bridging
• Minimizes network flooding through protocol-driven host MAC/IP route
distribution
• ARP suppression to minimize unnecessary flooding
• Peer discovery and authentication to improve security
• Optimal east-west and north-south traffic forwarding
Why a New Approach 18
Looking Ahead
Even though VXLAN technology has attained a considerable degree of maturity in a
very short time, the industry is already designing the next evolution of this technology.
Generic Protocol Encapsulation (VXLAN-GPE)
VXLAN is one of many data plane encapsulations available. Examples of other UDPbased encapsulations are LISP (Locator/ID Separation Protocol) and OTV (Overlay
Transport Virtualization). These three encapsulations are very similar, the differences
lying in the overlay shim header. While all three use the same size header, the field allocation and the naming are slightly different. Within the encapsulation, there are also
variations. While VXLAN maintains an inner-MAC header, LISP only carries an inner-IP
header. It becomes evident that an approach for header extensions is needed to avoid
adding yet another UDP-based encapsulation.
VXLAN-GPE was invented to bring some consolidation in the UDP-based encapsulation
family. A major part of VXLAN-GPE is the inclusion of a protocol-type field to define
what is being encapsulated and set the meaning for the various flags and options in the
overlay shim header. This protocol type describes the packet payload; currently defined types include IPv4, IPv6, Ethernet, and Network Service Header (NSH).
A prominent example for the need of this flexible protocol extension is Service Chaining
and the related NSH approach.
NSH enables the possibility of dynamically specifying that certain network traffic is sent
through a chain of one or more network services. The goal of NSH is to create a topology-independent way of specifying a service path. NSH also includes a number of
mandatory, fixed-size context headers designed to capture network platform information. NSH even contains an optional variable length metadata field for additional extensibility and is designed to include all required information inside fixed-size fields.
Why a New Approach 19
Creation of yet another encapsulation protocol stands to add more confusion to the already crowded encapsulation protocol space. The extensibility of VXLAN-GPE and
NSH promises to both reduce the amount of encapsulation in the industry and accommodate future network encapsulation requirements. Geneve, VXLAN-GPE, and NSH
are all recent protocol drafts proposed to the IETF. The three protocols provide similar
approaches to achieve flexible protocol mappings. While Geneve uses variable length
options, VXLAN-GPE and NSH use fixed size options. Cisco supports open standards
and will continuously reevaluate support for future encapsulations.
Evolution of the EVPN Control Plane
The current implementation of the EVPN control plane is focused on delivering scalable
data center Fabrics with mobility and segmentation. As EVPN control plane implementations become more complete, the EVPN control plane may address additional usecases such as DCI. The complete theoretical definition of the EVPN control plane is
captured in a series of Internet drafts being worked on at the IETF. The general specification of EVPN accommodates use cases beyond the Data Center Fabric, including
Layer 2 Data Center Interconnect.
In order to properly address the DCI requirements, the EVPN control plane implementation must be expanded to include the multi-homing functionality defined in the EVPN
specification to deliver failure containment, loop protection, site-awareness, and optimized multicast replication.

Fundamental Concepts
Fundamental Concepts
Introduction
In the networking world, an overlay network is a virtual network running on top of a
physical network infrastructure. The physical network provides an underlay function,
offering the connectivity and services required to support the virtual network instances
delivered in the overlay. The virtual network allows for an independent set of network
services to be offered regardless of the underlay infrastructure, even though those services may be the same. As an example, it is possible to deliver Layer 2 connectivity services on top of a Layer 3 network infrastructure via an overlay network. A common example of this would be VPLS service offered over a carrier’s MPLS infrastructure.

An overlay network typically provides transport of network traffic between tunnel endpoints on top of the underlay by encapsulating and decapsulating traffic between tunnel endpoints. The tunnel endpoint may be delivered through a physical network device, and perform tunnel encapsulation/decapsulation in hardware. It also may be virtual, with the tunnel endpoint process running in a hypervisor. A hardware tunnel endpoint provides greater performance leveraging hardware-based forwarding, but has
less flexibility implementing new capabilities. In contrast, a software endpoint provides
increased flexibility but at the cost of limited performance.
This chapter provides an overview of the concepts required to have a basic understanding of the technology and how it works.
Fundamental Concepts

What is VXLAN?

Virtual Extensible LAN (VXLAN) as defined in RFC 7348 is an overlay technology designed to provide Layer 2 and Layer 3 connectivity services over a generic IP network.
IP networks provide increased scalability, balanced performance and predictable failure
recovery. VXLAN achieves this by tunneling Layer 2 frames inside of IP packets. VXLAN
requires only IP reachability between the VXLAN edge devices, provided by an IP routing protocol.
There are pros and cons to consider when selecting the underlay routing protocol and
these are discussed in more detail in the Single-POD VXLAN Design Chapter.
The VXLAN standard defines the packet format illustrated by the following diagram:
Figure: VXLAN Packet Format
Fundamental Concepts 23
VXLAN uses an 8-byte header that consists of a 24-bit identifier (VNID) and multiple reserved bits. The VXLAN header, along with the original Ethernet frame, is placed in the
UDP payload. The 24-bit VNID is used to identify Layer 2 segments and to maintain
Layer 2 isolation between the segments. With 24 bits allocated for the VNID, VXLAN can
support up to 16 million logical segments.
The terminology used when describing the key components of a VXLAN Fabric include:
• VTEP – Virtual Tunnel Endpoint: The hardware or software element at the edge of
the network responsible for instantiating the VXLAN tunnel and performing VXLAN
encapsulation and decapsulation
• VNI – Virtual Network Instance: a logical network instance providing Layer 2 or
Layer 3 services and defining a Layer 2 broadcast domain
• VNID – Virtual Network Identifier: a 24-bit segment ID that allows the addressing of
up to 16 million logical networks to be present in the same administrative domain
• Bridge Domain: A set of logical or physical ports that share the same flooding or
broadcast characteristics
The VXLAN tunnel endpoint function can be performed by a hardware device or by a
software entity such as a hypervisor. The main advantage of using a hardware-based
tunnel endpoint is the enhanced performance offered through the capabilities of the
switch ASICs.
Alternatively, a software-based VTEP removes the dependency from the hardware
switches, albeit at the expense of performance. Additionally, VXLAN deployments could
adopt hybrid approaches, where the VXLAN tunnels are established between hardware
and software VTEPs. More information on this can be found in the Software Overlays
chapter.
Fundamental Concepts 24
As discussed in the introduction, the use of VXLAN technology brings several benefits
to Data Center networking which include:
• Multi-tenancy: VXLAN Fabrics inherently support multi-tenancy both at Layer 2
(separate Layer 2 VNIs represent logically isolated bridging domains) and Layer
3 (by defining different VRFs for each supported tenant)
• Mobility: The overlay capability offered by VXLAN provides Layer 2 extension service across the data center to provide flexible deployment and mobility of physical
and virtual endpoints
• Increased Layer 2 segment scale: VLAN-based designs are limited to a maximum of
4,096 Layer 2 segments due to the use of a 12 bit VLAN ID. VXLAN introduces a 24-
bit VNID that theoretically supports up to 16 million distinct segments
• Multi-path Layer 2 support: Traditional Layer 2 networks support one active path
because Spanning Tree (STP) expects and enforces a loop-free topology by blocking
redundant paths. A VXLAN Fabric leverages a Layer 3 underlay network for the use
of multiple active paths

How does VxLAN Works?

To be continued…