Welcome to the modern data center. If you have been working in networking for a few years, you have probably noticed a massive shift in how we build architectures. The traditional three-tier model (core, distribution, and access) served us well for decades. But as applications grew more complex and virtualization became standard, that old hierarchical model started to show its limits. Spanning Tree Protocol blocked redundant links to prevent loops, wasting expensive bandwidth. Traffic shifted, moving east-to-west between servers rather than north-to-south. We needed a new way to design scalable networks.
Enter the Spine-Leaf architecture paired perfectly with VXLAN and EVPN. This powerful combination is the modern gold standard for enterprise data centers. It gives us high scalability, rapid convergence, and the ability to stretch Layer 2 securely over a Layer 3 foundation. But this technological leap has a learning curve. Terminology like VTEP, VNI, and EVPN Route Types can feel completely overwhelming. We will carefully break down exactly how a data packet travels through a VXLAN EVPN network and focus purely on operational behavior, control plane logic, and data plane mechanics. By the end, readers will understand exactly what happens when Host A talks to Host B across a modern fabric.
The Core Foundation: The Underlay and the Overlay
| The Underlay (The Physical Roads) | The Overlay (The Logical Tunnels) |
| This is the tangible network. It consists of physical spine and leaf switches, copper and fiber cables, and a routing protocol providing IP reachability. Every leaf connects to every spine, but leaves never connect directly to other leaves. This strict topology creates a predictable, non-blocking mesh. The underlay uses protocols like OSPF, IS-IS, or eBGP to route traffic. Importantly, the underlay knows absolutely nothing about your servers, virtual machines, or customer VLANs. It only knows about the loopback IP addresses of the switches. | The overlay is the logical network where the true magic happens. Built on top of the underlay using secure tunnels (VXLAN), the overlay is where your actual endpoint devices live. It handles MAC address tables, subnets, and tenant isolation through VRFs. When a server sends a packet, the overlay encapsulates it, hides it inside a UDP tunnel, and hands it to the physical underlay for delivery. This brilliant separation of physical duties and logical routing makes EVPN powerful. |
Meet the Players: Key Terminology
To successfully navigate the fabric, we need to clearly define the specific vocabulary driving this modern technology.
| VTEP (VXLAN Tunnel Endpoint): The crucial entrance and exit ramp to the VXLAN highway. Typically residing on the leaf switch, the VTEP encapsulates normal traffic into VXLAN tunnels and decapsulates VXLAN traffic back into normal traffic. Every VTEP has a unique underlay IP address. |
VNI (VXLAN Network Identifier): The scalable replacement for traditional VLANs. While standard VLANs are strictly limited to 4,094 unique segments, the VNI uses a 24-bit identifier allowing over 16 million network segments. Layer 2 VNIs stretch single subnets, while Layer 3 VNIs securely route traffic between different subnets. |
| MAC-VRF: A dedicated Virtual Routing and Forwarding instance holding MAC addresses and their associated IP addresses for a specific tenant. It provides strict isolation between different security zones. |
| BGP EVPN (Ethernet VPN): The highly advanced control plane of the network. Instead of relying on inefficient flooding to find MAC addresses, switches use BGP EVPN to proactively advertise endpoint locations to each other, similar to how traditional BGP routes internet traffic. |
| Route Distinguisher (RD) & Route Target (RT): Since BGP carries information for multiple isolated tenants, the RD mathematically ensures addresses remain unique, while the RT strictly controls which switches import and export specific routes. |
The Control Plane: Building the Network Map
In legacy networks, switches learn MAC addresses passively. When an unknown packet arrives, the switch floods it everywhere. This wastes bandwidth and causes network storms. VXLAN EVPN drastically changes this paradigm by intelligently moving MAC address learning completely into the control plane.
Before any user data flows, the leaf switches aggressively communicate using Multiprotocol BGP with the EVPN address family. Imagine a brand new server, Host A, boots up and connects to Leaf.
As soon as Host A sends its very first packet (like an ARP request), Leaf 1 inspects it and securely learns Host A’s MAC and IP address. Instead of keeping this a secret, Leaf 1 instantly generates a specialized BGP update message called an EVPN Route Type 2.
Leaf 1 packages Host A’s MAC address, IP address, the associated Layer 2 VNI, and Leaf 1’s own VTEP IP address into this BGP update. Leaf 1 transmits this update to the Spine switches. In a well-designed fabric, Spines act strictly as BGP Route Reflectors. They efficiently reflect this new route down to every other leaf switch in the entire network.
When Leaf 2 securely receives this BGP update, it immediately updates its local tables. Leaf 2 now explicitly knows that to reach Host A, it must securely encapsulate the packet and send it across the underlay straight to Leaf 1. The incredible advantage here is that Leaf 2 learned this vital information without a single drop of data plane flooding. The entire map was proactively built.
Step 1: The Layer 2 Packet Walk (Intra-Subnet)
/Now that the control plane has cleanly built our routing map, let us carefully trace a real data packet. Host A wants to communicate with Host B. Both hosts strictly reside in the exact same IP subnet and VLAN, but they connect to completely different leaf switches (Host A on Leaf 1, Host B on Leaf 2).
Ingress at Leaf 1: Host A generates a standard Ethernet frame targeting Host B’s destination MAC address. It transmits this frame into Leaf 1. Leaf 1 checks its MAC address table. Thanks to the brilliant BGP EVPN control plane, Leaf 1 already knows Host B lives remotely behind Leaf 2.
The Magic of Encapsulation: Leaf 1 acts as the VTEP. It cannot simply send the raw Ethernet frame into the spine. Therefore, Leaf 1 encapsulates the original frame. It meticulously adds a VXLAN header containing the specific VNI. Next, it adds a UDP header. Then, it adds an outer IP header setting the Source IP to Leaf 1’s VTEP address and the Destination IP to Leaf 2’s VTEP address. Finally, it adds an outer MAC header to reach the first-hop spine switch.
Traversing the Underlay: The massively encapsulated packet leaves Leaf 1 and hits the Spine. The Spine strictly looks at the outer destination IP address (Leaf 2). Using robust underlay routing protocols, the Spine performs a lightning-fast hardware lookup and routes the packet smoothly down to Leaf 2.
Decapsulation at the Egress Leaf: The encapsulated packet finally arrives at Leaf 2. Leaf 2 inspects the outer IP header and sees its own VTEP address. Leaf 2 skillfully strips off the outer MAC header, the outer IP header, the UDP header, and completely removes the VXLAN header. What remains is the exact original Ethernet frame. Leaf 2 securely forwards the frame directly out the physical interface to Host B.
Step 2: The Layer 3 Packet Walk (Inter-Subnet Routing)
Modern data centers primarily rely on robust Layer 3 routing. What specifically happens when Host A (in Subnet 10) needs to communicate with Server C (in Subnet 20)? This introduces the brilliant concept of the Distributed Anycast Gateway and Symmetric Integrated Routing and Bridging (IRB).
The Distributed Anycast Gateway Advantage
In legacy networks, the default gateway lived centrally. Traffic leaving the subnet needlessly traveled to this central choke point, got routed, and traveled back down. VXLAN EVPN elegantly uses a Distributed Anycast Gateway. Every leaf switch is configured with the same default gateway IP and MAC address for a given subnet. When Host A routes traffic, its default gateway is locally present on the first switch it touches (Leaf 1), ensuring routing happens immediately at the edge.
Symmetric Routing in Action
- Host A sees Server C is in a different subnet and sends an ARP request for its local default gateway. Leaf 1 instantly responds because it locally hosts the Anycast Gateway.
- Host A drafts the packet targeting the Anycast Gateway MAC on Leaf 1.
- Leaf 1 receives the packet, recognizes its own router MAC address, and immediately triggers a Layer 3 routing lookup in the tenant VRF.
- Leaf 1 confidently checks its EVPN routing table and finds a BGP route for Server C. The route dictates that Leaf 1 must use the specialized Layer 3 VNI and forward to Leaf 3.
- Leaf 1 strips off the original Ethernet header entirely. It expertly encapsulates the raw IP packet inside a new VXLAN header utilizing the Layer 3 VNI. It securely adds the UDP and outer IP headers targeting Leaf 3.
- The packet zips rapidly across the Spine underlay, securely traversing the physical mesh.
- Leaf 3 receives the packet and violently strips off the VXLAN encapsulation. Because the packet aggressively arrived on the Layer 3 VNI, Leaf 3 performs a final routing lookup.
- Leaf 3 generates a brand-new inner Ethernet header, inserting its own Anycast Gateway MAC as the source and Server C’s actual MAC as the destination.
Check 100% Job Guarantee Courses
- CCIE Enterprise Infrastructure V1.0/R&S Training in India
- CCIE Security V6.1 Course
- Cyber Security & Network Security Course
- Cloud Security + Network Security Expert
Frequently asked Questions:
The underlay is the physical Layer 3 network made of spine and leaf switches and provides IP reachability between VTEP addresses using protocols such as OSPF, IS-IS, or eBGP. The overlay is the logical VXLAN network built on top of the underlay and carries endpoint MAC addresses, IP subnets, and tenant segmentation.
A VTEP, or VXLAN Tunnel Endpoint, is typically located on a leaf switch. It encapsulates normal Ethernet traffic into VXLAN before sending it across the Layer 3 underlay and decapsulates VXLAN traffic when it reaches the destination leaf.
A VNI, or VXLAN Network Identifier, is a 24-bit identifier used to distinguish virtual network segments. It provides more than 16 million possible identifiers, allowing VXLAN to scale far beyond traditional VLAN limits. Layer 2 VNIs are used for Layer 2 segments, while Layer 3 VNIs are used for routing between subnets.
BGP EVPN acts as the control plane of the VXLAN fabric. Instead of relying only on flooding, leaf switches advertise endpoint MAC and IP information to other fabric nodes using EVPN routes. This allows remote leaf switches to learn where endpoints are located before normal user traffic needs to be forwarded.
When Host A sends a frame to Host B, Leaf 1 identifies that Host B is behind Leaf 2. Leaf 1 encapsulates the original Ethernet frame with a VXLAN header, UDP header, and outer IP header using the VTEP addresses. The spine forwards the packet based only on the outer destination IP, and Leaf 2 removes the VXLAN encapsulation and delivers the original frame to Host B.
For communication between different subnets, the ingress leaf performs a Layer 3 lookup in the tenant VRF. It selects the required Layer 3 VNI, encapsulates the traffic in VXLAN, and sends it across the underlay to the destination leaf. The destination leaf removes the VXLAN encapsulation, performs another routing lookup, creates a new Ethernet header, and forwards the packet to the destination endpoint.
