CCIE Service Provider Update

Cisco 7200 is in the house!
Read the announcement from Cisco here.

With the introduction of 7200 series routers to CCIE lab starting November 2006, all my calculation has been changed completely. Why taking CCIE now if you know after November you will have chance to have "real world" scenario in the lab? With 7200 most probably several features that were not tested before, i.e. L2VPN Martini, will be added to the lab.

Now we are back in business.

Things That Keep Me Alive

Cisco 7600, Supervisor 720-3B, MPLS VPN, 10 GE modules, QOS, Cisco 6509, Firewall Blade, NAM, Wireless LWAPP with WISM, OSPF stub, MP-BGP, Cat 4500 Sup V-10GE, XENPAX, IDS blade, ASA with IPS, Cisco 3845 voice gateway, OSPF Stub, MCS 7800, Cisco 6513,
Routing to the Edges, CSA, Call Manager, Unity, 802.1x, WPA-2, Video Phone, HDV, PVDM2, CiscoWorks, MARS, ACS, Radius, ATM E3, AAA, Layer 3 Roaming, VG224, Load Balancing, Transparent Proxy with WCCP, VTP mode Transparent, Route Target, Port Channel, OSPF Area Authentication, DHCP Option 150, Firewall vlan-group, NTP, High Availability, Dial Peer, Calling Searh Space, BGP Route Reflector, SSHv2, VRF, Call Park, LDAP, 3750 StackWise.
All in 25 days.

Wake Me Up When September Ends

I have just resigned from my company.
I still can’t believe it. Finally it really happens.
I still can’t believe finally I can tender my resignation letter.
After 4 years, 4 bloody years.
As per my contract, I have to stay in my company for another 3 months to complete all the projects or hand them over to my successor.

What’s wrong with my company?
Uhm, nothing.

It’s only that I’m not ready to end up here.
I feel like a big fish in a small pond.
I want to go to bigger pond. I want to go to the ocean.
I may go down drowning over there, but at least I have to try.

And it’s not only a bigger pond.
I want to go to a place where people judge me only based on my expertise and my capabilities to deliver the work. And nothing else.
I want to go to a place where I can grow.
Where I can’t see the end of the road.
The place that respects me for what I can do.

You may call me a dreamer.
That place may not even exist.
After 3 months, I may find myself standing in nowhere.
But instead of die in curiosity keep thinking about it, it’s better to try to find out.
And I still have time to dream until 1st October.

So please wake me up when September ends.
And if you know that such place really exists, please let me know.

Enterprise MPLS VPN - Howto

Some people said that I don’t know what I’m talking about when I told them I run MPLS VPN within Enterprise network. They said I was just trying to make a hype. I told them that I’m not. I have configured MPLS VPN on one of my customer with around 4000 users. Seeing is believing. So without further arguments, I would like to explain how I do it.

The picture shows the Campus Network building blocks model that is the most common topology in Enterprise Network. It contains Access or Edge Switches where the end users are connected, Distribution Switches as aggregation point for the Access Switches, Core Switches as the central of the network, and Server Farm Switches to connect all the servers.

Why do we need such blocks model? Because it’s modular and scalable. Most of the time we use duplicate hardware and multiple connection links on each block to provide redundancy. Connection to the Internet, through the firewall, can be facilitated by connecting the Internet building block to the Core Switches. And it applies to connection to branch offices as well, called Wide Area Network (WAN), the building block can be connected to Core Switches. I don’t draw both Internet and WAN blocks for the sake of simplicity.

Cisco Systems offers service module on its chassis-based switches. The most common modules that my customers opt to buy are Firewall and Intrusion Prevention System (IPS) blades that are installed on Server Farm Switches to protect the servers. Firewall modules or well known as Firewall blades will be one of the key of implementing MPLS VPN in my scenario.

So what are the requirements? My customers have 8 different users group that are separated into 8 different VLANs. All those VLANs shall communicate to each other without any restrictions except for the 8th VLAN: they must not see the other VLANs at all, but some selective users from different VLANs should be able to establish one-way communication to that 8th VLAN. The 8th VLAN will have its own Internet connection through ADSL, and not through the main Internet link and Internet building block, and it considered to be out of my customer administration control completely. So the main idea is just like having De Militarized Zone (DMZ) inside the internal network separated in different edge switches location!

In normal circumstances, I would configure inter-VLAN routing on the closest Layer 3 Devices to the end users, and put Access Control List (ACL) to provide the restriction. The problem with this approach: administration overhead to maintain the ACL. Any modification on the ACL requires any reconfiguration on all those Layer 3 Devices.

So I chose more elegant way by simply enabling MPLS VPN. Especially since the hardware used in this scenario are Cisco 6500/7600 model with Supervisor 720-3B module that supports MPLS in the hardware.

Following is the step-by-step how I accomplish my goal:

Step 1: Physical Connection
As it showed in the picture, there are multiple links to provide redundancy. Access Switches are connected to 2 different Distribution Switches, each Distribution Switch is connected to both Core Switches and they are connected to each other as well. Server Farms Switches are just like another distribution switches: connected to each other and to both Core Switches. The connection between Distribution – Core – Server Farms is utilizing high speed 10 Gigabit per second fiber links. Connection between Access to Distribution can use 1 Gigabit per second or more with Ether-Channel technology, and it depends on the over-subscription ratio: the ratio between number of end users and the uplink. Access Switches can be stacked, and with the new StackWise technology from Cisco on 3750 series switches, all access switches in 1 stack act as 1 switch with combined number of interfaces.

Step 2: Connectivity with Interior Gateway Protocol (IGP)
The next step is to provide connectivity with IGP Routing. I chose OSPFv2 and put Core, Distribution, and Server Farm Switches into Area 0 Backbone. Connection between Distribution to Access, most of the time it is the Switch Virtual Interface (SVI) or VLAN Interface, is placed into different area to facilitate Summarization into Area 0.

The 2 Firewall blades installed in 2 Server Farm switches are configured in Single Context mode and active-passive failover. We must configure 1 VLAN between Server Farms Switches and the Firewall blades, and this VLAN acts as the “Outside” network for the Firewalls. OSPF Totally Stub Area is configured between 2 Server Farm switches and the active Firewall, to inject only default route to the firewall blade pointing to the switches, and to get the routes to all the Servers networks behind the Firewall blade.

For connection between Distribution to Access, if I terminate Layer 2 VLAN in Distribution Switches with SVI, Distribution Switches will be the routing gateways for all the end users. But if I want to have the same VLAN spans across multiple Access Switch stacks, then I need to run Hot Standby Routing Protocol (HSRP) on both Distribution Switches and I must have Layer 2 Link or Trunk between Distribution Switches. Having layer 2 Trunk between Distribution Switches, and from Distribution to Access switches, can forms Layer 2 loop between Distribution – Access – Distribution and it forces me to rely on Spanning Tree Protocol (STP) to break this loop. Now I have 3 different protocols running in my Distribution Switches: IGP, HSRP, STP and I require to sync the configuration on all those 3 protocols.

I don’t want to get into that complexity, and since my Access switches are Cisco 3750 with EMI software, I decided to run Layer 3 Routing between Distribution and Access. So Access Switches are the gateways for all the end users now. Having Routing to the Access model provides several benefits: there is only 1 protocol for connectivity within the network which is the IGP, we can use Layer 3 tools such as Ping and Traceroute to verify end-to-end connectivity and not bother to check all Layer 2 parameters such as STP root bridge etc, and by default IGP provides equal cost load balancing to utilize better of all the uplinks from the Access to Distribution.

The link from Distribution to Access can use Ether-Channel to provide more than 1 Gbps connection. It’s a Layer 2 Trunk that allows only 1 VLAN to pass through and this VLAN is used as Layer 3 link from Access to Distribution. I can make the Ether-Channel interface as Layer 3 port directly but I would need another Layer 3 link for my MPLS VPN. It will be explained next in Step 4.

Step 3: Enable MPLS LDP on all MPLS-enabled devices
This step is straight forward. By default with current IOS version, Cisco enables Tag Distribution Protocol (TDP) instead of Label Distribution Protocol (LDP). So what I need to do is only defining 1 loopback interfaces as my Router ID and enabling LDP on all interfaces required.

Cisco 3750 access switches don’t support MPLS labeling. So in my scenario the MPLS cloud is formed between Distribution – Core – Server Farms. Core Switches act as P routers and both Distribution and Server Farms Switches act as PE routers.

Quick verification can be done by looking at the LDP neighborship on each MPLS device. Up to this step, we have already had our MPLS backbone ready for the real application: MPLS Layer 3 VPN.

Step 4: Virtual Routing Forwarding (VRF) and PE-CE links
It’s time to enable VRF on each Distribution. Define the Route Distinguisher (RD) and Route Target (RT) and assign the PE-CE links into the VRF. If I chose the Routing to Distribution model, where Distribution Switches are the routing gateways for all end users, the PE-CE links are the SVI interfaces.
But since I decide to have Layer 3 Routing between Distribution and Access, then I need to create another VLAN for Layer 3 link from Distribution to Access. So now I have 2 VLANs for Layer 3 links between Distribution and Access: 1 for the global routing and 1 for the VRF.

Cisco 3750 switch with EMI software supports multi-VRF or VRF-lite feature. Basically with this feature we still can’t do MPLS labeling but it can extend the VRF from Distribution to Access switches. So in any Access switches where I have the 8th VLAN, what I need to do: create the VLAN, assign particular ports into the VLAN, create SVI or VLAN interface as the default gateway for the end users, create VRF with Route Distinguisher, then assign the SVI into the VRF. The same VRF will be assigned to one of the VLAN for layer 3 links to Distribution. Now I have VRF all the way from Distribution, Layer 3 Link between Distribution and Access, and the SVI in Access switches.

Since the 8th VLAN Interface will be part of VRF, the subnet will not show up in global routing table in any Access Switches hence it won’t be able to communicate to any other VLANs even in the same Access switch.

Communication between PE – CE can utilize Static, RIP, OSPFv2, EIGRP and even BGP. If there is only 1 VLAN just like in my scenario, I can use Static Routing for the sake of simplicity. So Distribution will have static route for the 8th VLAN pointing to the Access Switch, and the Access Switch can have Static default route pointing to both Distribution Switches. If I want to use OSPFv2 and on each Distribution it has to run in different Process ID than the OSPFv2 that provides connectivity for global routing.

All the users in 8th VLAN can reach each other within the same Distribution Switches. Now it’s the time to connect them to another Distribution Switches and Server Farms.

Step 5: Multi-Protocol BGP (MP-BGP) and Route Reflectors
MP-BGP is used to transmit the VRF routes from one PE to another PE. The first thing we need to do is to make both Core Switches as BGP Route Reflectors, to avoid having a fully mesh topology. All PEs are required to establish the communication to Route Reflectors only. Remember, with MP-BGP we need to configure Address Family VPNv4 under BGP Routing configuration, activate the neighbors and enable BGP Extended Community to transfer the Route Target parameters. Route Target is used to define with route will be exported and installed on each PE router.

On each Distribution Switch, all Static or OSPFv2 routes that is used in PE-CE connection need to be redistributed into BGP, and all BGP VPNv4 routes achieved from another PE need to be redistributed into the VRF OSPFv2. If we use Static Routing, default gateway has to be configured on each Access Switches pointing to Distribution.

Once we complete this step, all Distribution and Server Farm Switches should be able to see all 8th VLAN routes inside the VRF routing table.

Step 6: Connecting the VPN to the Global Network
Connectivity between 8th VLAN in different Distribution Switches has been achieved, now it’s time to connect this VPN to the rest of the network that I call Global Network. Firewall blade is the key here. It protects all the servers and at the same time it acts as the meeting point between 8th VLAN VRF and the rest of the network.

Between Server Farm Switches and Firewall blades, we have already configured 1 VLAN as the Outside network for the Firewalls. So any traffic to the Servers from all user VLANs, except the 8th VLAN, get into the Firewalls through this Outside network VLAN. Now we need to create another VLAN between Server Farm Switches and Firewall Blades, and assign this VLAN into the VRF. This VLAN will act as DMZ network connected to Firewall Blades.

By default Cisco Firewall modules only allow to have 1 SVI or VLAN interfaces in single context mode to act as Outside interfaces. To circumvent this problem, we need to enable firewall multiple-vlan-interfaces feature. Use this feature with caution! Wrong configuration may lead to the traffic bypassing the Firewalls to reach the servers.

Once we have another VLAN acting as DMZ for the Firewalls, we can setup the Access Control List in the Firewall blades to allow communication from the VPN to the servers, or communication between all other VLANs to the 8th VLAN.

Static routing for traffic to the servers or any other VLANs can be configured in Server Farm Switches VRF pointing to Firewall blades DMZ interface, and this static route must be redistribute into the MP-BGP so all 8th VLANs know how to reach all servers and any other VLANs. We should do the same trick for global network so all other VLANs know how to reach the 8th VLAN through the Firewall blade Outside interface.

Step 7: Network Ready For Use testing
It’s time to verify our setup. We should test the connectivity with step-by-step approach: verify the IGP for global routing, verify MPLS LDP in all MPLS-enabled devices, check the PE-CE connectivity, test the connection between 8th VLAN in different Access Switches but still connected to the same Distribution Switches, verify the VPNv4 routes, and test connectivity between 8th VLAN in different Distribution Switches, and connectivity to the Firewall blade and Server Farms Switches.
The last verification, check the ACL on Firewall blades to make sure 8th VLAN can connect to the servers but not to any other VLANs in global network, and selected users from any other VLANs are allowed to communicate to the 8th VLAN through the Firewalls.


As you can see, one of the benefit of using MPLS Layer 3 VPN instead of distributed ACL on each Distribution Switch is to cut the administration overhead to maintain the network. We can have a single infrastructure to provide different isolated users group or network on top of it, and the policy to control the communication between different users group can be centralized using Firewall Blade.

I’m using MPLS VPN to segregate the 8th VLAN. One day I may come across the requirements to segregate all those 8 VLANs into 8 different isolated networks, and allow the communication between each other only through centralized Firewall. That will be the day I would say Thank You, Once Again to all the guys who invented MPLS Layer 3 VPN.

MPLS in the Enterprise

I knew it. I have been talking about it from the past 2 years.
It’s coming. I knew it, and it’s coming.

When people talk about MPLS, they always associate it with Service Provider. By adding label to the packet between layer 2 and layer 3, MPLS can provide so many services such as Layer 3 VPN, Layer 2 VPN, Traffic Engineering and so on.

Why Enterprise customers need MPLS?
MPLS was invented originally to optimize and increase the switching performance by not doing Layer 3 lookup, but MPLS label lookup instead. Nowadays switching performance in network device has been increased and it is equal for either Layer 3 lookup or layer 2 lookup. So increasing the performance is not the answer we are looking for.

Okay, it’s really good to consolidate ATM backbone and Frame-Relay backbone into single IP infrastructure with Layer 2 VPN. But which Enterprise customer maintains the physical layer for its backbone? And forget about Traffic Engineering for time being.

The most applicable MPLS service for Enterprise is: Layer 3 VPN. But wait, why in single Enterprise network you need to run MPLS L3 VPN?

Well, the answer is obvious if you have any third party vendors or consultants working in several places in your network. You want them to use your network transparently: they can connect to each other but they can’t see your infrastructure. MPLS L3 VPN is the answer.

Now I want to push it even further. I have one project to build big campus network, with core, distribution, and access switches topology. And the users are divided into several departments. The policy from my customer: within one department, regardless of the physical location, users should be able to connect to each other. But they should not be able to connect to any other departments. And all of them share the same data center, and share the same Internet connection.

I have two options for this: put Access Control List (ACL) in any distribution switches or the gateways. This is the most common option that any Network Engineers would choose. The second option is to have VPN within each department so they will not be able to communicate to each other. VPN can be provided with normal GRE tunnel, IPSec, and.. MPLS.
Compare to the other two, MPLS configuration is easier and more fit to address the above requirement.

The MPLS cloud will start from distribution switches. So at minimum, distribution should run hardware that can support MPLS. From Cisco this can be Catalyst 6500 series with Supervisor 720-3B. We can terminate each VPN into the firewall, one VRF for one VLAN connected to the firewall. With this way, we can have all the MPLS VPN connected to firewall as DMZ. Later on, if it’s required to provide specific access between VPN, all the connection can be inspected and filtered through the firewall.

Things get interesting if you want to extend MPLS to the access switches. Cisco encourages to have Routing terminated into the access nowadays, to eliminate the requirement of Spanning-Tree Protocol and HSRP. If we can run routing in access switches, the gateway for all users will be the access switch itself. No STP and HSRP required. And to extend MPLS to the access, we need to have VRF-Lite feature to bind each user VLAN to dedicated uplink to Distribution switches. In distribution switches, each uplink from access switch will be placed into designated VRF.

So it’s coming, everyone. MPLS is already here, and it’s inevitable.
All Enterprise customers hear me now: MPLS time has come.

Choked

I’m overloaded.

From the past couple of months I have been involved in 4 major projects in my company. Some of them are the largest that we have ever dealt with. This promotes my company as the hottest Cisco partner in the country, and we do the hottest project in town together with Cisco directly. And how about my roles? From technical project manager, lead engineer, designer, consultant, implementation supervisor, to logistic manager. Working starts from pre-implementation until training and project hand-over.

I’m under pressure.

All the projects have similar time frame. All of them started from last month, and 2 will start the installation during this summer, while the other 2 will be still in design process.
Meeting, meeting, meeting. Design workshop. Presentation. High Level and Low Level Design. Implementation Plan. Network Ready for Use. Site Readiness. Material Delivery. Staging. Material inspection. Site survey. Testing procedure. Documentation.
So many things to do, so less time to have.

I’m overjoyed.

Four different customers from airport, shopping mall, residential and university. Different technology on each place. From MPLS Layer 3 VPN, very high availability design, 10 Gigabit to the edges, triple play with Multicast and QOS, OSPF multi area, Wireless network with LWAPP and layer 3 roaming, IP Telephony, Firewall and Intrusion Prevention System, up to network management. Different design and customer requirements. Different expectations. Different rules.

I’m choked. I’m choked, to the limit.

I can’t breath. All the workloads. All the responsibilities.
It’s hell a lot of fun, but it’s not worth it anymore.
Keep working days and nights, even during the weekends.
No complaints, until I started getting phone calls from my daughter:
“Daddy, where are you? I want to have my dinner with you.”

I’m choked. And I believe it’s not worth it.
Especially since I still haven’t got my respect.

I need to do something about it.

Respect

Due to the option that I took, I have to give up one of the thing that I like the most: my 325i. Yup, it’s BMW 2002 model with 2.5L engine that can bring me 0 to 100 km/h in 7 seconds. Sunroof and navigation system. Pretty white chick.

I knew it since beginning that the 3rd option would not be easy. It is the red pill that Neo had to take in order to know what the meaning of The Matrix is. Well, it may not be that hard, but it is still not a straight trough highway where I can see the end of the road. Again, the power of uncertainty is something that can really make our life so dynamic, and so painful at the same time.

Anyway, when I sold my car last week and started driving a rental Toyota Corolla, my friend told me that I would lose something that he called BMW Respect.
What the heck is that? It is a respect that other car owners give you in the street, he replied. Everytime you try to change your lane with you BMW or any expensive cars, people tend to give you more room.
That’s silly, I though, there is no such thing.

So here I was, driving my Toyota happily in one of Dubai busy streets. Okay, I was in the slow lane and I wanted to increase my speed. It was time for me to change my lane.
What the…???!!! This guy almost hit my car! He didn’t reduce his speed at all to give me some room to enter the lane.
What’s your problem, dude? Maniac.

I started thinking about my friend’s respect. Nah, it’s just a coincidence. There’s always speedy maniac everywhere.

I kept trying to convince myself until I got the same experience over and over again. Everytime I tried to change the lane, I really had to fight for it. I never faced this issue with my previous car.
What’s wrong with Toyota, guys?
So that kind of respect really exists?

In another day, I went to one store in shopping mall to buy something. I noticed that I was left alone for quite some time, none of the store attendants tried to approach me to ask what I want or offer me services.
Were all of them busy? Not really. That guy was standing in the corner and did nothing other than watching the whole store. What? There was a couple who wore decent clothes and the guy went to them with a big smile and offered his favor.

Okay, I was wearing only normal shirt and jeans. But does it mean I’m not a potential buyer? In fact I was ready to spend my money but the story ended up by me walking out the store due to the way its employee treated me.

My thought about this respect started bugging me. Yesterday I drove my car to my office which is located in one of five star hotels in Sheik Zayed Road, Dubai’s main road.
Normally everytime I drive through the hotel atrium with my BMW, there’s always one hotel officer who offers me valet parking.
Hey, what happens today? Where’s the valet parking guy?
With my company policy I can’t use hotel valet service anyway, but it’s still good to see at least those guys try to show me some respect.

Wait. Did I say respect?
So is my friend definition about respect really true?

Why does such standard exist?
So people must drive expensive car to get respect?
So people must wear decent clothing to get service?

Suddenly I feel vulnerable. I feel insecure.
Not because of my Toyota looks like made from the cheap material just like normal Coke can and it makes me feel really insecure if I ever get into even a small accident.
I feel vulnerable about my life. About the way some people treat other people with their own definition of respect.

I feel insecure about my job.
Is this the reason why even I have been with my current company for 4 years now and never get any raise? Even I believe I have delivered some of my company largest project successfully but as a person I'm still not within the standard to get my respect?

And is this the reason why I still can’t join Cisco ME until now?
No way. Cisco Systems is an Equal Opportunity Employer. At least that’s what written in the website.
And isn’t it clear from the last Gulf partner summit that they would not hire people from partner? But wait. Isn’t one of my CCIE colleagues who used to work with my company joining Cisco recently?
Is it because he has more experience and expertise than me or because some other reasons?

Another hopeless thought.

I just want to work in a place where people respect me only because of my expertise and performance in delivering the job.
And nothing else.
Please let me know if such place exists.