Sunday, December 6, 2009

MPLS vs ATM vs legacy switching guys....

back when I started preparing for my ccip. the first exam and training I attended relevant to the certification was MPLS. it was an intense two week course and the instructor took almost a whole week on ATM/cell-mode/cell switching and generally telling us how important it was to fully understand ATM including popular hardware that supported it. Boy am I glad I paid attention, and that course is paying up big time.

Today I find myself in an interesting position. My employer is jumping headlong in the mpls bandwagon at the core (various departments run mpls for their own little' networks:-))...being a telco you can imagine the fear the 'old school' C7/ss7/ATM legacy type switching guys bring to our meetings. If any one there has gone through this transitions please tell me how you went about re-assuring them that apart from some added complexities:-) life moves on.....

either way Africa and some parts of Asia will probably be the last frontier for any vendor that hasn't managed to penetrate the US and european markets at the core. We have them all cisco,juniper,Huawei,alcatel,ECI etc etc to evaluate..some make it very easy to knock off...I can see at least 3 that will require a little bit of work..

Either way the traditional telco vendor's with their closed model of working centered around the operators is definately coming to an end. The focus is slowly turning to the customer/employee and their fun stuff...oh I also see a post on a similar issue over at

Wednesday, December 2, 2009

what next...

So after the lab, I had to come back to work, all my pending projects, any pending school work, my neglected girlfriend all had to be caught up with and told that it's not over yet.

The girlfriend and other friends were very understanding, my bank statement and employer tended to disagree. So im catching up on projects now till January.

Im in the middle of interesting/challenging projects. Some are fun like implementing a multivendor mpls core (Huawei/Cisco) first in a test environment and later on our main network. (I'll probably post a list of all relevant commands later). Huawei is becoming an interesting 'partner' for african networks. Im also in an interesting bet with a friend on the future of Wimax with LTE hot on its heels. Im making notes elsewhere but will share once i migrate to my own website some time in the future.

the guys at Internetwork expert have me covered on the training and in January the plan is to take a full bootcamp to sort out the the troubleshooting for the R&S ccie.

21st sept 2009....

So I went out to brussels took the lab and was immediately humbled by the experience. Everything was falling into place very well until the last 3 hours of the lab when I discovered a 'fatal' mistake made immediately after lunch. I have a few tips after that experience. All have been mentioned, Im just stressing their importance:

1: read the entire lab before you type a single command.
2: verify after each section. This turned out to be most important for me.
3: The OEQ's are a no brainer, at least for me I didn't see anything out of this world.

So its the troubleshooting section for the new format that has me a bit jittery.

I picked on the ones that really affected me. I currently feel like Gollum/Smeagol, we want my precious ring 'back':-) will take the 4.0 R&S (for everyone that i had told Im moving out CCIE SP, that will have to come after RS, i've invested too much on this to give up now:-)...

see you around....

Monday, May 11, 2009

Interesting turn of events

so after the mandatory 72 Hr wait I get to know whether my two years of toil paid up. I say this because I can't book a lab till the results are posted....otherwise I passed.... And that has to be one of the most challenging 2+ hrs sitting for an exam ever...can't wait for an 8hr lab...though at least i can now concentrate on the lab, first i'll take an assessor from cisco, then map out a plan.

I can only imagine what I'll fell like after passing the lab... The race to actually get a lab slot by October is on, Im pretty sure most if not all are taken.

Things i hated about the exam:
- can't go back to previous questions...arghhhh...
- OSPF and multicast really had me at a corner. I spent too much time on those areas for my comfort.
- I woke up late, didn't have breakfast and I was not even allowed to chew gum during the exam:-)

My wish list:
- that for the full lab fee, I get to do the open ended qustions from home (prometric)and not have to travel so far.
- that i get a lab slot in Dubai or brussels before Oct 18.

oh well...let me go for lunch....

Wednesday, April 15, 2009

ohh for the life of me!!

ignorance more frequently begets confidence than does knowledge: it is those who
know little, not those who know much, who so positively assert that this or that
problem will never be solved by science.

charles darwin - the descent of man

so just in case I ever wonder - in the future of course- why i havent updated this blog for such a long time:

well for starters studying for the ccie reminded me of how little I know - seriously - Im not being modest, I've worked in this field for close to 5 years and I think i've learnt more this year than any experience ever taught me - the professional certs helped but the RS material just blew it up, expanded it to greater horizons, a different cloud..., I just hope in the future I'll be able to put all this knowledge on a network - bring it to life so to say -......either way its mine and Im happy with my progress so far.... Aaanyhow... having to sit for other exams doesn't help much either. Either way my head is i believe well above water - eyes are waterly from lack of sleep- and Im beginning to suspect the ratio of coffee to blood in my veins is skewed.....ohh and dont forget the easter beer:-)

I doubt another post will show up till mid may. by then i should be done with all my exams including the written ccie .... (mainly to renew my ccnp/ccip) and hopefully start getting ready for the lab later this year...or very early next year....cheerszz

Monday, March 16, 2009

IPv6 From the Ground Up : Part - II

ICMP for IPv6 is identified by a header value of 58 in the IPv6 next header field. ICMPv6 is used to report errors and perform internet layer functions eg ping for diagnostics. It's the base protocol for IPv6 and has to be fully implemented and understood by aspiring engineers.

Diagram used for this article is:

IPv6 Neighbor discovery and unicast routing.

Unicast routing is off by default, remember to enable it to allow ICMpv6 neighbor discovery that replaces ARP.

Router0(config)#ipv6 unicast-routing
Router0(config)#int f0/0
Router0(config-if)#ipv6 enable
Router0(config-if)#no shutdown

Router0#sh int f0/0
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is cc00.1368.0000 (bia cc00.1368.0000)

Router0#sh ipv6 interface fastEthernet 0/0
FastEthernet0/0 is up, line protocol is up
IPv6 is enabled, link-local address is FE80::CE00:13FF:FE68:0
No global unicast address is configured
Joined group address(es):
MTU is 1500 bytes
ICMP error messages limited to one every 100 milliseconds
ICMP redirects are enabled
ND DAD is enabled, number of DAD attempts: 1
ND reachable time is 30000 milliseconds

Router0#ping FE80::CE01:13FF:FE68:0
Output Interface: FastEthernet0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to FE80::CE01:13FF:FE68:0, timeout is 2 seconds:
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/40/164 ms

note the expanded 0's (zeroes below), they mean the same thing.

Router0#ping FE80:0000:0000:0000:CE01:13FF:FE68:0
Output Interface: FastEthernet0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to FE80::CE01:13FF:FE68:0, timeout is 2 seconds:
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/32/96 ms

note similar commands have to be run on router 1.
Now lets take some debug to observe this process of enabling th elink local address, but first cover a few basics:
  • IPv6 host adresses are generated from interface mac addresses. from the previouse post (partI), mac addresses are 48 bits and need conversion to 64bit to make a EUI-64 address.
  • ICMPv6 Neighbor discovery is used to resolve layer 3 address to Layer 2 address. in case of ethernet, that would be a mac address to an IP address, or frame relay dlci to an address, or pvc to an ip address etc etc...
  • This is not necessary for point to point links. the router knows that any traffic resolving/recursing to the interface based on the routing table will use whatever layer 2 circuit is assigned to the circuit.
  • no inverse neighbor discovery yet. so all routes should be mapped incase of frame relay (frame relay map ipv6).
ICMPv6 replaces IPV4 ARP for neighbor discovery.
  1. Solicitations - asking other neighbors for info.
  • Neighbor Solicitations - By any general hosts eg desktops and other hosts.
  • Router Solicitations - Devices routing IPV6 eg a default gateway. eg router to router segments.
  • Used to decide what the remote L2 address is of hosts and routers. The two types are there because there is additional info apart from the L2 address. eg routers can tell hosts the network prefix - this way a host just needs to enable IPV6, start sending neighbor solicitations to find out the neighbor, and router solicitation to find out the routers. The router sends back the network bit and the host - stateless autoconfiguration is built into ipv6 protocol stack.
  1. Advertisements - sending informations.

  • Neighbor advertisements
  • Router Advertisemens.
debug ipv6 packets and ipv6nd
ICMPv6: Received ICMPv6 packet from ::, type 135
ICMPv6: Received ICMPv6 packet from FE80::CE00:13FF:FE68:0, type 136
ICMPv6-ND: Sending NS for FE80::CE01:16FF:FE0C:0 on FastEthernet0/0
!note the NS (neighbor solicitation) this is basically like asking' can I use this address?"
IPV6: source :: (local)
dest FF02::1:FF0C:0 (FastEthernet0/0)
!solicited node multicast address...used for duplicate address detection (DAD). ie essentially we ask 'is anyone using this address? in the segment.)
traffic class 224, flow 0x0, len 64+16, prot 58, hops 255, originating
IPv6: Sending on FastEthernet0/0
ICMPv6-ND: DAD: FE80::CE01:16FF:FE0C:0 is unique.

!Note chances of having a conflict are rare in this case since the address is derived from your mac address.and ICMPv6 acknowledges that the address is indeed unique.

ICMPv6-ND: Sending NA for FE80::CE01:16FF:FE0C:0 on FastEthernet0/0
!next we are advertising that we're an IPV6 neighbor with the address above.

IPV6: source FE80::CE01:16FF:FE0C:0 (local)
dest FF02::1 (FastEthernet0/0)
traffic class 224, flow 0x0, len 72+8, prot 58, hops 255, originating
IPv6: Sending on FastEthernet0/0
ICMPv6-ND: Address FE80::CE01:16FF:FE0C:0/0 is up on FastEthernet0/0

ICMPv6-ND: Sending RA to FF02::1 on FastEthernet0/0
ICMPv6-ND: MTU = 1500
IPV6: source FE80::CE00:16FF:FE0C:0 (local)
dest FF02::1 (FastEthernet0/0)
traffic class 224, flow 0x0, len 72+1428, prot 58, hops 255, originating
IPv6: Sending on FastEthernet0/0

Here - above- R0 sends then sends out an RA - router advertisement

ICMPv6: Received ICMPv6 packet from FE80::CE01:16FF:FE0C:0, type 134
ICMPv6-ND: Received RA from FE80::CE01:16FF:FE0C:0 on FastEthernet0/0
and receives an advertisement from R1. Please note no network addresses are set yet, so what you receive is the routers link local address.
Router0#show ipv6 neighbors
IPv6 Address Age Link-layer Addr State Interface
FE80::CE01:16FF:FE0C:0 0 cc01.160c.0000 REACH Fa0/
note the routers above only have link local processing

other commands that show output for different IP versions:
Router0#sh ipv6 int brief
FastEthernet0/0 [up/up]

!shows the link local addresses on our interfaces.

Router0#sh ip int br
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 YES manual up up

Router0#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet - cc00.160c.0000 ARPA FastEthernet0/0

Router0#sh ipv6 neighbors
IPv6 Address Age Link-layer Addr State Interface
FE80::CE01:16FF:FE0C:0 2 cc01.160c.0000 STALE Fa0/0
Router0#sh ipv6 route
IPv6 Routing Table - 2 entries
Codes: C - Connected, L - Local, S - Static, R - RIP, B - BGP
U - Per-user Static route
I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary
O - OSPF intra, OI - OSPF inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
L FE80::/10 [0/0]
via ::, Null0
L FF00::/8 [0/0]
via ::, Null0
!note the Null0, this is because the traffic is local (remember this are not global addresses yet).
FE80::/10 is the entire range of link local addresses.

Router0#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is not set is subnetted, 1 subnets
C is directly connected, FastEthernet0/0

Router0#sh ipv6 int
FastEthernet0/0 is up, line protocol is up
IPv6 is enabled, link-local address is FE80::CE00:16FF:FE0C:0
No global unicast address is configured
Joined group address(es):
!all host multicast, this is where the advertisements are sent to for autoconfiguration.
MTU is 1500 bytes
ICMP error messages limited to one every 100 milliseconds
ICMP redirects are enabled
ND DAD is enabled, number of DAD attempts: 1
ND reachable time is 30000 milliseconds
ND advertised reachable time is 0 milliseconds
ND advertised retransmit interval is 0 milliseconds
ND router advertisements are sent every 200 seconds
ND router advertisements live for 1800 seconds
Hosts use stateless autoconfig for addresses.

Router0#sh ip int
FastEthernet0/0 is up, line protocol is up
Internet address is
Broadcast address is
Address determined by setup command
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound access list is not set
all other commands, telnet etc also work but you need to be specific. Is there get a way to make ipv6 default IPversion

another question:
can you disable IPv4 processing and only have IPV6 processing on a router?

Interesting question came up during this writing:
can you disable IPv4 processing and only have IPV6 processing on a router?

IPv6 From the Ground Up : Part - I Contd.......

From RFC2460, the following information jumps out at you immediately:

- IPv6 is supposedly a succesor to IPv4 (RFC 791). It's uptake (at least in kenya) is a bit discouraging though I get the feeling this will be forced on networks when V4 resources run out.
-Expands the address size from 32 to 128 bits supporting more levels of addressing, more addressable nodes and autoconfiguration. Multicast routing is scaled by using 'scope' and a new address called anycast is defined.
-Header format is simplified to make the packet handling better and limit bandwidth costs.
-Improved support for extensions and other improvements. IPv6 has less stringent limits on length of options.
-another key thing is extension support for authentication and other privacy measures eg confidentiality and integrity can be extended on the header.
-flow labelling capability takes qos to a whole /nother level. eg you can label a flow for which the sender requests special handling eg real time traffic.

IPv6 Header Format as seen on the rfc - modified by my notes:

|Version-4bits|Traffic Class| Flow Label |
| Payload Length | Next Header | Hop Limit |
| |
+ +
| |
+ Source Address +
| |
+ +
| |
| |
+ +
| |
+ Destination Address +
| |
+ +
| |
Extension headers:
Optional information is encoded in separate headers placed between the IPv6 header and the upper layer header. this have to be identified by a distinct header value.
  • Headers are not processed until they reach the destination/s. As such a node has to process extension headers strictly in the order they appear in the packet.
  • if the hop-by-hop header , which must be immeadeately following the IPv6 header is present, then it will be processed by all nodes. its presence is indicated by a value zero (0) in the next field header.
  • ICMP code 1 is sent to a sender if a parameter is not understood ie unrecognized. the same is apparently sent to a packet originator if a value zero is found on any header other than the IPv6 header.
  • If more than one extension header is used, there's a specific order they should appear in. (please read the rfc) i definately don't expect this on an exam. However, IPv6 header, hop by hop options, Destination options, routing and fragmentation followed by authentication, encapsulation and security and a destinations headed then an upper layer header is what's listed.
  • Note the two destinations, one is the final destination and the other is the destination listed in the routing header. ie (once before a Routing header and once before the upper-layer header) That still does confuse me.
  • IPv6 will accept extension headers in any order apart from the hop-by-hop optiones one that has to follow the IPv6 header.
  • A destination header is not examined or processed until it reaches the node identified in the Destination Address field of the IPv6 header.
  • Note, the frgment header is just like in IPv4 used to send a packet larger than the path mtu. However this is only done by the source nodes not routers along the path. the value identifying it is 44.
  • each fragment will have an unfragmentable part, a fragment header and the fragment itself. The unfragmentable part has the payload length of the original v6 header changed to contain the fragmented packet length only, the fragment header id's the first header of the fragmentable header of the original packet. Then obviously the fragment lengths must have a resulting MTU of the path being taken to the destination.
  • For reassembly, the packets must have the same source & destination address and fragment identification
  • note, the fragment header doesn't show up in the final reassembled packet.
  • IPv6 requires that the MTU on each link be either equal to or more than 1280 octets. and fragmentation and or reassembly must be procided by a layer below the IPv6.
  • its recommended that IPv6 nodes implement PMTU.
  • IPv4 ttl is renamed to hop-limit . this is because IPv6 nodes are not required to enforce a packets lifetime. applications curently relyin gon the internet layer for ttl purposes have to be upgraded to have a mechanism to detect and discard obsolete packets
Im not generally a fan of RFC's but for topics I don't understand, I'd rather start from there and work out towards the actual configuration with some sound theories behind me...(there's the added fact that Im out of books budget and RFC's are pretty much the source of most info)

further reading:
Encapsulating Security Payload

Tuesday, March 10, 2009

IPv6 From the Ground Up : Part - I

The Basics

**Please go through rfc2373 in its entirety. Life will be much easier after that.

Our learning topology is a simple: two routers just to show neighbor discovery: This will mainly be used on PartII and any others that follow.

Understanding IPv6 without using it is might not be easy, however playing around with it while planning a CCIE should set you on the right path.

To get a proper grasp of IPv6, you need to understand:
- A link-local address, site-local and global IPv6 address.
- The loopback address (::1) for the loopback interface
- The multicast addresses of joined groups
- Number of bits on an IPv6 address (128 - bits, 16 bytes)

Also very important, what is a modified EUI-address, its purpose and how its generated. I found it also very important to know and understand IEEE 802 addresses.

Basics on MAC addressing:
The IEEE 802 address consist of 24 bit company identifier and a 24 bit extension ID. this is uniquely assigned and gives you a 48-bit address. This 48-bit address is also called the physical, hardware, or media access control (MAC) address.

EUI-64 Addresses
This addressing extends the '24-bit' extension ID on a MAC address to 40 bits. The company/manufacturer ID is still left at 24-bits. This 64 bits are then used to identify the host/node. This is what is called a link local address. Routers do not forward this addresses.

To convert a MAC address to an EUI address, I use the following method. Note this only gives us the link local address, in part 2 or 3 we'll discuss how the rest of the address is completed/generated....lets use an example:

Host X has a MAC address of 12-34-56-78-90-12
on a router, this would be the burnt in address (bia) or the mac address.

First we insert FFFE between the 3rd and 4th bytes ie between the vendor ID and extension ID which results to 12-34-5F-FF-E6-78-90-12/1234-5678-9012. You can easilly do this by slicing the address into two halves.

Next take the first byte (two characters=1 byte) so in our case the first byte is 12 (note this is in hex) and convert it to binary - 0001 0010.. Take the 7th most significant bit and flip it/or invert it, this gives 0001 0000. Convert this back to hex and you get :
put this in proper notation for IPv6 and get:

In case you get hang up on wording

The IEEE now considers the label MAC-48 to be an obsolete term which was previously used to refer to a specific type of EUI-48 identifier used to address hardware interfaces within existing 802-based networking applications and should not be used in the future. Instead, the term EUI-48 should be used for this purpose.

RFC2373 - IP Version 6 Addressing Architecture
My friend tells me most of what he learnt on IPv6 was solidified at an internetwork experts boot camp so go over to their site and grab some work book, have no idea which one in particular.

Part II
IPv6 Neighbor Discovery:

Sunday, March 8, 2009

CCIE motivation

today is a dark dark day, last week was a bit of an anti climax for me, trying to sort things out so I can afford the LAB, figure out the best place to take it (down to brussels, India and or Dubai)... Im thinking some place I havent been and a sibling might be visiting India hmm why not India for the ccie?

anyway''''my morale is pretty low today, I think its just the thought of getting up to go to work tomorrow, I'm actually starting to look at the CCIE as a means to just switch employers, and do something more intense, challenging...anyway.. cant concentrate much so re-running lord of the rings...hah I also went to an asian fruit market and bought a bunch of froots...., ahh yes also confirmed the order and hopeful delivery of a triathlon kit .

Im done rumbling....oh yes also made payment for the written in April:-)

Tuesday, March 3, 2009

CSM module and a 7613 woes...or fun it depends

So i tried to bring a csm/csg module up on another chassis. All modules, including the Supervisor Engine (if you have redundant Supervisor Engines),
support online insertion and removal (OIR). You can add, replace, or remove modules without interrupting the system power or causing other software
or interfaces to shut down. So no worries there. the colors changed as expected on the LED....then went off........

SUP1234#sh module 13
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
13 4 Content Services Gateway WS-SVC-CSG-1 SAD094906MP

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
13 0013.c39f.1270 to 0013.c39f.1277 1.5 Unknown Unknown PwrDown

Mod Online Diag Status
---- -------------------
13 Not Applicable

so from the above we can tell there's a problem. Hopefully not too serious.

SUP1234#hw-module module 13 reset
Proceed with reload of module?[confirm]
% module 13 is operationally off (Module Failed SCP dnld)

I only ever had this SCP dnld issue with MWAMS never a CSG/CSM module. I cant even trace back to a bug showing this as a problem...YET.
I tried to power it up manually:

SUP1234#conf t
Enter configuration commands, one per line. End with CNTL/Z.
SUP1234(config)#power enable module 13

still not shows up as failed SCP dnld....
so resetting the module with hw-module module 13 reset and power enable module 13 both fail to power up my module.
If this were an MWAM i'd assume something buggy, but this is a new module, granted the hardware is a bit newer than the other csg's..

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 000f.342c.5318 to 000f.342c.531f 1.4 3.1(3)C7(7) Ok
2 0011.93b4.5698 to 0011.93b4.569f 1.4 3.1(3)C7(7) Ok
3 0011.5c81.1e6c to 0011.5c81.1e9b 6.1 6.3(1) 8.5(0.46)RFW Ok
4 0011.5c81.157c to 0011.5c81.15ab 6.1 6.3(1) 8.5(0.46)RFW Ok
5 0001.c9dd.0f5e to 0001.c9dd.0f65 1.4 3.1(3)C7(7) Ok
6 0002.fcc1.f844 to 0002.fcc1.f84b 1.4 3.1(3)C7(7) Ok
7 0016.46f9.0c58 to 0016.46f9.0c5b 5.3 8.4(2) 12.2(18)SXF7 Ok
8 0016.c85e.a958 to 0016.c85e.a95b 5.3 8.4(2) 12.2(18)SXF7 Ok
9 0001.c9de.32a0 to 0001.c9de.32a7 1.7 3.1(8) Ok
10 001b.53bc.b038 to 001b.53bc.b03f 6.1 7.2(1) 2.1(3.0) Ok
11 0011.92b7.c748 to 0011.92b7.c74f 4.0 7.2(1) 2.1(3.0) Ok
12 001d.70c4.fc14 to 001d.70c4.fc43 3.0 12.2(18r)S1 12.2(18)SXF7 Ok
13 0013.c39f.1270 to 0013.c39f.1277 1.5 Unknown Unknown PwrDown

so check power:
I have enough power....and its the only one not working....Im out of slots on the chassis so I can't move it around....grrrr.....
I'll move it to a different chassis and see....maybe the module got 'corrupt' while being moved from one chassis to another ( i had it somewhere else before) and wanted to spice up my afternoon.

anyone have a clue on this one? id be interested.

RS blueprint

looking at the Expanded blueprint by IE , I think I understand why multicast is such a pain, I missed out on key areas.....Im re-doing/ re-reading most of th ematerial again.

I could sit for the written any time now but until Im pretty sure of the lab payments there's no need to rush it for now, it how ever means I start preparing for the lab and ensuring nothing gets forgotten.

I have recently renewed two professional certs and while that doesn;t guarantee my written pass all the reading i did after that should get me one...or very very close (its still an exam so I'll keep my fingers crossed:)

Friday, February 27, 2009

what was cooking today:

after the csg/ggsn exploits, the better part of my week is freed up. I like it when I do something and the impact is felt almost immediately by customers, the feedback is quite refreshing.

An assesor lab some time back showed me weak on IPv6, multicast and i need to polih up on L2 technologies.

Either way I think Im still on track for the CCIE. time to start redirecting my finances towards this goal. This is probably going to be the most painful one since money is hard to come by.

anyway,,...I was on the following links today for other reasons mainly work related, and since its slowly becoming clear that data center networking might be my next bread and butter:-)

**Yes I had nowhere to book mark the pages and needed a quick reference point...:-)

Configuring Enhanced Service-Aware Billing - on the ggsn ( i deal with mobility so ggsn's sgsn,s are sort of my main dish:-)
Technical Resources for the Enterprise - Cisco Design Zone
Data Center Assurance Program (DCAP) 3.0
Protocol Compliance Statements for the CSG 3.1(3)C7(1)
Cisco DCAP Data Mobility Manager (DMM) Design Guide (External)

Tuesday, February 24, 2009

CMX CSG upgrade:

Today had me doing an MOP for some upgrades. This is for a cisco CMX running on a 7613 chassis. It assumes you plug in a CSG module on slot 13 and it doesn't have the right software installed but you have it on disk0 on the supervisor.

Next I will show how to put it together with some ggsn's, configure and test billing for mobile users.

Directory of disk0:/
5 -rw- 4736628 Jun 25 2008 02:33:06 +03:00 c6csg-apc.31-3.C7.7.bin

The above shows the image we want is in disk0
Now make the file above accessible via tftp as follows:

CMX1(config)#tftp-server bootflash: c6csg-apc.31-3.C7.7.bin

Now the file above can be reached and picked via tftp by the csg. Another alternative is to use our tftp server.

CMX1#session slot 13 processor 0

The default escape character is Ctrl-^, then x.

You can also type 'exit' at the remote prompt to end the session

Trying ... Open


www.C o n t e n t w

www.S e r v i c e s w

www.G a t e w a y w


CSG> dir


upgrade slot0:|server-ip-addr filename

ping ip-addr

show ...

copy coredump tftp|rcp ip-addr filename [rcp-user]

capture [on|off]

pktlog ...


CSG> upgrade slot0: c6csg-apc.31-3.C7.7.bin

Upgrading System Image 1

CSG ExImage Nov 8 2007

R/W| Reading:lam_ppc.bin..DONE Writing:lam_ppc.bin..DONE

Read 13 files in download image. (13,0,0)

Saving image state for image 1...done.

CSG> exit

Good Bye.

[Connection to closed by foreign host]

You should get output similar to the above.

CMX1# hw-module mod 13 reset

When the module comes up sh mod should give the output almost exactly like below:

CMX1#sh mod

5 0001.c9dd.0f5e to 0001.c9dd.0f65 1.4 3.1(3)C7(7) Ok

That’s it your CSG is upgraded.

Thursday, February 19, 2009

stil on track

Ahhh Im still on track for the ccie, had an assessor lab - still need a bit of lab time- though work has really intruded on my time table. However the work has mainly been fun appart from a few 'people' related political annoyances. I have also learnt how important some politics is to get some jobs done so I won't complain much.

Im rerouting my finances to cover for the ccie. Giving it a single shot and I don't see why it should not be passable by year end.

Im done with most of the books, so labs from now on, probably upgrading some hardware (ive been at work alot lately and the current laptop can't hack more than 5 routers on dynamips). so a new one is in order (pre-exam gift):-)

Sunday, January 25, 2009

My left knee is a traitor!!

So i took a break from work, reading and all things indoors today (sunday). One other goal I have this year is to get fitter, swim better - longer, jump a higher bungee, run a 21Km (half marathon) without something on or in me giving up.

So to test my current status, I went for a triathlon:-)....The swim almost killed me, so bad was I my event was changed to a duathlon - didn't finish the swim....:-(I'm yet to take the ccie lab but i suredo hope my chances with it are better than what I had in water...

I transitioned after the last swimmer - to be fair- and off we went cycling....I think apart from laziness, and probably terrible eating habits, I'm a strong, good cyclist...the trails at KU-the university are not bad at all, parts of it are single track, brief vegetation then its you and open ground....

Ahhh then came the run, after about 4Km, my left knee totally gave up on me, the rest of the body definately felt betrayed, there was this awfully sharp pain from the back of my knee (I don't run much) and I swear I could hear my heart bouncing with every ended well, heck they gave me a medal - probably to console me:-) the running kicked my ass though...This knee thing! not sure whether to get it checked out, so if you get by here and have a clue let me know...

Im dedicating more hours to the run and general fitness stuff every week, not sure what to do with the swim, need a complete a full triathlon by end of year I need to probably put in as much time as I'll do the ccie lab, It will be very satisfying to accomplish both by end year-something I plan on doing-....either way I find that getting out every once in a while works wonders for my concentration..... ahhh now back to IPV6.....

Thursday, January 22, 2009

Etherchannel Load Balancing and Forwarding Methods

a couple of WS-C3750E's - had a strange problem (mainly with my assumptions) on how etherchannel load balances. I thought it was automatic:-) it should feel what i need and do it!!!...

Imagine two ports bound to form one port channel giving 200Mbps. Now imagine on one end you have two hosts/servers that generate/carry a lot of traffic (an ftp server for instance) to multiple destinations on the other end - internet, auth servers etc etc...

One of the hosts has more traffic than the other, infact alot more then 98Mb. so when traffic hit 100Mbps, i started noticing random packet drops. why why why...I thought this is a 200mbps interface???

upon further checks we discovered that one of the interfaces within the bundle was dropping packets/frames.

so the checks started:

Gitau-Switch-01-Sw#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:

src-mac was the default.
Now according to cisco :
EtherChannel load balancing can use either source-MAC or destination-MAC address forwarding.

With source-MAC address forwarding, when packets are forwarded to an EtherChannel, they are distributed across the ports in the channel based on the source-MAC address of the incoming packet. Therefore, to provide load balancing, packets from different hosts use different ports in the channel, but packets from the same host use the same port in the channel (and the MAC address learned by the switch does not change).

With destination-MAC address forwarding, when packets are forwarded to an EtherChannel, they are distributed across the ports in the channel based on the destination host's MAC address of the incoming packet. Therefore, packets to the same destination are forwarded over the same port, and packets to a different destination are sent on a different port in the channel.

So obviously the default load balancing was not working for me.

Since this was a 3750, I correctly figured that it can also use IP. playing around with the setup on the end that was dropping packets, the following sort of sorted me out:

Gitau-Switch-01-Sw#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:

the command to make this change is:

port-channel load-balance src-ip

you can play around with:

Gitau-Switch-01-Sw#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr

you can run a test by:

test etherchannel load-balance interface port-channel [#] ip [src] [dst]



Gitau-Switch-01-Sw#sh int g1/0/24 | include drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 20191

now :

Gitau-Switch-01-Sw#sh int g2/0/24 | include drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Gitau-Switch-01-Sw#sh int g1/0/24 | include drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

I figure after some time I'll come across a few drops..because the network hates me!!!

Tuesday, January 20, 2009

Understanding OSPF contd...

After about 20 hours lab and roughly 10 hours going through OSPF, I still can't say I have the confidence to go through OSPF without going down belly fast in the CCIE lab. More practice is definately called for to get at least to 70% confidence levels.

So i decided to just make some notes, go through BGP and start on Multicast as the information digested. Plus Im curious how much of all this material I'll have to cover again.

Apparently OSPF is the most widely used IGP. It brings in the concept of Areas. If you imagine a midsized no actually a midsized company in Europe is probably a LARGE company in Kenya, so if you imagine a LARGE:-) company, an area would probably be a building or a department.

Router ID's, Neighbors, Adjacencies, LSA's *,Hello protocol,Areas, election (DR,BDR) are terms you'll come across and if the books you're reading don't address them:-), google will sort you out.

OSPF sort of goes around the whole network and maps out what you ask it to giving you paths to destinations, its a link state protocol.

OSPF order of operation:
1:A router sends Hello packets, discovers neighbors and elects a Designated Router. Link-state information and a list of neighbors is included in the packet.

*Mar 1 00:15:08.803: OSPF: Send hello to area 1 on Serial1/0 from
*Mar 1 00:15:10.903: OSPF: Rcv hello from area 1 from Serial1/0
*Mar 1 00:15:10.911: OSPF: End of hello processing
*Mar 1 00:15:11.071: %OSPF-5-ADJCHG: Process 1, Nbr on Serial1/0 from LOADING to FULL, Loading Done
*Mar 1 00:15:18.807: OSPF: Send hello to area 1 on Serial1/0 from

*Mar 1 00:15:20.907: OSPF: Rcv hello from area 1 from Serial1/0
*Mar 1 00:15:20.911: OSPF: End of hello processing

This is an initial hello.

Debugging ip packet detail shows the messages a bit clearly:

*Mar 1 00:20:48.815: IP: s= (local), d= (Serial1/0), len 80, sending broad/multicast, proto=89
*Mar 1 00:20:50.899: IP: s= (Serial1/0), d=, len 80, rcvd 0, proto=89

Protocol 89 is OSPF, is the ALLSPFRouters multicast address, from th esource you can also tell whether we originated the hello or not. Since this are serial links, no DR/BDR election takes place.

Let me swap out the links with ethernet to see how that works out....on my next post. Seems the lab is inaccessible....this is a simple lab, I think I'll dynamip on the laptop for a while:...

Sunday, January 18, 2009

back...yes Im back and it feels good....

Yani, that was one looong busy buuusy week.....Didn't get much done...however this weekend sort of saved it. I got through almost everything to get me sort of set for the February deadline for the written exam. Hopefully some projects at work won't come in the way, and if they do they better be fun..On the solaris+squid+wccp, my advise is don't run it in a busy production environment, far too many things coud go wrong...

OSPF from the ground up from tomorrow.....

Friday, January 9, 2009

new day, new solaris + squid :-challenges

Great another morning, my solaris 10, squid installed, but in comes another requirement. Run wccp2 on it. Im happy because I get to mess around with a router, sad because Ive done this countless times on linux and freeebsd but never on solaris. So quickly go to sunfreeware; not a thing when i search for wccp. hmm back to basics, maybe we can work around this by first understanding the requirements ( I always try to know the way a protocol works first before adapting it for whatever system I need it for), this way If i ever need to configure lets say OSPF on juniper, so long as I know how SPF works, i can run it on anything:

I also decided to take some debug output for future reference:

Commands you'll use on the router:

show ip wccp web-cache detail
show ip wccp web-cache view
debug ip wccp events - display WCCP events
debug ip wccp packets - display WCCP packet information

Here's how wccp works with squid:

Once squid is started with the wccp option, the router sees this and sends an I see you message as follows (I had always assumed the here I am message comes first - I was obviously wrong:

Jan 9 08:57:51.584 NAIROBI: WCCP-PKT:S00: Sending I_See_You packet to 196.201.xx.xx w/ rcv_id 00006C9C

and you get this log below on the router - this is not a debug output.

Jan 9 08:58:01.593 NAIROBI: %WCCP-5-SERVICEFOUND: Service web-cache acquired on WCCP client 196.201.xx.xx

Squid then sends back HERE_I_AM packets, and carries on a conversation with the router.

Jan 9 08:58:01.593 NAIROBI: WCCP-PKT:S00: Received valid Here_I_Am packet from 196.201.xx.xx w/rcv_id 00006C9C

If you were also debugging events, you get the following:

Jan 9 08:58:01.593 NAIROBI: WCCP-EVNT:S00: Assignment wait timer started
Jan 9 08:58:01.593 NAIROBI: WCCP-EVNT:S00: Built new router view: 1 routers, 1 usable WCCP clients, change # 0000004D

From then on traffic gets redirected to squid.

and if you disconnect you get the following event/log:

Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Assignment wait timer started
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Built new router view: 0 routers, 1 usable WCCP clients, change # 0000004B
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Router removed.
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Assignment wait timer started
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Built new router view: 0 routers, 0 usable WCCP clients, change # 0000004C
Jan 9 08:57:37.279 NAIROBI: %WCCP-1-SERVICELOST: Service web-cache lost on WCCP client 196.201.xx.xx

On the router, the basic config is widely available on the internet....create an access-list for traffic to redirect, enable wccp etc etc...

This conversation (between router and squid) is totally separate from the ip_wccp or ip_gre module --these packets never go through those channels. For so long as the router is receiving WCCP HERE_I_AM packets (with the proper ID), the router will send traffic to the cache IP, encapsulated in a GRE packet, and there in lies the problem or the gotcha as ccie candidates like saying. In a production if you do this without confirming the tunnels, all your redirected traffic will be blackholed by squid.

The GRE decapsulation is a separate process from squid. Squid doesn't talk to the gre/ip_wccp module, The gre decapsulation occurs at the network layer in your kernel, and then the packets enter the normal routing table - on bsd/linux it involves iptables/chains
to hijack any packets destined for port 80 on the internet - In my case thats traffic coming from the router in GRE packets and passes it over to the cache engine's port.

I wonder what happens if I let squid listen on port you still need something to 'jack' the packets? I'll try that in a while and let you know....

So now with that understanding, you'd think I fix this on solaris pretty fast yes??? nope, nada, my cluelessness gets in the way....AGAIN!!

So again back to google, I can't get an ip_wccp for solaris module but Im sure I will get gre and the ip redirect working...and get more CLUEFUL overall then we can go back to regular work, If you stumble upon this and have a clue...bail me out:-)pleasee....I still have about 5 bgp, 5 ospf LABS to get through before Sunday....

Thursday, January 8, 2009

Grrr...Solaris 10 install

I have generally always HATED installing operating systems. Unfortunately today caught me doing just that. Installing Solaris 10 on a v490 that just refused to cooperate with me. So there I was huddled in the data center at temperatures my body doesn't agree with; an installation that took a minimum 2 hours (had to do it twice - resized the disks wrongly the first time) so 4 hours....grrr.....a fun fun......If I curse or throw a shoe at you in traffic blame SUN:-) or solaris...and forgive me!!

and it still bloody refused to boot in the end. damn thing couldnt initialize the boot disk without my help , kumbe the boot sequence was wrong:-(

Retrying network initialization

Can't open boot device

{3} ok boot disk <<---this resolved my first problem, finally booting - 4 hrs down the line
Boot device: /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@0,0 File and args: Loading ufs-file-system package 1.4 04 Aug 1995 13:02:54. FCode UFS Reader 1.12 00/07/17 15:48:16. Loading: /platform/SUNW,Sun-Fire-V490/ufsboot Loading: /platform/sun4u/ufsboot

then came installing squid, which is what i wanted to do to begin with. If you've installed solaris you know how bare minimal instal gets done. So we always keep a set of standard packages to install to get things going:

## pwd
# ls
apache-2.2.6-sol10-sparc-local libintl-3.4.0-sol10-sparc-local php-5.2.5-sol10-sparc-local
curl-7.18.0-sol10-sparc-local libxml2-2.6.31-sol10-sparc-local php-5.2.6-sol10-sparc-local
expat-2.0.1-sol10-sparc-local make-3.81-sol10-sparc-local python-2.5.1-sol10-sparc-local
flex-2.5.33-sol10-sparc-local mysql-5.0.51-sol10-sparc-local sed-4.1.5-sol10-sparc-local
fping-2.4b2-sol10-sparc-local ncurses-5.6-sol10-sparc-local thttpd-2.25b-sol10-sparc-local
gcc-3.4.6-sol10-sparc-local netsnmp-5.4.1-sol10-sparc-local wget-1.11.1-sol10-sparc-local
libiconv-1.11-sol10-sparc-local openldap-2.4.11-sol10-sparc-local zip-2.32-sol10-sparc-local
libidn-1.6-sol10-sparc-local openssl-0.9.8f-sol10-sparc-local zlib-1.2.3-sol10-sparc-local
so we keep those around and use the standard pkgadd'er:

pkgadd -d

so how long do you think that sort of took?....

oh yes I also had missing paths:

export PATH=/usr/bin:/usr/local/bin:usr/sbin

setenv PATH /usr/bin:/usr/local/bin:usr/sbin

adding me as a user so we can ssh and leave this cold place...:

bash-3.00# mkdir /export/home/jgitau
bash-3.00# useradd -d /home/jgitau -m -s /usr/bin/bash -c "John Gitau" jgitau
bash-3.00# passwd jgitau
New Password:
Re-enter new Password:
passwd: password successfully changed for jgitau

scp/ftp the files or use whichever method makes you for apache

scp apache-2.2.6-sol10-sparc-local jgitau@196.201.xx.xx:/export/home/jgitau/SUNFREEWARE

aaanywayz....i finally got to compile squid and it was a happy day, just not a single ccie sort of reading will get done today...

Ha...yes this post is mainly just to remind me what packages to get downloaded next time, and simple commands I sometimes forget...and general noise for the blog...:-)...

seriously though, you can let me know if you do get a problem with this sort of thing, live around nairobi, and you promise not to take me to a data center with working air conditioners - because then it would be a nice warm environment-) for me:-) hmm I didn't see any DustPuppies, I relax over at userfriendly alot..probably too much

Tuesday, January 6, 2009

Understanding OSPF....

After going through ospf, and to properly understand some concepts, I hit the intanet looking for pre-done dynamips labs:-) to save on time. during my cyber walk, I came across the article
Configuring Basic OSPF (Dynamips). It was excellent and had a very well detailed instruction on things.

However one of the requirements for the lab was:

That the ISP router always be the DR, one of the Remote sites always be a BDR and one remote site never participates in the election process. (Please go to for the diagram and full article - but it looks something like below all routers connected through switches.


I redid the whole lab with emphasis on meeting the designated router (DR)/backup designated router (BDR) requirements

ISP#sh ip int br | exclude un
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 YES manual up up
Loopback0 YES manual up up

Remote1#sh ip int br | ex una
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 YES manual up up
Loopback0 YES manual up up
Loopback1 YES manual up up
Remote2#sh ip int br | ex una
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 YES manual up up
Loopback0 YES manual up up
Loopback1 YES manual up up

After bringing up the interfaces, I tested connectivity as evilrouter dude had done it - well almost:-):

ISP(tcl)#foreach address {
+>} {ping $address}

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Success rate is 80 percent (4/5), round-trip min/avg/max = 32/58/96 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Success rate is 80 percent (4/5), round-trip min/avg/max = 16/45/88 ms

Now we get OSPF going, just as he had done it, I'll just show the neighbors here:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 1 FULL/BDR 00:00:34 FastEthernet0/0 1 FULL/DROTHER 00:00:38 FastEthernet0/0
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 1 FULL/DR 00:00:31 FastEthernet0/0 1 FULL/BDR 00:00:39 FastEthernet0/0
Remote2#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 1 FULL/DR 00:00:39 FastEthernet0/0 1 FULL/DROTHER 00:00:31 FastEthernet0/0

-we have proper basic ospf

Since Im interested in the DR/BDR, I'll skip the details here since the original
at still has them.

lets go over the requirements again in proper detail for clarity:
1: ISP - will always be the DR
2: Remote1 - will always be the BDR
3: remote2 - never participates in an election.
Remote2 is easy:

Remote2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Remote2(config)#interface fastethernet 0/0
Remote2(config-if)#ip ospf priority 0

ISP is also easy:

ISP(config)#interface fastethernet 0/0
ISP(config-if)#ip ospf priority ?
<0-255> Priority
ISP(config-if)#ip ospf priority 255

The neighbors currently look like this:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
ISP#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 0 FULL/DROTHER 00:00:32 FastEthernet0/0 1 FULL/DR 00:00:35 FastEthernet0/0
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 255 FULL/BDR 00:00:36 FastEthernet0/0 0 FULL/DROTHER 00:00:38 FastEthernet0/0
Remote2#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 255 FULL/BDR 00:00:33 FastEthernet0/0 1 FULL/DR 00:00:38 FastEthernet0/0

so according to the above output: - is fine, it won't participate in the election process.

Immediately something is wrong with the BDR/DR election . Since I reset the Remote2 OSPF process first, it got to be DR first and the ISP router had to be the BDR. That I'm sure starts to make my point but, just to ensure we go through all the evil router guy's/guyette's??:-) steps: I'll go ahead and change the priority of the RemoteRouter to 254 and clear the ISP router's ospf process first - !so it gets to be DR, followed by the others and the output is as follows:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 0 FULL/DROTHER 00:00:30 FastEthernet0/0 254 FULL/BDR 00:00:34 FastEthernet0/0
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 255 FULL/DR 00:00:32 FastEthernet0/0 0 FULL/DROTHER 00:00:34 FastEthernet0/0
Remote2#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 255 FULL/DR 00:00:30 FastEthernet0/0 254 FULL/BDR 00:00:36 FastEthernet0/0

However, If I now reset the ISP router's ospf interface, or if the link to the ISP were to go off:

Remote1#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 0 FULL/DROTHER 00:00:38 FastEthernet0/0

*Mar 1 01:06:44.087: %OSPF-5-ADJCHG: Process 1, Nbr on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expiredsh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 254 FULL/DR 00:00:39 FastEthernet0/0

So now we have a situation where the former BDR was upgraded to DR. The reason I think your requirement for the ISP to always be a BDR cant be met with
this configuration is so long as there's a BDR that gets upgraded to a DR and no pre-emption happens, then your ISP will always be either a DR or BDR even
under normal circumstances. The priority only kicks in if the processes are reset at the same time.

Here's the output after bringing up the ISP router.

*Mar 1 01:12:17.647: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar 1 01:12:18.647: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
*Mar 1 01:12:25.631: %OSPF-5-ADJCHG: Process 1, Nbr on FastEthernet0/0 from LOADING to FULL, Loading Done
*Mar 1 01:12:25.719: %OSPF-5-ADJCHG: Process 1, Nbr on FastEthernet0/0 from LOADING to FULL, Loading Done
ISP(config-if)#do sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface 0 FULL/DROTHER 00:00:37 FastEthernet0/0 254 FULL/DR 00:00:34 FastEthernet0/0

ISP#sh ip ospf interface f0/0
FastEthernet0/0 is up, line protocol is up
Internet Address, Area 0
Process ID 1, Router ID, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State BDR, Priority 255
Designated Router (ID), Interface address
Backup Designated router (ID), Interface address
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:05
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 2/2, flood queue length 0
Next 0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 1
Last flood scan time is 0 msec, maximum is 4 msec
Neighbor Count is 2, Adjacent neighbor count is 2
Adjacent with neighbor
Adjacent with neighbor (Designated Router) !<<<------ note the DR!
Suppress hello for 0 neighbor(s)

So the question is, how do you ensure that a router is always a BDR? Is that even possible? How to go about it?

Lastly, many thanks to whoever runs for this article

Sunday, January 4, 2009

confused laziness.....and expect:-) (sunday) caught me trying to catch up with OSPF but I just couldn't stand the books, sites - and a friend who'd come for an apparently 'intelligent discussion on nssa's and routers, or sgsn's etc etc', in the end we opened a couple of beers, fed the fish (the only other living thing in my house is 4 fish) and yapped about current affairs, state of african economies and how soon we're going to be travelling all over the place setting up networks sampling local brews:-) - Please note kenya and the general (east) african region has relied on satellite communication for so long, however we have submarine fiber coming in by 2010-2012 so yes I see a very bright future for networkers:-) no bootcamps, no authorized training centers for high end certs, no internetwork expert, no get certified here you really have to lab up and put in the time...ahh the joys.

Anyway after several hours of that 'drinking', the neurons were perfectly fired up to engage in some creative boredom activity. I thought of one of the most annoying repetitive tasks my team has to undertake daily and decided to get rid of it.

Every once in a while we get customers calling our 1st line support guys just to check if their remote sites have established a pdp session successfully. On a cisco GGSN the command would be:

show gprs gtp pdp-context msisdn 2547221x1x2xx

To get the following:

36xx72xxxxxxxxx 196.xx.xx.01 LOCAL 2547221x1x2xx safaricom
In most cases the next request could be to clear the pdp context so they can re-establish it again

clear gprs gtp pdp-context tid
So in comes EXPECT:
this is an extraordinarilly great tool for writing scripts for the lazy sysadnmin to drive other programs. It recognizes prompts and sends keystrokes in response. It was written by Don Libes of NIST, and you can find papers on it in Usenix LISA (Large Systems Administration) conference proceedings, as well as on the Internet.

My drunken goal was to create a web interface where the users can check this for themselves.

The expect script itself was pretty straight forward:

#!/usr/local/bin/expect -f

set force_conservative 0 ;# set to 1 to force conservative mode even if
;# script wasn't run conservatively originally
if {$force_conservative} {
set send_slow {1 .1}
proc send {ignore arg} {
sleep .1
exp_send -s -- $arg

set msisdn [lindex $argv 0]
set timeout 3
spawn /usr/bin/bash
match_max 100000
send -- "telnet 196.x.x.x\r" # your GGSN/cisco routers IP address
expect -exact "telnet 196.x.x.x\r # your GGSN/cisco routers IP address
Trying 196.x.x.x...\r
Connected to 196.x.x.x.\r
Escape character is '^\]'.\r
User Access Verification\r
Username: "
send -- "drunkenmaster\r"
expect -exact "drunkenmaster\r
Password: "
send -- "jedimaster\r"
expect -exact "\r
send -- "en\r"
expect -exact "en\r
Password: "
send -- "jedimaster\r"
expect -exact "\r
send -- "show gprs gtp pdp-context msisdn $msisdn"
expect -exact "show gprs gtp pdp-context msisdn $msisdn"
send -- "\r"
send -- "exit\r"
send -- ""
expect eof

Next do a simple perl,php or whatever makes you happy script, put it on a web server somewhere and guys can access it with a url. The handiwork can be viewed here:
* i just modified an old script we use for users to change unix passwords still using expect and adapted it for my needs.

Please note the above url is only active as a demo - in any case it would only be applicable to our users but im sure it explains the concept

On putting in a valid msisdn the output should show up in follows - or you can use the url above:
Please check from LINE 17 (seventeen) thats where your output will be

GGSN returned following information:
[0] => spawn /bin/bash
[1] => telnet 196.x.x.x
[2] => www-data@monitor01:/var/www$ telnet 196.x.x.x
[3] => Trying 196.x.x.x...
[4] => Connected to 196.x.x.x.
[5] => Escape character is '\^]'.
[6] =>
[7] =>
[8] => User Access Verification
[9] => [10] => Username: drunkenmaster
[11] => Password:
[12] =>
[13] => GGSN-xx01>en
[14] => Password:
[15] => GGSN-xx01#show gprs gtp pdp-context msisdn
[16] => TID MS Addr Source SGSN Addr MSISDN APN
[17] => 36xx72xxxxxxxxx 196.xx.xx.01 LOCAL 2547221x1x2xx safaricom
[18] =>
[19] => GGSN-xx01#exit
[20] => Connection closed by foreign host. [
21] => www-data@gitaus-TestServer:/var/www$

please note this is a sample output and no script is actually run to get this output

Please contact the data team! To run another query, : Please click me to run another query! "

Ahh finally success, the php script just inserts the msisdn....if you're interested in it, leave a comment and I'll probably just upload it...

You can run this for virtually anything, show bgp neighbors, for users to change unix passwordsetc etc..just modify the commands as you see fit. You can for instance give your users access a router to check the status of one of their interfaces, or clear statistics or whatever makes you happy, its a bonus if you have the security guys screaming at you:-) no seriously ensure your corporate policy is followed when doing some of this things...

Also if you forget to modify the tty's on a router, users might lock you out of telnet access:-)

please note this could obviously work better, but i wasn't looking for better, i needed to kill some time and be productive at the same time. For instance we could post process the output to only show line 17 ( I also generally prefer if guys especially co-workers to at least have an idea of whats run and what the output would look like from a console)...but time's up, and I think I'll have an easier time tomorrow...maybe some day....

Saturday, January 3, 2009

Understanding EIGRP from the bottom up Part I I

EIGRP Metrics:
Routers 'discuss' their topology tables. So unlike other protocols, EIGRP has a table with all the routes to a destination.

show ip eigrp topology all-links
!--is the command that shows this table - all links is optional
This topology table is the one holding all the info to make decisions, the outcome is usually a distance and vector to each destination.

Minimum bandwidth and the total delay is the information used to compute the metric. The values are automatically picked from configured values on your interfaces. Generally bandwidth will be more critical for lower bandwidth interfaces while delay is more key where high speed interfaces are used.

Formula to get :
the bandwidth:
bandwidth = 10 000 000 / bandwidth * 256
The delay
delay = delay * 256

The values got for bandwidth and delay are then used in the metric computation. I won't go into detail, please check this cisco link or read a book.

the default behaviour is to just calculate as follows:
metric = bandwidth + delay

(remember to round off your figures after each calculation, I can't remember why but i think floating point math on cisco was the reason)

so if you have the following scenario:

bw 56k
/Delay 2000---Router4---| |
/ | |
Router1== |---------Router2------|
\ |bw10000 |
\__bw 128 |delay100 |Destination Network
Delay 1000-Router3---| |bw 10000
|delay 100

metric = bandwidth + delay
minimum bandwidth = 56k
Total Delay = 100 + 100 + 100 + 2000
= 2200

[(10 000 000/56) + 2200] x 256 = (178571 +2200) x 256
= 180771 x 256
= 46277376

10 000 000/56 is actually = 178571.42857142857142857142857143 but we round it off...
If you calculate the metric through router 3, you find the metric is 20307200

so router one will use the path via Router3 to get to the destination network.
  • The bandwidth is calculated from the configured interface through which the desired network is visible ie where the update is coming in through.
  • The delay is cumulative; ie each router adds a delay and sends it backwards...Im not sure how to explain this if you can't see it....just count the delay from the destination backwards

Feasible Distance (FD)
  • This is the best metric/best path to the destination network - includes the metric to the neighbor advertising the network - from our diagram thats Router2.
reported Distance (RD)
  • This is the total metric as advertised by an upstream router/neighbor. from the ascii diagram above, that will be the distance advertised by Router4 or Router3 (obviously one is going to be used as a FD
feasible Successor (FS)
  • This is the path whose reported distance is less than the feasible distance. This is usually installed in the topology table as a backup.
The reported distance is always calculated from the router advertising the route to the network. so the reported distance from router4 is the metric to get to the destination network from Router4 and the same goes from the reported network from Router3.

If the link between Router1 and Router3 goes down, the convergence is almost instant, users will probably not even notice it since the feasible successor/backup route is immediately picked.

Checking loops:
The FD,FS and RD concepts are used to breack loops using the logic that the reported distance CANNOT be higher than the feasible distance. A route/path with a higher Rd than the FD won't show up on the topology table.

Other loop 'taking care of mechanisms'

Split Horizon - A route WILL NEVER be advertised through the interface it was learnt from.

Poison reverse - after learning of a route through a certain interface, any advertisements back the same interface are sent us unreachable for the said route/network.

  • If a FD sends an update, queries regarding the same network are not sent to it.
  • Stuck in active (SIA) occurs if a query takes too lon gto be answered by a neighbor....
Redistribution: Ahh this is when I started having fun....I think I'll deal/write about redistribution after going throroughly through OSPF

now lets see if my trusty old motor bike survived the holidays:-).......happy new year....

Want to read some more.....go here

Friday, January 2, 2009

Understanding EIGRP from the bottom up Part I

When tackling the BSCI, I went through the EIGRP material, just enough to pass the exam. So I concentrated more on the theory end of things. I rarely if ever needed the certifications for my job ie it was'nt a requirement for promotions or anything. (telcos in kenya have very poor skill spotters in my opinion), either way we do run EIGRP so this time I took the time to understand it properly.

Key fields in the EIGRP header are as follows:

* The opcode field specifies the EIGRP packet type (update, query, reply, hello).
* The checksum applies to the entire EIGRP packet, excluding the IP header.
* The rightmost bit in the flags field is the initialization bit and is used in establishing a new neighbor relationship
* The sequence and ack fields are used to send messages reliably
* The AS number identifies the EIGRP process issuing the packet. The EIGRP process receiving the packet will process the packet only if the receiving EIGRP process has the same AS number; otherwise, the packet will be discarded.

EIGRP is very widely documented so googling for EIGRP header will robably give more details.

Other highlights:

  1. Its Hybrid and uses the DUAL (Diffused Update ALgorithm).
  2. Neighbor discovery and maintenance ensures only updates are sent when needed (hellos). The hello interval is 5 and 60 seconds, the default hold time is three times (15 and 180 seconds) the hello timer duration. Hold time is the amount of time a router will consider a neighbor alive without receiving a hello packet. The timers can be adjusted per interface with the ip hello-interval eigrp and ip hold-time eigrp.
Lets take a look:
We'll use the same topology we used for RIP to explore the EIGRP timers.

R0: :
R1: :

Network Addresses:
R0: Serial1/0 :
R1: Serial1/0 :

ffr Addresses:
R0: Serial1/1.105 :
R1: Serial1/1.501 :

Ethernet interfaces:
R0: Fastethernet0/0:
r1: FastEthernet0/0:

Test reachability - before continuing - If all is well, add in EIGRP:

On R0

R0#sh run | section eigrp
router eigrp 1
no auto-summary

on R1

router eigrp 1
no auto-summary

What pops up immediately:

*Mar 1 00:02:32.223: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/0) is up: new adjacency
*Mar 1 00:02:32.383: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/1.501) is up: new adjacency
*Mar 1 00:02:35.447: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/1.501) is resync: peer graceful-restart
*Mar 1 00:02:35.455: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/0) is resync: peer graceful-restartR0#sh ip eigrp neighbors

IP-EIGRP neighbors for process 1

H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 Se1/1.105 12 00:10:52 894 5000 0 15
0 Se1/0 12 00:10:52 127 762 0 16

The value of the hold column shouldn't be more than the timer unless you're losing packets. Also the Q(queue) count column should always be '0' unless there's a problem.

For instance, lets block EIGRP on R0 and see what shows up on R1:

R0(config)#access-list 100 deny  eigrp any any
R0(config)#access-list 100 permit ip any any
int s1/0
ip access-group 100 in

This pops up on R0
*Mar  1 00:38:34.547: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/0) is down: holding time expired
and on R1 we now have the following:
*Mar 1 00:38:34.899: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/0) is down: Interface Goodbye received
*Mar 1 00:38:39.623: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/0) is up: new adjacency
*Mar 1 00:39:59.147: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/0) is down: retry limit exceeded
*Mar 1 00:40:03.619: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor (Serial1/0) is up: new adjacency
R1#sh ip eigrp neighbors
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 Se1/0 11 00:00:36 1 5000 1 0
1 Se1/1.501 11 00:38:08 137 822 0 10
Note the queue count is 1 (one) , this implies an un-acknowledged hello.

On R0 the neighbor is removed:
R0#sh ip eigrp neighbors
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 Se1/1.105 13 00:38:54 894 5000 0 15
Other commands you can run to troubleshoot: I'll just BOLD key info from each command
R1#show ip eigrp interfaces
IP-EIGRP interfaces for process 1

Xmit Queue Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes
Se1/0 1 0/0 124 0/15 563 0
Se1/1.501 1 0/0 137 0/15 639 0

R1# show ip eigrp topology
IP-EIGRP Topology Table for AS(1)/ID(

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
r - reply Status, s - sia Status

P, 1 successors, FD is 2169856
via Connected, Serial1/0
P, 1 successors, FD is 2169856
via Connected, Serial1/1.501

R1# show ip eigrp traffic
IP-EIGRP Traffic Statistics for AS 1
Hellos sent/received: 1202/1179 <---remember we had blocked some hellos with access list 100
Updates sent/received: 109/9
Queries sent/received: 2/2
Replies sent/received: 4/2
Acks sent/received: 1/12
Input queue high water mark 4, 0 drops
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 174
PDM Process ID: 169
R1# show ip eigrp accounting
IP-EIGRP accounting for AS(1)/ID( <<--note teh ID is our loopback
Total Prefix Count: 2 States: A-Adjacency, P-Pending, D-Down
State Address/Source Interface Prefix Restart Restart/
Count Count Reset(s)
A Se1/0 1 0 0
A Se1/1.501 1 0 0
R1#show ip eigrp 100 ?
accounting  IP-EIGRP Accounting
interfaces IP-EIGRP interfaces
neighbors IP-EIGRP neighbors
topology IP-EIGRP Topology Table
traffic IP-EIGRP Traffic Statistics
!use this if running more than one AS on the same router

show ip eigrp neighbors
Already covered
!Note since these routes are learnt via connected, our routing table currently wont
show any eigrp routes. To get this we need to advertise our loopbacks via eigrp

R1#sh ip route
!--------edited----------! is subnetted, 1 subnets
C is directly connected, Serial1/0 is subnetted, 1 subnets
C is directly connected, Loopback0 is subnetted, 1 subnets
C is directly connected, Serial1/1.501 is subnetted, 1 subnets
C is directly connected, FastEthernet0/0

See!! no EIGRP routes, now lets add the loopback addresses:
R0#sh run | section eigrp
router eigrp 1
no auto-summary

and on R1:
R1#sh run | section eigrp
router eigrp 1
no auto-summary
our output:
R0#sh ip route eigrp is subnetted, 2 subnets
D [90/2297856] via, 00:01:05, Serial1/1.105
[90/2297856] via, 00:01:05, Serial1/0
R1#sh ip route eigrp is subnetted, 2 subnets
D [90/2297856] via, 00:01:23, Serial1/1.501
[90/2297856] via, 00:01:23, Serial1/0
*EIGRP doesnt build neighbor relationships over secondary addresses.
*Please note changing the hello interval using ip hello-interval eigrp doesn't adjust the hold. time timer for you.
*Load balancing: routing protocols install routes to your routing table, the switching fabric within the router/switch does the load balancing based on various things. eg per packet load balancing, per destination etc etc...all this depends on your switching (cef, fast switching)
let me make a quick demonstration:

If we try to reach R1's loopback with CEF enabled and the routing table as it is now (by default we use per destination loadbalancing) -
R1#debug ip icmp
ICMP packet debugging is on

*Mar 1 01:04:19.239: ICMP: echo reply sent, src, dst
*Mar 1 01:04:19.371: ICMP: echo reply sent, src, dst
*Mar 1 01:04:19.459: ICMP: echo reply sent, src, dst
*Mar 1 01:04:19.495: ICMP: echo reply sent, src, dst
*Mar 1 01:04:19.507: ICMP: echo reply sent, src, dst
Note the path taken is the same, all packets to the same destination will use
same path

now disable CEF (globally - no ip cef)on R0 and make the same 'ping'
*Mar 1 01:07:27.411: ICMP: echo reply sent, src, dst
*Mar 1 01:07:27.495: ICMP: echo reply sent, src, dst
*Mar 1 01:07:27.539: ICMP: echo reply sent, src, dst
*Mar 1 01:07:27.583: ICMP: echo reply sent, src, dst
*Mar 1 01:07:27.591: ICMP: echo reply sent, src, dst
Note the alternating paths, this is per Packet Load balancing
You can use sh ip interface to figure out what sort of switching is in use. I only did this to show how the routing protocol really only populates the routing database, how traffic is moved across networks is not its business.....I'll follow this up with EIGRP metrics,feasible distance, reported distance and feasible successors.....

Oreilly's IP routing by By Ravi Malhotra - I found chapter4 available on oreilly:-) good for you