Sunday, January 25, 2009

My left knee is a traitor!!

So i took a break from work, reading and all things indoors today (sunday). One other goal I have this year is to get fitter, swim better - longer, jump a higher bungee, run a 21Km (half marathon) without something on or in me giving up.

So to test my current status, I went for a triathlon:-)....The swim almost killed me, so bad was I my event was changed to a duathlon - didn't finish the swim....:-(I'm yet to take the ccie lab but i suredo hope my chances with it are better than what I had in water...

I transitioned after the last swimmer - to be fair- and off we went cycling....I think apart from laziness, and probably terrible eating habits, I'm a strong, good cyclist...the trails at KU-the university are not bad at all, parts of it are single track, brief vegetation then its you and open ground....

Ahhh then came the run, after about 4Km, my left knee totally gave up on me, the rest of the body definately felt betrayed, there was this awfully sharp pain from the back of my knee (I don't run much) and I swear I could hear my heart bouncing with every step...it ended well, heck they gave me a medal - probably to console me:-) the running kicked my ass though...This knee thing! not sure whether to get it checked out, so if you get by here and have a clue let me know...

Im dedicating more hours to the run and general fitness stuff every week, not sure what to do with the swim, need a plan..to complete a full triathlon by end of year I need to probably put in as much time as I'll do the ccie lab, It will be very satisfying to accomplish both by end year-something I plan on doing-....either way I find that getting out every once in a while works wonders for my concentration..... ahhh now back to IPV6.....

Thursday, January 22, 2009

Etherchannel Load Balancing and Forwarding Methods

a couple of WS-C3750E's - had a strange problem (mainly with my assumptions) on how etherchannel load balances. I thought it was automatic:-) it should feel what i need and do it!!!...

Imagine two ports bound to form one port channel giving 200Mbps. Now imagine on one end you have two hosts/servers that generate/carry a lot of traffic (an ftp server for instance) to multiple destinations on the other end - internet, auth servers etc etc...

One of the hosts has more traffic than the other, infact alot more then 98Mb. so when traffic hit 100Mbps, i started noticing random packet drops. why why why...I thought this is a 200mbps interface???

upon further checks we discovered that one of the interfaces within the bundle was dropping packets/frames.

so the checks started:

Gitau-Switch-01-Sw#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-mac

src-mac was the default.
Now according to cisco :
EtherChannel load balancing can use either source-MAC or destination-MAC address forwarding.

With source-MAC address forwarding, when packets are forwarded to an EtherChannel, they are distributed across the ports in the channel based on the source-MAC address of the incoming packet. Therefore, to provide load balancing, packets from different hosts use different ports in the channel, but packets from the same host use the same port in the channel (and the MAC address learned by the switch does not change).

With destination-MAC address forwarding, when packets are forwarded to an EtherChannel, they are distributed across the ports in the channel based on the destination host's MAC address of the incoming packet. Therefore, packets to the same destination are forwarded over the same port, and packets to a different destination are sent on a different port in the channel.

So obviously the default load balancing was not working for me.

Since this was a 3750, I correctly figured that it can also use IP. playing around with the setup on the end that was dropping packets, the following sort of sorted me out:

Gitau-Switch-01-Sw#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-ip

the command to make this change is:

port-channel load-balance src-ip

you can play around with:


Gitau-Switch-01-Sw#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr

you can run a test by:

test etherchannel load-balance interface port-channel [#] ip [src] [dst]

references:

http://www.edgenetworks.nl/etherchannel.html
http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml


before:

Gitau-Switch-01-Sw#sh int g1/0/24 | include drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 20191

now :

Gitau-Switch-01-Sw#sh int g2/0/24 | include drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Gitau-Switch-01-Sw#sh int g1/0/24 | include drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0


I figure after some time I'll come across a few drops..because the network hates me!!!

Tuesday, January 20, 2009

Understanding OSPF contd...

After about 20 hours lab and roughly 10 hours going through OSPF, I still can't say I have the confidence to go through OSPF without going down belly fast in the CCIE lab. More practice is definately called for to get at least to 70% confidence levels.

So i decided to just make some notes, go through BGP and start on Multicast as the information digested. Plus Im curious how much of all this material I'll have to cover again.

Apparently OSPF is the most widely used IGP. It brings in the concept of Areas. If you imagine a midsized no actually a midsized company in Europe is probably a LARGE company in Kenya, so if you imagine a LARGE:-) company, an area would probably be a building or a department.

Router ID's, Neighbors, Adjacencies, LSA's *,Hello protocol,Areas, election (DR,BDR) are terms you'll come across and if the books you're reading don't address them:-), google will sort you out.

OSPF sort of goes around the whole network and maps out what you ask it to giving you paths to destinations, its a link state protocol.

OSPF order of operation:
1:A router sends Hello packets, discovers neighbors and elects a Designated Router. Link-state information and a list of neighbors is included in the packet.

*Mar 1 00:15:08.803: OSPF: Send hello to 224.0.0.5 area 1 on Serial1/0 from 1.1.1.1
*Mar 1 00:15:10.903: OSPF: Rcv hello from 2.2.2.1 area 1 from Serial1/0 1.1.1.2
*Mar 1 00:15:10.911: OSPF: End of hello processing
*Mar 1 00:15:11.071: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.1 on Serial1/0 from LOADING to FULL, Loading Done
*Mar 1 00:15:18.807: OSPF: Send hello to 224.0.0.5 area 1 on Serial1/0 from 1.1.1.1

*Mar 1 00:15:20.907: OSPF: Rcv hello from 2.2.2.1 area 1 from Serial1/0 1.1.1.2
*Mar 1 00:15:20.911: OSPF: End of hello processing

This is an initial hello.

Debugging ip packet detail shows the messages a bit clearly:

*Mar 1 00:20:48.815: IP: s=1.1.1.1 (local), d=224.0.0.5 (Serial1/0), len 80, sending broad/multicast, proto=89
*Mar 1 00:20:50.899: IP: s=1.1.1.2 (Serial1/0), d=224.0.0.5, len 80, rcvd 0, proto=89


Protocol 89 is OSPF, 224.0.0.5 is the ALLSPFRouters multicast address, from th esource you can also tell whether we originated the hello or not. Since this are serial links, no DR/BDR election takes place.

Let me swap out the links with ethernet to see how that works out....on my next post. Seems the lab is inaccessible....this is a simple lab, I think I'll dynamip on the laptop for a while:...

Sunday, January 18, 2009

back...yes Im back and it feels good....

Yani, that was one looong busy buuusy week.....Didn't get much done...however this weekend sort of saved it. I got through almost everything to get me sort of set for the February deadline for the written exam. Hopefully some projects at work won't come in the way, and if they do they better be fun..On the solaris+squid+wccp, my advise is don't run it in a busy production environment, far too many things coud go wrong...

OSPF from the ground up from tomorrow.....

Friday, January 9, 2009

new day, new solaris + squid :-challenges

Great another morning, my solaris 10, squid installed, but in comes another requirement. Run wccp2 on it. Im happy because I get to mess around with a router, sad because Ive done this countless times on linux and freeebsd but never on solaris. So quickly go to sunfreeware; not a thing when i search for wccp. hmm back to basics, maybe we can work around this by first understanding the requirements ( I always try to know the way a protocol works first before adapting it for whatever system I need it for), this way If i ever need to configure lets say OSPF on juniper, so long as I know how SPF works, i can run it on anything:

I also decided to take some debug output for future reference:

Commands you'll use on the router:

show ip wccp web-cache detail
show ip wccp web-cache view
debug ip wccp events - display WCCP events
debug ip wccp packets - display WCCP packet information

Here's how wccp works with squid:

Once squid is started with the wccp option, the router sees this and sends an I see you message as follows (I had always assumed the here I am message comes first - I was obviously wrong:

Jan 9 08:57:51.584 NAIROBI: WCCP-PKT:S00: Sending I_See_You packet to 196.201.xx.xx w/ rcv_id 00006C9C

and you get this log below on the router - this is not a debug output.

Jan 9 08:58:01.593 NAIROBI: %WCCP-5-SERVICEFOUND: Service web-cache acquired on WCCP client 196.201.xx.xx

Squid then sends back HERE_I_AM packets, and carries on a conversation with the router.

Jan 9 08:58:01.593 NAIROBI: WCCP-PKT:S00: Received valid Here_I_Am packet from 196.201.xx.xx w/rcv_id 00006C9C

If you were also debugging events, you get the following:

Jan 9 08:58:01.593 NAIROBI: WCCP-EVNT:S00: Assignment wait timer started
Jan 9 08:58:01.593 NAIROBI: WCCP-EVNT:S00: Built new router view: 1 routers, 1 usable WCCP clients, change # 0000004D

From then on traffic gets redirected to squid.

and if you disconnect you get the following event/log:

Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Assignment wait timer started
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Built new router view: 0 routers, 1 usable WCCP clients, change # 0000004B
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Router 196.201.xxx.xxx removed.
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Assignment wait timer started
Jan 9 08:57:37.279 NAIROBI: WCCP-EVNT:S00: Built new router view: 0 routers, 0 usable WCCP clients, change # 0000004C
Jan 9 08:57:37.279 NAIROBI: %WCCP-1-SERVICELOST: Service web-cache lost on WCCP client 196.201.xx.xx

On the router, the basic config is widely available on the internet....create an access-list for traffic to redirect, enable wccp etc etc...

This conversation (between router and squid) is totally separate from the ip_wccp or ip_gre module --these packets never go through those channels. For so long as the router is receiving WCCP HERE_I_AM packets (with the proper ID), the router will send traffic to the cache IP, encapsulated in a GRE packet, and there in lies the problem or the gotcha as ccie candidates like saying. In a production if you do this without confirming the tunnels, all your redirected traffic will be blackholed by squid.

The GRE decapsulation is a separate process from squid. Squid doesn't talk to the gre/ip_wccp module, The gre decapsulation occurs at the network layer in your kernel, and then the packets enter the normal routing table - on bsd/linux it involves iptables/chains
to hijack any packets destined for port 80 on the internet - In my case thats traffic coming from the router in GRE packets and passes it over to the cache engine's port.

I wonder what happens if I let squid listen on port 80....do you still need something to 'jack' the packets? I'll try that in a while and let you know....

So now with that understanding, you'd think I fix this on solaris pretty fast yes??? nope, nada, my cluelessness gets in the way....AGAIN!!


So again back to google, I can't get an ip_wccp for solaris module but Im sure I will get gre and the ip redirect working...and get more CLUEFUL overall then we can go back to regular work, If you stumble upon this and have a clue...bail me out:-)pleasee....I still have about 5 bgp, 5 ospf LABS to get through before Sunday....

Thursday, January 8, 2009

Grrr...Solaris 10 install

I have generally always HATED installing operating systems. Unfortunately today caught me doing just that. Installing Solaris 10 on a v490 that just refused to cooperate with me. So there I was huddled in the data center at temperatures my body doesn't agree with; an installation that took a minimum 2 hours (had to do it twice - resized the disks wrongly the first time) so 4 hours....grrr.....a fun fun......If I curse or throw a shoe at you in traffic blame SUN:-) or solaris...and forgive me!!

and it still bloody refused to boot in the end. damn thing couldnt initialize the boot disk without my help , kumbe the boot sequence was wrong:-(

Retrying network initialization
Evaluating:

Can't open boot device

{3} ok boot disk <<---this resolved my first problem, finally booting - 4 hrs down the line
Boot device: /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@0,0 File and args: Loading ufs-file-system package 1.4 04 Aug 1995 13:02:54. FCode UFS Reader 1.12 00/07/17 15:48:16. Loading: /platform/SUNW,Sun-Fire-V490/ufsboot Loading: /platform/sun4u/ufsboot

then came installing squid, which is what i wanted to do to begin with. If you've installed solaris you know how bare minimal instal gets done. So we always keep a set of standard packages to install to get things going:

## pwd
/export/home/jgitau/SUNFREEWARE
# ls
apache-2.2.6-sol10-sparc-local libintl-3.4.0-sol10-sparc-local php-5.2.5-sol10-sparc-local
curl-7.18.0-sol10-sparc-local libxml2-2.6.31-sol10-sparc-local php-5.2.6-sol10-sparc-local
expat-2.0.1-sol10-sparc-local make-3.81-sol10-sparc-local python-2.5.1-sol10-sparc-local
flex-2.5.33-sol10-sparc-local mysql-5.0.51-sol10-sparc-local sed-4.1.5-sol10-sparc-local
fping-2.4b2-sol10-sparc-local ncurses-5.6-sol10-sparc-local thttpd-2.25b-sol10-sparc-local
gcc-3.4.6-sol10-sparc-local netsnmp-5.4.1-sol10-sparc-local wget-1.11.1-sol10-sparc-local
libiconv-1.11-sol10-sparc-local openldap-2.4.11-sol10-sparc-local zip-2.32-sol10-sparc-local
libidn-1.6-sol10-sparc-local openssl-0.9.8f-sol10-sparc-local zlib-1.2.3-sol10-sparc-local
so we keep those around and use the standard pkgadd'er:

pkgadd -d

so how long do you think that sort of took?....

oh yes I also had missing paths:
so:

Bash:
export PATH=/usr/bin:/usr/local/bin:usr/sbin

sh:
setenv PATH /usr/bin:/usr/local/bin:usr/sbin


adding me as a user so we can ssh and leave this cold place...:

bash-3.00# mkdir /export/home/jgitau
bash-3.00# useradd -d /home/jgitau -m -s /usr/bin/bash -c "John Gitau" jgitau
bash-3.00# passwd jgitau
New Password:
Re-enter new Password:
passwd: password successfully changed for jgitau

scp/ftp the files or use whichever method makes you happy....eg for apache

scp apache-2.2.6-sol10-sparc-local jgitau@196.201.xx.xx:/export/home/jgitau/SUNFREEWARE

aaanywayz....i finally got to compile squid and it was a happy day, just not a single ccie sort of reading will get done today...

Ha...yes this post is mainly just to remind me what packages to get downloaded next time, and simple commands I sometimes forget...and general noise for the blog...:-)...

seriously though, you can let me know if you do get a problem with this sort of thing, live around nairobi, and you promise not to take me to a data center with working air conditioners - because then it would be a nice warm environment-) for me:-) hmm I didn't see any DustPuppies, I relax over at userfriendly alot..probably too much

Tuesday, January 6, 2009

Understanding OSPF....

After going through ospf, and to properly understand some concepts, I hit the intanet looking for pre-done dynamips labs:-) to save on time. during my cyber walk, I came across the article
Configuring Basic OSPF (Dynamips). It was excellent and had a very well detailed instruction on things.

However one of the requirements for the lab was:

That the ISP router always be the DR, one of the Remote sites always be a BDR and one remote site never participates in the election process. (Please go to evilrouters.net) for the diagram and full article - but it looks something like below all routers connected through switches.

Remote1===\
======ISP
Remote2===/

I redid the whole lab with emphasis on meeting the designated router (DR)/backup designated router (BDR) requirements
Configurations:

ISP#sh ip int br | exclude un
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 10.10.10.1 YES manual up up
Loopback0 188.46.37.254 YES manual up up

------
Remote1#sh ip int br | ex una
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 10.10.10.3 YES manual up up
Loopback0 192.168.1.1 YES manual up up
Loopback1 192.168.1.161 YES manual up up
-------
Remote2#sh ip int br | ex una
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 10.10.10.2 YES manual up up
Loopback0 192.168.1.65 YES manual up up
Loopback1 192.168.1.129 YES manual up up

After bringing up the interfaces, I tested connectivity as evilrouter dude had done it - well almost:-):

ISP#tclsh
ISP(tcl)#for
ISP(tcl)#forea
ISP(tcl)#foreach address {
+>10.10.10.2
+>10.10.10.3
+>} {ping $address}

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 32/58/96 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.3, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 16/45/88 ms
ISP(tcl)#

Now we get OSPF going, just as he had done it, I'll just show the neighbors here:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
192.168.1.129 1 FULL/BDR 00:00:34 10.10.10.2 FastEthernet0/0
192.168.1.161 1 FULL/DROTHER 00:00:38 10.10.10.3 FastEthernet0/0
------
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
188.46.37.254 1 FULL/DR 00:00:31 10.10.10.1 FastEthernet0/0
192.168.1.129 1 FULL/BDR 00:00:39 10.10.10.2 FastEthernet0/0
-----
Remote2#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
188.46.37.254 1 FULL/DR 00:00:39 10.10.10.1 FastEthernet0/0
192.168.1.161 1 FULL/DROTHER 00:00:31 10.10.10.3 FastEthernet0/0

-we have proper basic ospf

Since Im interested in the DR/BDR, I'll skip the details here since the original
post
at evilrouters.net still has them.

lets go over the requirements again in proper detail for clarity:
1: ISP - 10.10.10.1 will always be the DR
2: Remote1 - 10.10.10.3 will always be the BDR
3: remote2 - never participates in an election.
Remote2 is easy:

Remote2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Remote2(config)#interface fastethernet 0/0
Remote2(config-if)#ip ospf priority 0
Remote2(config-if)#


ISP is also easy:

ISP(config)#interface fastethernet 0/0
ISP(config-if)#ip ospf priority ?
<0-255> Priority
ISP(config-if)#ip ospf priority 255

The neighbors currently look like this:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
ISP#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:32 10.10.10.2 FastEthernet0/0
10.10.10.3 1 FULL/DR 00:00:35 10.10.10.3 FastEthernet0/0
-------
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/BDR 00:00:36 10.10.10.1 FastEthernet0/0
10.10.10.2 0 FULL/DROTHER 00:00:38 10.10.10.2 FastEthernet0/0
-------
Remote2#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/BDR 00:00:33 10.10.10.1 FastEthernet0/0
10.10.10.3 1 FULL/DR 00:00:38 10.10.10.3 FastEthernet0/0


so according to the above output:

10.10.10.2 - is fine, it won't participate in the election process.

Note:
Immediately something is wrong with the BDR/DR election . Since I reset the Remote2 OSPF process first, it got to be DR first and the ISP router had to be the BDR. That I'm sure starts to make my point but, just to ensure we go through all the evil router guy's/guyette's??:-) steps: I'll go ahead and change the priority of the RemoteRouter to 254 and clear the ISP router's ospf process first - !so it gets to be DR, followed by the others and the output is as follows:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:30 10.10.10.2 FastEthernet0/0
10.10.10.3 254 FULL/BDR 00:00:34 10.10.10.3 FastEthernet0/0
-------
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/DR 00:00:32 10.10.10.1 FastEthernet0/0
10.10.10.2 0 FULL/DROTHER 00:00:34 10.10.10.2 FastEthernet0/0
-------
Remote2#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/DR 00:00:30 10.10.10.1 FastEthernet0/0
10.10.10.3 254 FULL/BDR 00:00:36 10.10.10.3 FastEthernet0/0

However, If I now reset the ISP router's ospf interface, or if the link to the ISP were to go off:

Remote1#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:38 10.10.10.2 FastEthernet0/0

Remote2#
*Mar 1 01:06:44.087: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.10.1 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expiredsh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.3 254 FULL/DR 00:00:39 10.10.10.3 FastEthernet0/0

So now we have a situation where the former BDR was upgraded to DR. The reason I think your requirement for the ISP to always be a BDR cant be met with
this configuration is so long as there's a BDR that gets upgraded to a DR and no pre-emption happens, then your ISP will always be either a DR or BDR even
under normal circumstances. The priority only kicks in if the processes are reset at the same time.

Here's the output after bringing up the ISP router.


ISP(config-if)#
*Mar 1 01:12:17.647: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar 1 01:12:18.647: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
*Mar 1 01:12:25.631: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.10.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
*Mar 1 01:12:25.719: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.10.3 on FastEthernet0/0 from LOADING to FULL, Loading Done
ISP(config-if)#do sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:37 10.10.10.2 FastEthernet0/0
10.10.10.3 254 FULL/DR 00:00:34 10.10.10.3 FastEthernet0/0

ISP#sh ip ospf interface f0/0
FastEthernet0/0 is up, line protocol is up
Internet Address 10.10.10.1/29, Area 0
Process ID 1, Router ID 10.10.10.1, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State BDR, Priority 255
Designated Router (ID) 10.10.10.3, Interface address 10.10.10.3
Backup Designated router (ID) 10.10.10.1, Interface address 10.10.10.1
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:05
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 2/2, flood queue length 0
Next 0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 1
Last flood scan time is 0 msec, maximum is 4 msec
Neighbor Count is 2, Adjacent neighbor count is 2
Adjacent with neighbor 10.10.10.2
Adjacent with neighbor 10.10.10.3 (Designated Router) !<<<------ note the DR!
Suppress hello for 0 neighbor(s)

So the question is, how do you ensure that a router is always a BDR? Is that even possible? How to go about it?

Lastly, many thanks to whoever runs http://evilrouters.net for this article

Sunday, January 4, 2009

confused laziness.....and expect:-)

...today (sunday) caught me trying to catch up with OSPF but I just couldn't stand the books, sites - and a friend who'd come for an apparently 'intelligent discussion on nssa's and routers, or sgsn's etc etc', in the end we opened a couple of beers, fed the fish (the only other living thing in my house is 4 fish) and yapped about current affairs, state of african economies and how soon we're going to be travelling all over the place setting up networks sampling local brews:-) - Please note kenya and the general (east) african region has relied on satellite communication for so long, however we have submarine fiber coming in by 2010-2012 so yes I see a very bright future for networkers:-).....so no bootcamps, no authorized training centers for high end certs, no internetwork expert, no narbik...to get certified here you really have to lab up and put in the time...ahh the joys.

Anyway after several hours of that 'drinking', the neurons were perfectly fired up to engage in some creative boredom activity. I thought of one of the most annoying repetitive tasks my team has to undertake daily and decided to get rid of it.

Every once in a while we get customers calling our 1st line support guys just to check if their remote sites have established a pdp session successfully. On a cisco GGSN the command would be:

show gprs gtp pdp-context msisdn 2547221x1x2xx

To get the following:

TID MS Addr Source SGSN Addr MSISDN APN
36xx72xxxxxxxxx 196.xx.xx.01 LOCAL 196.20x.xxx.xx 2547221x1x2xx safaricom
In most cases the next request could be to clear the pdp context so they can re-establish it again

clear gprs gtp pdp-context tid
So in comes EXPECT:
this is an extraordinarilly great tool for writing scripts for the lazy sysadnmin to drive other programs. It recognizes prompts and sends keystrokes in response. It was written by Don Libes of NIST, and you can find papers on it in Usenix LISA (Large Systems Administration) conference proceedings, as well as on the Internet.

My drunken goal was to create a web interface where the users can check this for themselves.

The expect script itself was pretty straight forward:


#!/usr/local/bin/expect -f

set force_conservative 0 ;# set to 1 to force conservative mode even if
;# script wasn't run conservatively originally
if {$force_conservative} {
set send_slow {1 .1}
proc send {ignore arg} {
sleep .1
exp_send -s -- $arg
}
}


set msisdn [lindex $argv 0]
set timeout 3
spawn /usr/bin/bash
match_max 100000
send -- "telnet 196.x.x.x\r" # your GGSN/cisco routers IP address
expect -exact "telnet 196.x.x.x\r # your GGSN/cisco routers IP address
Trying 196.x.x.x...\r
Connected to 196.x.x.x.\r
Escape character is '^\]'.\r
\r
\r
User Access Verification\r
\r
Username: "
send -- "drunkenmaster\r"
expect -exact "drunkenmaster\r
Password: "
send -- "jedimaster\r"
expect -exact "\r
\r
GGSN-xx01>"
send -- "en\r"
expect -exact "en\r
Password: "
send -- "jedimaster\r"
expect -exact "\r
GGSN-xx01#"
send -- "show gprs gtp pdp-context msisdn $msisdn"
expect -exact "show gprs gtp pdp-context msisdn $msisdn"
send -- "\r"
send -- "exit\r"
send -- ""
expect eof


Next do a simple perl,php or whatever makes you happy script, put it on a web server somewhere and guys can access it with a url. The handiwork can be viewed here:
* i just modified an old script we use for users to change unix passwords still using expect and adapted it for my needs.


Please note the above url is only active as a demo - in any case it would only be applicable to our users but im sure it explains the concept

On putting in a valid msisdn the output should show up in full....as follows - or you can use the url above:
Please check from LINE 17 (seventeen) thats where your output will be

GGSN returned following information:
Array
(
[0] => spawn /bin/bash
[1] => telnet 196.x.x.x
[2] => www-data@monitor01:/var/www$ telnet 196.x.x.x
[3] => Trying 196.x.x.x...
[4] => Connected to 196.x.x.x.
[5] => Escape character is '\^]'.
[6] =>
[7] =>
[8] => User Access Verification
[9] => [10] => Username: drunkenmaster
[11] => Password:
[12] =>
[13] => GGSN-xx01>en
[14] => Password:
[15] => GGSN-xx01#show gprs gtp pdp-context msisdn
[16] => TID MS Addr Source SGSN Addr MSISDN APN
[17] => 36xx72xxxxxxxxx 196.xx.xx.01 LOCAL 196.20x.xxx.xx 2547221x1x2xx safaricom
[18] =>
[19] => GGSN-xx01#exit
[20] => Connection closed by foreign host. [
21] => www-data@gitaus-TestServer:/var/www$

please note this is a sample output and no script is actually run to get this output

Please contact the data team! To run another query, : Please click me to run another query! "


Ahh finally success, the php script just inserts the msisdn....if you're interested in it, leave a comment and I'll probably just upload it...

You can run this for virtually anything, show bgp neighbors, for users to change unix passwordsetc etc..just modify the commands as you see fit. You can for instance give your users access a router to check the status of one of their interfaces, or clear statistics or whatever makes you happy, its a bonus if you have the security guys screaming at you:-) no seriously ensure your corporate policy is followed when doing some of this things...

Also if you forget to modify the tty's on a router, users might lock you out of telnet access:-)

please note this could obviously work better, but i wasn't looking for better, i needed to kill some time and be productive at the same time. For instance we could post process the output to only show line 17 ( I also generally prefer if guys especially co-workers to at least have an idea of whats run and what the output would look like from a console)...but time's up, and I think I'll have an easier time tomorrow...maybe some day....

http://oreilly.com/catalog/expect/chapter/ch03.html
http://www.marcelgagne.com/node/582
ftp://ftp.cisco.com/pub/cisco-expect.shar

Saturday, January 3, 2009

Understanding EIGRP from the bottom up Part I I

EIGRP Metrics:
Routers 'discuss' their topology tables. So unlike other protocols, EIGRP has a table with all the routes to a destination.

show ip eigrp topology all-links
!--is the command that shows this table - all links is optional
This topology table is the one holding all the info to make decisions, the outcome is usually a distance and vector to each destination.

Minimum bandwidth and the total delay is the information used to compute the metric. The values are automatically picked from configured values on your interfaces. Generally bandwidth will be more critical for lower bandwidth interfaces while delay is more key where high speed interfaces are used.

Formula to get :
the bandwidth:
bandwidth = 10 000 000 / bandwidth * 256
The delay
delay = delay * 256

The values got for bandwidth and delay are then used in the metric computation. I won't go into detail, please check this cisco link or read a book.

the default behaviour is to just calculate as follows:
metric = bandwidth + delay

(remember to round off your figures after each calculation, I can't remember why but i think floating point math on cisco was the reason)


so if you have the following scenario:


bw 56k
/Delay 2000---Router4---| |
/ | |
Router1== |---------Router2------|
\ |bw10000 |
\__bw 128 |delay100 |Destination Network
Delay 1000-Router3---| |bw 10000
|delay 100


metric = bandwidth + delay
minimum bandwidth = 56k
Total Delay = 100 + 100 + 100 + 2000
= 2200

[(10 000 000/56) + 2200] x 256 = (178571 +2200) x 256
= 180771 x 256
= 46277376

10 000 000/56 is actually = 178571.42857142857142857142857143 but we round it off...
If you calculate the metric through router 3, you find the metric is 20307200

so router one will use the path via Router3 to get to the destination network.
  • The bandwidth is calculated from the configured interface through which the desired network is visible ie where the update is coming in through.
  • The delay is cumulative; ie each router adds a delay and sends it backwards...Im not sure how to explain this if you can't see it....just count the delay from the destination backwards

Feasible Distance (FD)
  • This is the best metric/best path to the destination network - includes the metric to the neighbor advertising the network - from our diagram thats Router2.
reported Distance (RD)
  • This is the total metric as advertised by an upstream router/neighbor. from the ascii diagram above, that will be the distance advertised by Router4 or Router3 (obviously one is going to be used as a FD
feasible Successor (FS)
  • This is the path whose reported distance is less than the feasible distance. This is usually installed in the topology table as a backup.
The reported distance is always calculated from the router advertising the route to the network. so the reported distance from router4 is the metric to get to the destination network from Router4 and the same goes from the reported network from Router3.

If the link between Router1 and Router3 goes down, the convergence is almost instant, users will probably not even notice it since the feasible successor/backup route is immediately picked.

Checking loops:
The FD,FS and RD concepts are used to breack loops using the logic that the reported distance CANNOT be higher than the feasible distance. A route/path with a higher Rd than the FD won't show up on the topology table.

Other loop 'taking care of mechanisms'

Split Horizon - A route WILL NEVER be advertised through the interface it was learnt from.

Poison reverse - after learning of a route through a certain interface, any advertisements back the same interface are sent us unreachable for the said route/network.

Notes:
  • If a FD sends an update, queries regarding the same network are not sent to it.
  • Stuck in active (SIA) occurs if a query takes too lon gto be answered by a neighbor....
Redistribution: Ahh this is when I started having fun....I think I'll deal/write about redistribution after going throroughly through OSPF

now lets see if my trusty old motor bike survived the holidays:-).......happy new year....

Want to read some more.....go here

Friday, January 2, 2009

Understanding EIGRP from the bottom up Part I

When tackling the BSCI, I went through the EIGRP material, just enough to pass the exam. So I concentrated more on the theory end of things. I rarely if ever needed the certifications for my job ie it was'nt a requirement for promotions or anything. (telcos in kenya have very poor skill spotters in my opinion), either way we do run EIGRP so this time I took the time to understand it properly.

Key fields in the EIGRP header are as follows:

* The opcode field specifies the EIGRP packet type (update, query, reply, hello).
* The checksum applies to the entire EIGRP packet, excluding the IP header.
* The rightmost bit in the flags field is the initialization bit and is used in establishing a new neighbor relationship
* The sequence and ack fields are used to send messages reliably
* The AS number identifies the EIGRP process issuing the packet. The EIGRP process receiving the packet will process the packet only if the receiving EIGRP process has the same AS number; otherwise, the packet will be discarded.

EIGRP is very widely documented so googling for EIGRP header will robably give more details.

Other highlights:

  1. Its Hybrid and uses the DUAL (Diffused Update ALgorithm).
  2. Neighbor discovery and maintenance ensures only updates are sent when needed (hellos). The hello interval is 5 and 60 seconds, the default hold time is three times (15 and 180 seconds) the hello timer duration. Hold time is the amount of time a router will consider a neighbor alive without receiving a hello packet. The timers can be adjusted per interface with the ip hello-interval eigrp and ip hold-time eigrp.
Lets take a look:
We'll use the same topology we used for RIP to explore the EIGRP timers.

Loopbacks
R0: : 172.20.1.1/32
R1: : 172.20.2.1/32

Network Addresses:
R0: Serial1/0 : 192.168.10.1/30
R1: Serial1/0 : 192.168.10.2/30

ffr Addresses:
R0: Serial1/1.105 : 192.168.20.1/30
R1: Serial1/1.501 : 192.168.20.2/30

Ethernet interfaces:
R0: Fastethernet0/0: 20.20.20.1/24
r1: FastEthernet0/0: 10.10.10.1/24

Test reachability - before continuing - If all is well, add in EIGRP:

On R0

R0#sh run | section eigrp
router eigrp 1
network 192.168.0.0 0.0.255.255
no auto-summary

on R1

router eigrp 1
network 192.168.0.0 0.0.255.255
no auto-summary

What pops up immediately:


*Mar 1 00:02:32.223: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is up: new adjacency
*Mar 1 00:02:32.383: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.20.1 (Serial1/1.501) is up: new adjacency
*Mar 1 00:02:35.447: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.20.1 (Serial1/1.501) is resync: peer graceful-restart
*Mar 1 00:02:35.455: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is resync: peer graceful-restartR0#sh ip eigrp neighbors

IP-EIGRP neighbors for process 1

H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 192.168.20.2 Se1/1.105 12 00:10:52 894 5000 0 15
0 192.168.10.2 Se1/0 12 00:10:52 127 762 0 16

The value of the hold column shouldn't be more than the timer unless you're losing packets. Also the Q(queue) count column should always be '0' unless there's a problem.

For instance, lets block EIGRP on R0 and see what shows up on R1:

RO
R0(config)#access-list 100 deny  eigrp any any
R0(config)#access-list 100 permit ip any any
int s1/0
ip access-group 100 in

This pops up on R0
*Mar  1 00:38:34.547: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.2 (Serial1/0) is down: holding time expired
and on R1 we now have the following:
R1#
*Mar 1 00:38:34.899: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is down: Interface Goodbye received
*Mar 1 00:38:39.623: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is up: new adjacency
*Mar 1 00:39:59.147: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is down: retry limit exceeded
*Mar 1 00:40:03.619: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is up: new adjacency
R1#sh ip eigrp neighbors
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 192.168.10.1 Se1/0 11 00:00:36 1 5000 1 0
1 192.168.20.1 Se1/1.501 11 00:38:08 137 822 0 10
Note the queue count is 1 (one) , this implies an un-acknowledged hello.

On R0 the neighbor is removed:
R0#sh ip eigrp neighbors
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 192.168.20.2 Se1/1.105 13 00:38:54 894 5000 0 15
Other commands you can run to troubleshoot: I'll just BOLD key info from each command
R1#show ip eigrp interfaces
IP-EIGRP interfaces for process 1

Xmit Queue Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes
Se1/0 1 0/0 124 0/15 563 0
Se1/1.501 1 0/0 137 0/15 639 0

R1# show ip eigrp topology
IP-EIGRP Topology Table for AS(1)/ID(172.20.2.1)

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
r - reply Status, s - sia Status

P 192.168.10.0/30, 1 successors, FD is 2169856
via Connected, Serial1/0
P 192.168.20.0/30, 1 successors, FD is 2169856
via Connected, Serial1/1.501


R1# show ip eigrp traffic
IP-EIGRP Traffic Statistics for AS 1
Hellos sent/received: 1202/1179 <---remember we had blocked some hellos with access list 100
Updates sent/received: 109/9
Queries sent/received: 2/2
Replies sent/received: 4/2
Acks sent/received: 1/12
Input queue high water mark 4, 0 drops
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 174
PDM Process ID: 169
R1# show ip eigrp accounting
IP-EIGRP accounting for AS(1)/ID(172.20.2.1) <<--note teh ID is our loopback
Total Prefix Count: 2 States: A-Adjacency, P-Pending, D-Down
State Address/Source Interface Prefix Restart Restart/
Count Count Reset(s)
A 192.168.10.1 Se1/0 1 0 0
A 192.168.20.1 Se1/1.501 1 0 0
R1#show ip eigrp 100 ?
accounting  IP-EIGRP Accounting
interfaces IP-EIGRP interfaces
neighbors IP-EIGRP neighbors
topology IP-EIGRP Topology Table
traffic IP-EIGRP Traffic Statistics
!use this if running more than one AS on the same router


show ip eigrp neighbors
Already covered
!Note since these routes are learnt via connected, our routing table currently wont
show any eigrp routes. To get this we need to advertise our loopbacks via eigrp

R1#sh ip route
!--------edited----------!

192.168.10.0/30 is subnetted, 1 subnets
C 192.168.10.0 is directly connected, Serial1/0
172.20.0.0/32 is subnetted, 1 subnets
C 172.20.2.1 is directly connected, Loopback0
192.168.20.0/30 is subnetted, 1 subnets
C 192.168.20.0 is directly connected, Serial1/1.501
10.0.0.0/24 is subnetted, 1 subnets
C 10.10.10.0 is directly connected, FastEthernet0/0

See!! no EIGRP routes, now lets add the loopback addresses:
R0#sh run | section eigrp
router eigrp 1
network 172.20.1.0 0.0.0.255
network 192.168.0.0 0.0.255.255
no auto-summary

and on R1:
R1#sh run | section eigrp
router eigrp 1
network 172.20.2.0 0.0.0.255
network 192.168.0.0 0.0.255.255
no auto-summary
our output:
R0#sh ip route eigrp
172.20.0.0/32 is subnetted, 2 subnets
D 172.20.2.1 [90/2297856] via 192.168.20.2, 00:01:05, Serial1/1.105
[90/2297856] via 192.168.10.2, 00:01:05, Serial1/0
R1#sh ip route eigrp
172.20.0.0/32 is subnetted, 2 subnets
D 172.20.1.1 [90/2297856] via 192.168.20.1, 00:01:23, Serial1/1.501
[90/2297856] via 192.168.10.1, 00:01:23, Serial1/0
Notes:
*EIGRP doesnt build neighbor relationships over secondary addresses.
*Please note changing the hello interval using ip hello-interval eigrp doesn't adjust the hold. time timer for you.
*Load balancing: routing protocols install routes to your routing table, the switching fabric within the router/switch does the load balancing based on various things. eg per packet load balancing, per destination etc etc...all this depends on your switching (cef, fast switching)
let me make a quick demonstration:

If we try to reach R1's loopback with CEF enabled and the routing table as it is now (by default we use per destination loadbalancing) -
R1#debug ip icmp
ICMP packet debugging is on

R1#
*Mar 1 01:04:19.239: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.371: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.459: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.495: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.507: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
Note the path taken is the same, all packets to the same destination will use
same path

now disable CEF (globally - no ip cef)on R0 and make the same 'ping'
R1#
*Mar 1 01:07:27.411: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:07:27.495: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.10.1
*Mar 1 01:07:27.539: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:07:27.583: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.10.1
*Mar 1 01:07:27.591: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
Note the alternating paths, this is per Packet Load balancing
You can use sh ip interface to figure out what sort of switching is in use. I only did this to show how the routing protocol really only populates the routing database, how traffic is moved across networks is not its business.....I'll follow this up with EIGRP metrics,feasible distance, reported distance and feasible successors.....

References:
Cisco
Oreilly's IP routing by By Ravi Malhotra - I found chapter4 available on oreilly:-) good for you