Tuesday, December 7, 2010

lab dates set

Well...big project done, some free time coming up, mobile lab right here at home:-)

Lab Details
Lab Type Routing and Switching
Lab Date 07-MAR-11
Lab Location Nairobi

I wish me well:-)
sleepless nights coming up....

Thursday, November 25, 2010

CRS1 snmp and other counters

Sooo Im almost done with a very interesting project and wanted to share a little something on what happened when we wanted to monitor the optical interfaces.

First off the core network has a bunch of CRS1's interconnected using 3 x 10G DWDM links to the rest of the network giving us what someone fondly called a 10Gig core, or 30 depending on how/what you calculate.

Now the dwdm link characteristics are very important to ensure the deck of cards above it doesnt come crashing down. Some fiber was not very clean and you'd get parameters like:

RP/0/RP0/CPU0:xxxxxxx1#sh controllers dwdm 0/0/0/2
Thu Nov 25 14:10:16.303 EAT

Port dwdm0/0/0/2

Controller State: up

Loopback: None

G709 Status

LOS = 0 LOF = 0 LOM = 0
BDI = 0 IAE = 0 BIP = 0
BEI = 0 TIM = 0

AIS = 0 BDI = 0 OCI = 0
LCK = 0 BIP = 0 BEI = 0
PTIM = 0 TIM = 0

FEC Mode: Enhanced FEC(default)
EC(current second) = 3217480 EC = 602132861570 UC = 43432861570
pre-FEC BER = 2.89E-4 Q = 3.42 Q Margin = 1.74

Remote FEC Mode: Enhanced FEC

Detected Alarms: None
Asserted Alarms: None
BER Thresholds: OTU-SF = E-2 OTU-SD = E-5

OTU TTI Sent String ASCII: Tx TTI Not Configured
OTU TTI Received String ASCII: Rx TTI Not Recieved
OTU TTI Expected String ASCII: Exp TTI Not Configured

ODU TTI Sent String ASCII: Tx TTI Not Configured
ODU TTI Received String ASCII: Rx TTI Not Recieved
ODU TTI Expected String ASCII: Exp TTI Not Configured

Optics Status

Optics Type: Cisco 10Gb DWDM
Wavelength Info: C-Band, XXX ITU Channel=27, Frequency=1xx.80THz, Wavelength=1534.976nm
TX Power = 4.45 dBm
RX Power = -10.90 dBm
TDC Info

TDC Not Supported on the Plim

So any way we needed a way to graph and send alerts based on output like that.

IOS/XR's MIB's do not seem to have anything I could work with.

*if you know the MIB/OID to get me TX/RX power,UC,EC and prefec values I'll be very grateful.

*If you also happen to have an explanation on how to interpret the pre fec values eg:

pre-FEC BER = 2.89E-4

I'd also be very happy.

Enter some scripts:
to get this info we used a combination of expect and perl.

Basically we wrote a script that connects to each node and runs runs the command and dumps it on a text file. our perl-fu then comes on along and picks out the bits and pieces we need and dumps it on a database.

Zabbix (our mrtg'like monitoring system) graphs those. A few other scripts send us sms and email alerts.

Sample expect script:

set timeout 3
spawn /bin/bash
match_max 100000
send -- "telnet\r"
expect -exact "telnet\r
Connected to\r
Escape character is '^\]'.\r
User Access Verification\r
Username: "
send -- "yadada\r"
expect -exact "durangor\r
Password: "
send -- "django\r"
expect -exact "\r
send -- "show controllers dwdm 0/0/0/2"
expect -exact "show controllers dwdm 0/0/0/2"
send -- "\r"
send -- " exit\r"
send -- " "
expect eof

its a simple one that one.

other helpful commands for graphing other things:
sh snmp mib object-name
sh snmp interface
sh snmp interface tenGigE 0/0/0/0 ifindex

I am having alot of fun with IOS xr, the crs1 has some interesting features/utilities too....more on this later...

Sunday, November 21, 2010

the matrix & stupidity

sunday afternoon, nice sunny day re-running 'The Matrix'....while listening to the line with morpheus saying the matrix is all around us , it immediately jumped to mind and I completed the sentence, just like stupid people....

the matrix is all around us, just like stupid people -

* one of my biggest fears is that i become stupid or thick or......


Thursday, November 11, 2010

Seneca: Moral letters to Lucilius

Ahhh the folly of road construction and fiber cuts:-) while I totally thoroughly don't mind the infrastructure build going on in the country (Kenya), the near total outages we're suffering due to fiber cuts are raising very interesting questions....and halted some of my work:-)

How do you build a redundant national fiber infrastructure?
how come the builders didn't know or allow for this sort of thing happening?
Is there room for new players?
whats my role in all this?
will the local counties benefit from this ?
and many many more

so I drifted to Moral letters to Lucilius/Letter 22 mainly just to kill time and maybe get in touch with my inner self.....I basically have nothing to write today....but I felt I had to fill the space:-) no the letters and the fiber cuts are not related either:-)

anyway interesting words from the moral letters:
11. My dear Lucilius; there are a few men whom slavery holds fast, but there are many more who hold fast to slavery.
12. But if you keep turning round and looking about, in order to see how much you may carry away with you, and how much money you may keep to equip yourself for the life of leisure, you will never find a way out. No man can swim ashore and take his baggage with him.

*PS I read these letters during boring endless time suck's called meetings...

Monday, November 8, 2010

Openflow - why its an exciting game changer for me

When I started my career (8 years ago) in networking, life was easy. I worked at a startup and had alot of freedom to experiment. I did some sysadmin stuff, lots of networking stuff and life was great.

Over that time something struck me as odd: when choosing for instance on what to run Bind, I had a choice of hardware, CPU's, memory and many many operating systems to chose from. This coupled with opensourse stuff meant we could run an ISP very cheaply with commodity hardware. I think we ran DNS on a regular Dell PC mounted on a shelf for the more than 3 yrs I worked there.

What we sort of accepted without question was the networking gear. We'd call a vendor test their products and buy it in full. The hardware, processor/central logic manufacturer,operating systems -- everything was by the same company - in our case cisco.

When I moved on to a mobile network, I noticed an uncanny similarity to this scenario with the GSM industry. Apple Android and other manufacturers have done alot to open up an industry that was shrouded in mystery. GSM products and specifically the core electronic components were as far as I know only done by a few companies: classic suppliers were Texas Instruments, ST/Ericsson, ADI/MediaTek), In neon, Freescal. Herald welte has an excellent article on this.

For each of the other things I needed to learn, there was always a simulator and cheap hardware to learn on. Openbsc and osmocombb (I had lots of fun with this on a motorolla C117) really helped me 'kill some two months eaarly this year. I killed that phone, doubt it'll ever make another call though maybe I'll have some time to mess around with it. its definately my fault that it doesn't work, not the software.

When I started on mobile data I used opensgsn extensively for testing the core network to verify flows without worrying about the access network,coupled with open bsc I could create very interesting test scenarios without worrying about the radio. It also made me sound quite clever while having discussions with the 'core' network guys.

I saw the openggsn project is now back to life. I never played around with this. I was lucky to have actual hardware - Cisco + Huawei.

Where am I going with this and what does openflow have to do with it:

Well I installed openflow over the last weekend after reading James Hamilton 's Datacenter networks are in my way document. I also have lots of hardware (PC's) to play around with. It addresses my earlier observation that in the networking world, it’s a vertically integrated stack and this slows innovation and artificially holds margins high. It’s a business model makes services unnecesarily expensive for everyone apart from the vendor and It lags behind in 'conforming to the rest of the pack ie servers and other general technology areas which are open.

Enter http://www.openflowswitch.org

The important thing for me is where this could lead to. Cheaper products, excellent learning platform (especially if cisco insist on making IOS inaccessible for educational purposes) and the ability to run networks on cheap hardware. Our small businesses require this. To this end I havt to also thank the folks running quagga,vyatta (I had to shift our entire NAT for millions of customers to Vyatta at some point - the free version) et all....I have had alot of fun...

If interested, my openflow is currently on a standalone PC, next I'll play with multiple PC's, vmware and see where this wave leads to....all in all a very productive use of my free time...

Friday, November 5, 2010

Gender Violence Recovery Center

Third world networker will be mixing it up at The Safaricom Foundation and the Gender Violence Recovery Center (GVRC) annual fundraising dinner this evening which will be at the Carnivore Grounds.

The Foundation will be matching every shilling raised so I figure it's good use of my money. The foundations results are very visible so I try and support their projects whenever/howevre I can.

why: because third world networker believes in a few things, he also tries to have a life outside the lab and office.

Thursday, November 4, 2010

stp, sctp and tcp

It was during a discussion about the migration strategies for the R4/R99 and some LTE test gear that it was disclosed a new STP (Signalling transfer point) would be integrated at around the same time the new core would be coming 'live'. Some of the MPLS vendor's guys have not worked on a mobile core before so it's a learning experience for everyone.

It got more interesting when a 'new' protocol SCTP was mentioned as being in extensive use. One of the guys is a new ccie and as you can imagine his TCP knowledge is still fresh in his mind. It thoroughly bewildered him why SCTP was even necessary in the first place. We white boarded quite a number of reasons with explanations (it makes it easier for the migration if we're all semi clueful of what's going on on the network) so we tend to stop 'everything' and explain alot.

rfc2960 (http://www.ietf.org/rfc/rfc2960.txt) gives a very clear explanation. Please head over there to see why SCTP is necessary for some applications.

I also expect alot more implementations and use of SCTP over the coming years.well at least I expect it at work. It has a mature socket API so writing applications for this is not a big problem for experienced and novice programmers who dont mind putting in some time.

At work I've changed some apps to sctp to gain the 'multihomning advantage', other than that there's really nothing to get excited about. It's however good to have an idea that such a protocol exists, if nothing else, it makes for exciting 'beer' talk.

Wednesday, November 3, 2010

weird language blues

So the final stages of our core modernization project have had me communicating in very funny acronym like terms: NRFU, FTS,NOS,SFP,XFP,DWDM,Lambda, etc etc add that to all the acronyms we go through for the ccie bgp,ospf,rip,lisp,trill -last two not really ccie stuff but our data center team has them rolling off their tongues too easilly lately and you've got a nice recipe for confusion. Regular project managers (the sort that only ask about time, money (when when when!)) have their own AFAIK,IMO,IMHO...I think we're slowly coming up with a new language....

ok back to TS:-)

Tuesday, November 2, 2010

Why I got back on the ccie wagon!

Well my written had expired, pursuing it (the ccie) was no longer fun -I have had more exciting projects this year- and other areas of my life were not keeping up with the extensive study required to get the ccie lab done.

However: I find that there are now several good, quite good reasons with actual practical application to go back to it. So I re-took the written v4 -twice (I had underestimated and grossed over some topics! assembled an actual lab and I think Im in pretty good form to go for the lab now.

And yes the main motivation is now a job swap or go independent. After this project Im working on (a huge major mobile core redesign/migration from the good old tdm to the IP wonders featureing a no small number of CRS-1's (P) and 7609's (PE)) - throw in multivendor CE's and the fun just never stops. I can only foresee boredom and trouble tickets after this :-( unless something changes within the organization (from structure to compensation policies).


Wednesday, August 11, 2010

African Peering and Interconnection Forum: Unlocking Africa’s Regional Interconnection

The internet society is for 3 days sponsoring a conference :
The African Peering and Interconnection Forum: Unlocking Africa’s Regional Interconnection

Now for most that know me, I started my career working for a startup ISP that has since grown way bigger than I had imagined. I also happen to have been around when we had to technically justify peering and answer questions like what is internet peering, why peer, how does it work etc etc...and finally the KIXP was set up.

Im interestingly caught up in another transformation (IP/Buusiness transformation for a large telco). so see you there...

I moved on quite a bit, but the opportunities to catch up with old friends is always welcome.


registration is still open, slides will be availed and live streaming is available here:

IM conference Service and Email

(a) Those with jabber clients can join the AfPIF conference room,

A list of jabber clients can be found at

(b) Those without can send their comments and/or queries to
mwangi@isoc.org, karen@isoc.org and morris@isoc.org


Monday, April 12, 2010

Cisco SAMI blades....and ASA 5580's

I have worked for the last 3 or so years now with the cisco GGSN/CSG aka CMX (Cisco Mobile Exchange) supporting gsm,gprs and UMTS on the 7613 platform.

The old boxes were getting sort of old and tired, They also have the new ASR5000 (starent acquisition) that I believe to be by far more superior than what they were pushing.

Anyway pending some decisions, cisco being the nice guys they are actually sent some brand new toys for me to test. Please see attached....this is going to be a great week:-) I get to lab up the entire topology and integrate the new toys:-)

**Last three images are the sami blades

I also get to play around with the Cisco ASA 5580 with GTP inspection licence. Now this has been on my wish list for a while now. Swapping out the almost old firewalls for re-deployment will be easy. The fun is mainly in the higher number of supported interfaces and their capacity (10G), the gtp inspection will save our GGSN's quite alot of resources....oh well some photos (yes I opened them up to see what's in there).....

I'll post a review of both next time.....
I have some requests for easy to understand terms when dealing with gsm/gprs/umts mobility....I'll see about that too...Its just so wide I don't know where to start so if you have some ideas please let me know....

Tuesday, April 6, 2010

New direction?

Every once in a while I remind myself to update this blog. Trouble is most of the material Im working on for the CCIE has been posted somewhere. So apart from having some sort of online diary to keep track of stuff, I rarely see sense in making posts. so I don't. I keep detailed notes though...

I however realize that third world networkers can be about something else. How we do things here. I have been lucky to see/visit other networks for good enough comparisons. I can tell we are lagging behind in some areas .... I also happen to work on a large mobile network.

I can make this about how we run networks a bit differently on a tighter budget to satisfy totally different customer needs.

It can be about how we don't have a terrestial network to speak of and how we manage to get by.

It can be about how since I set up my first network until late last year (8years now), all international traffic was on satellite.

It can be about how a straight forward PCEF <==> pcrf<==>IN implementation can suddenly become complicated due to vendor use of proprietary protocols and how that has (a/no) place on tomorrows network.....

It can be tips on how to scale a CSG to carry more traffic than what Cisco originally posted on their page. Or how the Huawei GGSN compares to the ASR 5000.

It can be about me and what Im thinking about and all the cool stuff Im playing with...yes it can be about the ccie which is more a means to an end now....there just is no excitement....just need the digits to move on....cisco have made too many changes and contradictory remarks about this program its just tiring ... but i've worked too hard to stop now so.....lets see where this leads to....see you around

Thursday, February 18, 2010

still on track...i think!!

oh well...figured I have a few minutes to type up.....yes Im still on, if financials allow I think Im good to go in April .... I like that i can now pretty much book the lab with a week to spare, heck i can book flights,visa's before the lab if i wanted to:-) R&S seats are not as full as late last year... -- so I won't rush the booking (I have found booking the lab three months in advance to be some sort of motivator in itself in the past)....

wish me well...