CRS1 snmp and other counters

Sooo Im almost done with a very interesting project and wanted to share a little something on what happened when we wanted to monitor the optical interfaces.

First off the core network has a bunch of CRS1's interconnected using 3 x 10G DWDM links to the rest of the network giving us what someone fondly called a 10Gig core, or 30 depending on how/what you calculate.

Now the dwdm link characteristics are very important to ensure the deck of cards above it doesnt come crashing down. Some fiber was not very clean and you'd get parameters like:

RP/0/RP0/CPU0:xxxxxxx1#sh controllers dwdm 0/0/0/2
Thu Nov 25 14:10:16.303 EAT

Port dwdm0/0/0/2

Controller State: up

Loopback: None

G709 Status


OTU
         LOS  = 0           LOF  = 0           LOM  = 0
         BDI  = 0           IAE  = 0           BIP  = 0
         BEI  = 0           TIM  = 0

ODU
         AIS  = 0           BDI  = 0           OCI  = 0
         LCK  = 0           BIP  = 0           BEI  = 0
         PTIM = 0           TIM  = 0

FEC Mode: Enhanced FEC(default)
         EC(current second) = 3217480     EC   = 602132861570  UC   = 43432861570
         pre-FEC BER = 2.89E-4            Q  = 3.42          Q Margin = 1.74

Remote FEC Mode: Enhanced FEC
         FECMISMATCH = 0

Detected Alarms: None
Asserted Alarms: None
Alarm Reporting Enabled for: LOS LOF LOM IAE OTU-BDI OTU-TIM OTU_SF_BER OTU_SD_BER ODU-AIS ODU-BDI OCI LCK PTIM ODU-TIM FECMISMATCH
BER Thresholds: OTU-SF = E-2  OTU-SD = E-5

OTU TTI Sent     String ASCII: Tx TTI Not Configured
OTU TTI Received String ASCII: Rx TTI Not Recieved
OTU TTI Expected String ASCII: Exp TTI Not Configured

ODU TTI Sent     String ASCII: Tx TTI Not Configured
ODU TTI Received String ASCII: Rx TTI Not Recieved
ODU TTI Expected String ASCII: Exp TTI Not Configured

Optics Status

         Optics Type:  Cisco 10Gb DWDM
         Wavelength Info: C-Band, XXX ITU Channel=27, Frequency=1xx.80THz, Wavelength=1534.976nm
         TX Power = 4.45 dBm
         RX Power = -10.90 dBm
TDC Info

        TDC Not Supported on the Plim

So any way we needed a way to graph and send alerts based on output like that.

IOS/XR's MIB's do not seem to have anything I could work with.

*if you know the MIB/OID to get me TX/RX power,UC,EC and prefec values I'll be very grateful.

*If you also happen to have an explanation on how to interpret the pre fec values eg:

pre-FEC BER = 2.89E-4 

I'd also be very happy.

Enter some scripts:
to get this info we used a combination of expect and perl.

Basically we wrote a script that connects to each node and runs runs the command and dumps it on a text file. our perl-fu then comes on along and picks out the bits and pieces we need and dumps it on a database.

Zabbix (our mrtg'like monitoring system) graphs those. A few other scripts send us sms and email alerts.

Sample expect script:

set timeout 3
spawn /bin/bash
match_max 100000
send -- "telnet 192.168.1.1\r"
expect -exact "telnet 192.168.1.1\r
Trying 192.168.1.1...\r
Connected to 192.168.1.1.\r
Escape character is '^\]'.\r
\r
\r
User Access Verification\r
\r
Username: "
send -- "yadada\r"
expect -exact "durangor\r
Password: "
send -- "django\r"
expect -exact "\r
\r
RP/0/RP0/CPU0:xxxxxxxx1#"
send -- "show controllers dwdm 0/0/0/2"
expect -exact "show controllers dwdm 0/0/0/2"
send -- "\r"
send -- " exit\r"
send -- " "
expect eof

its a simple one that one.

other helpful commands for graphing other things:
sh snmp mib object-name
sh snmp interface
sh snmp interface tenGigE 0/0/0/0 ifindex

I am having alot of fun with IOS xr, the crs1 has some interesting features/utilities too....more on this later...

the matrix & stupidity

sunday afternoon, nice sunny day re-running 'The Matrix'....while listening to the line with morpheus saying the matrix is all around us , it immediately jumped to mind and I completed the sentence, just like stupid people....

the matrix is all around us, just like stupid people -

* one of my biggest fears is that i become stupid or thick or......

Gitau

Seneca: Moral letters to Lucilius

Ahhh the folly of road construction and fiber cuts:-) while I totally thoroughly don't mind the infrastructure build going on in the country (Kenya), the near total outages we're suffering due to fiber cuts are raising very interesting questions....and halted some of my work:-)

How do you build a redundant national fiber infrastructure?
how come the builders didn't know or allow for this sort of thing happening?
Is there room for new players?
whats my role in all this?
will the local counties benefit from this ?
and many many more

so I drifted to Moral letters to Lucilius/Letter 22 mainly just to kill time and maybe get in touch with my inner self.....I basically have nothing to write today....but I felt I had to fill the space:-) no the letters and the fiber cuts are not related either:-)

anyway interesting words from the moral letters:
11. My dear Lucilius; there are a few men whom slavery holds fast, but there are many more who hold fast to slavery.
12. But if you keep turning round and looking about, in order to see how much you may carry away with you, and how much money you may keep to equip yourself for the life of leisure, you will never find a way out. No man can swim ashore and take his baggage with him.

*PS I read these letters during boring endless time suck's called meetings...

Openflow - why its an exciting game changer for me

When I started my career (8 years ago) in networking, life was easy. I worked at a startup and had alot of freedom to experiment. I did some sysadmin stuff, lots of networking stuff and life was great.

Over that time something struck me as odd: when choosing for instance on what to run Bind, I had a choice of hardware, CPU's, memory and many many operating systems to chose from. This coupled with opensourse stuff meant we could run an ISP very cheaply with commodity hardware. I think we ran DNS on a regular Dell PC mounted on a shelf for the more than 3 yrs I worked there.

What we sort of accepted without question was the networking gear. We'd call a vendor test their products and buy it in full. The hardware, processor/central logic manufacturer,operating systems -- everything was by the same company - in our case cisco.

When I moved on to a mobile network, I noticed an uncanny similarity to this scenario with the GSM industry. Apple Android and other manufacturers have done alot to open up an industry that was shrouded in mystery. GSM products and specifically the core electronic components were as far as I know only done by a few companies: classic suppliers were Texas Instruments, ST/Ericsson, ADI/MediaTek), In neon, Freescal. Herald welte has an excellent article on this.

For each of the other things I needed to learn, there was always a simulator and cheap hardware to learn on. Openbsc and osmocombb (I had lots of fun with this on a motorolla C117) really helped me 'kill some two months eaarly this year. I killed that phone, doubt it'll ever make another call though maybe I'll have some time to mess around with it. its definately my fault that it doesn't work, not the software.

When I started on mobile data I used opensgsn extensively for testing the core network to verify flows without worrying about the access network,coupled with open bsc I could create very interesting test scenarios without worrying about the radio. It also made me sound quite clever while having discussions with the 'core' network guys.

I saw the openggsn project is now back to life. I never played around with this. I was lucky to have actual hardware - Cisco + Huawei.

Where am I going with this and what does openflow have to do with it:

Well I installed openflow over the last weekend after reading James Hamilton 's Datacenter networks are in my way document. I also have lots of hardware (PC's) to play around with. It addresses my earlier observation that in the networking world, it’s a vertically integrated stack and this slows innovation and artificially holds margins high. It’s a business model makes services unnecesarily expensive for everyone apart from the vendor and It lags behind in 'conforming to the rest of the pack ie servers and other general technology areas which are open.

Enter http://www.openflowswitch.org

The important thing for me is where this could lead to. Cheaper products, excellent learning platform (especially if cisco insist on making IOS inaccessible for educational purposes) and the ability to run networks on cheap hardware. Our small businesses require this. To this end I havt to also thank the folks running quagga,vyatta (I had to shift our entire NAT for millions of customers to Vyatta at some point - the free version) et all....I have had alot of fun...

If interested, my openflow is currently on a standalone PC, next I'll play with multiple PC's, vmware and see where this wave leads to....all in all a very productive use of my free time...

Gender Violence Recovery Center

Third world networker will be mixing it up at The Safaricom Foundation and the Gender Violence Recovery Center (GVRC) annual fundraising dinner this evening which will be at the Carnivore Grounds.

The Foundation will be matching every shilling raised so I figure it's good use of my money. The foundations results are very visible so I try and support their projects whenever/howevre I can.

why: because third world networker believes in a few things, he also tries to have a life outside the lab and office.

stp, sctp and tcp

It was during a discussion about the migration strategies for the R4/R99 and some LTE test gear that it was disclosed a new STP (Signalling transfer point) would be integrated at around the same time the new core would be coming 'live'. Some of the MPLS vendor's guys have not worked on a mobile core before so it's a learning experience for everyone.

It got more interesting when a 'new' protocol SCTP was mentioned as being in extensive use. One of the guys is a new ccie and as you can imagine his TCP knowledge is still fresh in his mind. It thoroughly bewildered him why SCTP was even necessary in the first place. We white boarded quite a number of reasons with explanations (it makes it easier for the migration if we're all semi clueful of what's going on on the network) so we tend to stop 'everything' and explain alot.

rfc2960 (http://www.ietf.org/rfc/rfc2960.txt) gives a very clear explanation. Please head over there to see why SCTP is necessary for some applications.

I also expect alot more implementations and use of SCTP over the coming years.well at least I expect it at work. It has a mature socket API so writing applications for this is not a big problem for experienced and novice programmers who dont mind putting in some time.

At work I've changed some apps to sctp to gain the 'multihomning advantage', other than that there's really nothing to get excited about. It's however good to have an idea that such a protocol exists, if nothing else, it makes for exciting 'beer' talk.

weird language blues

So the final stages of our core modernization project have had me communicating in very funny acronym like terms: NRFU, FTS,NOS,SFP,XFP,DWDM,Lambda, etc etc add that to all the acronyms we go through for the ccie bgp,ospf,rip,lisp,trill -last two not really ccie stuff but our data center team has them rolling off their tongues too easilly lately and you've got a nice recipe for confusion. Regular project managers (the sort that only ask about time, money (when when when!)) have their own AFAIK,IMO,IMHO...I think we're slowly coming up with a new language....

ok back to TS:-)
Gitau

Why I got back on the ccie wagon!

Well my written had expired, pursuing it (the ccie) was no longer fun -I have had more exciting projects this year- and other areas of my life were not keeping up with the extensive study required to get the ccie lab done.

However: I find that there are now several good, quite good reasons with actual practical application to go back to it. So I re-took the written v4 -twice (I had underestimated and grossed over some topics! assembled an actual lab and I think Im in pretty good form to go for the lab now.

And yes the main motivation is now a job swap or go independent. After this project Im working on (a huge major mobile core redesign/migration from the good old tdm to the IP wonders featureing a no small number of CRS-1's (P) and 7609's (PE)) - throw in multivendor CE's and the fun just never stops. I can only foresee boredom and trouble tickets after this :-( unless something changes within the organization (from structure to compensation policies).

JGitau