At the Hadrian Hotel

At the Hadrian Hotel

Monday, March 02, 2015

The End of an Era, The Beginning of a New Adventure

For the past 27 years, I have been involved with the running of major network infrastructure for Princeton University.  Way back in 1988, on my birthday, I took a position with CIT's Network Systems group, working with Cisco AGS, MGS, and IGS routers, Applitek CATV-to-10Base5 Ethernet bridges, and Cabletron MMAC switches.  I wrote a whole bunch of Perl code to implement a host "database," worked with everybody and his brother to re-subnet the campus, and launched the first BOOTP service to make that re-subnetting task easier.

Along the way I moved from using X10 on an IBM RT to X11 on a Sun 4 (or was it a 3?) while the campus moved from AUI connections to proprietary twisted-pair ethernet to 10Base-T.  We deployed Hughes LAN Systems CATV-to-Ethernet bridges for the main campus network and moved the Appliteks over to the new DormNet network.  We also dabbled in ATM for the campus and ISDN out to remote sites.  The dial-up terminal servers just weren't fast enough for some folks, I suppose.

After just 7 years and 9 months, I left the central computing organization for a position on the Technical Staff of the Computer Science department.  The group was smaller than CIT, unsurprisingly, but was called upon to provide many of the same services, albeit for a smaller and much more demanding group of customers.  While my duties and tasks there were more varied, I was still, at heart, a "network guy."  In CS, I oversaw the move from 10Base2 to 10baseT using Cisco Catalyst 5000 switches, and then to a shiny new Packet Engines switch.  When Alcatel bought Packet Engines, we followed, and even tried their next generation architecture.  I'd tell you what it was, but I don't remember as we were not too happy with it and moved to a Foundry FastIron 1500 as soon as we could.  While we were mucking about with all of these wired network changes, we also deployed the first sizable wireless installation on campus, using (pre-Cisco-acquisition) Aironet access points.  I also wrote a new host database application just for the CS department.

After Cisco bought Aironet and before we bought the Foundry, Cisco added MAC-based Authentication to the access points.  Nobody seemed to have this feature on the wired side of the house - at least it didn't exist on the FastIron 1500 we just installed.  I proposed to our Foundry team that they add this as a feature, and they worked with me, and others, to do just that.  As part of this collaboration with Foundry and members of the CS Tech Staff, I was able to present a paper at the 2004 USENIX LISA Conference.  I stayed a bit longer with CS than I did CIT, but in May of 2011 I moved from CS to OIT - while I was gone they renamed the department.

My position at OIT was the Manager of the Network Systems group, the same group I had left almost 16 years before.  For the past 3 years and 11 months, I managed a small, dedicated team responsible for the architecture and overall operation of the campus network.  While my team didn't do the hard field work of dealing with switches in wiring closets, if something goes wrong on the network, the buck stopped with us.

During my tenure, we collapsed multiple Catalyst 6513 routers at the core down to a Nexus 7018, and are in the process of configuring the Nexus 7018 in the on-campus data center as a redundant switch/router for all of the buildings on the campus.  We also brought online a 7018 at the University's High-Performance Computing Research Center, NATed the campus-wide wireless infrastructure, and helped our colleagues in Enterprise Infrastructure Services move from layer 2 load balancers and firewalls to layer 3 infrastructure, as well as a number of other campus network improvements.

As of today, all of that is behind me.  The network team is under new (very capable) management, and I am moving on to become the Senior Architect for Advanced Networking in OIT.  I have been tasked, along with a number of other newly-minted Senior Architects, to divine and define the direction for computing and network infrastructure for the campus, moving forward.

I am moving from the Support Service group to Research Computing.  We expect research to drive the needs for advanced network infrastructure on campus, so putting me in the group that directly serves the research community makes quite a bit of sense.  I expect to be spending a fair amount of time meeting with researchers across many disciplines as we try to figure out exactly what problems we need to solve to provide them with world-class infrastructure.  However, we will not be designing the next generation of the campus network only for the researchers.  I firmly believe that SDN, and other new and emerging network technologies will allow us to improve the network for everyone at Princeton.

For the first time in 27+ years, I will not have a day-to-day operational role in network infrastructure.  I cannot say that I will necessarily miss it, but I will miss working so closely with the dedicated people that keep the bedrock of the campus infrastructure stable.  However, exciting times are ahead, and I am thrilled that I will be at the fore as we build the next generation of computing and network infrastructure at Princeton.

Monday, May 23, 2011

The Times, They Are A-Changin'

Friday marked the end of my last week as a member of the Computer Science Department. For over 15 years I worked with dedicated individuals who saw to the needs of the faculty, students and staff of the department. I was fortunate to to work around some very smart people and learned quite a bit.

Today I start a new adventure, as the manager of the Network Systems group within the Office of Information Technology. I've known the people in my group for almost all of the time I've been at the University, and look forward to working with them. The scope of my new job will be a bit more narrow than my last, but the scale will be huge. All-in-all, a fair balance, I'd say. :-)

I'm excited to see where all of this will take me.

Tuesday, October 06, 2009

Running Simplify Media's Software on CentOS 5

I love Simplify Media's music sharing system. It lets me have access to all of the music I have on one computer from my iPhone or any other computer I use. So, I no longer need multiple copies of every song, and I can listen to the entirety of my collection from the iPhone, regardless of the amount of memory available on the phone. Now, I don't have a ton of music, but I do have over 30 GB. Clearly, that would be a bit tough to shoe-horn on to a 16 GB iPhone.

Now, in addition to the iPhone app, Simplify has software for the Mac, MS-Windows, and Ubuntu Linux. My home fileserver, where all of my music lives, is a Linux system, but for a number fo reasons I am running CentOS, not Ubuntu. Unfortunately, the Linux code provided by Simplify will not "just run" on CentOS because of a system library problem.

Simplify's Linux software includes the SimplifyMedia client and the SimplifyPeer server. These programs are dynamically linked and written in C++, and therefore require the library. The latest library version available on CentOS is libstdc++.6.0.8, but the Simplify programs require I found all of this out when I tried to run SimplifyPeer and got this error message:
./SimplifyPeer: /usr/lib/ `GLIBCXX_3.4.9' not found (required by ./SimplifyPeer)
Being the geek that I am, I ran the 'strings' command against the library and looked for "GLIBCXX." The highest number I found was "3.4.8," which clearly was not going to cut it. I did a number fo different Google searches, and most of the results I found spoke of replacing the file, but very few of them actually told me how to go about getting a copy. I finally stumbled across a forum post on that included a link to download a RAR file containing the libstdc++.6.0.9 library file.

Great! Now I had the library file I needed to run the program. The problem I now had was that I was a bit nervous about replacing /usr/lib/ I wasn't sure if just replacing it with a later version would break anything. So I started looking at options for the dynamic linker, starting with LD_LIBRARY_PATH. Unfortunately, none of the various permutations of LD_LIBRARY_PATH values I could think of worked. It was time to check out the man page....

In the man page on my system, the first environment variable listed is LD_LIBABRY_PATH, and the second one is LD_PRELOAD. Herein lies the required magic. Files listed in this variable are "ELF shared libraries to be loaded before all others." This allows you to "selectively override functions in other shared libraries." It turns out this was just what I needed.

The SimplifyPeer program is normally invoked by a bash script that sets a number of arguments in addition to those required from the user. Since I had unpacked the .tgz file into /usr/local, all I needed to do was put the file in the same directory as the program, and add this line to the bash script before the SimplifyPeer program was run:
export LD_PRELOAD=/usr/local/simplifymedia/
That done, I fired up the script with the appropriate arguments for my account, machine name and MP3 directory, and it ran! My family and I could now use Simplify Media to access all of the music on my fileserver from my laptop, the various PCs around the house and, most importantly, the short-on-memory iPhone. This will allow for more varied musical choices on those long car rides to DC when transporting my oldest to and from college.

Wednesday, March 18, 2009

Google Voice: First Impressions

After waiting for what felt like forever, last night I finally got my invitation to upgrade from GrandCentral to Google Voice. Following are some of my first impressions....

Custom ring-back tones are gone - my British friends used to get something that sounded like home.

"Call Presentation" is now settable by group and address book entry - finally, I no longer need to press "1" to accept a call from members of my family. When I pick up the call is immediately connected. This is different form the GrandCentral call screening options. There, I could either screen everybody, screen blocked number, or screen nobody. This finer-grained control is something I (and others, I suspect) have been waiting for. There does seem to be a downside, though. If I have call presentation off (aka connect on answer), there's a chance that my home answering machine, my work voicemail, or my mobile voicemail will answer even if I don't, and then the advantage of Google Voice's voicemail is lost. I'll need to check up to 3 places if I don't pick up fast enough.

Custom greetings seem to still be available.

Voicemail messages now get transcribed to text. By default the transcribed messages is e-mailed to your GMail account. You can still also get notification of new voicemail via SMS to your cell phone, but now the SMS message has the transcribed message (at least if it's short enough). Nicely, the voicemail notification now comes from my Google Voice number, rather than some seemingly random SMS number. That allows me (at least on the iPhone) to call that number and then pick up the message, in case the entire thing didn't get sent in the SMS message or if the transcription looks strange. I've left myself a few simple voicemail messages that were not transcribed all too well.

Text messages are interesting... If somebody sends a SMS message to my Google Voice number, it shows up on my phone as coming from some number in the 406 area code. If I reply to that number, they get my reply. If somebody else sends a SMS message to that number, it is not delivered to the sender of the original message. Clearly there is some sort of mapping such that messages from me to that number get re-routed back to the originator. This mapping appears to last at least 2.5 hours - maybe it's permanent.

In addition, if I get a text message from somebody in my Google address book, their name shows up as a prefix in every message I get from them. If they aren't in my address book, then their originating phone number shows up there instead.

That's it for now. Lunch is over and I really should get back to doing what I get paid for. :-)

Technorati Tags: , ,

Tuesday, March 10, 2009

ETech Notes from "Refactor Your Wetware"

My morning ETech 09 tutorial was "Refactor Your Wetware," presented by Andy Hunt, the author of the book "Pragmatic Thinking & Learning." Below are some notes I took during the tutorial. Perhaps they will be useful to others.

There is a big difference between typing a note and hand-writing because of the way the brain processes during both activities.

The Dreyfus Model:

If you force experts to follow the rules laid out for novices, you can degrade their performance signifcantly (up to 100%?).

Experts are more likely to see themselves as part of the system. Novices feel that they are outside the system.

There is a cultural bias against intuition, which sprins from deeply ingrained knowledge. Experts operate on intuition.

The brain can be imagined as 2 different types of "CPUs" with shared access to the memory, where only 1 CPU can access the memory at a time. CPU #1 is more "von Neumann" and linear and slower in operation. CPU #2 is more like a DSP, non-linear and fast. CPU #1 is sometimes referenced as the left brain, and CPU #2 is sometimes referenced as the right brain. We'll use L-Mode (for linear mode) and R-Mode (for rich mode).

N. Negraponte: If you want to learn about a frog, don't disect one, build one.

Pretty (or aesthetically pleasing) things are actually easier to use.

When you're typing notes, L-Mode gets preference and shuts down R-Mode. You're forcing symbolic processing.

Dream state imagery is more R-Mode, which is why a dream evaporates as you you try to explain it, which utilizes L-Mode. You can't read signs in a dream because it would require L-Mode.

Math prodigies have better coordinated L-Mode and R-Mode processing, rather than having to switch back and forth like the rest of us.

Check into Lozanov Séances from the 1970s. He would immerse students in a rich R-Mode experience related to a given task. They would perform better than students who did not have the R-Mode experience.

Leading with an R-Mode experience before an L-Mode "lecture" gives a context for the L-Mode staff to better stick to.

Investigate Image Streaming and Morning Pages.

A "whack on the side of the head" can help to clarify thinking. Looking at a problem differently, such as "in reverse" can be helpful.

If you don't keep track of great ideas that you have, you'll stop noticing that you have them, and then you'll stop having them. Carry a notebook of some type so you can write down ideas as you have them.

"Education" comes from "edu" and "care" which translates to "drawn forth." Dumping a load of knowledge on somebody is less educational than getting them to realize and work things out.

Getting Things Done:

- scan a queue once and process what you can, catagorize other items
- work each pile
- don't keep mental lists, they will distract you -- write your lists
- join the "inbox 0" crowd

- survey: scan the ToC and chapter summaries for an overview
- question: note any questions you have
- read: read in its entirety
- recite: summarize, take notes, and put things in your own words
- review: reread, expand notes, and discuss with colleagues

SQ3R can help you use books more effectively

Do Mind Maps, but do them by hand instead of using any software packages. Doing them by hand will use R-Mode, which can be more effective.

Affinity Grouping - have your team make notes on post-its and group them on a whiteboard. Use markers to show relationships.

Learn by Teaching - try to explain to others what you do, in terms they can understand

Gain Experience:
(tennis example) Place a chair in a tennis court. Hit balls from the other sid, but don't try to hit the chair. Instead, just hit the ball and verbalize where it goes in relation to the chair. This sets up a feedback loop. See "The Inner Game of Tennis" and other "The Inner Game of" books.

Prevent brain lock-up. Give your brain permission to fail. Your brain can lock-up when you arein a panic. Try to minimize deadline pressures.

Beware of e-mail apnea. Breathe!


Sit alert, with a straight back. Notice and release tension. Focus attention on just 1 thing at a time.

Managing the information torrent:

Keep a personal wiki. (eclipse has a wiki package)

Look into Sense Tuning

Checking e-mail too often can drop your effictive IQ by 10 points. Smoking a joint only drops it by 4.

E-Mail checking behavior can be driven by "variable intermittant reward" motivation.

More screen real estate can make you more productive by allowing more items to be visible, removing the need to switch applications (ALT-TAB), requiring a more overt context switch.

Use virtual desktops (eg Mac Spaces) to group related tasks, tools and applications. For example: put all the disruptive things such as e-mail and IM on their own desktop.

When interrupted (by a phone call, an office visitor, etc), leave yourself a breadcrumb in order to get bak to the interrupted task more quickly. Otherwise, you may need to re-create state from scratch.

New habits can take 3-4 weeks to gel.

Belief is physical. Belief can make changes to your brain. If you believe that someting is possible, your brain re-wires itself to make it easier to do the task. If you believe something is impossible, your brain will make it so.

Saturday, December 06, 2008


My son Eric neglected to tell me that he had a duet at last night's winter concert (his last one of high school). If I had known, I might have brought something better than my Canon PowerShot A710 to record this. Oh, Eric is the boy on the right. :-)

Wednesday, October 22, 2008

Cluster Node-Locking with Torque and Maui

These are mostly notes to myself so that I can figure out how to do this more quickly next time...

We needed to add some nodes to a Rocks 4.1 cluster where members of a particular lab were to have exclusive use of the nodes for a period of time. So, we had to find a way to allow these folks to submit jobs that would run only on the new nodes and to also prevent anybody else from running on the nodes. We chose a belt-and-suspenders approach using features of both Torque (PBS) and Maui.

Previously, we had a single "default" queue for all users of this cluster. We added a "vision" queue for the users of the new machines so that they would be able to explicitly request that their jobs run on the new hardware. This queue specifies ACLs for the node list as well as the users allowed to submit jobs to the queue. In addition, there is a "neednodes" resource specified that gives Maui a clue as to where any jobs in this queue can be run. Here are the commands we ran to set up the queue:

qmgr -c "create queue vision queue_type=execution"
qmgr -c "set queue vision resources_default.neednodes = vision"
qmgr -c "set queue vision acl_hosts=compute-0-22+compute-0-23+compute-0-24"
qmgr -c "set queue vision acl_host_enable = false"
qmgr -c "set queue vision acl_users=user1"
qmgr -c "set queue vision acl_users+=user2"
qmgr -c "set queue vision acl_users+=user3"
qmgr -c "set queue vision acl_user_enable=true"
qmgr -c "set queue vision enabled = True"
qmgr -c "set queue vision started = True"

The acl_host_enable = false setting causes Torque to use the acl_hosts list as nodes on which jobs should be queued, rather than as nodes that can run the qsub command. Note that there does not appear to be a way to set multiple acl_users in a single command. While a "list queue" command will show the users in a comma-separated list, if you try to set the ACL that way you get a syntax error. The same can be said for the method of using a plus sign as is done for the hosts ACL.

In addition to setting up the vision queue, a change was needed for the default queue and to the Torque nodes file which, in our case, was /opt/torque/server_priv/nodes but generically would be found at $TORQUE_HOME/server_priv/nodes. We added a "neednodes" resource to the default queue as we did for the vision queue:
qmgr -c "set queue default resources_default.neednodes = general"

For each of the 3 new machines, we appended the word "vision" to the line defining the node like so:
compute-0-22.local np=4 vision

For the rest of the nodes in the file, we added the word "general" like so:
compute-0-0.local np=4 general

After restarting the pbs_server and maui daemons, the end result was that anybody could submit jobs to the default queue and they would run on any node except the 3 nodes dedicated to the vision lab. Only specific users could submit jobs to the vision queue and those jobs would only run on the 3 new machines. This is just what we were looking for. If we ever want to allow everybody to use the new nodes from the default queue, I believe that it should be as simple as appending the word "general" to the "vision" nodes in the server_prive/nodes file.

Technorati Tags: , , , ,