Wednesday, March 18, 2009
Google Voice: First Impressions
After waiting for what felt like forever, last night I finally got my invitation to upgrade from GrandCentral to Google Voice. Following are some of my first impressions....
Custom ring-back tones are gone - my British friends used to get something that sounded like home.
"Call Presentation" is now settable by group and address book entry - finally, I no longer need to press "1" to accept a call from members of my family. When I pick up the call is immediately connected. This is different form the GrandCentral call screening options. There, I could either screen everybody, screen blocked number, or screen nobody. This finer-grained control is something I (and others, I suspect) have been waiting for. There does seem to be a downside, though. If I have call presentation off (aka connect on answer), there's a chance that my home answering machine, my work voicemail, or my mobile voicemail will answer even if I don't, and then the advantage of Google Voice's voicemail is lost. I'll need to check up to 3 places if I don't pick up fast enough.
Custom greetings seem to still be available.
Voicemail messages now get transcribed to text. By default the transcribed messages is e-mailed to your GMail account. You can still also get notification of new voicemail via SMS to your cell phone, but now the SMS message has the transcribed message (at least if it's short enough). Nicely, the voicemail notification now comes from my Google Voice number, rather than some seemingly random SMS number. That allows me (at least on the iPhone) to call that number and then pick up the message, in case the entire thing didn't get sent in the SMS message or if the transcription looks strange. I've left myself a few simple voicemail messages that were not transcribed all too well.
Text messages are interesting... If somebody sends a SMS message to my Google Voice number, it shows up on my phone as coming from some number in the 406 area code. If I reply to that number, they get my reply. If somebody else sends a SMS message to that number, it is not delivered to the sender of the original message. Clearly there is some sort of mapping such that messages from me to that number get re-routed back to the originator. This mapping appears to last at least 2.5 hours - maybe it's permanent.
In addition, if I get a text message from somebody in my Google address book, their name shows up as a prefix in every message I get from them. If they aren't in my address book, then their originating phone number shows up there instead.
That's it for now. Lunch is over and I really should get back to doing what I get paid for. :-)
Custom ring-back tones are gone - my British friends used to get something that sounded like home.
"Call Presentation" is now settable by group and address book entry - finally, I no longer need to press "1" to accept a call from members of my family. When I pick up the call is immediately connected. This is different form the GrandCentral call screening options. There, I could either screen everybody, screen blocked number, or screen nobody. This finer-grained control is something I (and others, I suspect) have been waiting for. There does seem to be a downside, though. If I have call presentation off (aka connect on answer), there's a chance that my home answering machine, my work voicemail, or my mobile voicemail will answer even if I don't, and then the advantage of Google Voice's voicemail is lost. I'll need to check up to 3 places if I don't pick up fast enough.
Custom greetings seem to still be available.
Voicemail messages now get transcribed to text. By default the transcribed messages is e-mailed to your GMail account. You can still also get notification of new voicemail via SMS to your cell phone, but now the SMS message has the transcribed message (at least if it's short enough). Nicely, the voicemail notification now comes from my Google Voice number, rather than some seemingly random SMS number. That allows me (at least on the iPhone) to call that number and then pick up the message, in case the entire thing didn't get sent in the SMS message or if the transcription looks strange. I've left myself a few simple voicemail messages that were not transcribed all too well.
Text messages are interesting... If somebody sends a SMS message to my Google Voice number, it shows up on my phone as coming from some number in the 406 area code. If I reply to that number, they get my reply. If somebody else sends a SMS message to that number, it is not delivered to the sender of the original message. Clearly there is some sort of mapping such that messages from me to that number get re-routed back to the originator. This mapping appears to last at least 2.5 hours - maybe it's permanent.
In addition, if I get a text message from somebody in my Google address book, their name shows up as a prefix in every message I get from them. If they aren't in my address book, then their originating phone number shows up there instead.
That's it for now. Lunch is over and I really should get back to doing what I get paid for. :-)
Technorati Tags:
Tuesday, March 10, 2009
ETech Notes from "Refactor Your Wetware"
My morning ETech 09 tutorial was "Refactor Your Wetware," presented by Andy Hunt, the author of the book "Pragmatic Thinking & Learning." Below are some notes I took during the tutorial. Perhaps they will be useful to others.
----
There is a big difference between typing a note and hand-writing because of the way the brain processes during both activities.
----
The Dreyfus Model:
If you force experts to follow the rules laid out for novices, you can degrade their performance signifcantly (up to 100%?).
Experts are more likely to see themselves as part of the system. Novices feel that they are outside the system.
----
There is a cultural bias against intuition, which sprins from deeply ingrained knowledge. Experts operate on intuition.
----
The brain can be imagined as 2 different types of "CPUs" with shared access to the memory, where only 1 CPU can access the memory at a time. CPU #1 is more "von Neumann" and linear and slower in operation. CPU #2 is more like a DSP, non-linear and fast. CPU #1 is sometimes referenced as the left brain, and CPU #2 is sometimes referenced as the right brain. We'll use L-Mode (for linear mode) and R-Mode (for rich mode).
----
N. Negraponte: If you want to learn about a frog, don't disect one, build one.
----
Pretty (or aesthetically pleasing) things are actually easier to use.
----
When you're typing notes, L-Mode gets preference and shuts down R-Mode. You're forcing symbolic processing.
----
Dream state imagery is more R-Mode, which is why a dream evaporates as you you try to explain it, which utilizes L-Mode. You can't read signs in a dream because it would require L-Mode.
----
Math prodigies have better coordinated L-Mode and R-Mode processing, rather than having to switch back and forth like the rest of us.
----
Check into Lozanov Séances from the 1970s. He would immerse students in a rich R-Mode experience related to a given task. They would perform better than students who did not have the R-Mode experience.
Leading with an R-Mode experience before an L-Mode "lecture" gives a context for the L-Mode staff to better stick to.
----
Investigate Image Streaming and Morning Pages.
----
A "whack on the side of the head" can help to clarify thinking. Looking at a problem differently, such as "in reverse" can be helpful.
----
If you don't keep track of great ideas that you have, you'll stop noticing that you have them, and then you'll stop having them. Carry a notebook of some type so you can write down ideas as you have them.
----
"Education" comes from "edu" and "care" which translates to "drawn forth." Dumping a load of knowledge on somebody is less educational than getting them to realize and work things out.
----
Getting Things Done:
- scan a queue once and process what you can, catagorize other items
- work each pile
- don't keep mental lists, they will distract you -- write your lists
- join the "inbox 0" crowd
----
SQ3R:
- survey: scan the ToC and chapter summaries for an overview
- question: note any questions you have
- read: read in its entirety
- recite: summarize, take notes, and put things in your own words
- review: reread, expand notes, and discuss with colleagues
SQ3R can help you use books more effectively
----
Do Mind Maps, but do them by hand instead of using any software packages. Doing them by hand will use R-Mode, which can be more effective.
----
Affinity Grouping - have your team make notes on post-its and group them on a whiteboard. Use markers to show relationships.
----
Learn by Teaching - try to explain to others what you do, in terms they can understand
----
Gain Experience:
(tennis example) Place a chair in a tennis court. Hit balls from the other sid, but don't try to hit the chair. Instead, just hit the ball and verbalize where it goes in relation to the chair. This sets up a feedback loop. See "The Inner Game of Tennis" and other "The Inner Game of" books.
----
Prevent brain lock-up. Give your brain permission to fail. Your brain can lock-up when you arein a panic. Try to minimize deadline pressures.
----
Beware of e-mail apnea. Breathe!
----
Breathing:
Sit alert, with a straight back. Notice and release tension. Focus attention on just 1 thing at a time.
----
Managing the information torrent:
Keep a personal wiki. (eclipse has a wiki package)
----
Look into Sense Tuning
----
Checking e-mail too often can drop your effictive IQ by 10 points. Smoking a joint only drops it by 4.
E-Mail checking behavior can be driven by "variable intermittant reward" motivation.
----
More screen real estate can make you more productive by allowing more items to be visible, removing the need to switch applications (ALT-TAB), requiring a more overt context switch.
Use virtual desktops (eg Mac Spaces) to group related tasks, tools and applications. For example: put all the disruptive things such as e-mail and IM on their own desktop.
----
When interrupted (by a phone call, an office visitor, etc), leave yourself a breadcrumb in order to get bak to the interrupted task more quickly. Otherwise, you may need to re-create state from scratch.
----
New habits can take 3-4 weeks to gel.
----
Belief is physical. Belief can make changes to your brain. If you believe that someting is possible, your brain re-wires itself to make it easier to do the task. If you believe something is impossible, your brain will make it so.
----
There is a big difference between typing a note and hand-writing because of the way the brain processes during both activities.
----
The Dreyfus Model:
If you force experts to follow the rules laid out for novices, you can degrade their performance signifcantly (up to 100%?).
Experts are more likely to see themselves as part of the system. Novices feel that they are outside the system.
----
There is a cultural bias against intuition, which sprins from deeply ingrained knowledge. Experts operate on intuition.
----
The brain can be imagined as 2 different types of "CPUs" with shared access to the memory, where only 1 CPU can access the memory at a time. CPU #1 is more "von Neumann" and linear and slower in operation. CPU #2 is more like a DSP, non-linear and fast. CPU #1 is sometimes referenced as the left brain, and CPU #2 is sometimes referenced as the right brain. We'll use L-Mode (for linear mode) and R-Mode (for rich mode).
----
N. Negraponte: If you want to learn about a frog, don't disect one, build one.
----
Pretty (or aesthetically pleasing) things are actually easier to use.
----
When you're typing notes, L-Mode gets preference and shuts down R-Mode. You're forcing symbolic processing.
----
Dream state imagery is more R-Mode, which is why a dream evaporates as you you try to explain it, which utilizes L-Mode. You can't read signs in a dream because it would require L-Mode.
----
Math prodigies have better coordinated L-Mode and R-Mode processing, rather than having to switch back and forth like the rest of us.
----
Check into Lozanov Séances from the 1970s. He would immerse students in a rich R-Mode experience related to a given task. They would perform better than students who did not have the R-Mode experience.
Leading with an R-Mode experience before an L-Mode "lecture" gives a context for the L-Mode staff to better stick to.
----
Investigate Image Streaming and Morning Pages.
----
A "whack on the side of the head" can help to clarify thinking. Looking at a problem differently, such as "in reverse" can be helpful.
----
If you don't keep track of great ideas that you have, you'll stop noticing that you have them, and then you'll stop having them. Carry a notebook of some type so you can write down ideas as you have them.
----
"Education" comes from "edu" and "care" which translates to "drawn forth." Dumping a load of knowledge on somebody is less educational than getting them to realize and work things out.
----
Getting Things Done:
- scan a queue once and process what you can, catagorize other items
- work each pile
- don't keep mental lists, they will distract you -- write your lists
- join the "inbox 0" crowd
----
SQ3R:
- survey: scan the ToC and chapter summaries for an overview
- question: note any questions you have
- read: read in its entirety
- recite: summarize, take notes, and put things in your own words
- review: reread, expand notes, and discuss with colleagues
SQ3R can help you use books more effectively
----
Do Mind Maps, but do them by hand instead of using any software packages. Doing them by hand will use R-Mode, which can be more effective.
----
Affinity Grouping - have your team make notes on post-its and group them on a whiteboard. Use markers to show relationships.
----
Learn by Teaching - try to explain to others what you do, in terms they can understand
----
Gain Experience:
(tennis example) Place a chair in a tennis court. Hit balls from the other sid, but don't try to hit the chair. Instead, just hit the ball and verbalize where it goes in relation to the chair. This sets up a feedback loop. See "The Inner Game of Tennis" and other "The Inner Game of" books.
----
Prevent brain lock-up. Give your brain permission to fail. Your brain can lock-up when you arein a panic. Try to minimize deadline pressures.
----
Beware of e-mail apnea. Breathe!
----
Breathing:
Sit alert, with a straight back. Notice and release tension. Focus attention on just 1 thing at a time.
----
Managing the information torrent:
Keep a personal wiki. (eclipse has a wiki package)
----
Look into Sense Tuning
----
Checking e-mail too often can drop your effictive IQ by 10 points. Smoking a joint only drops it by 4.
E-Mail checking behavior can be driven by "variable intermittant reward" motivation.
----
More screen real estate can make you more productive by allowing more items to be visible, removing the need to switch applications (ALT-TAB), requiring a more overt context switch.
Use virtual desktops (eg Mac Spaces) to group related tasks, tools and applications. For example: put all the disruptive things such as e-mail and IM on their own desktop.
----
When interrupted (by a phone call, an office visitor, etc), leave yourself a breadcrumb in order to get bak to the interrupted task more quickly. Otherwise, you may need to re-create state from scratch.
----
New habits can take 3-4 weeks to gel.
----
Belief is physical. Belief can make changes to your brain. If you believe that someting is possible, your brain re-wires itself to make it easier to do the task. If you believe something is impossible, your brain will make it so.
Technorati Tags: , ETech09
Saturday, December 06, 2008
Shosholoza
My son Eric neglected to tell me that he had a duet at last night's winter concert (his last one of high school). If I had known, I might have brought something better than my Canon PowerShot A710 to record this. Oh, Eric is the boy on the right. :-)
Wednesday, October 22, 2008
Cluster Node-Locking with Torque and Maui
These are mostly notes to myself so that I can figure out how to do this more quickly next time...
We needed to add some nodes to a Rocks 4.1 cluster where members of a particular lab were to have exclusive use of the nodes for a period of time. So, we had to find a way to allow these folks to submit jobs that would run only on the new nodes and to also prevent anybody else from running on the nodes. We chose a belt-and-suspenders approach using features of both Torque (PBS) and Maui.
Previously, we had a single "default" queue for all users of this cluster. We added a "vision" queue for the users of the new machines so that they would be able to explicitly request that their jobs run on the new hardware. This queue specifies ACLs for the node list as well as the users allowed to submit jobs to the queue. In addition, there is a "neednodes" resource specified that gives Maui a clue as to where any jobs in this queue can be run. Here are the commands we ran to set up the queue:
The acl_host_enable = false setting causes Torque to use the acl_hosts list as nodes on which jobs should be queued, rather than as nodes that can run the qsub command. Note that there does not appear to be a way to set multiple acl_users in a single command. While a "list queue" command will show the users in a comma-separated list, if you try to set the ACL that way you get a syntax error. The same can be said for the method of using a plus sign as is done for the hosts ACL.
In addition to setting up the vision queue, a change was needed for the default queue and to the Torque nodes file which, in our case, was /opt/torque/server_priv/nodes but generically would be found at $TORQUE_HOME/server_priv/nodes. We added a "neednodes" resource to the default queue as we did for the vision queue:
For each of the 3 new machines, we appended the word "vision" to the line defining the node like so:
For the rest of the nodes in the file, we added the word "general" like so:
After restarting the pbs_server and maui daemons, the end result was that anybody could submit jobs to the default queue and they would run on any node except the 3 nodes dedicated to the vision lab. Only specific users could submit jobs to the vision queue and those jobs would only run on the 3 new machines. This is just what we were looking for. If we ever want to allow everybody to use the new nodes from the default queue, I believe that it should be as simple as appending the word "general" to the "vision" nodes in the server_prive/nodes file.
We needed to add some nodes to a Rocks 4.1 cluster where members of a particular lab were to have exclusive use of the nodes for a period of time. So, we had to find a way to allow these folks to submit jobs that would run only on the new nodes and to also prevent anybody else from running on the nodes. We chose a belt-and-suspenders approach using features of both Torque (PBS) and Maui.
Previously, we had a single "default" queue for all users of this cluster. We added a "vision" queue for the users of the new machines so that they would be able to explicitly request that their jobs run on the new hardware. This queue specifies ACLs for the node list as well as the users allowed to submit jobs to the queue. In addition, there is a "neednodes" resource specified that gives Maui a clue as to where any jobs in this queue can be run. Here are the commands we ran to set up the queue:
qmgr -c "create queue vision queue_type=execution"
qmgr -c "set queue vision resources_default.neednodes = vision"
qmgr -c "set queue vision acl_hosts=compute-0-22+compute-0-23+compute-0-24"
qmgr -c "set queue vision acl_host_enable = false"
qmgr -c "set queue vision acl_users=user1"
qmgr -c "set queue vision acl_users+=user2"
qmgr -c "set queue vision acl_users+=user3"
qmgr -c "set queue vision acl_user_enable=true"
qmgr -c "set queue vision enabled = True"
qmgr -c "set queue vision started = True"
The acl_host_enable = false setting causes Torque to use the acl_hosts list as nodes on which jobs should be queued, rather than as nodes that can run the qsub command. Note that there does not appear to be a way to set multiple acl_users in a single command. While a "list queue" command will show the users in a comma-separated list, if you try to set the ACL that way you get a syntax error. The same can be said for the method of using a plus sign as is done for the hosts ACL.
In addition to setting up the vision queue, a change was needed for the default queue and to the Torque nodes file which, in our case, was /opt/torque/server_priv/nodes but generically would be found at $TORQUE_HOME/server_priv/nodes. We added a "neednodes" resource to the default queue as we did for the vision queue:
qmgr -c "set queue default resources_default.neednodes = general"
For each of the 3 new machines, we appended the word "vision" to the line defining the node like so:
compute-0-22.local np=4 vision
For the rest of the nodes in the file, we added the word "general" like so:
compute-0-0.local np=4 general
After restarting the pbs_server and maui daemons, the end result was that anybody could submit jobs to the default queue and they would run on any node except the 3 nodes dedicated to the vision lab. Only specific users could submit jobs to the vision queue and those jobs would only run on the 3 new machines. This is just what we were looking for. If we ever want to allow everybody to use the new nodes from the default queue, I believe that it should be as simple as appending the word "general" to the "vision" nodes in the server_prive/nodes file.
Technorati Tags:
Monday, September 22, 2008
Space Shuttles and Rainbows
A co-worker of mine pointed me to this picture that he saw on digg. That prompted me to go off and find the whole series, which I tagged on del.icio.us.
Technorati Tags:
Wednesday, August 20, 2008
It's the Little Things
The other day, I discovered yet another little useful thing about the iPhone 3G. I don't know if the original iPhone did this, but the new one remembers volume settings in the iPod application. I normally use a cassette adapter to listen to podcasts when I drive into work. The first time I plugged in the iPhone earbuds/mic, I lowered the volume from the "full" I needed for the cassette adapter. A little later, I tried the built-in speaker and had to adjust the volume again.
The next time I went to listen in the car, I plugged in the adapter and the volume jumped all the way to full on its own. I unplugged the adapter and the volume dropped back to the value for the built-in speaker. I then tried the earbuds and the volume slider moved again. I know this may seem like a little thing, but it's kinda nice that I don't have to muck with the volume every time I want to listen a different way....
The next time I went to listen in the car, I plugged in the adapter and the volume jumped all the way to full on its own. I unplugged the adapter and the volume dropped back to the value for the built-in speaker. I then tried the earbuds and the volume slider moved again. I know this may seem like a little thing, but it's kinda nice that I don't have to muck with the volume every time I want to listen a different way....
Technorati Tags:
Drinking the Kool-Aid
It looks like the iPhone Kool-Aid has been flowing in my department, and it seems that everybody but my manager has imbibed. At first, it was just Steve who really wanted one. Then, I think he slipped something into my coffee one morning and I decided to give it a go as well. One-by-one, 4 more guys in the group decide to take the plunge (cue the big red pitcher saying "Oh yeah!"). Even the guy who really wanted the Blackberry Bold decided to go for it. So far, Shazam and SimplifyMedia are the hit applications....
Technorati Tags:
Wednesday, June 11, 2008
How Not to Upgrade Your Computer Room UPS
The Computer Science Department at Princeton University needed to replace its computer room UPS. The old unit was a PowerWare 9330-40, a 208V 40 kVA UPS, installed in July 2005. When this box was first installed, the load was around 60%. Over time this has crept up to over 90%, causing us to believe that an upgrade was in order.
The new UPS is a PowerWare 9390-80, a 480V 80 kVA UPS that fills two 6 foot tall racks and weighs in at something near 4000 lbs -- we needed to add additional supports under the raised floor to handle the increased weight load. The UPS and its wall-mounted bypass switch cabinet arrived a few weeks ago, were put into the computer room, and were wired together in preparation for the big day.
That day was Tuesday, June 10, when we had scheduled a full computer room downtime from 6 AM until 2 PM. We figured that this should be plenty of time to unwire the old 208V UPS, re-wire our Liebert power distribution center back to factory specs (we had to change things around in order to accommodate a 208V UPS in a room with a 480V power feed), and get the new UPS and the computers and network gear up and running. Well as they say, "Two out of three ain't bad."
We had everything in the room shut down and powered off somewhere between 6:00 and 6:30, allowing the electrician to get started re-wiring at 7 AM. The PowerWare technician arrived around 9 and started his prep and check work on the new UPS. He needed to connect all of the batteries together and test voltages, check over all of the pre-wiring work, as well as a bunch of other stuff required before the unit could be turned on.
Most of the morning, everything seemed to be going along fine. The electrician finishes ripping out the old UPS wiring hacks and gets the new UPS and bypass switched wired in the way they should be. The UPS technician does his initial power-on and starts testing the system. Some time was lost when the electrician had trouble getting lugs installed on a breaker, but that was only about a 30 minute diversion. Then the first problem hit...
There are three panel boards in the Liebert system for power distribution, each of which has a 225A main breaker. When we went to put some test load on the UPS, one of those main breakers refused to stay on. It didn't trip, the handle just slowly moved from "on" to "off" on its own. It never made that satisfying "click" when you turned it on. Since about half of our load was on that panel board, this was going to be a problem.
The electrician kept working at it, and eventually the breaker clicked into the on position and stayed there. We're planing on having it replaced the next time we take the room down, but we decided to live with it for now. If the breaker goes, we'll just be replacing it a bit sooner than we planned.
At some point, the PowerWare technician decided that he didn't like the path that some of the signal wires took between the two UPS cabinets and decided they should be re-routed. During that re-wiring process, he discovered that he could no longer get the battery breaker to turn on, rendering the UPS essentially useless. He puzzled over this for quite a while (hours, in fact) and decided that we needed a replacement circuit breaker for the battery cabinet. Since this was a new UPS start-up and not a failure situation in the eyes of the company, the earliest we could get a replacement breaker was Wednesday morning at 8:30 AM. Even getting this required quite a bit of the technician's time on the phone.
We had underestimated a little on how long the electrician would need to complete his tasks, but if that and the 225A breaker were the only issues, we would have only missed our downtime window by about an hour. However, the problems with the new UPS stretched our downtime until 6 PM, four additional hours (for 12 hours of downtime). At this point, we had to decide whether or not to bring up the computer room without a UPS. As there were thunderstorms predicted overnight, we decided to leave everything off. Considering that the lights blinked out later that night, I believe we made the correct decision.
Fast forward to Wednesday at 8:30 AM (26.5 hours of downtime). Two replacement battery cabinet circuit breakers were delivered by FedEx. The PowerWare technician arrived around 9:15 and got to work. He was not able to find the hex wrench he needed to remove the wires from the breaker, so one of us went to track one down. By the time we found a wrench, the technician had found his and got to work. The replacement breaker was installed and when the technician tried to turn it on, it didn't work either. By 11 AM (29 hours of downtime), the cause of the problem was still unknown. By 1 PM (31 hours of downtime) , after examining his schematics (which didn't appear to exactly match the UPS we had) and making a number of phone calls, the technician finally figured out that the breaker required 48V available in order to be switched on and that the fuse protecting that 48V circuit had blown. This is an $11 part that had taken, by this time, over 8 hours of effort to identify as the source of the problem. The total cost of this UPS is around $80,000.
Of course, the technician didn't have one of these fuses in his van. Two of my co-workers called around to a number of electrical houses and eventually found some. By around 3 PM (33 hours of downtime), the technician had replaced the fuse and was running his final checks. We decided to test the EPO (Emergency Power Off) switch before we put any load on the system. It did absolutely nothing. The wire from the EPO switches had never been connected to the UPS control input. Once again the UPS was opened up and the wire connected. We test again and the UPS only went into bypass, but didn't shut down. This is not exactly what we would have expected a "power off" button to do. The technician did some studying of the documentation, changed some settings and we tried again, with the same result. It turns out that we need to hold EPO switch for at least 3 seconds for the UPS to actually shut down. None of us like this, but I guess we'll have to live with it for now.
We got our operational training so that we could properly take the UPS in and out of maintenance bypass, and we started to bring up the room. It was now around 5 PM (35 hours of downtime). With everything down for so long, we decide that caution was the order of the day when we brought the world back up. What normally takes 45 minutes to an hour took 90 minutes. By 6:30 PM (36.5 hours after our adventure started), everything was back up and we were starting to receive all of the EMail that had been queued up for us around the world. All of us in the Computer Science Department are hoping that we never have to do this again.
The new UPS is a PowerWare 9390-80, a 480V 80 kVA UPS that fills two 6 foot tall racks and weighs in at something near 4000 lbs -- we needed to add additional supports under the raised floor to handle the increased weight load. The UPS and its wall-mounted bypass switch cabinet arrived a few weeks ago, were put into the computer room, and were wired together in preparation for the big day.
That day was Tuesday, June 10, when we had scheduled a full computer room downtime from 6 AM until 2 PM. We figured that this should be plenty of time to unwire the old 208V UPS, re-wire our Liebert power distribution center back to factory specs (we had to change things around in order to accommodate a 208V UPS in a room with a 480V power feed), and get the new UPS and the computers and network gear up and running. Well as they say, "Two out of three ain't bad."
We had everything in the room shut down and powered off somewhere between 6:00 and 6:30, allowing the electrician to get started re-wiring at 7 AM. The PowerWare technician arrived around 9 and started his prep and check work on the new UPS. He needed to connect all of the batteries together and test voltages, check over all of the pre-wiring work, as well as a bunch of other stuff required before the unit could be turned on.
Most of the morning, everything seemed to be going along fine. The electrician finishes ripping out the old UPS wiring hacks and gets the new UPS and bypass switched wired in the way they should be. The UPS technician does his initial power-on and starts testing the system. Some time was lost when the electrician had trouble getting lugs installed on a breaker, but that was only about a 30 minute diversion. Then the first problem hit...
There are three panel boards in the Liebert system for power distribution, each of which has a 225A main breaker. When we went to put some test load on the UPS, one of those main breakers refused to stay on. It didn't trip, the handle just slowly moved from "on" to "off" on its own. It never made that satisfying "click" when you turned it on. Since about half of our load was on that panel board, this was going to be a problem.
The electrician kept working at it, and eventually the breaker clicked into the on position and stayed there. We're planing on having it replaced the next time we take the room down, but we decided to live with it for now. If the breaker goes, we'll just be replacing it a bit sooner than we planned.
At some point, the PowerWare technician decided that he didn't like the path that some of the signal wires took between the two UPS cabinets and decided they should be re-routed. During that re-wiring process, he discovered that he could no longer get the battery breaker to turn on, rendering the UPS essentially useless. He puzzled over this for quite a while (hours, in fact) and decided that we needed a replacement circuit breaker for the battery cabinet. Since this was a new UPS start-up and not a failure situation in the eyes of the company, the earliest we could get a replacement breaker was Wednesday morning at 8:30 AM. Even getting this required quite a bit of the technician's time on the phone.
We had underestimated a little on how long the electrician would need to complete his tasks, but if that and the 225A breaker were the only issues, we would have only missed our downtime window by about an hour. However, the problems with the new UPS stretched our downtime until 6 PM, four additional hours (for 12 hours of downtime). At this point, we had to decide whether or not to bring up the computer room without a UPS. As there were thunderstorms predicted overnight, we decided to leave everything off. Considering that the lights blinked out later that night, I believe we made the correct decision.
Fast forward to Wednesday at 8:30 AM (26.5 hours of downtime). Two replacement battery cabinet circuit breakers were delivered by FedEx. The PowerWare technician arrived around 9:15 and got to work. He was not able to find the hex wrench he needed to remove the wires from the breaker, so one of us went to track one down. By the time we found a wrench, the technician had found his and got to work. The replacement breaker was installed and when the technician tried to turn it on, it didn't work either. By 11 AM (29 hours of downtime), the cause of the problem was still unknown. By 1 PM (31 hours of downtime) , after examining his schematics (which didn't appear to exactly match the UPS we had) and making a number of phone calls, the technician finally figured out that the breaker required 48V available in order to be switched on and that the fuse protecting that 48V circuit had blown. This is an $11 part that had taken, by this time, over 8 hours of effort to identify as the source of the problem. The total cost of this UPS is around $80,000.
Of course, the technician didn't have one of these fuses in his van. Two of my co-workers called around to a number of electrical houses and eventually found some. By around 3 PM (33 hours of downtime), the technician had replaced the fuse and was running his final checks. We decided to test the EPO (Emergency Power Off) switch before we put any load on the system. It did absolutely nothing. The wire from the EPO switches had never been connected to the UPS control input. Once again the UPS was opened up and the wire connected. We test again and the UPS only went into bypass, but didn't shut down. This is not exactly what we would have expected a "power off" button to do. The technician did some studying of the documentation, changed some settings and we tried again, with the same result. It turns out that we need to hold EPO switch for at least 3 seconds for the UPS to actually shut down. None of us like this, but I guess we'll have to live with it for now.
We got our operational training so that we could properly take the UPS in and out of maintenance bypass, and we started to bring up the room. It was now around 5 PM (35 hours of downtime). With everything down for so long, we decide that caution was the order of the day when we brought the world back up. What normally takes 45 minutes to an hour took 90 minutes. By 6:30 PM (36.5 hours after our adventure started), everything was back up and we were starting to receive all of the EMail that had been queued up for us around the world. All of us in the Computer Science Department are hoping that we never have to do this again.
Technorati Tags:
