Fun with Upgrading your Technology Business Info Servers

So, we migrated from VMWare 5.0 to 5.1. Plus we had to update our processor minimums due to some Cisco firmware bug. It required to basically split our servers into 2 separate clusters. they weren't truly separate clusters, but in our primary it behaved that way due to the processor settings. Once the physically server had its firmware upgrade, any servers moved to it had to be cold moved...ie shutdown, then moved. Anyhow...we migrated our 80+ servers except our Business info servers. I like getting my paycheck. Its after the 20 of the month. cool, lets schedule with the BIS IT person. I get the green light Tuesday night at 7. She actually likes me to let me start that early, so thank you my friend! She does her back up. it goes long. no problem, 7:30 pm. i get to go. shutdown the vm! no problem. change host. no problem. Boot. good. update vmware tools. good. reboot. still good. shutdown again. Update firmware, yep v9 good. start it back up. Hrm, lets peak at windows update.seriously, 82 updates. text, patching. (forgiveness, not permission). 45 minutes later..a reboot. she's having kittens. I text it is all good, just slow. All comes back. Our exchange starts.

her: I thought you said 30 minutes.
me: you hadn't patched since dec 2011.
her: i know...
me: did anyone notice the longer outage
her: no
me: did it come back and work perfectly?
her: yes.
me: Greatness.
Her: I am glad you did the ugprade, not anyone else
me: (down ego, down ego internally). thank you. you know i wouldn't let anything happen to your server.

Now, i'm begging her assistance to get a flat-file out so i can send it to generate user accounts for a 3rd party. Her system has some data i don't in AD.

Moral of the story: Take care of your peers!

Generators and UPS Fun

Okay, i'm a dork and geek. Learning random shituff from your peers in other fields is fun to me. especially when they let me ask my noob questions and answer them minus the sarcasm. onward to the story.

We've installed a generator at one of our sites. The generator powers both my MDF and IDFs along with the freezers for food service. (for those of you going why freezer too...if that food spoils, we could be out of $500k...yeah, my network gear at $250k is important, but thats a lot of food too. Yeah yeah yeah, down time, loss productivity, etc are costs too, but that 500k versus streaming a youtube khan academy lesson is harder to quantify in k-12). So, the generator is in. our gear is also protected by a UPS which we left in place since generators need 1-30 seconds to come on even with an ATS. Line cleaning of the voltage and all that other wonderful stuff is good too for the day-to-day operation. One day we lose power. The generator kicks on. the freezers go on to the generator. the UPS in my closets light up like an X-mas tree. wtf does that error code mean. other than the UPS isn't happy and we are on battery and the battery say less than 15 minutes. So, post mortem begins. (thank god it was during the summer while 90% of staff was out). Yeah, everyone blamed my UPS. I get the UPS codes based on the error light. Line voltage error. still blaming the UPS. Get our master electrician in. show him the UPS works on standard line power, but not generator. ask him to prove me an idiot or prove me right. I say its still an generator problem. 2 minutes later (yeah, he's pretty swift and smart), he has the problem IDed. I'm not an idiot! (yay?!) Standard building power is 240, generator is 208. I have deer eyes. I ask my questions. explain. well, most UPSes won't accept more than a 10% change in voltage. If it is bigger than 10% change you get the line voltage errors. so, 240-24 is 216 and 208 is less than 216 which is more than 10% delta. UPS stayed pissed off and never accepted the generator as a valid power input. Battery drained and we were offline. a step up transformer is being installed to fix the issue to fix the issue. 

Moral 1 of the story: Make sure your generator and utility voltage match.

Moral 2 of the story: respect your peers and what they know outside your trade! Never know when the electrician or HVAC or alarm guy may help you out too.

Moral 3: School districts store a lot of food in the central freezers. A lot. 

iPad Deployment

I've been dormant. Lets restart. Catharsis, right?

What the heck, lets cover a 3rd rail of Technology, iPads! 

We are in the midst of trying to deploy 2000+ ipads to our teachers. Fun! Most people think Technology support people don't want iPads on their network on in the staff hands because we don't like them. If that were only true. Many of us use the them day to day for both work and play. Its an excellent device. It simply runs. Its intuitive for most folks after 20 minutes. After that, there are apps that do damn near anything you can dream of. Even configure switches if you are network guy, or manage vmware, or anything. So, yes, we like the portability and functionality of the device. However...ask us to deploy them and manage them...ah, that gets us worked up.

Deployment: Apple doesn't have good tools for mass deployment. Period. IPCU and Apple configurator leave a lot to be desired. They can help deploy. However, the degrees of suck depend on your level of deployment customization. Examples: 

1) So, we want to pre-deploy our apps for our staff as part of our base. Great! use configurator. Months later, there is an update and itunes is giving the red 1. Guess which itunes account it asks for when updating pre-deployed apps. Yep, the administrative account used in prep. As any good admin will tell you, yeah, right, over my dead body do you get our admin/deploy account information. Bad things happen when admin or deploy accounts get into the wild.

2) Oh, btw, you used Apple configurator. You went fully managed. (wow, either you have some serious somethings or a lot of time). 2000 devices were configured off of one machine running Configurator. Its a weekend, 30 teachers went to some conference and did something creative to their machines. You have to reimage since the device doesn't work. Guess what! that single apple laptop (you did use a laptop right?) has the only valid restore of those 30 devices. So, you truly have a centralized deployment and management tool! that single laptop! Awesome. now get out there and touch all 30 devices! 

Application roll-out: So, we want to give all our teachers the latest coolest must have app! Awesome! That app costs $3.99. yeah, so. 8k later to Apple (and is our rep driving another new car? -- i kid). we have a pot of money to deploy this app. Magically, we get the application out to our staff using our favorite MDM (we've tried 3, they are all decent). Teacher x leaves the district. Oh, btw, we are out 4 bucks for that one app. We have 20 apps with an average cost of $2.50 so we are out $50.00 (*2000 devices, is our rep driving another new vehicle?)? Why right? can't we just recover the app back into our pool? Ah, not if you decided you wanted to let the teachers customize the device and use their own itunes account. Yep, that $50 worth apps just went to that personal itunes account of the teacher. Hrm, giving away tax-payer money. Not good. So, the moral of the story, is recognize if you allow individual itunes account and push out apps, get all parties on board (finance, hr, technology) and recognize the cost and loss of apps due to employment changes of people. The cost creep can get expensive and have some legal ramifications.

Bonjour: Ok, as a long time network person who cut his teeth on apple devices and networks, gawd, it looks like the revived a portion of the team who wrote of appletalk. non-routable. Bonjour is crap crap crap, not a good enterprise protocol. So, if you have any sort of industry standard wireless network (802.11n/a/g), you probably have a centralized controller. Most like your LAN networks aren't the same as your wireless networks. There's a high probability, these two routers may be multiple hops away from each other. So, you want to remote control that apple desktop using a bonjour enabled on your ipad . yeah, i know it works great at home right? but not at work. my network people are a-holes and don't know what they are doing. I can't speak to that :), but they may not be entirely at fault. In the enterprise (yeah, your 20 school district is considered an enterprise), those router "hops" stop bonjour and that remote control functionality. I know a lot of the bigger vendors are coming up with solutions to resolve this issue, but some of those are quite buggy. In addition, 802.11ac will force resolutions to come and marry up local lans and wlans since tunneling 1G across a WAN will be a huge bandwidth strain. We aren't there yet. (plus, my wifi infrastructure sales person needs a new vehicle too).

Anyhow, there are more items, and each task dreamed up seems to introduce another set of obstacles. We need to get all parties to recognize the uniqueness of the device compared to where the world was 4-5 years ago. That windows 7 box isn't an ipad. The tools aren't the same. The management capabilities are less on the ipad. The deployment capabilities are less. However, the customer can use the ipad quickly and most like the unit. Support and deployment and manageability is -different-. Don't ask us to provide the same customer service experience. It's all different. As the tools of our trade change, so do the expectations.