Fun with Upgrading your Technology Business Info Servers

So, we migrated from VMWare 5.0 to 5.1. Plus we had to update our processor minimums due to some Cisco firmware bug. It required to basically split our servers into 2 separate clusters. they weren't truly separate clusters, but in our primary it behaved that way due to the processor settings. Once the physically server had its firmware upgrade, any servers moved to it had to be cold moved...ie shutdown, then moved. Anyhow...we migrated our 80+ servers except our Business info servers. I like getting my paycheck. Its after the 20 of the month. cool, lets schedule with the BIS IT person. I get the green light Tuesday night at 7. She actually likes me to let me start that early, so thank you my friend! She does her back up. it goes long. no problem, 7:30 pm. i get to go. shutdown the vm! no problem. change host. no problem. Boot. good. update vmware tools. good. reboot. still good. shutdown again. Update firmware, yep v9 good. start it back up. Hrm, lets peak at windows update.seriously, 82 updates. text, patching. (forgiveness, not permission). 45 minutes later..a reboot. she's having kittens. I text it is all good, just slow. All comes back. Our exchange starts.

her: I thought you said 30 minutes.
me: you hadn't patched since dec 2011.
her: i know...
me: did anyone notice the longer outage
her: no
me: did it come back and work perfectly?
her: yes.
me: Greatness.
Her: I am glad you did the ugprade, not anyone else
me: (down ego, down ego internally). thank you. you know i wouldn't let anything happen to your server.

Now, i'm begging her assistance to get a flat-file out so i can send it to generate user accounts for a 3rd party. Her system has some data i don't in AD.

Moral of the story: Take care of your peers!

Generators and UPS Fun

Okay, i'm a dork and geek. Learning random shituff from your peers in other fields is fun to me. especially when they let me ask my noob questions and answer them minus the sarcasm. onward to the story.

We've installed a generator at one of our sites. The generator powers both my MDF and IDFs along with the freezers for food service. (for those of you going why freezer too...if that food spoils, we could be out of $500k...yeah, my network gear at $250k is important, but thats a lot of food too. Yeah yeah yeah, down time, loss productivity, etc are costs too, but that 500k versus streaming a youtube khan academy lesson is harder to quantify in k-12). So, the generator is in. our gear is also protected by a UPS which we left in place since generators need 1-30 seconds to come on even with an ATS. Line cleaning of the voltage and all that other wonderful stuff is good too for the day-to-day operation. One day we lose power. The generator kicks on. the freezers go on to the generator. the UPS in my closets light up like an X-mas tree. wtf does that error code mean. other than the UPS isn't happy and we are on battery and the battery say less than 15 minutes. So, post mortem begins. (thank god it was during the summer while 90% of staff was out). Yeah, everyone blamed my UPS. I get the UPS codes based on the error light. Line voltage error. still blaming the UPS. Get our master electrician in. show him the UPS works on standard line power, but not generator. ask him to prove me an idiot or prove me right. I say its still an generator problem. 2 minutes later (yeah, he's pretty swift and smart), he has the problem IDed. I'm not an idiot! (yay?!) Standard building power is 240, generator is 208. I have deer eyes. I ask my questions. explain. well, most UPSes won't accept more than a 10% change in voltage. If it is bigger than 10% change you get the line voltage errors. so, 240-24 is 216 and 208 is less than 216 which is more than 10% delta. UPS stayed pissed off and never accepted the generator as a valid power input. Battery drained and we were offline. a step up transformer is being installed to fix the issue to fix the issue. 

Moral 1 of the story: Make sure your generator and utility voltage match.

Moral 2 of the story: respect your peers and what they know outside your trade! Never know when the electrician or HVAC or alarm guy may help you out too.

Moral 3: School districts store a lot of food in the central freezers. A lot. 

iPad Deployment

I've been dormant. Lets restart. Catharsis, right?

What the heck, lets cover a 3rd rail of Technology, iPads! 

We are in the midst of trying to deploy 2000+ ipads to our teachers. Fun! Most people think Technology support people don't want iPads on their network on in the staff hands because we don't like them. If that were only true. Many of us use the them day to day for both work and play. Its an excellent device. It simply runs. Its intuitive for most folks after 20 minutes. After that, there are apps that do damn near anything you can dream of. Even configure switches if you are network guy, or manage vmware, or anything. So, yes, we like the portability and functionality of the device. However...ask us to deploy them and manage them...ah, that gets us worked up.

Deployment: Apple doesn't have good tools for mass deployment. Period. IPCU and Apple configurator leave a lot to be desired. They can help deploy. However, the degrees of suck depend on your level of deployment customization. Examples: 

1) So, we want to pre-deploy our apps for our staff as part of our base. Great! use configurator. Months later, there is an update and itunes is giving the red 1. Guess which itunes account it asks for when updating pre-deployed apps. Yep, the administrative account used in prep. As any good admin will tell you, yeah, right, over my dead body do you get our admin/deploy account information. Bad things happen when admin or deploy accounts get into the wild.

2) Oh, btw, you used Apple configurator. You went fully managed. (wow, either you have some serious somethings or a lot of time). 2000 devices were configured off of one machine running Configurator. Its a weekend, 30 teachers went to some conference and did something creative to their machines. You have to reimage since the device doesn't work. Guess what! that single apple laptop (you did use a laptop right?) has the only valid restore of those 30 devices. So, you truly have a centralized deployment and management tool! that single laptop! Awesome. now get out there and touch all 30 devices! 

Application roll-out: So, we want to give all our teachers the latest coolest must have app! Awesome! That app costs $3.99. yeah, so. 8k later to Apple (and is our rep driving another new car? -- i kid). we have a pot of money to deploy this app. Magically, we get the application out to our staff using our favorite MDM (we've tried 3, they are all decent). Teacher x leaves the district. Oh, btw, we are out 4 bucks for that one app. We have 20 apps with an average cost of $2.50 so we are out $50.00 (*2000 devices, is our rep driving another new vehicle?)? Why right? can't we just recover the app back into our pool? Ah, not if you decided you wanted to let the teachers customize the device and use their own itunes account. Yep, that $50 worth apps just went to that personal itunes account of the teacher. Hrm, giving away tax-payer money. Not good. So, the moral of the story, is recognize if you allow individual itunes account and push out apps, get all parties on board (finance, hr, technology) and recognize the cost and loss of apps due to employment changes of people. The cost creep can get expensive and have some legal ramifications.

Bonjour: Ok, as a long time network person who cut his teeth on apple devices and networks, gawd, it looks like the revived a portion of the team who wrote of appletalk. non-routable. Bonjour is crap crap crap, not a good enterprise protocol. So, if you have any sort of industry standard wireless network (802.11n/a/g), you probably have a centralized controller. Most like your LAN networks aren't the same as your wireless networks. There's a high probability, these two routers may be multiple hops away from each other. So, you want to remote control that apple desktop using a bonjour enabled on your ipad . yeah, i know it works great at home right? but not at work. my network people are a-holes and don't know what they are doing. I can't speak to that :), but they may not be entirely at fault. In the enterprise (yeah, your 20 school district is considered an enterprise), those router "hops" stop bonjour and that remote control functionality. I know a lot of the bigger vendors are coming up with solutions to resolve this issue, but some of those are quite buggy. In addition, 802.11ac will force resolutions to come and marry up local lans and wlans since tunneling 1G across a WAN will be a huge bandwidth strain. We aren't there yet. (plus, my wifi infrastructure sales person needs a new vehicle too).

Anyhow, there are more items, and each task dreamed up seems to introduce another set of obstacles. We need to get all parties to recognize the uniqueness of the device compared to where the world was 4-5 years ago. That windows 7 box isn't an ipad. The tools aren't the same. The management capabilities are less on the ipad. The deployment capabilities are less. However, the customer can use the ipad quickly and most like the unit. Support and deployment and manageability is -different-. Don't ask us to provide the same customer service experience. It's all different. As the tools of our trade change, so do the expectations. 

Fun with Certificates

Fun with certificates. Wow. Certificates in an enterprise are a delicate item to undertake, even just end-point certificates such as users, workstations and SCEP/NDES devices.I finally issued my first certificate to an iPad as a -computer- account following a lot of the blogs on the internet. Now it is time to see if that cert can be used for device authentication on our new wireless overlay. Yeah, we could do generic accounts on the device which is the fall back plan. We've found anything we have to rely on the end point user tends to require a certain amount of support costs. I'll fully document everything this weekend if it works and we are to deploy a successful test (10-100 units). 1 is easy. 10 is work. 100 needs automation. Have to get it relatively automated if possible. No comment on the lack of documentation except on microsoft oriented sites. To those who have created the content, I thank you! My basic sites I visited and followed.

• http://www.kurtdillard.com/StudyGuides/70-640/6.html
• http://marckean.wordpress.com/2010/07/28/build-an-offline-root-ca-with-a-subordinate-ca/
• http://social.technet.microsoft.com/wiki/contents/articles/2900.offline-root-certification-authority-ca.aspx 
• bounty of sub-links to read and understand
• http://security-24-7.com/windows-2008-r2-certification-authority-installation-guide/
• http://blogs.technet.com/b/pki/archive/2012/02/27/ndes-and-ipads.aspx

Notice the lack of apple content. there is a reason! Bad Apple! You will need to download the iphone configuration utility too. The issue I had is with the X.500 name. There is hints and direction, but people are stingy on this one. My entry looked like

O=mycompany.com, CN=iPad123456

O is our domain listed in the certificate signing piece. If your CA signs certs for devices in mycompany.com, then put O=mycompany.com.

CN is the devices name you gave it. We use our asset tag system. This doesn't have to match much along the way. Until I fixed those I was getting 0x800094001. The other flavored the request subject name is invalid or too long is addressed by the authors above.

The O should be capitalized. So should the CN. including a comma and space might be necessary. Haven't chased the rabbit that far down the hole yet.

Onto the next fun part. Cisco ISE. Oh boy! 1.1 release software is fun.

Quick update

We've been busy! The summer projects are about to start. We ran school late this year. Short list of items:

• When deploying workstation certificates, check which servers can issue the certificates.Things you learn along the way.
• Having new team members is fun. New blood. New perspective. New questions. Nice thing to have. 
• Having issues with whatsupgold "crashing". Trying to have it read some WMI from windows servers and it just stops scanning. Everything. Not good.
• Good is the VM discovery features in whatsup. Check that discover vm environment!
• Updating, verifying, 3500 edge switches is a pain in the ... well, its a pain. Once verified, whatsconfigured will take over and rule. Voice config and security and a few other configs to update.
• Emergency construction is fun. And smelly. Broken water lines, of all flavors! Have to move our equipment (workstations, network stuff, others) out of the way. 
• Every 3rd party application wants to do upgrades this week such as our bus route software, special ed *2, energy management, food service, and a couple of others. Not a bad thing, but they asked -- Yesterday. Saying no isn't a real option. At least most of these the vendors will webex/remote-in and do the upgrade. So Veeam snapshot before starting in case they -aw !@#$- it. 
• Google Calendar is awesome. Sharing a calendar with vendors/resellers regarding schedules at schools is beneficial. Having to train them how to sign up and use it, but all in all it is good. Better than having to continually export a ics from exchange. 

That has been the last week. And my people are busier than me. Its good to have a team to run $1M mostly on their own.

Nothing Technical

The only downside to summer planning is all that happens is meetings. And once those are done, you have to schedule more meetings. Nature of the beast. The kids are going. The teachers are going. The buildings are about to empty. Facilities, Construction, Training, Summer School, Technology all want a chance at the empty buildings to do tasks, whether it be painting, waxing, moving AC units, continuing ed credits, credit recovery, and whatever we are supposed to do (phones, wireless, ipads, nooks, etc). Time to get on the same page to start. We all know best laid plans, etc, but at least if you start on the same page you can adjust. Here's hoping we get to the same page. Thats why the lack of technical fun recently. And I'm tired of getting errors with ipads trying to register as a device with NDES/SCEP. grrr. Need those 4-6 hours uninterrupted to turn up the debugging to really see what the errors are.

Coordination

Woo! Summer is almost here! Welcome to the beginning of an organized train wreck called summer project time. Well, not quite a train wreck, but does it ever feel like it. Trying to cram all work that was put off during the year cause it was too intrusive, too expensive, too time intensive into 8-10 weeks. I think we need a bigger shoe horn sometimes. Here are the few items we are trying to coordinate.

1. Construction at 6 sites
2. Summer School
3. Summer Cleaning/Waxing Schedule
4. Summer painting schedules
5. Technology projects
6. Start of new school year stuff
7. Vacations
8. Politics 

Ok. It is only 7 items. Each has anywhere between 2-20 sub-items beneath it. For example, who cares about summer cleaning, right? Well, everyone in the district does. The floors gotta -shine- the first day those kids come back. First impressions matter. And a shine with my big ol' hoof print in it isn't going to win me any points. So we have to schedule around that process which takes about 5 days per school with 4 schools going at a time. 

I'll even break out item 5, cause, well, that's what I do for a living.

• 5a -- deploy new phones to 40 sites!
• Pick up old phones
• What do we with the wall where the old phone was mounted? Yikes!
• deploy new phones
• verify extensions
• verify e911
• configure a fax solution now that we are all IP
• 5b -- deploy 1000 access points to 40 sites!
• Verify student, staff, and guest ssid's work right
• install about 100 switches to light up said APs
• 5c -- implement new email policy
• 5d -- deploy out 300 ipads
• 5e -- deploy out 300 nooks
• 5f -- deploy out 1200 new workstations/laptops
• 5g -- Upgrade internet pipe to larger size
• 5h -- implement new content filter

I'm sure if i polled my group, they'd add 2 or 3 more each. 

Anyhow that's what we do during the summer. Working with the other departments is fun. Teamwork matters.