Deploying 600 iPads to Teachers in a Single Day (& 1800 in 6 days)

We deployed 600 ipads in a single day and 1800 in a week. Here's a brief outline. 

1. Project Scope Prep
1. Our deployment event was only to hand-out the device with a minimal configuration. Training would come later. The basic items we would do were
1. Connect the iPad/Customer to wireless via user account (no SCEP). We wanted location services on wireless to be useful
2. Connect the iPad/Customer to our MDM solution. 
3. Connect the iPad/Customer to mail
4. Connect the customer to his/her own iTunes account
5. Document Customer and iPad association along with CSR who took care of the customer. 
2. iPad Prep (4 weeks)
1. Inventory Control
1. Received and inventoried 2300 ipads.
2. Brought asset tags into asset management system for tracking
2. "Image Prep"
1. Non-managed deployment model so individual itunes accounts can be used.
2. Only app loaded was work order app and MDM catalog
3. Minimal security prep (we trust 'em with our kids, so a piece of electronics is a no brainer)
4. Charge the iPads to over 85% before storing. 2 weeks storage didn't drop the battery much.
3. Location Prep (1 day physically prepping, weeks diagramming)
1. We obtained one of middle schools gyms. Seating capacity was 420.
2. Parking lot could hold about 120 vehicles
3. We diagramed the traffic flow. It was white-boarded. and re-whiteboarded. 
1. Since it was August in Texas -- kept everyone inside as much as possible
2. Found the "slow" part would be the actual customer to (customer service rep) CSR interaction on the configuration. We would go at our customers pace. We allocated 20 CSR here.
3. We borrowed stantions (crowd control pole things with ropes) from food service for customer routing within the cafe.
4. We had it catered on day 1. Hey, if they wait, feed and water 'em! Plus if they had to bring kids, we could send the kid home on a sugar high.
5. We brought 3 temporary APs to connect since the most customers we had per half hour was 50. 3+2 in the ceiling is 5 APs. Should handle 100ish clients. If you are a wireless geek saying what about everyone bringing phones and your CSR stations, yeah we know
6. We hard cabled our CSR locations with temporary switches and connections. 
7. The staff at the building we used were AMAZING! custodians, office staff, food service, coaches, everyone was great. 
4. Staffing Prep
1. We did just-in time training. Literally the day before. Took 4 hours to prep the CSRs.
1. How to enter a work order
2. How to connect wifi
3. how to connect to MDM
4. how to check email
5. how to connect itunes
6. what to do when something goes wrong
1. We had 2 "senior floaters" that could _hopefully_ resolve 90% of the issues. Typical issues were
1. Customer (teacher) wasn't in work order system. (import in)
2. Customer password -- reset at CSR location.
1. CSR's were granted pwd reset abilities
3. Customer was dropping wifi (quick solution -- wifi white list)
4. MDM certificate corruption (new ipad, or profile removal)
5. iPad flaked -- replace.
6. Customer asks for training or more hands-on or beyond the scope of the deployment questions.
1. PUNT -- we had our instructional tech team available to answer these questions. Again. these people were awesome during the roll-out.
7. Customer doesn't have iTunes account. We create. Use internal email. The customer had 3 weeks to get it done if they wanted a separate account. 
2. Instructional Staff Prepping
1. Gave them assigned stations. The lead for this team took this off my plate after we walk-through the diagram several times. She allocated her staff to the right spots.
1. Greeters
2. Q & A tables
3. Floaters (help where needed)
3. Misc Staff Prepping (not much here to prep)
1. Paperwork runners -- ran tickets to CSR
2. Restockers -- made sure there were iPads for the CSR
3. Clean up folks -- 600 ipads and case generate trash. Boxes, film, ..just trash.
5. Advertising & Customer Scheduling
1. Used our teacher training registration tool to advertise to teachers
2. Also sent email to campus admins in case some teachers didn't receive email.
3. Locked sessions at 50 per half hour starting at 9AM. Left 90 minute break for lunch. 12 sessions total. 
1. Walk-ins were permitted. We actually did 622 day 1. 
6. Work Flow Processing
1. Greet the customer at the doors near the gym and cafe! Tell them where to go next. (1 person)
2. Hand customer Acceptable use policy/device loan policy. Actual signature page was part of work order (1 person)
1. There is food & water here if we get back logged along with seating. We found it was used by spouses and kids. 
3. Verify employment and/or enrollment. I'm trusting...to a point. (2 people)
1. State ID
2. District ID so we can tell job roles. Teachers were the primary recipient, not admins. Turning away principals and APs is always fun. They were actually understanding so it wasn't bad.
4. Hostess station (2 people). Watches CSR stations and maintains customer list. Hostesses would seat customers and do a simple list like a restaurant. When a CSR came available, called the next customer. We had welcome to your iPad documents along with the Device Agreement documents to read while waiting.
1. We had "flags" for the CSR people to raise when they were done and awaiting another person.
5. CSR stations (16 -20 people). Configure the iPad as listed above.
1. Generate and print work order
1. Backside of work order has signature page for device agreement (sneaky?)
2. Configure iPad
3. Place iPad into case
4. Give customer iPad and Device Agreement page.
6. Q & A stations (2-4 people)
1. Staffed by instructional tech folks at tables.
7. Signature and copy area.
1. Mixed with the Q&A area. 
2. Sign the Usage agreement and make copies if customer wants
8. Departure station
1. Receive signed agreement paperwork
2. Give customer power cables for iPad

That's all it took. Advertising up and down the food chain was critical.

1) There were a lot of people who ensured our message was consistent. This is for teachers was a big deal. 

2) What to expect was a big deal. -- iPad deployment only, not training

3) Ability to put other projects on "hold" for 2 days

Fun with Upgrading your Technology Business Info Servers

So, we migrated from VMWare 5.0 to 5.1. Plus we had to update our processor minimums due to some Cisco firmware bug. It required to basically split our servers into 2 separate clusters. they weren't truly separate clusters, but in our primary it behaved that way due to the processor settings. Once the physically server had its firmware upgrade, any servers moved to it had to be cold moved...ie shutdown, then moved. Anyhow...we migrated our 80+ servers except our Business info servers. I like getting my paycheck. Its after the 20 of the month. cool, lets schedule with the BIS IT person. I get the green light Tuesday night at 7. She actually likes me to let me start that early, so thank you my friend! She does her back up. it goes long. no problem, 7:30 pm. i get to go. shutdown the vm! no problem. change host. no problem. Boot. good. update vmware tools. good. reboot. still good. shutdown again. Update firmware, yep v9 good. start it back up. Hrm, lets peak at windows update.seriously, 82 updates. text, patching. (forgiveness, not permission). 45 minutes later..a reboot. she's having kittens. I text it is all good, just slow. All comes back. Our exchange starts.

her: I thought you said 30 minutes.
me: you hadn't patched since dec 2011.
her: i know...
me: did anyone notice the longer outage
her: no
me: did it come back and work perfectly?
her: yes.
me: Greatness.
Her: I am glad you did the ugprade, not anyone else
me: (down ego, down ego internally). thank you. you know i wouldn't let anything happen to your server.

Now, i'm begging her assistance to get a flat-file out so i can send it to generate user accounts for a 3rd party. Her system has some data i don't in AD.

Moral of the story: Take care of your peers!

Generators and UPS Fun

Okay, i'm a dork and geek. Learning random shituff from your peers in other fields is fun to me. especially when they let me ask my noob questions and answer them minus the sarcasm. onward to the story.

We've installed a generator at one of our sites. The generator powers both my MDF and IDFs along with the freezers for food service. (for those of you going why freezer too...if that food spoils, we could be out of $500k...yeah, my network gear at $250k is important, but thats a lot of food too. Yeah yeah yeah, down time, loss productivity, etc are costs too, but that 500k versus streaming a youtube khan academy lesson is harder to quantify in k-12). So, the generator is in. our gear is also protected by a UPS which we left in place since generators need 1-30 seconds to come on even with an ATS. Line cleaning of the voltage and all that other wonderful stuff is good too for the day-to-day operation. One day we lose power. The generator kicks on. the freezers go on to the generator. the UPS in my closets light up like an X-mas tree. wtf does that error code mean. other than the UPS isn't happy and we are on battery and the battery say less than 15 minutes. So, post mortem begins. (thank god it was during the summer while 90% of staff was out). Yeah, everyone blamed my UPS. I get the UPS codes based on the error light. Line voltage error. still blaming the UPS. Get our master electrician in. show him the UPS works on standard line power, but not generator. ask him to prove me an idiot or prove me right. I say its still an generator problem. 2 minutes later (yeah, he's pretty swift and smart), he has the problem IDed. I'm not an idiot! (yay?!) Standard building power is 240, generator is 208. I have deer eyes. I ask my questions. explain. well, most UPSes won't accept more than a 10% change in voltage. If it is bigger than 10% change you get the line voltage errors. so, 240-24 is 216 and 208 is less than 216 which is more than 10% delta. UPS stayed pissed off and never accepted the generator as a valid power input. Battery drained and we were offline. a step up transformer is being installed to fix the issue to fix the issue. 

Moral 1 of the story: Make sure your generator and utility voltage match.

Moral 2 of the story: respect your peers and what they know outside your trade! Never know when the electrician or HVAC or alarm guy may help you out too.

Moral 3: School districts store a lot of food in the central freezers. A lot. 

iPad Deployment

I've been dormant. Lets restart. Catharsis, right?

What the heck, lets cover a 3rd rail of Technology, iPads! 

We are in the midst of trying to deploy 2000+ ipads to our teachers. Fun! Most people think Technology support people don't want iPads on their network on in the staff hands because we don't like them. If that were only true. Many of us use the them day to day for both work and play. Its an excellent device. It simply runs. Its intuitive for most folks after 20 minutes. After that, there are apps that do damn near anything you can dream of. Even configure switches if you are network guy, or manage vmware, or anything. So, yes, we like the portability and functionality of the device. However...ask us to deploy them and manage them...ah, that gets us worked up.

Deployment: Apple doesn't have good tools for mass deployment. Period. IPCU and Apple configurator leave a lot to be desired. They can help deploy. However, the degrees of suck depend on your level of deployment customization. Examples: 

1) So, we want to pre-deploy our apps for our staff as part of our base. Great! use configurator. Months later, there is an update and itunes is giving the red 1. Guess which itunes account it asks for when updating pre-deployed apps. Yep, the administrative account used in prep. As any good admin will tell you, yeah, right, over my dead body do you get our admin/deploy account information. Bad things happen when admin or deploy accounts get into the wild.

2) Oh, btw, you used Apple configurator. You went fully managed. (wow, either you have some serious somethings or a lot of time). 2000 devices were configured off of one machine running Configurator. Its a weekend, 30 teachers went to some conference and did something creative to their machines. You have to reimage since the device doesn't work. Guess what! that single apple laptop (you did use a laptop right?) has the only valid restore of those 30 devices. So, you truly have a centralized deployment and management tool! that single laptop! Awesome. now get out there and touch all 30 devices! 

Application roll-out: So, we want to give all our teachers the latest coolest must have app! Awesome! That app costs $3.99. yeah, so. 8k later to Apple (and is our rep driving another new car? -- i kid). we have a pot of money to deploy this app. Magically, we get the application out to our staff using our favorite MDM (we've tried 3, they are all decent). Teacher x leaves the district. Oh, btw, we are out 4 bucks for that one app. We have 20 apps with an average cost of $2.50 so we are out $50.00 (*2000 devices, is our rep driving another new vehicle?)? Why right? can't we just recover the app back into our pool? Ah, not if you decided you wanted to let the teachers customize the device and use their own itunes account. Yep, that $50 worth apps just went to that personal itunes account of the teacher. Hrm, giving away tax-payer money. Not good. So, the moral of the story, is recognize if you allow individual itunes account and push out apps, get all parties on board (finance, hr, technology) and recognize the cost and loss of apps due to employment changes of people. The cost creep can get expensive and have some legal ramifications.

Bonjour: Ok, as a long time network person who cut his teeth on apple devices and networks, gawd, it looks like the revived a portion of the team who wrote of appletalk. non-routable. Bonjour is crap crap crap, not a good enterprise protocol. So, if you have any sort of industry standard wireless network (802.11n/a/g), you probably have a centralized controller. Most like your LAN networks aren't the same as your wireless networks. There's a high probability, these two routers may be multiple hops away from each other. So, you want to remote control that apple desktop using a bonjour enabled on your ipad . yeah, i know it works great at home right? but not at work. my network people are a-holes and don't know what they are doing. I can't speak to that :), but they may not be entirely at fault. In the enterprise (yeah, your 20 school district is considered an enterprise), those router "hops" stop bonjour and that remote control functionality. I know a lot of the bigger vendors are coming up with solutions to resolve this issue, but some of those are quite buggy. In addition, 802.11ac will force resolutions to come and marry up local lans and wlans since tunneling 1G across a WAN will be a huge bandwidth strain. We aren't there yet. (plus, my wifi infrastructure sales person needs a new vehicle too).

Anyhow, there are more items, and each task dreamed up seems to introduce another set of obstacles. We need to get all parties to recognize the uniqueness of the device compared to where the world was 4-5 years ago. That windows 7 box isn't an ipad. The tools aren't the same. The management capabilities are less on the ipad. The deployment capabilities are less. However, the customer can use the ipad quickly and most like the unit. Support and deployment and manageability is -different-. Don't ask us to provide the same customer service experience. It's all different. As the tools of our trade change, so do the expectations. 

Fun with Certificates

Fun with certificates. Wow. Certificates in an enterprise are a delicate item to undertake, even just end-point certificates such as users, workstations and SCEP/NDES devices.I finally issued my first certificate to an iPad as a -computer- account following a lot of the blogs on the internet. Now it is time to see if that cert can be used for device authentication on our new wireless overlay. Yeah, we could do generic accounts on the device which is the fall back plan. We've found anything we have to rely on the end point user tends to require a certain amount of support costs. I'll fully document everything this weekend if it works and we are to deploy a successful test (10-100 units). 1 is easy. 10 is work. 100 needs automation. Have to get it relatively automated if possible. No comment on the lack of documentation except on microsoft oriented sites. To those who have created the content, I thank you! My basic sites I visited and followed.

• http://www.kurtdillard.com/StudyGuides/70-640/6.html
• http://marckean.wordpress.com/2010/07/28/build-an-offline-root-ca-with-a-subordinate-ca/
• http://social.technet.microsoft.com/wiki/contents/articles/2900.offline-root-certification-authority-ca.aspx 
• bounty of sub-links to read and understand
• http://security-24-7.com/windows-2008-r2-certification-authority-installation-guide/
• http://blogs.technet.com/b/pki/archive/2012/02/27/ndes-and-ipads.aspx

Notice the lack of apple content. there is a reason! Bad Apple! You will need to download the iphone configuration utility too. The issue I had is with the X.500 name. There is hints and direction, but people are stingy on this one. My entry looked like

O=mycompany.com, CN=iPad123456

O is our domain listed in the certificate signing piece. If your CA signs certs for devices in mycompany.com, then put O=mycompany.com.

CN is the devices name you gave it. We use our asset tag system. This doesn't have to match much along the way. Until I fixed those I was getting 0x800094001. The other flavored the request subject name is invalid or too long is addressed by the authors above.

The O should be capitalized. So should the CN. including a comma and space might be necessary. Haven't chased the rabbit that far down the hole yet.

Onto the next fun part. Cisco ISE. Oh boy! 1.1 release software is fun.

Quick update

We've been busy! The summer projects are about to start. We ran school late this year. Short list of items:

• When deploying workstation certificates, check which servers can issue the certificates.Things you learn along the way.
• Having new team members is fun. New blood. New perspective. New questions. Nice thing to have. 
• Having issues with whatsupgold "crashing". Trying to have it read some WMI from windows servers and it just stops scanning. Everything. Not good.
• Good is the VM discovery features in whatsup. Check that discover vm environment!
• Updating, verifying, 3500 edge switches is a pain in the ... well, its a pain. Once verified, whatsconfigured will take over and rule. Voice config and security and a few other configs to update.
• Emergency construction is fun. And smelly. Broken water lines, of all flavors! Have to move our equipment (workstations, network stuff, others) out of the way. 
• Every 3rd party application wants to do upgrades this week such as our bus route software, special ed *2, energy management, food service, and a couple of others. Not a bad thing, but they asked -- Yesterday. Saying no isn't a real option. At least most of these the vendors will webex/remote-in and do the upgrade. So Veeam snapshot before starting in case they -aw !@#$- it. 
• Google Calendar is awesome. Sharing a calendar with vendors/resellers regarding schedules at schools is beneficial. Having to train them how to sign up and use it, but all in all it is good. Better than having to continually export a ics from exchange. 

That has been the last week. And my people are busier than me. Its good to have a team to run $1M mostly on their own.

Nothing Technical

The only downside to summer planning is all that happens is meetings. And once those are done, you have to schedule more meetings. Nature of the beast. The kids are going. The teachers are going. The buildings are about to empty. Facilities, Construction, Training, Summer School, Technology all want a chance at the empty buildings to do tasks, whether it be painting, waxing, moving AC units, continuing ed credits, credit recovery, and whatever we are supposed to do (phones, wireless, ipads, nooks, etc). Time to get on the same page to start. We all know best laid plans, etc, but at least if you start on the same page you can adjust. Here's hoping we get to the same page. Thats why the lack of technical fun recently. And I'm tired of getting errors with ipads trying to register as a device with NDES/SCEP. grrr. Need those 4-6 hours uninterrupted to turn up the debugging to really see what the errors are.