Writing Policy

Okay, i am techie. I don't like writing policies. Its simple as that. I find policies difficult. Not necessarily to write, but to find the balance the institutions requirements along with the "humans" who will be interacting with the things which i'm writing policy. My human side always plays the what-ifs. the grey areas. ugh.

So, we are writing our lost and stolen and replacement device policy. Finance liked it minus some Financial language! easily remedied. waiting on legal and HR. Something about with holding a last paycheck until we get our gear back. The goal is to either get our gear back or withhold the cost of the units said staff member failed to return. Simple enough. However, that means our paperwork has to be in order. teacher A is owner of laptop X and ipad y. Fun! more paperwork! (yes, we have tools to tie the devices to the users. YEAH KACE!).

Week 1 Totals

Well, I ran our last 7 days of work orders totals. OMG! Wow.

1240 work orders closed in a week.
High water was around 1000 open work orders
current total is around 650 open requests
highest daily request was 400 work orders. doing the total on this day was sad. we close 275 requests and got 400.

(for those of you who don't know the pattern of k-12..here goes. In early June, turn everything off and store it...somewhere. and that somewhere is likely only conditioned down to 90-95 degrees. then, in the last week of august, find said stored items. turn them on and hope they work. Remember, some of the teachers turning these items on don't "know" technology. its a magic box. we are ok with that. getting the little 2nd grader to complete a task can be magic to me too. she does her magic with the kids, my team does our magic with the devices).

Thats pretty good from what for us. 38 sites, 3500 staff users. new ipads. We have a pretty good support staff who understands getting everyone up and running is job 1. We are on the second wave of requests, its works, but i can't do X. Access youtube, run adobe pro, etc. these are the more complex problems where unfortunately "no" becomes an answer.

Inheritances

I didn't know what to title this...inheritances or victims of our success. Anyhow, I might brag or show some bravado, but I think my team is pretty good at what we do with what we have.

Last year we inherited badge access control our buildings. All I got was another help desk staff. I think I made out like a champ. Sad? Maybe, but she's really good! anyhow, Security access, awesome?! Why? Well, people have this notion we solve problems in our department and 3 months to change a badge access was too long. Who'dathunkit? So now this is my beast to manage as of last year.

Its now year 1+ of owning access control. It is the start of school, and badge access is my disaster to deal with. Its not technically a disaster, it just is something we have to take care of. we hire 10-15% new staff per year and move another 10-25% around. 40 sites, 3500 staff. Yep, its not pretty right now. So, our access control systems don't work well for our customers. Yeah, we've identified the underlying problem(s), our connections to HR systems, etc, but the customer doesn't care. Ms. Teacher gets pissed her badge doesn't work on the door. We GET it. we get pissed too. Hell, i set off the alarm and call security when my doesn't work. (hello, security, i'm the the building. i set off the alarm. i'll be here for 2 hrs. I'm guessing my badge doesn't work on the way out either so you'll have to set it). We get 100 work order per day for badge access now. They get resolved quickly, but having a door not open when you badge it pisses everyone off. Its that simple. Trying to educate bosses and customers, that 700 open active work orders aren't -that bad- is a task. Last year we peaked at 1200. I'm guessing 2000 this year will be the peak.

I'm sure there is a moral to the story. I think it is integrate your systems. I've met with HR. They are willing to help. there are good software solutions to help update all systems quickly. There is budget. Now, can we get the people part to work.

Deploying 600 iPads to Teachers in a Single Day (& 1800 in 6 days)

We deployed 600 ipads in a single day and 1800 in a week. Here's a brief outline. 

1. Project Scope Prep
1. Our deployment event was only to hand-out the device with a minimal configuration. Training would come later. The basic items we would do were
1. Connect the iPad/Customer to wireless via user account (no SCEP). We wanted location services on wireless to be useful
2. Connect the iPad/Customer to our MDM solution. 
3. Connect the iPad/Customer to mail
4. Connect the customer to his/her own iTunes account
5. Document Customer and iPad association along with CSR who took care of the customer. 
2. iPad Prep (4 weeks)
1. Inventory Control
1. Received and inventoried 2300 ipads.
2. Brought asset tags into asset management system for tracking
2. "Image Prep"
1. Non-managed deployment model so individual itunes accounts can be used.
2. Only app loaded was work order app and MDM catalog
3. Minimal security prep (we trust 'em with our kids, so a piece of electronics is a no brainer)
4. Charge the iPads to over 85% before storing. 2 weeks storage didn't drop the battery much.
3. Location Prep (1 day physically prepping, weeks diagramming)
1. We obtained one of middle schools gyms. Seating capacity was 420.
2. Parking lot could hold about 120 vehicles
3. We diagramed the traffic flow. It was white-boarded. and re-whiteboarded. 
1. Since it was August in Texas -- kept everyone inside as much as possible
2. Found the "slow" part would be the actual customer to (customer service rep) CSR interaction on the configuration. We would go at our customers pace. We allocated 20 CSR here.
3. We borrowed stantions (crowd control pole things with ropes) from food service for customer routing within the cafe.
4. We had it catered on day 1. Hey, if they wait, feed and water 'em! Plus if they had to bring kids, we could send the kid home on a sugar high.
5. We brought 3 temporary APs to connect since the most customers we had per half hour was 50. 3+2 in the ceiling is 5 APs. Should handle 100ish clients. If you are a wireless geek saying what about everyone bringing phones and your CSR stations, yeah we know
6. We hard cabled our CSR locations with temporary switches and connections. 
7. The staff at the building we used were AMAZING! custodians, office staff, food service, coaches, everyone was great. 
4. Staffing Prep
1. We did just-in time training. Literally the day before. Took 4 hours to prep the CSRs.
1. How to enter a work order
2. How to connect wifi
3. how to connect to MDM
4. how to check email
5. how to connect itunes
6. what to do when something goes wrong
1. We had 2 "senior floaters" that could _hopefully_ resolve 90% of the issues. Typical issues were
1. Customer (teacher) wasn't in work order system. (import in)
2. Customer password -- reset at CSR location.
1. CSR's were granted pwd reset abilities
3. Customer was dropping wifi (quick solution -- wifi white list)
4. MDM certificate corruption (new ipad, or profile removal)
5. iPad flaked -- replace.
6. Customer asks for training or more hands-on or beyond the scope of the deployment questions.
1. PUNT -- we had our instructional tech team available to answer these questions. Again. these people were awesome during the roll-out.
7. Customer doesn't have iTunes account. We create. Use internal email. The customer had 3 weeks to get it done if they wanted a separate account. 
2. Instructional Staff Prepping
1. Gave them assigned stations. The lead for this team took this off my plate after we walk-through the diagram several times. She allocated her staff to the right spots.
1. Greeters
2. Q & A tables
3. Floaters (help where needed)
3. Misc Staff Prepping (not much here to prep)
1. Paperwork runners -- ran tickets to CSR
2. Restockers -- made sure there were iPads for the CSR
3. Clean up folks -- 600 ipads and case generate trash. Boxes, film, ..just trash.
5. Advertising & Customer Scheduling
1. Used our teacher training registration tool to advertise to teachers
2. Also sent email to campus admins in case some teachers didn't receive email.
3. Locked sessions at 50 per half hour starting at 9AM. Left 90 minute break for lunch. 12 sessions total. 
1. Walk-ins were permitted. We actually did 622 day 1. 
6. Work Flow Processing
1. Greet the customer at the doors near the gym and cafe! Tell them where to go next. (1 person)
2. Hand customer Acceptable use policy/device loan policy. Actual signature page was part of work order (1 person)
1. There is food & water here if we get back logged along with seating. We found it was used by spouses and kids. 
3. Verify employment and/or enrollment. I'm trusting...to a point. (2 people)
1. State ID
2. District ID so we can tell job roles. Teachers were the primary recipient, not admins. Turning away principals and APs is always fun. They were actually understanding so it wasn't bad.
4. Hostess station (2 people). Watches CSR stations and maintains customer list. Hostesses would seat customers and do a simple list like a restaurant. When a CSR came available, called the next customer. We had welcome to your iPad documents along with the Device Agreement documents to read while waiting.
1. We had "flags" for the CSR people to raise when they were done and awaiting another person.
5. CSR stations (16 -20 people). Configure the iPad as listed above.
1. Generate and print work order
1. Backside of work order has signature page for device agreement (sneaky?)
2. Configure iPad
3. Place iPad into case
4. Give customer iPad and Device Agreement page.
6. Q & A stations (2-4 people)
1. Staffed by instructional tech folks at tables.
7. Signature and copy area.
1. Mixed with the Q&A area. 
2. Sign the Usage agreement and make copies if customer wants
8. Departure station
1. Receive signed agreement paperwork
2. Give customer power cables for iPad

That's all it took. Advertising up and down the food chain was critical.

1) There were a lot of people who ensured our message was consistent. This is for teachers was a big deal. 

2) What to expect was a big deal. -- iPad deployment only, not training

3) Ability to put other projects on "hold" for 2 days

Fun with Upgrading your Technology Business Info Servers

So, we migrated from VMWare 5.0 to 5.1. Plus we had to update our processor minimums due to some Cisco firmware bug. It required to basically split our servers into 2 separate clusters. they weren't truly separate clusters, but in our primary it behaved that way due to the processor settings. Once the physically server had its firmware upgrade, any servers moved to it had to be cold moved...ie shutdown, then moved. Anyhow...we migrated our 80+ servers except our Business info servers. I like getting my paycheck. Its after the 20 of the month. cool, lets schedule with the BIS IT person. I get the green light Tuesday night at 7. She actually likes me to let me start that early, so thank you my friend! She does her back up. it goes long. no problem, 7:30 pm. i get to go. shutdown the vm! no problem. change host. no problem. Boot. good. update vmware tools. good. reboot. still good. shutdown again. Update firmware, yep v9 good. start it back up. Hrm, lets peak at windows update.seriously, 82 updates. text, patching. (forgiveness, not permission). 45 minutes later..a reboot. she's having kittens. I text it is all good, just slow. All comes back. Our exchange starts.

her: I thought you said 30 minutes.
me: you hadn't patched since dec 2011.
her: i know...
me: did anyone notice the longer outage
her: no
me: did it come back and work perfectly?
her: yes.
me: Greatness.
Her: I am glad you did the ugprade, not anyone else
me: (down ego, down ego internally). thank you. you know i wouldn't let anything happen to your server.

Now, i'm begging her assistance to get a flat-file out so i can send it to generate user accounts for a 3rd party. Her system has some data i don't in AD.

Moral of the story: Take care of your peers!

Generators and UPS Fun

Okay, i'm a dork and geek. Learning random shituff from your peers in other fields is fun to me. especially when they let me ask my noob questions and answer them minus the sarcasm. onward to the story.

We've installed a generator at one of our sites. The generator powers both my MDF and IDFs along with the freezers for food service. (for those of you going why freezer too...if that food spoils, we could be out of $500k...yeah, my network gear at $250k is important, but thats a lot of food too. Yeah yeah yeah, down time, loss productivity, etc are costs too, but that 500k versus streaming a youtube khan academy lesson is harder to quantify in k-12). So, the generator is in. our gear is also protected by a UPS which we left in place since generators need 1-30 seconds to come on even with an ATS. Line cleaning of the voltage and all that other wonderful stuff is good too for the day-to-day operation. One day we lose power. The generator kicks on. the freezers go on to the generator. the UPS in my closets light up like an X-mas tree. wtf does that error code mean. other than the UPS isn't happy and we are on battery and the battery say less than 15 minutes. So, post mortem begins. (thank god it was during the summer while 90% of staff was out). Yeah, everyone blamed my UPS. I get the UPS codes based on the error light. Line voltage error. still blaming the UPS. Get our master electrician in. show him the UPS works on standard line power, but not generator. ask him to prove me an idiot or prove me right. I say its still an generator problem. 2 minutes later (yeah, he's pretty swift and smart), he has the problem IDed. I'm not an idiot! (yay?!) Standard building power is 240, generator is 208. I have deer eyes. I ask my questions. explain. well, most UPSes won't accept more than a 10% change in voltage. If it is bigger than 10% change you get the line voltage errors. so, 240-24 is 216 and 208 is less than 216 which is more than 10% delta. UPS stayed pissed off and never accepted the generator as a valid power input. Battery drained and we were offline. a step up transformer is being installed to fix the issue to fix the issue. 

Moral 1 of the story: Make sure your generator and utility voltage match.

Moral 2 of the story: respect your peers and what they know outside your trade! Never know when the electrician or HVAC or alarm guy may help you out too.

Moral 3: School districts store a lot of food in the central freezers. A lot. 

iPad Deployment

I've been dormant. Lets restart. Catharsis, right?

What the heck, lets cover a 3rd rail of Technology, iPads! 

We are in the midst of trying to deploy 2000+ ipads to our teachers. Fun! Most people think Technology support people don't want iPads on their network on in the staff hands because we don't like them. If that were only true. Many of us use the them day to day for both work and play. Its an excellent device. It simply runs. Its intuitive for most folks after 20 minutes. After that, there are apps that do damn near anything you can dream of. Even configure switches if you are network guy, or manage vmware, or anything. So, yes, we like the portability and functionality of the device. However...ask us to deploy them and manage them...ah, that gets us worked up.

Deployment: Apple doesn't have good tools for mass deployment. Period. IPCU and Apple configurator leave a lot to be desired. They can help deploy. However, the degrees of suck depend on your level of deployment customization. Examples: 

1) So, we want to pre-deploy our apps for our staff as part of our base. Great! use configurator. Months later, there is an update and itunes is giving the red 1. Guess which itunes account it asks for when updating pre-deployed apps. Yep, the administrative account used in prep. As any good admin will tell you, yeah, right, over my dead body do you get our admin/deploy account information. Bad things happen when admin or deploy accounts get into the wild.

2) Oh, btw, you used Apple configurator. You went fully managed. (wow, either you have some serious somethings or a lot of time). 2000 devices were configured off of one machine running Configurator. Its a weekend, 30 teachers went to some conference and did something creative to their machines. You have to reimage since the device doesn't work. Guess what! that single apple laptop (you did use a laptop right?) has the only valid restore of those 30 devices. So, you truly have a centralized deployment and management tool! that single laptop! Awesome. now get out there and touch all 30 devices! 

Application roll-out: So, we want to give all our teachers the latest coolest must have app! Awesome! That app costs $3.99. yeah, so. 8k later to Apple (and is our rep driving another new car? -- i kid). we have a pot of money to deploy this app. Magically, we get the application out to our staff using our favorite MDM (we've tried 3, they are all decent). Teacher x leaves the district. Oh, btw, we are out 4 bucks for that one app. We have 20 apps with an average cost of $2.50 so we are out $50.00 (*2000 devices, is our rep driving another new vehicle?)? Why right? can't we just recover the app back into our pool? Ah, not if you decided you wanted to let the teachers customize the device and use their own itunes account. Yep, that $50 worth apps just went to that personal itunes account of the teacher. Hrm, giving away tax-payer money. Not good. So, the moral of the story, is recognize if you allow individual itunes account and push out apps, get all parties on board (finance, hr, technology) and recognize the cost and loss of apps due to employment changes of people. The cost creep can get expensive and have some legal ramifications.

Bonjour: Ok, as a long time network person who cut his teeth on apple devices and networks, gawd, it looks like the revived a portion of the team who wrote of appletalk. non-routable. Bonjour is crap crap crap, not a good enterprise protocol. So, if you have any sort of industry standard wireless network (802.11n/a/g), you probably have a centralized controller. Most like your LAN networks aren't the same as your wireless networks. There's a high probability, these two routers may be multiple hops away from each other. So, you want to remote control that apple desktop using a bonjour enabled on your ipad . yeah, i know it works great at home right? but not at work. my network people are a-holes and don't know what they are doing. I can't speak to that :), but they may not be entirely at fault. In the enterprise (yeah, your 20 school district is considered an enterprise), those router "hops" stop bonjour and that remote control functionality. I know a lot of the bigger vendors are coming up with solutions to resolve this issue, but some of those are quite buggy. In addition, 802.11ac will force resolutions to come and marry up local lans and wlans since tunneling 1G across a WAN will be a huge bandwidth strain. We aren't there yet. (plus, my wifi infrastructure sales person needs a new vehicle too).

Anyhow, there are more items, and each task dreamed up seems to introduce another set of obstacles. We need to get all parties to recognize the uniqueness of the device compared to where the world was 4-5 years ago. That windows 7 box isn't an ipad. The tools aren't the same. The management capabilities are less on the ipad. The deployment capabilities are less. However, the customer can use the ipad quickly and most like the unit. Support and deployment and manageability is -different-. Don't ask us to provide the same customer service experience. It's all different. As the tools of our trade change, so do the expectations.