Weird Days

Days where 90% of the job is awesome, but the 10% does it in and tanks it are no fun. That was one of those. Just when you have a handle on what you can do and will do as a team, poof, it changes. I know it happens, but this was far beyond the ordinary. I'll do the fun parts.

Phone calls with your vendors, resellers, and others that are actually fun, useful and on target. Had a conference call with our implementor, wireless hardware provider and us. Our implementation team from our reseller stepped up and spoke up on our behalf. The hardware vendor themselves worked to help resolve our issue. Those are good calls. Even with 7 people on a con-call with no true "organizer". 5 tech people with 2 sales creatures on the same call are interesting. I genuinely like our implementation person. Smart, sharp, savvy, and in my corner. good cloning material.

Before that I found out my zoning was 23/24 correct. freaking misclicks. Easily corrected and non-production storage. Anyhow. created the servers onto the ISE so i could provision storage. Next step to do, create adequate storage and offload busier arrays onto these units. However, adding the array to our management system...whats this yellow hardware error (and my previous storage person took another job 3 weeks ago). ah, crap. support call. We've had it happen before. And my boss wondered why i built-in redundancy at every layer possible, hardware software, cabling. CCR will save the day when we have to go offline on the array with error for the rebuild. About the only issue our users will see....is none unless they happen to be do something that 1-4 minutes we failover the ccr on sunday morning.

During the lunch hour we got to play host to our eRate company. It was informational for them so one of their new people could put the terms to the physical items. either they were polite or i was informative. 90 minutes of me doing a data center tour. i'm going with the former.

Anyhow, those were the fun parts.

KACE & Skyward - Part 2

I really did more research into this project. Kinda hit a sticking point. So our Skyward system sends email to a Reforma printer spooler (by Fabsoft) that put data in certain locations ..into a pdf which is emailed. Well, crap. as stated before KACE wants emails with
@'fieldname' blah blah blah.
I can direct the email to our system based on key fields in the print stream. that's not a problem. I'm stuck on how to elegantly get the data from the print stream (or pdf) into a formatted email.
That's where i'm stuck, but it's moved down on the priority list. From my poking around in reforma and other products I can pull field locations from pdfs and put them into a flat file or similar. that file or document can then be sent as contents of an email. As I told my boss...good programmer, 2 weeks to 2 months depending on how good we want it. damn. Filed under the list of doable, but no time.

following on this. I learned whatsupgold let us format the emails however we want. Yep. it can create the work order automagically into KACE. Just have create the rules to send email to the KACE smtp listener when an event or status changes to something bad. Can even set status as open or closed, but that can be a headache. having the system open a work order and close it without human interaction shouldn't count. such as...custodial staff unplugs device from wall for vacuum. work order create! custodial plugs device back in! close work order. (I should add the assignment to me so I get the credit right? :) The hard part on this is writing the policy and rules so we get work orders for what is "real" for our environment. having 1000 switches our users can touch makes this a challenge. A layer 8-12 (your environment, politics, religion, money!) decision.

interviewing people

I've always found this interesting process. meeting people. Trying to judge them both technically and personally in 60 minutes is a challenge. A few notes from the interviewer side of the table.

1) Please don't put down something on your resume as a talent or skill if you've never really done it. Especially if it is in the job descriptions or requirement. wow. If you have familiarity with administering exchange, be prepped for some decent questions or quickly define what you've done! What have you done? what are the server types? ever done clustering? how about policy rules? what have you done in powershell? those are the starter questions. I'm going to try to gauge your depth of knowledge. The thought process is, do you have enough basic knowledge to start. ok, how much do i have to train you on our products. what skillset do you bring that i don't have on my team. <--extra bonus points for you here.  

2) Be ready to tell about your successes on a recent job or project. Why was it a success? To the company? to you?

3) Be ready to tell about where you either failed, or not done your best and what you learned. Seriously. we are all in tech. If you administer, design, maintain, or implement large systems its likely you've had a few learning moments. An example: I've had my contractors send a directional bore for a fiber run through geo-thermal system on the coldest day of the year knocking out the heat. Not good. Lesson I learned was to gather exact details! the info i had wasn't precise about -which- northside parking lot had geos under them. I also learned that geos go up and down between 36-48" and our bores at 48". last, i got the best pick me up that day from our construction sup..."if your 20k issue is the biggest we encounter in this project, I'll be estatic"

4) I'd say be ready for an oddball question or two, but prepping for that is hard. Just be prepared for an awkward, weird, unusual question. Its to put you on your toes. catch your reaction. 

5) have questions for us. Anything. Ask about the work environment? how often are weekends? how often unscheduled weekends? Do you like working here? what do you do to assist with family life such as work from home or comp time, etc? Does your office do anything to keep it fun and friendly? Remember, you want to find out if you want to work for us too!

Construction Meetings

These are fun. Seriously. Where else do you get to listen to the architect talk how the color of the walls and carpet will affect "x". Followed by how this type of toilet flusher thingie won't work cause kids beat the bejesus out of stuff causing it to break or flood? or as you are working on your iPad hear that the are swapping out all AC units at a campus this summer. That causes a pucker factor. hey, wait. no ac. what about our mdfs? idfs? so, these new units, the go...on the roof. awesome. so, are you doing anything about ceiling. Oh, just removing the entire grid.All my APs are going to go offline. and cameras. and speaker. crap. When do you expect to finish. Oh the friday before all staff returns. uh, houston, we have a problem. no security and paging will cause some uh, issues and not just for my staff. Then we have to work coordinate teams to get this stuff done. take down, put back up. I actually like listening to a couple of the guys from the construction company cause how well they can do logistics in their heads. quite impressive talent to behold. And plan and schedule is born.

then back to work. The most favorite word uttered during any meeting is..asbestos. the fact i know the general idea of the law in the state is...sad? 10 days to post notice. no access during the removal. anything left in the area is gone once the abatement start. Then pray they don't find anything else. 10 days doesn't seem like much, until you factor in summer is effectively 10-12 weeks long. Plus the abatement time. Losing 2-4 weeks out of that is a huge hit.

That was the highlight of the day.

Technology Companies Failing to Embrace New Technologies

After having, lets call them discussions, with some of our vendors, I've started to wonder why some "technology companies" fail to embrace new technologies and methodologies. For example, one of our vendors is -requiring-  us to have them fly a goober (technical dork) to our site to install the new version of their program on a new server which is on a virtual machine. So, they are requiring the person to be here, take up space in order to remote into a console onto a server that they could have access to remotely. To click next a bunch of time. and watch the vm restart really fast. Example 2, another vendor does not support backup tools on a vm machine. thats right. no veeam or third party product. only vmware's back utility. So, since we a re a school district, i see a script coming up to avoid this issue. Followed by the lecture from the vendor, that's not best practice and yeah, i'm not made out of money.

We are in technology damn it! move forward. If someone questions your reason for doing something (especially as a company) and the best you got is, 'its the way we've always done it'. you need to re-examine your practices. We do it all the time. If we get called on that statement, and you have a possibly better idea, flesh it out, explain it, show me.

Explaining New Services Impact to Non-Tech Peers

Trying to explain to peers when they want to implement a new service how it affect us. Yeah, our process are lacking so bear with me. The conversation typical goes,
My Peer: The vendor said we can turn on x feature and let all these people register, login and do all sorts of cool stuff. Its all web based and doesn't require anything from your group. It even integrates with ldap and you have to do nothing. Plus everyone else is doing it.

Me: red flags. Vendor said...vendors will say anything for a sale. And everyone else is doing it? seriously? in a k-12 environment using that argument. wow. Just wow. Yellow flags: Logins = username and password issues, who are our customers going to call? Who is going to fix the logins? When do the customers login?
(There are good answers to these questions, but I'm not sharing). Next, Please god, tell me they use ssl and some decent security standards on passwords. LDAP. Ah, share my directory environment across the internet. yeah, thats a bit scary. (non-verbal, yes, i know i can limit access to ip, require a certain username and password {preferrably something that looks like the cat stepped on the keyboard}scope the ldap searches and a bunch of other things, but my group has to do this). Last, who and when do you want to this? I have to dedicate a resource to contact the vendor, contact the vendors tech people to ask questions (and yes those are 2 separate steps. We usually have to call the product AM. Then be polite to get the tech people so we can ask nuts and bolts questions).

The part that is hard to hiding your cards so to speak. Sometimes, it really is a good idea and product, but the questions need to be asked and answered. Trying to balance the questions so I don't look like I'm out and out saying no, but getting them to understand vendors, uh oh hell, lets be blunt, LIE, and it will have some impact on us anytime a new service turned on cause something will break and all technology items go through my help desk to start. Whether we "fix" the problem or route the call appropriately is something to discuss.

Frayed at the edges

Ah, people and processes and other things that go crazy during the day. I love my group. I love our version of crazy, our version of insanity within our little sanctum, but wow some days are trying. Not on me necessarily, but on my folks. Its spring. the reality of crazy season is hitting everyone within my group. Millions of dollars in equipment coming in and having to be deployed during the next 4 months. (may, june, july, aug if I counted right). Edges are fraying. So its manager time. And no, i have no delusions of being superman. like i said, it is my weakest suit and requires the most attention. Thats what kept me busy today. No more sharing on that cause personnel are personal. and probably most of tomorrow. Ok. one thing...

Day in the life

Fun day...Lets recap! Sad when you google for your own blog for documentation.

Interview for application support specialist. I love acronyms! What an A.S.S.! we wanted Imaging and App Support Specialist from Im an ASS as the title. No go. Damn. 

zoned out 2 new storage arrays for access by 6 servers on a Cisco MDS 9509. Lovely. In the GUI. Taking notes so I can train someone else on how to do it. Usually, I just console in and paste a script with the pieces and parts. Give each device an alias. make sure you have a convention since most of the HBA's are duals. We do server name P0 and P1 for servers. The storage arrays have lots more ports and controllers. We use XIO so the "friendly" names look like Xio2MRC1P1 and Xio2MRC2P5. MRC has ports 1-4, MRC has ports 5-8. We only patch port 1&5 for now. Create the zone. and the zones to the zoneset, activate. wait for the zoneset to update. listen for screaming. nothing, good. scan for hosts on the arrays. done! well, not really, but done enough for now.

install Cisco ISE on 3 VMs from item 2. ISE on ISE. yeah. now talk to your group about it. I've gotta work on the ISE. what? the arrays are down? What? the policy engines are messed up? Confusing! And I love Cisco, but they need to kick whoever is keeping them from officially support VMWare 5.x on their applications. Unreal. 

Provide documentation to a reseller/implementer for wireless IP schemes. Spreadsheets. eye bleed. Making sure you have enough IPs to scale from 150 to 4000 nodes at a campus is fun. Do you break it up? what if the require layer 3 mobility. Not like users walk-around or anything. Do you anchor? do you do something else. choices! This outta make tomorrow a good conversation with the reseller.

End of year budgeting! We should have a few bucks to roll back to the general fund. yay my team. I just click ok on order approvals. Or not. Deny is a good one too!

Then people. Dealing with the human side of the job. We are emotional beasts. This is actual the most time consuming part of the job. People need. period. they just need. Need you. Need backing. Need a pat on the back. Need an ear. Need a moment of your time. Need feedback. Need guidance. Need. None of these things are bad. Some need more than others, but all people have their needs. This is the part of the job that I continue to learn. Simple visits with everyone. Actually listening to them talk about their problems (and as a tech person this is hard. just listening. give me a problem, i'll solve it. Not always what folks are looking for!). 

And that's a day without meetings! I'll doc one of those up some day.