Amy Lewis (@CommNimja) recently tweeted about mentoring:

“Good mentorship is like physical therapy: no pain, no gain.”

“Just had amazing conversation about mentors, and I have a question. Do you actively seek mentors and what do you do with them?”

The resulting discussion got me thinking about the mentoring process, and particularly where I fit into the role of a mentor and mentee. As a manager of people in the technology field and a technologist myself, I have the responsibility of mentoring those on my team in a number different ways.

The easiest way for me to mentor someone is in the area of technology. I have very talented people on my team, but we cover all IT disciplines (OS, servers, storage, virtualization, networking, security, compliance, project management, etc) with a small number of people. We have a very diverse set of application groups we support, each with their own set of requirements and nuances. We need to have a lot of depth and breadth in our knowledge, so there is always opportunity to mentor each other. One of my skills is the ability to see the bigger picture and build solutions that are more universal for the environment, rather than something very specific to one team, so that is something I try to mentor into my team.

Communication is another area where I am called to mentor my team. The company I work for has a diverse personality mix. With talent comes some interesting personalities… Knowing how to communicate with each person isn’t easy and something I struggle with myself. This also happens to be the hardest area where I am a mentee. I live deep in a lot of different technology and am called upon to give high level strategic presentations to the owners, sometimes with very little preparation. For me, I need to spend more time in preparation for these discussions and extracting myself from the technology. When I took the management role, I specifically asked for a mentor specifically to address this aspect and my boss has provided a lot of good feedback.

This topic has made me think about other areas that I need mentorship. These includes presentations,  HR aspects, and from a non-work related aspect, being a better Dad.

Where can I mentor others? For starters, I am a local VMUG leader and I could search out people who want to learn more. I am assisting one of the guys on my team as he studies for his VCAP exams and I could do more of that with the local VMUG members.



VMworld 2014 Wrap-Up

One of my favorite weeks of the year is almost complete. I’m at the airport heading home, looking forward to being back home with my family. VMworld was simply amazing this year. I have always enjoyed the experience, but the community brought the whole experience to another level.

I had some objectives to accomplish this year:

  1. Research SIEM tools
  2. DevOps integration
  3. OpenStack integration
  4. Get out of my predominantly introverted shell
  5. VCDX–do I really want to do it?

Continue reading

VMworld 2014

VMworld is coming up fast at the end of August and I’m getting pretty excited. From a professional perspective, it is the highlight of my year. This will be my 4th time attending and will be one of my busiest schedules yet.

For me, it is important to have a hotel close to the venue, the Moscone Center. In order for this to happen, a reservation needs to be made well in advance of VMworld registration, or make the reservation by registering very quickly after registration opens–I registered within a couple of hours of registration being open to get a reservation at a reasonably priced hotel in Union Square. My coworker who is attending with me ended up at a different hotel since her registration was a few hours later. One of the benefits of being a VMUG leader is that they offer perks associated with volunteer time at VMworld–one of which is getting into the Marriott right by the Moscone Center. I was able to make that switch and it will be a huge benefit. The swag that can be obtained at VMworld is second to none and being able to drop it off at the room means I can travel light during the day.

The process I use for planning my overall schedule has been refined over the last few years. Once the general schedule is released, I start blocking off my calendar with the general sessions, VMworld party, hall crawl, lunches, etc. For things like lunch, I use a different color and mark the it as available instead of busy. Where this helps is when I start to schedule sessions. I know I need to eat, so I don’t want to fill my schedule to the point of not being able to eat lunch. The 30 minutes between sessions is sufficient for lunch.

Schedule Builder is now live and the sessions are filling up fast. This is another one of those things that should be tackled as soon as possible after it is available. The best sessions can be filled up within a couple of hours. The most popular sessions might be repeated, and/or the room assignments will get adjusted to bigger rooms. Aside from my first year, I’ve had one or two focus areas, and this year is no different. At $job, we are building a long term SDDC strategy and my goal is to get a better handle on the technologies that will be part of it–OpenStack and storage are the primary areas. I build my schedule first based on Advanced Technical sessions in the focus topics, then specific speakers who have helped me with their blog posts and Twitter posts (Chris Collotti, Duncan Epping, Scott Lowe, William Lam, Rawlinson Rivera, Cormac Hogan, Chad Sakac, Jason Nash, Josh Odgers, etc), and finally, I look for other VCDXs that are presenting. The value of VMworld is that I can get the equivalent of many expensive training courses compressed into one week by listening to the right people. As a whole, I try to avoid purely vendor sessions. I’ve been to some sessions that say they won’t talk about their product and then espouse the benefits of their approach over everyone else for an hour–I can get that in the vendor hall.

The Welcome Reception is when I scope out vendors for swag and determine which ones warrant additional time talking to engineers–the challenge is getting past the sales people. I may have a management role at $job, but I also do hands on administration and design, so I want to talk to the people who develop and/or support the products we are interested in. I’m also coming at things from a different perspective this year… I want to know more about technologies that don’t apply directly to $job.

Current Session Schedule:

  • SDDC1580 – OpenStack in the Enterprise
  • STO2996-SPO – The vExpert Storage Game Show
  • STO1491 – From Clouds to Bits: Exploring the Software Defined Storage Lifecycle
  • SDDC1176 – Ask the Expert vBloggers
  • SDDC1600 – Art of IT Infrastructure Design: The Way of the VCDX – Panel
  • STO3162 – Software Defined Storage: Satisfy the Requirements of Your Application at the Granularity of a Virtual Disk with Virtual Volumes (VVols)
  • STO1965 – Virtual Volumes Technical Deep Dive
  • INF1736 – Designing Next Generation Software-Defined Data Centers: A Panel with VMware Certified Design Experts
  • STO2480 – Software Defined Storage – The VCDX Way Part II : The Empire Strikes Back
  • VAPP2305 – Extreme Performance Series – Understanding Applications that Require Extra TLC for Better Performance on vSphere – Deep Dive
  • SDDC2492 – How the New Software-defined Paradigms Will Impact Your vSphere Design
  • NET1592 – Under the Hood: Network Virtualization with OpenStack Neutron and VMware NSX
  • OPT2668 – DevOps Demystified! Proven Architectures to Support DevOps Initiatives
  • INF1552 – SDDC: Buzzword to Reality. Discussions with SDDC Architects that Goes Well Beyond Markitecture
  • HBC1534 – Recovery as a Service (RaaS) with vCloud Hybrid Service
  • STO3247 – VMware VVOL Technical Preview with Dell Storage

Other Stuff:

  • VMUG Leader Reception and Lunch
  • VCDX/vExpert Reception
  • VMworld Party
  • Welcome Reception
  • Hall Crawl
  • vBreakfast

Storage Functional Testing

I’ve spent a fair amount of time in the last couple of months testing storage. At $job, we have some very interesting datasets and workflows. These result in large bursts of intense I/O. This can lead to I/O latency issues when multiple databases attempt to write to disk at once. In the physical server world, these workloads were isolated, not impacting each other. In the virtual server world, shared storage can lead to bottlenecks that impact many systems at once. For this reason, storage selection is critical in our environment.

We have a mix of storage arrays, primarily iSCSI with some fiber channel, and a small amount of NFS. At this point, my favorite storage arrays are the Nimble Storage arrays. We have 4 of the CS240G-X4 models, two each in two data centers. The performance is stellar, analytics are awesome, and the reliability on every front is unmatched in our environment. I’ve done firmware updates under load in our DR environment with absolutely zero downtime and no real noticeable impact in performance. They just work… <shameless plug over>

Now for the main topic… when testing storage, what should we do to validate a new array, particularly one from a new-to-us vendor, and one that isn’t as mature as EMC, NetApp or HP? I deal with a VMware-based environment, so that’s my focus, but most of this directly applies regardless of environment.

I’ll outline my steps…

  1. Manual Controller Failover Testing – both quiesced and under heavy load
  2. Controller Software Upgrades – this should be done under load
  3. Simulated Drive Failures – this applied to both spinning and flash
  4. Power Supply Removal – they say it is hot swap, so test it before you have to bet your data on it
  5. Forced Controller Failure – remove one and see what happens
  6. Network Connectivity – start pulling a cable or two while under load
  7. Multipath Testing – know how things work in the recommended configuration (MRU vs ALUA) and what happens when the opposite is used
  8. Performance – iometer is very useful, not only to generate load for the aforementioned testing, but also for knowing what the limits are for performance

Storage is physical and involves a lot of interconnects. There are many aspects where issues can appear. Test them all before you put production critical data on the array. Know what the impact is to performance when something fails.

When testing, I use the I/O Analyzer fling from VMware. Deploy, configure the IP, and start running tests. You can deploy a fairly large number of them on one host, or across many hosts. The options are practically endless.

I’ll go into more details on how I do performance testing at some point in the future. I know my process isn’t perfect, but I try to know what the limitation are and testing the relevant settings (NMP, queue depth, etc) to know the impact.


OpenStack Summit

I have the privileges of heading to Atlanta today for a few days. This is a dual-purpose trip, first to do some work at $job’s data center, but, more exciting, is the second reason, which is the OpenStack Summit.

My role at $job is a mix of management and technical, with my primary technical aspect being the virtual infrastructure. VMware was selected as the hypervisor because of how simple it is to implement and manage and the amazing options for automation. Whenever possible, we will use open source options if they exist and don’t increase the workload significantly. My team is made up of 5 guys (including me) and manage >500 physical and virtual systems, with a wide variety of technologies (Linux, MySQL, Spacewalk, Puppet, VMware, storage, networking, security, etc).

My goal with attending the keynotes and wandering the vendor hall is to understand how OpenStack has been evolving over the past couple of years and where we could or should be using it within our infrastructure. I tend to be someone who tries to keep up on what is happening with competing and complementary solutions to what we use, and OpenStack has been maturing nicely where I don’t think it would be a significant management burden to implement and maintain.

One area of particular interest is how we could leverage OpenStack with Docker and Vagrant to provide a better environment for the software developers. We’re considering using vCloud for our developer environment, and I think the OpenStack ecosystem might have a comparable offering.

I’m also looking forward to meeting some of the people I follow on Twitter.