VMWare ESX 3.0 Monitoring Options in MOM 2005

Recently I was tasked with conducting a trial to find a workable solution for VMWare ESX 3.0 monitoring in a MOM 2005 enterprise environment. When searching for viable solutions, there were three primary criteria we felt were most important. First, I was looking for a product that can collect a broad range of event, state and performance data and also Deliver the information with some built-in knowledge (threshold and state rules), And finally, deliver well-formatted events to MOM, so wouldn’t have to spend hours writing scripts to transform nasty variable binding from raw SNMP events. The three competing products I put through the paces were: eXc VMWare Management Pack Quest Management eXtensions for MOM nWorks VMWare Events MP for MOM Of the three products in the trial, I found only one really met both of my criteria. Below are some technical insights into the products I tested, some high level details on how data is extracted and presented in MOM, some of shortcomings identified in the trial. 1) ESX VMWare MP The first product I tested was eXc's VMWare management pack. Unfortunately, the most attractive quality of this solution was the licensing. You will pay somewhere in the ballpark of $399(US) per DNS name which means you can run as many of their MP’s as you want against that machine. The first real test I put the test through was on UP/DOWN notifications. The product uses SNMP traps sent from the ESX Host machine and compares those traps against an Access DB and alert based on what the SNMP trap contained. Unfortunately the trap contained a string that included the physical path to the VMX file. This can render the alerts unusable if the ESX administrator does not follow appropriately descriptive naming conventions. The performance collection wasn't much better. It would basically initiate a telnet or SSH session (depending on your configuration) and gather performance data gleaned from output of the ESXTOP utility. This is okay but the MP only collected data on the ESX server as a whole which really doesn't help you that much when trying to figure out which VM is hogging the resources. It also failed the built-in knowledge criterion, because of the poorly formatted alerts, coupled with the fact that all UP/DOWN alerts come across as critical errors. I was also disappointed to learn that eXc doesn’t support their product on a MOM MS running on a virtual server front-end….the very scenario we run in our production environment. 2) Quest Management eXtensions for MOM The next product I tested was Quest Management eXtensions for MOM. This test didn't last very long because the version I was testing at the time didn't even support a VMWare ESX 3.0 host. Unfortunately it took a support call to Quest and three days of waiting for an answer to find this out. Not great response time for a company trying to sell me something 3) nWorks VMWare Events MP for MOM The final product tested was nWorks VMWare Events MP for MOM. nWorks offers two versions of their VMWare Management Pack. One version covers only VMWare events (called VMWare Events Only MP for MOM) at a cost of approx $500 (US) for a 2 CPU license, and a second version covers both events and performance logging called (VMWare Only MP for MOM), at a cost of approx $1300(US) for a 2 CPU license. Upon installation, I was impressed with the amount of data it collected from the ESX Host. It not only collected information about the ESX Host as a whole, but also data on each VM on that host as well. I think the big advantage is that the MP used the VMWare API to pull some of this data, avoiding the decidedly unfriendly SNMP varbind dump to MOM, saving me some scripting time. Granted this application needs to run some VMWare console based utilities using a SSH connection, but the amount of data it returns is very nice. The UP/DOWNS were a refreshing change from previous products because it actually pulled the name of the VM guest from the API and the alert was readable. The nWorks met all my criteria for selecting a ESX Monitoring solution. It provided a broad range of data about the ESX host and provided built-in knowledge through its alerts. nWorks Homepage http://www.nworks.com/index.php eXc Software http://www.excsoftware.com/version3/version3/Default.aspx

Monitoring SNMP-enabled Devices in Operations Manager 2007

I added an interim location for Operations Manager 2007 content  to MOMResources.org while the new site is being developed, where we'll be adding content regularly as we evaluate the RC2 release.   Among the first items is an excellent walkthrough on how to monitor SNMP-enabled applications in Operations Manager 2007,  by newly designated MOM MVP Daniel Matey. Get it HERE.   Daniel is an active blogger on MOM / SCOM, but in Spanish... http://dmatey.spaces.live.com

Overcoming Errors when Compiling MIBs in MOM 2005 (via smi2smir.exe)

Overcoming Errors when Compiling MIBs in MOM 2005 (via smi2smir.exe)   This post is for MOM administrators using MOM as an SNMP Trap-catcher, as defined in http://support.microsoft.com/kb/555455   In this post, I am going to offer some insight into addressing a common issue when compiling MIBs into WMI (MOF) format for MOM 2005 when using MOM as an SNMP trap-catcher, and offer a couple examples of an alternative that allows one to skip this process altogether.     Plan A – Identifying and Addressing Dependencies for MIB compilation to MOF Format   Windows has its own "SNMP" space in WMI, and by converting MIBs to MOF and loading into the WMI namespace (\.\root\snmp\smir ) puts them in a format MOM can understand. It's not uncommon when compiling MIBs to WMI format (MOF) with the smi2smir utility, you may run into an error, which is often assumed to be a syntax error. (Cisco seems to be a common source of problems). This is actually not always the case. It's sometimes a matter of identifying the MIBs upon which a given MIB is dependent. Consider the following example:   All vendors have the own SNMP OID (for example VMWARE is OID 1.3.6.1.4.1.687 or iso.org.dod.internet.private.enterprises.vmware). To successfully compile VMWARE MIB ( VMWARE-SYSTEM-MIB.mib) to MOF  - I need to add all MIBs on which this one is dependent (up to SNMP hierarchy - iso.org.dod.internet.private.enterprises ) in my smi2smir statement in order to compile successfully.   To identify the dependencies of a given MIB, you need to examine the IMPORTS section of the header file of the MIB you wish to import.     For example, see head of VMWARE-SYSTEM-MIB:   --------begin VMWARE-SYSTEM-MIB: file-------   VMWARE-SYSTEM-MIB DEFINITIONS ::= BEGIN IMPORTS     OBJECT-TYPE       FROM RFC1155-SMI    DisplayString       FROM RFC1213-MIB    vmwSystem  -- OBJECT IDENTIFIER ::= { vmware 1 }       FROM VMWARE-ROOT-MIB; --------end VMWARE-SYSTEM-MIB: file-------     In this example, the smi2smir utility reads ahead to retrieve objects FROM RFC1155-SMI, RFC1213-MIB, and VMWARE-ROOT-MIB. So if you do not first compile the MIBS on which this one is dependent, you'll receive an error message. You can compile these one at a time in order, or simply address all in a single command with smi2smir.exe, as shown in the example below. EXAMPLE smi2smir.exe /g VMWARE-SYSTEM-MIB.mib RFC1155-SMI.mib RFC1213-MIB.mib VMWARE-ROOT-MIB.mib IANAifType-MIB-V1SMI.my > VMWARE-SYSTEM-MIB.MOF   C:\WINDOWS\system32\wbem\snmp>smi2smir.exe /g OLD-CISCO-CPU-MIB.mib RFC-1212.mib CISCO-SMI-V1SMI.mib SNMPv2-SMI-V1SMI.mib > OLD-CISCO-CPU.MOF   Option B – Custom MOM Scripts If you do get your MIB loaded, the data displayed in the MOM alert window is simply a data dump of the varbind (*variable binding) list, which is a REALLY ugly and hard to work with. The other option is to not bother with importing the MIB at all. With MOM, you can simply write a script to parse the raw SNMP data to create a separate well-formatted event in MOM. I could spend an entire book chapter on this topic, so I'll simply point to a couple good examples of this you can read on your own:   Parsing UPS SNMP Events by John Hann http://msmvps.com/blogs/jfhann/pages/69388.aspx   And this 2nd one is a bit long, but I do step through the process of parsing an event parameter to transform   Parsing Nortel SNMP Events by Pete Zerger http://www.it-jedi.net/2006/12/day-6-telecom-monitoring-with-snmp-and_365.html     *DEFINITION: Variable Binding is a list of MIB objects that allows a requests recipient to see what the host issuing the SNMP GET wants to know     Thanks to Serhiy Lysan, sales specialist for Juniper Networks in the Ukraine for his input on this post.

Day 6 - Telecom Monitoring with SNMP and MOM 2005

It's rare that I get to do anything truly important with MOM (read "anything anyone other than me gives a damn about"). I'm not talking about monitoring that e-commerce website or really important ERP application, but an honest-to-God real-life, non-technical mission critical operational function in a large organization.   So, on this, the 6th day of Christmas, I am going to demonstrate the extensibility the current version of MOM in monitoring non-Windows systems (starring MOM in the role of SNMP trap catcher), as well as provide some samples on how to take raw SNMP events in MOM and format the data into something user-friendly. In this particular example, I am going to show you how we used MOM to capture 911 call data from telecom switch a  multi-site enterprise to report the source and location details of the call back to site administrators within seconds of the event…( the 411 on 911, so to speak)   Read the entire article here…

Book Review: Microsoft Operations Manager 2005 Field Guide

This book operates on a different paradigm from any other book yet published on MS Operations Manager 2005 in that is doesn't waste a single breathe on the basics. The authors assume you have the functional aspects of MOM securely in hand before you turn the first page. If you don't know a management server from agent-managed server, put this book down and move on. No space is wasted on this obvious in this guide. The value here obvious….best practices from experts with vast experience in the field. So if you're already grounded in the essentials, this book is an excellent desk reference for an enterprise administrator, as they address the topics I so frequently see in the newsgroups (and my inbox). I think the author's train of thought was spot on in their choice of subject matter, and this is the first book on MOM where I found myself wanting to add a comment here, a SQL statement there just to add my two cents J Attention is given to often overlooked planning tasks, such as the process for estimating expected volume of collected data, as well as a walkthrough of the System Center Capacity Planner, the preferred tool for modeling enterprise MOM deployments…The guys dig into all manner of topologies, supported configurations, capacity limits, etc…but it's really coverage of the configuration details that makes this book worth the price. I'll point out a few observations here. SNMP - For heterogeneous environments, the guide offers excellent coverage of the SNMP configuration process, allowing MOM to receive and process SNMP traps from non-Windows hosts (Something that is currently only addressed in a short community KB article). Events and Alerts – Good overage of event rule creation, with details on event parameters. Good chapter on rule tuning as well. Management Packs – While the book doesn't go into details of management pack configuration (this is after all documented at length in the MP guides), the guys do point out some common gotchas with AD, Exchange and IIS MPs. Here again, the value-add is in best practices for MP deployment and tuning via a sandbox environment. DTS - A short chapter is dedicated to the DTS Package, with succinct and relevant coverage of how the data flows, how the package is constructed, and how to dig your way out after extended failures…well done, gentlemen. MCF – The chapter on multi-tiered management group scenarios is possibly the best in the book. This chapter is exceptionally well structured, and walks through the configuration process for a multi-tiered environment in great detail, as well as some great troubleshooting bits. Few engineers on the planet have more experience in this area than Andy Dominey…I am guessing he wrote this chapter. If you're deploying a tiered management group topology, this chapter is a must-read. Troubleshooting – There are separate chapters on troubleshooting and troubleshooting scripts. The script troubleshooting is especially good. If you're not familiar with the ResponseTest utility or unfamiliar with remote debugging with Visual Studio, you'll find this chapter particularly informative. Scripting - No coverage of scripting per se, just troubleshooting. For scripting guidance, see the 4-part "Scripting MOM "series by Brian Wren, available on the Microsoft website. There are plenty of sample scripts on the MS site and mine ( www.momresources.org) to get anyone started. Security – There is also a chapter dedicated to security. The authors offer security recommendations for accounts on all services. Excellent guidance I don't think you'll find in the online MS documentation. Concepts of mutual authentication and proxying – both common points of confusion, are also given due attention. Reporting - Reporting coverage over 2 chapters is good. There's a nice example on how to use an existing report as a template, so to speak. This requires some Visual Studio experience, but a decent starter guide. Hit the Microsoft MSDN site for the SQL 2005 step-by-step report creation tutorials as well for some additional grounding in the basic concepts. No book is all things to all people, but all in all, if you're an experienced MOM administrator with more to do than care for MOM, this is a great book to have on the shelf a troubleshooting reference. Although if you're not already grounded in MOM 2005 concepts, I'd suggest you start with Essential MOM by Chris Fox. Get your copy of the Microsoft Operations Manager Field Guide HERE.

Monitoring WAN Latency in MOM

My client tells me it costs approximately $10K per month for a T1 in many countries in South America. So much for stepping up to a broadband connection. L   We had a few days awaiting the implementation window to deploy an SNMP-based solution for monitoring network connectivity between a US company and it's parent company's offices in South America, but in the interim had to come up with a quick stop-gap measure for monitoring WAN availability and performance . So a buddy and I put together a script-based mechanism for measuring, charting and alerting on WAN performance...a Network Latency Monitor (of sorts).   This script utilizes Win32_Pingstatus object in WMI and native MOM functionality to measure, chart, and alert on availability and performance. We tried to account for a variety of scenarios including total failure and exceptionally low ping times (to facilitate accurate reporting on faster links). Here are the details:   Script Functions:  Measures the average ping response time (in milliseconds) of target host Optionally, creates a performance object to chart performance. Generates an alert when ping responses breaches a user defined threshold. Optionally, logs an event in MOM (for testing). Script Parameters:   Host - Name or IP address of target host. ResponseThreshold - Threshold (in milliseconds) above which the script generates an alert of WARNING severity. LogPerformanceData - 1 creates a performance object for charts & reporting.  0 turns this feature off GenerateEvent - 1 generates an event (for testing) 0 turns this off.   The script is available for download at momresources.org. To get a copy, click HERE   Bug reports, feedback, etc. welcome at administrator@momresources.org

Several New MOM KBs This Week

Busy week for KBs. Here's a rundown: 918278 How to uninstall a MOM Management Pack 918481 How to install Microsoft Operations Manager (MOM) 2005 Service Pack 1 databases and MOM 2005 Reporting on a computer that has Microsoft SQL Server 2005 installed 912704 The Snmp.exe process generates an access violation when the event is converted to an SNMP trap in MOM 2005 917666 The MOM 2005 Notification Workflow Solution Accelerator is not supported on a SQL Server 2000 cluster (A PSS call we opened a couple months ago prompted this one) 917664 Error message when you try to install Microsoft Operations Manager 2005 Reporting: "Error code: -2147467259 (Unspecified error)" (This is the DisableLoopback fix)

HOWTO: Configure Microsoft Operation Manager 2000 or 2005 to monitor devices by using SNMP

Here is a handy solution for monitoring non-Windows devices with SNMP using MOM 2005. http://support.microsoft.com/?kbid=555455&SD=tech