1. Is the Counter specified for collection, but not available for collection on the machine?

An Event ID 604 Warning message in the Application Event log (or the <computername>.ntsmf.log file) documents performance Counters that were specified in the DCS Data Definition, but are not available for collection on the local machine. This happens because not all the Counters associated with a given performance Object are available for collection across all versions of the associated Performance Library DLL that is responsible for maintaining them. While the developers of the operating system base Counters, like Processor and System, are pretty good about maintaining upward compatibility between versions of their Perflib DLLs, not all developers of Perflib DLLs are as circumspect.

For instance, if you specify that you want to collect all the Counters in the Internet Information Services Global Object and you are running IIS version 5.0, you are likely to receive a Warning message similar to the following:

10/30/02-09:06:00 – Event ID: 604, Category: DCS, Severity: Warning

The following global counters were specified in the DCS but not found on the local machine and will be skipped.

Object, Internet Information Services Global (L2596, G4022)

Counter, Cache Size (L0, G4024)

Counter, Cache Used (L0, G4026)

Counter, Cached File Handles (L0, G4028)

Counter, Directory Listings (L0, G4030)

Counter, Objects (L0, G4032)

Counter, Cache Flushes (L0, G4034)

Counter, Cache Hits (L0, G4036)

Counter, Cache Misses (L0, G4038)

Counter, Cache Hits % (L0, G4040)

Counter, Measured Async I/O Bandwidth usage/ (L0, G4052)

These are Counters that used to be in the Internet Information Services Global Object, but were dropped in later versions of IIS.

This situation is not an error condition (which is why the Collection service only warns you about it), but you need to abandon any hope you may have of collecting those specific Counters on this machine.

1. Is the Performance Library DLL associated with the missing data missing?

Sometimes the Performance Library DLL that is defined is simply missing on the local machine. When the Performance Sentry collection service attempts to load the Library module referenced in the HKLM\SYSTEM\CurrentControlSet\Services\<servicename>\Performance key, the module is not found. In this event, you should see an Application Event log similar to the following:

11/05/02-17:48:08 – Event ID: 210, Category: Discovery, Severity: Warning
The Performance Library, “Perfctrs.dll”, for service “TCP/IP Protocol Driver” (Tcpip), failed to load.
The system error was “The specified module could not be found (126)”.
Performance data for this service will not be available.
The Event Viewer (source Perflib) may have more details

In this example, the perfctrs.dll module identified in the Library field of the /Services/Tcpip/Performance subkey was missing. When the collection service tried to load it, the Load operation failed, for the following reason: “The specified module could not be found.” When a problem like this occurs, using either the Extended Counter List applet or the Registry Editor, determine where the Library is supposed to be and verify that the Perflib DLL is, in fact, missing. If the Library field of the /Performance subkey points to a fully qualified filename, look for the module in the folder specified. If only the module name is referenced, then look for it in the %system32% folder in the Windows <root> directory.

Sometimes this occurs when the application that installed the Performance Library was removed, but the corresponding Performance subkey was not deleted when the uninstall procedure executed. In this case, it is usually safe to remove the Performance subkey manually using the Registry Editor. The Event ID 210 Warning message will disappear after that.

However, if you feel that the missing Performance Library DLL is a mistake, try to correct the problem. Sometimes uninstalling and re-installing the application will restore the Perflib. Alternatively, try to locate a working version of the Performance Library DLL on a different machine running the same or a similar level of software. Copy that Perflib DLL module to this machine and then restart the collection service to see if that solves the problem.

1. Is the Performance Library DLL associated with the missing data defective?

When the collection service encounters a defective Perflib DLL, it generates Application Event log messages that help to document the data collection problem. Common problems that you might see reported include:

1. a Perflib DLL that does not initialize successfully,

2. a Perflib DLL that fails to return any Objects and Counters, and

3. a Perflib DLL that returns zero values for all the Counter values that it reports.

Sometimes, the Perflib DLL has a bug which causes an unhandled Exception or other error condition that causes the collection thread calling the Module to terminate. Prior to terminating, the collection service (assuming you are running the “fat” version of the collector) will generate a Stack trace showing the precise status of the collection thread at the time the error occurred. This diagnostic trace information should be forwarded to the party responsible for the Perflib DLL that failed for resolution.

An example diagnostic trace message generated by an unhandled exception in a Perflib DLL is illustrated below:

As a result of encountering a serious error condition related to calling a defective Performance Library DLL, the collection service will recover and continue to gather data from whatever Perflib DLLs that are functioning correctly for the duration of the collection cycle. While the collection service bypasses collecting data from the defective Perflib for the remainder of the collection cycle, it will automatically retry the module in the initialization phase at the beginning of the next collection cycle. Consequently, the Warning messages that document the problem are likely to recur regularly at the beginning of each collection cycle until you decide to investigate and resolve the problem.

Only the most serious Perflib DLL defects trigger a diagnostic trace message. You are much more likely to encounter less serious defects like the ones described below.

Sometimes the collection service is able to locate and load the Perflib DLL without problems, but cannot gather the performance data the Perflib was designed to collect. Consider that many Perflib DLLs associated with extended Counters are designed to gather data from inside servers application processes like sqlserver.exe, store.exe, and inetinfo.exe. A frequent cause of data collection problems is that the process that a Perflib DLL needs to acquire performance data from is not running. For example, if the sqlserver.exe process is shutdown, the sqlctr80.dll Perflib DLL designed to gather SQL Server statistics will report an error condition, which the collection thread will post to the Event log.

If sqlserver.exe stops running or is hung for some reason, failed attempts to gather data SQL performance data from the SQL Performance Library DLL (e.g., sqlctr80.dll) will generate Warning messages like the following for each SQL performance Object the collector has been instructed to collect:

11/05/02-22:31:00 – Event ID: 2200, Category: Collection, Severity: Warning
Object, SQLServer:General Statistics (L2346, G17300), failed to return data this interval

11/05/02-22:31:00 – Event ID: 2200, Category: Collection, Severity: Warning
Object, SQLServer:Databases (L2370, G17700), failed to return data this interval

11/05/02-22:31:00 – Event ID: 2200, Category: Collection, Severity: Warning
Object, SQLServer:SQL Statistics (L2470, G18300), failed to return data this interval

Errors can also arise when the process that the Perflib DLL must communicate with to gather performance data is simply unavailable. In this example, the w3ctrs.dll Performance Library DLL cannot access the Inetinfo Web server publishing service that is the source for the Web service performance data that it reports.

11/05/02-16:50:16 – Event ID: 1309, Category: PerfData, Severity: Warning
World Wide Web Publishing (Global): no objects were found in the performance data buffer

11/05/02-16:51:16 – Event ID: 96, Category: N/A, Severity: Warning
The following performance libraries failed to return data.
Data from objects from these libraries will be unavailable for this cycle.
World Wide Web Publishing (W3SVC), “w3ctrs.dll”

11/05/02-16:51:16 – Event ID: 2200, Category: Collection, Severity: Warning
The following objects were specified in the DCS but failed to return data.
These objects will be skipped this cycle.
Web Service (L0, G8300)

11/05/02-16:51:16 – Event ID: 2200, Category: Collection, Severity: Warning
Object, Web Service Web Service (L0, G8300), failed to return data this interval

To rectify this situation, you need to resolve the underlying problem that is preventing the process the Perflib DLL depends on from running.

Version 2.4.7 of the Performance SeNTry (NTSMF) data collector will not execute if DEP is active. This does not apply to versions after 2.4.7

For details and options, see this Technical Note.

The main dispatcher thread of the Collection service generates a separate collection thread for each Perflib DLL that it finds installed on the local machine during the Discovery phase at the beginning of every collection cycle. (Follow this link for more .) Each Open and Collect call to a Perflib DLL is performed under a dedicated collection thread. With the threaded collector, it is possible to detect and recover from most failures involving direct calls to Perflib DLLs, sometimes even when calls via the Microsoft PDH and WMI plumbing that other products use, including System Monitor, fail. Under normal circumstances, the NTSMF collection service continues to gather performance data even in the face of buggy, third party Perflib DLL modules.

There are extenuating circumstances when it is desirable for the NTSMF collection agent to ignore completely specific Performance Library DLLs that are defective and perform no processing on these modules. Performance Library DLLs that malfunction in the following ways are candidates for exclusion from all NTSMF processing :

• The Perflib fails to return properly from any Open or Collect call and causes the collection agent, the application being monitored, or the system to hang.
• When NTSMF successfully opens the Performance Library DLL, the module cannot be shared properly with other applications that also attempt to Open them.

When defective Perflibs are excluded from processing, the NTSMF collection agent does not attempt to load the DLLs or call their Open routines during the Discovery phase at the beginning of every collection cycle. You exclude these defective Perflibs from all NTSMF-related processing using entries in the DmPerfss.cfg configuration file.

Note: the use of the the DmPerfss.cfg configuration file is optional. A default DmPerfss.cfg configuration file is installed automatically when you run the standard software Setup routine. The default DmPerfss.cfg configuration file contains entries for Performance Library DLLs that we have seen cause severe problems in some of our customer sites. We recommend that you install and use the default DmPerfss.cfg configuration file.  You can modify the defaultDmPerfss.cfg configuration file using any text editor if you determine there are additional Performance Library DLLs that need to be excluded from NTSMF processing. We recommend that you contact Customer Support before you make modifications to the DmPerfss.cfg configuration file for your environment. If you are having a problem, there may well be other solutions to consider. If the Performance Library DLL is defective, we might like to let other customers know about it.

The <Perflibs> section of the DmPerfss.cfg configuration file can contain an Exclude list and an Include section. These sections identify specific Performance Library DLLs by module name (and optionally by version) that are to be excluded or included for processing at the beginning of a collection cycle.

Exclude list. The Exclude list causes the collector to exclude the modules listed (by version, if specified) from all NTSMF collection agent processing. The collection agent will not attempt to load a Perflib DLL module on the Exclude list at the beginning of a collection cycle. The Version specification is optional. Module entries without a version specification cause all versions of the Perflib DLL to be excluded. If specific Perflib DLL module versions are to be excluded, then they must be listed in the Version section. If there are duplicate entries, the Exclude list takes precedence over modules on the Include list .

Code a <Module FileName=”filename“></Module> section for each Performance Library DLL that you want to exclude from processing. Optionally, you can add one or more <Version> specifications that qualify which versions of the Performance Library DLL should be excluded. If the <Version> specification does not exist, any version of the Performance Library DLL that is found is excluded.

An example DmPerfss.cfg configuration file with an Exclude list is coded below:

<PerfLibs>

<Exclude>
<! — Sybase ASA object –>
<Module FileName=”dbctrs8.dll”></Module>

<!– .NET CLR Data, NET CLR Networking –>
<Module FileName=”netfxperf.dll”><Version>1.0.3215.11</Version></Module>

<!– .NETFramework –>
<Module FileName=”mscoree.dll”><Version>1.0.3215.11</Version></Module>

<!– FileReplicaConn, FileReplicaSet –>

<Module FileName=”NTFRSPRF.dll”><Version>5.1.3590.0</Version></Module>

<!– WmiApRpl –>
<Module FileName=”wmiaprpl.dll”><Version>5.1.3590.0</Version></Module>

<!– Lotus Notes –>
<Module FileName=”nnotes.dll”> </Module>

</Exclude></PerfLibs>

Text inside a <! This is a comment > Comment section is ignored.

During the Discovery phase at the start of every collection cycle, the DmPerfss.cfg configuration file is processed to determine what Performance Library DLLs should be excluded from processing during that cycle. An Event ID 214 Informational message is written to the <computername>.ntsmf.log file that documents the Performance Library DLLs that were specifically excluded from processing, as illustrated below:

08/02/04-00:00:11 – Event ID: 214, Category: Discovery, Severity: Info

The following Performance Libraries are excluded from processing as specified in DmPerfss.cfg. Performance data from these libraries will not be available this cycle.
.NET CLR Data (.NET CLR Data), “netfxperf.dll”, version 1.1.4322.573
.NET CLR Networking (.NET CLR Networking), “netfxperf.dll”, version 1.1.4322.573
.NETFramework (.NETFramework), “mscoree.dll”, version 1.1.4322.573
WmiApRpl (WmiApRpl), “C:\WINDOWS\System32\wbem

Error Messages. In the case of errors accessing the DmPerfss.cfg configuration file, the configuration file is ignored and collection cycle processing proceeds as if theDmPerfss.cfg configuration file did not exist.

If the the DmPerfss.cfg configuration file is invalid, then the following Warning message is written:

08/02/03-00:50:13 – Event ID: 96, Category: None, Severity: Warning
The configuration file failed to open.
The Inclusion/Exclusion list will not be used this cycle.

If the DmPerfss.cfg configuration file contains syntax errors, then the following Warning message is written:

08/02/03-00:50:13 – Event ID: 96, Category: None, Severity: Warning
Failed to load, “E:\DmPerfss 2.4.6\bin\Debug\DmPerfss.cfg”.
Parse error, 0xC00CE503, encountered on line, 25, at position, 14.
Reason = Incorrect syntax was used in a comment.

 Figure 1.

 Simply divide Free Megabytes by Free Space (remember this is a percentage, so the displayed value must be divided by 100 to get the true numeric value.  In the above example, the calculation will be 5,541 / .02338 = 236,997.43.  So the disk size is 236,997MB or (dividing by 1024) 231.44GB.  This logical drive is 231GB!

 However, customers using Performance Sentry get this information automatically by collecting ‘base’ or ‘denominator’ counters.

 Some compound counters contain a Denominator/Base value that is used in calculating the current counter value. By default Performance Sentry writes the Denominator/Base value in the SMF file, unless the user chooses not to. This parameter is found in Performance Sentry Administration under the “Parameter Administration” pane under “Write Denominator Counters? (Y/N)”. See Figure 2.

 Figure 2.

 A good example of a compound counter that uses a denominator/base value is the Logical Disk/% Free Space counter. This counter is a PERF_RAW_FRACTION counter type. The counter type defines a compound structure supplying the numerator (Free Megabytes) and denominator/base (Free Space Base).

 Denominator/Base counters are identified by a unique counter name, constructed by appending the characters “Base” to the counter name. The counter named % Free Space Base is the denominator counter associated with the % Free Space counter.

 So the calculation formula is defined as:

 % Free Space = Free Megabytes / % Free Space Base (Total Available Mbytes in System)

 The Collection Service is responsible for calculating the fraction, which is the value, reported for the counter each interval. The Write Denominator Counters option instructs the Collection Service to supply the denominator value used in the calculation, too, which is otherwise a null value. In this case, the denominator value is a useful metric to collect because it supplies the capacity of the Logical Disk in bytes.

 The Summarization Utility of Performance Sentry makes use of the associated Denominator counters to summarize several Hit % counters in the Cache object correctly over the summarization interval. If denominator values for these PERF_SAMPLE_FRACTION counters are present, the Summarization Utility correctly produces the weighted average Hit % for the interval. If denominator values are not present; the Summarization Utility calculates the simple arithmetic mean, which is subject to error.

04/14/08-13:44:04 – Event ID: 2900, Category: Perf Text, Severity: Error Insufficient data in performance name table

Concerning the message “Insufficient data in performance name table” refers to the file perfc009.dat that resides in the <Windows>\system32 folder. This file contains the names of all installed objects and counters. The name table can become corrupted during installation of new or modified Counter definitions using the lodctr utility.  Even though Microsoft enhanced the procedure to rebuild a  damaged perfc009.dat file in Windows Server 2003 ( KB 300956)  and these problems typically impact all consumers of performance counters, the Performance Sentry Collection Service was modified to recover from severe damage to the perfc009.dat that it finds during Discovery and continue processing.

As an alternative, to the Microsoft KnowledgeBase entry, we sometimes suggest to our customers that they simply copy a Perfc009.dat file from a similarly configured server (after renaming the existing corrupt Perfc009.dat file – for backup or supporting documentation for the error).You can verify the fix by starting Perfmon (Performance Monitor) and looking at the performance counters.

Then stop and restart the Performance Sentry Collection Service so that service will re-initialize using the new, or repaired, file.

For additional repaired options you can read Microsoft’s knowledge base article: KB 300956

Ever notice all those Messages that the Performance Sentry collection service writes to the Application Event log and the <computername>.ntsmf.log file? These messages are designed to help diagnose data collection problems. By reviewing the Perfomrance Sentry Collection service Application Event log Warning messages that document most of the more common data collection anomalies that occur, you can usually identify and resolve a data collection problem quickly. This FAQ tells you how to use these Warning messages to solve data collection problems.

Please follow this simple procedure whenever you have a problem collecting data. First, answer the question below, and proceed to the next page of instructions based on your answer.

Is the Counter you need visible using the Microsoft System Monitor?

Start an interactive System Monitor session and see if the Counter or Counters you need are visible.
If the Counter you need is visible in System Monitor, the Performance Sentry Collection Service should also be able to collect it.

If the Counter value you need is not visible in System Monitor, the Performance Sentry Collection Service probably cannot collect it either. However, it is quite possible that Performance Sentry will generate a diagnostic  error or Warning message that will help you understand why the data you are interested in cannot be collected.

Note that beginning in Performance Sentry version 2.4.5, the Performance Sentry Collection Service does not honor the Disable Performance Counters Registry flag when it is set. Neither will any action performed by the Performance Sentry Collection Service ever cause the Disable Performance Counters Registry flag to be set. However, when the Disable Performance Counters Registry flag is set, it may indicate there is an underlying problem with a Performance Library DLL that could effect Performance Sentry’s ability to gather the associated performance data.

Instead of Disable Performance Counters, Performance Sentry version 2.4.6 and higher utilizes a comparable facility that allows you to specifically exclude Performance Library DLLs that are troublesome. Specific Performance Library DLL excludes and includes are noted in a configuration file in the root NTSMFxx folder named DmPerfss.cfg.

Running Performance Sentry under MS Cluster Server

To run Performance Sentry under a Windows cluster environment:

1. Install the Performance Sentry Collection service in each member of the cluster. There is no need to define any Cluster Resource as a “Generic Application” or “Generic Service”. Performance Sentry will automatically begin to read all shared disks and resources defined by the owner of the cluster on the next collection interval when a failover occurs. MS Cluster Server is started as a service and it may take longer to start than other services in a cluster.

It is recommended that the Performance Sentry service starts after the MS Cluster Server is started so that Performance Sentry can “see” the presence of cluster share drives and logical disk in the respective nodes. This service dependency can be accomplished by adding a “DependOnService” registry value for the particular cluster server software under:
HKLM\System\CurrentControlSet\Services\Dmperfss

1. Prepare the machine you want to use to administer Performance Sentry.

Run the Setup program contained on the installation disk to install the Sentry Administration program on the Windows workstation or server you intend to use to administer Performance Sentry. Sentry Administration is used to define and activate performance Data Collection Sets (DCSes) on the local machine and any remote computers that you want to monitor.

A Data Collection Set defines (1) which of the available Performance Counters you want to collect, (2) the data collection interval and other runtime parameters that control the operation of the collection agent, and (3) the filtering options that ensure that you are collecting the right amount of information. The program ships with standard defaults that are appropriate for most environments. We also provide you with a large number of pre-defined DCSes that are appropriate for different application servers. With Sentry Administration, you can assign any of the DCSes we provided, use the DCS Editor to modify them in a jiffy, or create your own.

2. Prepare the machine you want to monitor.

Install the Performance Sentry Collection engine as a service on any Windows computer that you wish to collect information about and start it. The Performance Sentry Collection Service collects the performance data you specify on the Windows computer on which it is installed. You can install as many copies of the Performance Sentry service as you are licensed to run. By default, DMPerfss.exe writes an NTSMF format data file to a \Data subdirectory of the installation directory on the monitored system’s local hard drive. You can change the Data directory to point anywhere you want.

3. Automate the process for consolidating Performance Sentry collection files for processing by SAS ITRM, MXG, or other performance reporting packages.

Once installed, the Collection Service runs automatically from the time your Windows machine starts up until it shuts down. At the end of the collection cycle, Performance Sentry closes the current .smf data logging file to free it up for processing. It immediately opens a new collection file so no collection intervals are lost. At this time you can schedule a process to copy the older collection file to a central location for consolidation and processing. This may involve setting up a connection to a remote drive on a machine you designate as a central collection file consolidation point and scheduling a program to move the old collection files to that machine at regularly scheduled times.

If the machine being monitored is behind a firewall, it may be necessary to run a simple ftp script to consolidate the daily data files.

The standard installation package that we ship contains three separate Setup routines that can be run separately or together. One setup routine is used to install the Sentry Administration GUI which is used to administer NTSMF data collection. You can install as many copies of the Sentry Administration program as necessary to administer NTSMF across your network. (It is often helpful to run separate copies of Sentry Administration in front of and behind your firewall, for example.

The second setup routine is for the Performance Sentry Collection Service. You can run this setup routine right from the distribution or copy it to disk and run it separately. There is even a silent mode setup option so that you can install the collection agent unattended. Finally, the User Manual shows how to create a simple installation script that you can use to roll out NTSMF across your network of Windows machines. A scripted installation of the collection service has the advantage of being able to assign a Data Collection set (DCS) of your own during installation and also allows you to assign a User Account to be impersonated during Cycle End processingThe User Manual also provides a sample script to use to distribute the NTSMF collection agent using SMS.

For those customers who develop their own distribution and installation procedures, the required set of files needed to install the current version of the collection service is documented on page 17 of the User Manual.

Download the full User Manual (pdf)

Unable to copy the PSSMsg.dll file during installation. Running an installation script to upgrade the collection service, you are likely to find that the PSSMsg.dll file is in use and cannot be replaced. The PSSMsg.dll file is an application Event log message dll that is loaded and used by the Event Log service. Any other utility programs that you run that access the Event log are likely also to have opened the PSSMsg.dll file. In order to copy the latest PSSMsg.dll file, you may have to shut down the Event Log service and any other service that accesses Event Log message dlls. Make sure you restart these services after you complete the file copy successfully.

Please do not agonize over the fact that your script cannot replace the PSSMsg.dll file because it is in use. It is not the end of the world to run the most current collection service with a slightly out-of-date Event Log Message dll.

Optional runtime files. Diagnostic symbols are no longer included in the program’s binary executables like DMPerfss.exe and DTSFnd.dll due to Microsoft compiler changes. Diagnostic symbols are now found in filenames ending with a .pdb suffix. Installing these debug symbol files is optional, but highly recommended so that we can maintain the high quality of the collection service code.

Installing the enclosed version of PKZIP25.exe is also optional.

The runtime files can be found in the \Program Files\NTSMF31 folder following the installation of the collection service following setup, or in the folder \Program Files\Performance Sentry\Collection Sevice Files following the installation of Performance Sentry Administration component.

The third setup routine is for Performance Sentry Portal and Performance Database.   The Performance Database is a Microsoft SQL Server based database and the Performance Sentry Portal is an ASP.NET graphical interface to the database.

As Windows grew and became more robust, so did NTSMF and its capabilities to the point that it was no longer just a performance data collector, but a set of components built around the NTSMF data collection service.  NTSMF was rebranded as Performance SeNTry with the emphasis on ‘NT’.  With the arrival of Windows 2000 and especially Windows Server 2003, the emphasis on ‘NT’ was dropped and the software is now simply called ‘Performance Sentry’. 

You’ll still see references to NTSMF in some of our program and folder descriptions in homage to our long history with Windows from its origins as ‘NT’ to its present versions of ‘Vista’ and ‘Server 2008′.