Will you please explain Cycle End Processing?

admin — Thu, 04 Mar 2010 16:05:21 +0000

This discussion of Cycle End processing also considers the following subtopics:

At the end of each collection cycle, the Collection service closes the current collection file to make it available for processing. The end of a cycle normally is used to initiate processing of the collection files in SAS or MXG. Cycle end processing normally consists of scheduling a process to consolidate Performance SeNTry data files at a central location. This section discusses automating cycle end processing.

The Collection service stores data collection logs in the \Data subdirectory under the NTSMF24 program directory by default. The location of the data directory can also be changed using the SeNTry Administration program by modifying the Directory where SMF files are stored start-up parameter.

Hierarchical file management sets up Current, Previous and Archive Folders under the \Data directory and manages collection files according to the criteria you specify. The hierarchical file management option is specifically designed to simplify your cycle end procedures. By default, old collection files are moved immediately from the Current Folder to the Previous Folder. After the number of days you specify, they are then migrated to the Archive Folder. Files are kept in the Archive Folder for an additional period of time and then deleted automatically according to your specifications. Because all automatic file management is performed prior to issuing the Cycle End Command, the cycle end procedure you execute can be as simple as a COPY *.* command pointing to the Previous Folder.

Unique collection data log file names are automatically generated of the form

systemname.yyyymmddhhmmCustomQualifier.smf

where yyyymmddhhmm is a date/time stamp identifying when the file was created. During the collection cycle, the Collection service can lock the current data file so that no other application can access it. Or the current data file can be Shared so that it can be processed by other Windows NT/2000 programs or commands while collection is occurring and the file is open. The collection file is closed at Cycle End and is available for processing.

The Collection service automatically launches the Cycle End Command you specify after all file management actions have been completed. You can specify commands directly or execute a .BAT file script, PERL script, or other command processing program. By default, the Cycle End command is launched immediately. Or you can specify that the Cycle End command be launched at a random time within a defined processing window (up to 60 minutes in duration).

Processing multiple files.

If the Collection service was restarted (due to a re-boot, for example) during the last collection cycle, there will be multiple .smf collection files in the\data\Previous folder at the time the Cycle End Command you specified is launched. Consequently, the Cycle End processing you perform should allow for the fact that multiple collection files may be available for processing. A wildcard in your filespec normally accomplishes this.

For ease of processing in MXG, you will normally consolidate collection files representing many different machines into one file. Use the COPY command available in Windows NT/2000 with the /B option to consolidate multiple Performance SeNTry collection data files into one large file for processing:

COPY /B C:\NTSMF24\data\Previous\*.SMF C:\NTSMF\Daily\NTDaily.SMF

The /B option of the COPY command concatenates multiple files without terminating each file with a x’3F’ end of file character. In the above example the file named NTDaily.SMF is associated with the MXG NTSMF input file name, for example. To process this data file in MXG, you would code a SAS language FILENAME NTSMF specification that references C:\NTSMF\Daily\NTDaily.SMF if you are running SAS under Windows NT/2000.

If you are copying files to send to an MVS mainframe for processing using SAS, use a combination of /A and /B switches, as follows to prevent embedded EOFs in the concatenated file:

copy /a f1.smf+f2.smf+f3.smf bigfile /b

Using the /a will cause the EOF mark to be removed from each of the source files f1.smf through f3.smf as they are concatenated into the destination file bigfile. the /b following the destination file bigfile ensures that an EOF mark is not appended to bigfile and subsequently transmitted to MVS.

More generally, to create a concatenated file usable under MVS:

copy /a *.smf bigfile /b

Using ftp

If you need to send daily .smf files to a non-Windows host machine for processing or the machine you need to transfer the files to is separated by a firewall, you may need to use the ftp utility to copy the files. Here is an example of a simple .bat command file that utilizes ftp:

COPY /B C:\NTSMF24\Data\Previous\*.SMF C:\NTSMF24\Data\Consolidated\Consolidated.SMF

ftp -v -i -s:Transfer.ftp

exit

The following example script connects to a remote server, uses a valid Username and Password to log into the remote system. Once logged unto the remote system, the program changes directory (CD) to the remote directory C:\Consolidated and sends the file from the originating folderC:\NTSMF24\Data\Consolidated\Consolidated.smf to the remote file Consolidated.smf on the remote server. Both origination and remote servers have to have FTP enabled, with common Usernames and Passwords defined. In the script, replace the keywords Remote, Username, and Password with parameters that are valid for your environment.

open Remote

Username

Password

cd C:\Consolidated

send C:\NTSMF24\Data\Consolidated\Consolidated.SMF Consolidated.SMF

Bye

You can also using ftp wild cards to transfer multiple files in a single command execution stream. The easiest way to do this to create an ftp script that uses the mput subcommand to send multiple .smf files contained in the \Previous folder.

CD C:\NTSMF24\DATA\PREVIOUS\

ftp -v -i -s:C:\FTPROCESS\transfersmf.txt

exit

where transfersmf.txt is a text file containing the following ftp script:

Rem Transfersmf.ftp sends multiple files to a remote central server

open Remote

Username

Password

prompt

mput *

Bye

The prompt subcommand turns off ftp interactive mode. The * wildcard filespec means that all data files located in the \Previous folder will be transferred.

Invoking the Summarization utility

The Summarization utility is designed to run at Cycle End, although it can also execute in a standalone environment. Letting the Summarization utility perform aggregation of the detailed collection files, you can distribute a good deal of the load involved in post-processing .smf files across your network.

In this example, the Summary utility is invoked in the Cycle End command script after a step that consolidates multiple detail files, prior to invokingftp to transfer the .sum.smf summarized file to a consolidation folder on a remote server:

copy /b C:\NTSMFv24\Data\Previous\*.SMF C:\NTSMFv24\Data\Previous\Allprevious.smf

rem Next run summarization rtn to summarize in 60 minute intervals

cd ..

dmsumsmf.exe -fAllprevious.smf -s60

rem Next execute FTP transfer bat file to upload file

ftp -v -i -s:Transfer.ftp

exit

where Transfer.ftp consists of the following script:

open 10.0.0.7
ftpguest
password
cd transfersmf
send
C:\NTSMFv24\Data\previous\Allprevious.sum.SMF
Allprevious.sum.SMF
Bye

Historical files

You can also take advantage of a roll-up feature in the Summarization utility to create and maintain historical data files that can streamline your reporting procedures. To use the roll-up feature, concatenate the current data file with the previous historical data file. When you invoke the Summary Utility on the combined file, specify the number of consecutive days worth of data that you want to keep. Any data older than the number of days you specify will be dropped from the summarized output file. Using the Summarization utility, you can create and maintain as many historical files as you want, as the example below illustrates. The sample script below maintains a daily historical roll-up file that spans the last 14 days, a weekly roll-up that spans 26 weeks, and a monthly roll-up that spans the last two years.

At Cycle End, issue the following command:

summary-example1.bat @LastDataFile:p @StartupDir

where summary-example1.bat is coded as follows:

rem Summay utility example

rem Cycle End command: summary-example1.bat @LastDataFile:p @StartupDir

rem Maintains daily, weekly, and monthly history files

rem Daily roll-up file: keeps 14 days of history, summarized to 10 minute intervals

rem Weekly roll-up file: keeps 26 weeks of history, summarized to 1 hour intervals

rem Monthly roll-up file: keeps 24 months of history, summarized to 4 hour intervals

rem ******************************************************************

rem

rem Utilizes the following Directory structure:

rem \data\

rem – \Archive

rem – \Backup — backup History files first

rem – \Current

rem – \History — current History roll-up files

rem – \Previous

rem – \Work — for temporary files

rem

rem summarize the contents of the \data\previous folder — yesterday’s smf data files

copy /B %1\*.smf %2\data\Work\%COMPUTERNAME%.combined.smf

%2\dmsumsmf.exe -f%2\data\Work\%COMPUTERNAME%.combined.smf -k2 -H”%2″ -s10

rem Delete old backups

del del %2\data\Backup\*.old.sum.smf

rem Backup History files

ren %2\data\Backup\%COMPUTERNAME%.daily.sum.smf %2\data\Backup\%COMPUTERNAME%.daily.old.sum.smf

ren %2\data\Backup\%COMPUTERNAME%.weekly.sum.smf %2\data\Backup\%COMPUTERNAME%.weekly.old.sum.smf

ren %2\data\Backup\%COMPUTERNAME%.monthly.sum.smf %2\data\Backup\%COMPUTERNAME%.monthly.old.sum.smf

copy %2\data\History\*.smf %2\data\Backup\

rem

rem Keep 14 days worth of Daily data

rem

rem Step 1: move the Daily summary file to \Work

move %2\data\History\%COMPUTERNAME%.daily.history.sum.smf %2\data\Work\

rem Step 2: Concatenate yesterday’s data with the current Daily history roll-up

copy /B %2\data\Work\%COMPUTERNAME%.combined.sum.smf+%2\data\Work\%COMPUTERNAME%.daily.sum.smf %2\data\History\%COMPUTERNAME%.daily.smf

rem Step 3: Run the Summarization utility on the combined file

%2\dmsumsmf.exe -f%2\data\History\%COMPUTERNAME%.daily.smf -k14 -H%2 -s10

rem

rem Keep 26 weeks worth of Weekly data

rem

move %2\data\History\%COMPUTERNAME%.weekly.sum.smf %2\data\Work\

copy /B %2\data\Work\%COMPUTERNAME%.combined.sum.smf+%2\data\Work\%COMPUTERNAME%.weekly.sum.smf %2\data\History\%COMPUTERNAME%.weekly.smf

%2\dmsumsmf.exe -f%2\data\History\%COMPUTERNAME%.weekly.smf -k182 -H%2 -s60

rem

rem Keep 24 months worth of Monthly data

rem

move %2\data\History\%COMPUTERNAME%.monthly.sum.smf %2\data\Work\

copy /B %2\data\Work\%COMPUTERNAME%.combined.sum.smf+%2\data\Work\%COMPUTERNAME%.monthly.sum.smf %2\data\History\%COMPUTERNAME%.monthly.smf

%2\dmsumsmf.exe -f%2\data\History\%COMPUTERNAME%.monthly.smf -k730 -H%2 -s240

rem

rem Clean up Work & History folders

del %2\data\Work\*.smf

del %2\data\History\%COMPUTERNAME%.daily.smf

del %2\data\History\%COMPUTERNAME%.weekly.smf

del %2\data\History\%COMPUTERNAME%.monthly.smf

Using WSH scripts

Windows Script Host (WSH) scripts written in either VBscript or Javascript provide a flexible alternative to shell scripts. You can specify that a WSH script be launched at Cycle End. Launching WSH scripts at Cycle End requires the 2.4.7.3 version of the collection agent.

The Script Center on the Microsoft TechNet web site is an excellent resource for anyone that wants to learn about WSH script basics. If you have access to the Microsoft Developer Network (MSDN) web, you can access many additional resources for developing WSH scripts.

WSH technology is built into Windows 2000 and above, so there is no extra software you need to start developing WSH scripts. Utilizing built-in scripting objects like Wscript, WshShell, WshEnvironment, and WshArguments, your script can execute other programs, access environment variables and retrieve script arguments. A file system scripting object allows your script to manipulate folders and files. Finally, using WMI, your script can access any of the configuration information that is accessible in the vast WMI repository.

WARNING: When you specify the execution of a WSH script at Cycle End, make sure you specify the //B runtime parameter when you invoke wscript.exe or cscript.exe. The //B runtime parameter runs your script in batch mode, suppressing all screen output and keyboard interaction. If you do not specify the //B runtime parameter and your WSH script issues a Wscript.Echo statement that writes to the display or initiates any other interaction with the screen, your script will hang.

For example, specify the following Cycle End command in your DCS parameters:

Wscript.exe //B “@StartupDir\CycleEndscript1.vbs” /datapath:@LastDataFile:p /rootdir:@StartupDir

For tips on configuring the collection service in order to debug a Cycle End WSH script, see the section entitled Debugging WSH scripts below.

The following VBscript example illustrates the functions typically performed in a Cycle End script using WSH. It performs the following processing steps:

It performs some string manipulation of today’s date to create a unique file name.
It validates the arguments /datapath and /rootdir arguments that are passed to it from the command line invocation.
It calls the NtDACmd utility to update the NTSMF PDB Sql Server database.
It then uses the Copy command with the /B option to consolidate multiple files into a single daily file, embedding the current date into the consolidated file’s name. (The file system scripting object has no comparable function.)
It calls the DmSumsmf utility to create a summarized version of the consolidated daily data file.
Finally, it copies the consolidated daily data file somewhere — Step 6 is the section of the Cycle End script that requires customization for your environment.

VBscript comment statements — any statement that starts with a ‘ character – are shown in an alternate color in this listing.

‘ VBScript source code
‘ Creates consolidated daily .smf file in /Previous folder
‘ Creates summarized daily .smf file from consolidated file
‘ Calls NtDACmd to initiate an NTSMF PDB database update
‘ Finally, copies the summarized file to a designated location
‘
‘ The script accepts two parameters:
‘ /datapath — the fully qualified path to the \data\Previous\ folder
‘ where yesterday’s .smf collection files are located
‘ /rootdir — the fully qualified path to the NTSMF root folder
‘ where the DmSumsmf and NtDACmd utilities are located
‘ Sample invokation at CycleEnd:
‘
‘ Wscript.exe //B “@StartupDir\CycleEndscript1.vbs” /datapath:@LastDataFile:p
‘ /rootdir:@StartupDir

‘Step 1 — Create a unique file name

Dim CreatedTodayDate
Dim CreatedTodayDateAsString
Dim EditDate ‘mmddyy format from standard Date function in VBscript
CreateTodayDate = Date
CreatedTodayDateAsString = CStr(CreateTodayDate)
‘Wscript.Echo “Today’s Date String is ” & CreatedTodayDateAsString

dim datestrlen
datestrlen = Len(CreatedTodayDateAsString)

dim loc
loc = InStr(CreatedTodayDateAsString,”/”)
EditDate = Left(CreatedTodayDateAsString,loc-1)
if len(EditDate) = 1 then
EditDate = “0″ & EditDate
End if
CreatedTodayDateAsString = Right(CreatedTodayDateAsString,datestrlen-loc)
datestrlen = datestrlen – loc
loc = InStr(CreatedTodayDateAsString,”/”)
EditDate = EditDate & “-” & Left(CreatedTodayDateAsString,loc-1)
CreatedTodayDateAsString = Right(CreatedTodayDateAsString,datestrlen-loc)
datestrlen = datestrlen – loc
EditDate = EDitDate & “-” & CreatedTodayDateAsString
‘Wscript.Echo “EditDate is ” & EditDate

‘Step 2 — Validate the arguments passed to the script
Set WshShell = CreateObject(”Wscript.Shell”)
Set objEnv = WshShell.Environment(”Process”)
LogonServer = objENV(”COMPUTERNAME”)

Dim namedArgs
Dim PrevFolder, RootFolder
Dim objFSO1,objPreviousFolder,fc1,allthefilenames

Set objFSO1 = CreateObject(”Scripting.FileSystemObject”)
Set namedArgs = WScript.Arguments.Named
‘WScript.Echo “datapath specified: ” & namedArgs.Item(”datapath”)
‘WScript.Echo “rootdir specified: ” & namedArgs.Item(”rootdir”)

PrevFolder = namedArgs.Item(”datapath”)
RootFolder = namedArgs.Item(”rootdir”)

‘Change the next assignment statement to generate a different name for consolidated data file
DailyFileName = PrevFolder & “\” & LogonServer & “.” & _
“consolidated_daily_filefor” & “.” & EditDate & “.smf”

‘WScript.Echo DailyFileName

If objFSO1.FolderExists(PrevFolder) then
Else
‘WScript.Echo “Invalid datapath argument: ” & PrevFolder
Return
End If

If objFSO1.FolderExists(RootFolder) then
Else
‘WScript.Echo “Invalid root directory argument: ” & RootFolder
Return
End If

‘Step 3 — Call the NtDACmd to update the NTSMF PDB database

command = RootFolder & “/” & “ntdacmd.exe -d ” & PrevFolder & ” -threads 2″
Dim execCommand2
Set execCommand2 = WshShell.Exec(command)

‘Wscript.Echo execCommand2.StdOut.ReadAll

‘Step 4 — Execute the Copy coammnd to consolidate the daily .smf data files

dim copyspec
copyspec = “”"” & PrevFolder & “\*.smf” & “”"”
‘Wscript.Echo “Copy .smf Files in Previous Folder to: ” & DailyFileName

dim command
command = “cmd.exe /c ” & “”"” & “copy /b ” & copyspec & ” ” & DailyFileName & “”"”
‘Wscript.Echo “Copy command string: ” & command

Dim execCommand
Set execCommand = WshShell.Exec(command)

‘Wscript.Echo execCommand.StdOut.ReadAll

‘Step 5 — Execute the DmSumsmf utility to summarize the daily .smf data files

command = RootFolder & “/” & “dmsumsmf -f” & DailyFileName & ” -h” & RootFolder
‘Wscript.Echo “Dmsumsmf command string: ” & command
Dim execCommand1
Set execCommand1 = WshShell.Exec(command)

‘Wscript.Echo execCommand.StdOut.ReadAll

‘Step 6 — Finally, copy the consolidated summary file somewhere ===>
‘Careful. Multiple sum.smf output files may be created
command = “cmd.exe /c ” & “”"” & “copy ” & PrevFolder & “\*.sum.smf \\STPUTL04\NTSMF” & “”"”
‘Wscript.Echo “Copy command string: ” & command
‘Dim execCommand3
‘Set execCommand3 = WshShell.Exec(command)
‘Wscript.Echo execCommand3.StdOut.ReadAll

Debugging WSH scripts. An old fashioned, but effective, way to debug WSH scripts is to use the Wscript.Echo method to write to the screen to trace the flow of program execution. This sample script is laced with Wscript.Echo statements that were used in debugging, but subsequently have been commented out. In order to view the output from Wscript.Echo statements while you test your script during Cycle End processing, you must

	remove the //B parameter that suppresses interactive output from the Wscript invocation statement on the Cycle End command line,
	allow the Performance SeNTry collection service to interact with the desktop (as illustrated below), and

restart the Performance SeNTry collection service after you have applied this Service Control Manager parameter change so that it can take effect.

This will allow you to view all Wscript.Echo output when the script executes during Cycle End command processing.

When you execute Wcsript.exe or ccript.exe to run your script, the //B parameter indicates batch execution mode. Batch mode suppresses command-line display of user prompts and script errors, which is important when you run a script during Cycle End command processing. Without the the //Bparameter, your script will hang, waiting for user input if you execute a Wscript.Echo statement that writes to the screen.

WARNING: When you have finished debugging your script, make sure you restore the //B parameter to the script invocation command line before you distribute to a production server. If you do not allow the collection service to interact with the desktop, as illustrated above, any attempts to write to the screen using Wscript.Echo statements will be invisible. In this case, not only will the script hang, but you will not be able to tell that it is hung.

The Demand Technology FAQ » SMF Files

Will you please explain Cycle End Processing?

Processing multiple files.

Using ftp

Invoking the Summarization utility

Historical files

Using WSH scripts

What is the format of the Performance Sentry data files?