MonitorWare Knowledge Base

We have configured a File Monitor to collect data produced by one of our applications. The application produces one file each day, where the date is part of the file name (e.g. Log_YYMMDD.txt). The File Monitor is configured with Log_%y%m%d. txt as file name.

Thursday and friday the Monitoragent did a fine job delivering the data. On saturday and sunday our application did not produce any log file (because there was simply no events to be logged). On monday a new log file was produced but this data was not delivered to the central database!!!

Some details: We could see mondays log file locally, but its contents were not delivered. We then manually created a new file for saturday using a text editor. As effect, mondays events (but not our manually added saturdays events) was delivered to the central database. After this everything was back to normal, i.e. subsequent events were promptly delivered (but we expect only until the next weekend). Note: The MWAgent was NOT restarted.

Suspicions: The local MWAgent was stubbornly trying to open saturdays log file disregarding any newer files or changes of date.

Please provide us with more information on how the File Monitor works:
1) Is it necessary to produce a log file each day even if there is nothing to be reported?
2) Exactly how does the current date affect which file that is read? Is there some kind of "time window" for when a log file must be produced in order to be read by the MWAgent? E.g. why did saturdays data not appear? A concern here is that no data must be lost. e.g. if produced near midnight. Will the MWAgent stop reading yesterdays file exactly at 00:00:00?
3) What happens if the date/time is changed in the system? Isn't there a risk that large time changes will hang the file monitor in the way described above?
4) What happens the first time the MWAgent is started? Will the File Monitor read all previous files up to the current date? (Once again, why were saturdays events not delivered?)

We are using MWAgent version 2.2 (Professional Workstation). No license for 3.x.

Dear Mats,

many thanks for contacting Adiscon support. What you describe is actually a "little" strange. At least I've never seen this. Anyhow, let's get on to your questions, first:

Mats Hjelte wrote:1) Is it necessary to produce a log file each day even if there is nothing to be reported?

Well, as it is designed, that should not be needed (and wasn't needed as far as I know).

Mats Hjelte wrote:2) Exactly how does the current date affect which file that is read? Is there some kind of "time window" for when a log file must be produced in order to be read by the MWAgent? E.g. why did saturdays data not appear? A concern here is that no data must be lost. e.g. if produced near midnight. Will the MWAgent stop reading yesterdays file exactly at 00:00:00?

The agent generates a new file based on the template you provide. Then, it compares that file name to the file name of the file that is currently being processed. If these two do NOT match, then one run is made over the previous file and all data in it is reported. Only after that the new file is opened and processed. The next time the agent checks, the file names (previous and current) are again identical. Thus, the agent processes the current file from where it was left.

Mats Hjelte wrote:3) What happens if the date/time is changed in the system? Isn't there a risk that large time changes will hang the file monitor in the way described above?

No, a hang does not occur (except, of course, if the same bug you experience bites...). However, if you change the data from e.g. friday to sunday, the saturday file will NOT be processed, because the previous file is then friday and the new file is sunday. So the previous one is processed, and then the current one. Saturday is never being generated as a name and thus not being processed. A key to understand this behaviour is that filemon is not working on calendar dates - it is just working on paresed template name and the resulting strings. In all real-world scenarios, this is no problem, because date changes of that magnitude do not happen.

Mats Hjelte wrote:4) What happens the first time the MWAgent is started? Will the File Monitor read all previous files up to the current date? (Once again, why were saturdays events not delivered?)

No, not at all. It knows it had no previous file and so it processes just the current one.

This all, however, seems to boil down to a program bug in your environment. Would it be possible that you enable debug logging (maximum level) and send us the resulting output text file. We can use that to track down what happens. Please submit the file to support@adiscon.com. Please ZIP it, as these files tend to become large. Debug logging can be configured in the general settings.

Looking forward to your reply,
Rainer Gerhards
Adiscon

Thank you for a very swift reply!

I agree with you that the "real" behavour we are experiencing does not look líke the "designed" behavour you have described. Looks like a bug - but not necessarily environment related. I will try to produce the detailed debug output you requested.

Some comments to your answers:

3) No, the MWAgent does not "hang". What I meant by "hang" was only that it refused to process mondays file unless saturdays file was provided.

4) Assume the application has produced a whole bunch of files before the MWAgent was started the first time. Wouldn't it be appropriate to collect all previous files instead of just collecting todays file? (Else data will be lost).

New: 5) What happens if contact is lost (e.g. no contact with SETP server) for say two days? Based on the description you have provided, I suspect there is a risk of data loss when the MWAgent moves on to a new date. Will the files produced while contact was lost be transmitted? My suspicion is that when contact is re-established, the MWAgent will skip directly from the currently read file (e.g. day1) to todays file (e.g. day3 - file name according to template) causing day2 to be lost.

In my humble opinion your "file name template" solution is too simplified. How can you possibly solve this problem without checking which matching files exist?

Mats Hjelte wrote:I agree with you that the "real" behavour we are experiencing does not look líke the "designed" behavour you have described. Looks like a bug - but not necessarily environment related.

Just to make sure I do not cause confusion: I do not intend to say there is something wrong with your environment. I just think this bug needs a specific environment to show up. I am pretty sure it is a program bug.

Mats Hjelte wrote:3) No, the MWAgent does not "hang". What I meant by "hang" was only that it refused to process mondays file unless saturdays file was provided.

OK, looks like we have the same perception of "hang"

Mats Hjelte wrote:4) Assume the application has produced a whole bunch of files before the MWAgent was started the first time. Wouldn't it be appropriate to collect all previous files instead of just collecting todays file? (Else data will be lost).

It depends. The way the file monitor is designed, this is not its task. You may argue that it might make sense to have a more sophisticated file name generator. One that actually looks at the dates and then computes all file names in between. This is a different beast than what we currently have in software. I agree this might be desirable and we will look in this issue - but this is neither advertised nor designed behaviour for the current builds.

Also, many customers do not like to receive the preexisting data. But, agreed, an option would be nice

Mats Hjelte wrote:New: 5) What happens if contact is lost (e.g. no contact with SETP server) for say two days? Based on the description you have provided, I suspect there is a risk of data loss when the MWAgent moves on to a new date. Will the files produced while contact was lost be transmitted? My suspicion is that when contact is re-established, the MWAgent will skip directly from the currently read file (e.g. day1) to todays file (e.g. day3 - file name according to template) causing day 2 to be lost.

You are right, in such a case the one day worth of data would be lost. This is one reason why I said above we think about another file name generator.

Mats Hjelte wrote:In my humble opinion your "file name template" solution is too simplified. How can you possibly solve this problem without checking which matching files exist?

Practice so far shows that the approach works well in all installations we know about.

So for now I would like to see what exactly causes your problem. Thanks for your cooperation with that.

Best regards,
Rainer Gerhards

To enable debug output, I had to restart the MWAgent. This caused "todays" data to be sent, i.e. the error situation was no longer present. It might take som time to recreate the situation. But you will hear from me later this week.

About a better file name generator, generating the in-between dates: Good idea since it solves the problem discussed in 5). However, to cope with the problem discussed in 4) checking for matching files is probably a better solution.

Sorry if i was too direct in the "In my humble opinion.. " part: I do not wish to argue about what works well in other installations. Just trying to solve MY problem.

By the way, I really like the way you openly discuss your product!

Best regards,
Mats Hjelte