Memory Leak from Nowhere

[JRL]

I have a compiled script I wrote in Feb 2010 that uses Macro Scheduler's telnet functions to check email accounts we have set up for our shop employees. It runs 24/7 on a dedicated computer. Out of the blue last week it stopped working. Killed the process and restarted it and it ran for two days and stopped working again. This time I noticed that Task Manager reports it is using 2 Gig of memory. On restart it is using 25 Meg of memory. The telnet processes cycle every 10 minutes. As I observe Task Manager I see the memory usage grow by 7 Meg each time the telnet processes run. 7 Meg times 144 processes per day is about 1 Gig per day which coincides with the program stopping every 2 days when it hits 2 Gig of memory usage.

Since the script has been running for almost 7 years and the Windows 7 automatic updates are turned off, I'm assuming there is something in the mail boxes causing this. Visually inspecting a few of them shows me nothing. The accounts are limited to only internal email and I don't see anything out of the ordinary. There's about a hundred mail boxes to go through and I have no clue what I'm looking for.

Yesterday I wrote a scheduled script that shuts down the process then restarts it every 24 hours, so I have things functional and working again. But I'm still curious about why this is happening. Anyone have any thoughts on how I could diagnose this? OR advice on why Telnet would grab and hold so much memory?

[CyberCitizen]

I got caught out with something similar to this, and found putting a simple Wait>1 in the script's loop process fixed the issue, otherwise it was processing to quickly and causing the mem usage to blow out.

Not sure if its the same here, but it was an old script as well.

[JRL]

Thanks Cy. I already have short waits between most lines in the telnet section of the script. Raising them all to 1 sec did not help. In any case my question is more generic. This particular issue was resolved by killing and restarting the exe every 24 hours.

[Marcus Tettmar]

If the macro was compiled in 2010 then it will have been compiled with an older version. It might be worth recompiling it with the latest version in case whatever is causing this memory leak has been fixed in the last 6 years. If problem persists we can then trouble shoot.

[JRL]

Thank you for your response.

As I said, already did that. Made longer waits and recompiled and it still consumed memory. At this point I don't have any more time to give this. Though I'd like to understand why the memory consumption is happening, I have a work around so I'm moving on.

Appreciate any effort anyone put into considering this issue.
Dick

[migro]

Hi All,

got similar findings with data processing and want yet open an new thread, when I find this post from Dick.

I used the following system for the test: HyperV-Client with 4 physical cores, 4 GB RAM on SSD-Raid with 2GB/s data throughput, Windows 8.1x64 Pro and MacroScheduler v14.3.04.

When I use the function CSVFileToArray with an file of 4,34 MB with around 225000 entries (contains only two columns with max. 10 digits each - see sample below) it takes 1:15 min. for reading the file into the array and consumes 200MB from system memory. The allocated memory isn't released even when I set "Let>csvData=".

Code: Select all

PatientNumber;PersonNumber
6560141;31129943344
11678;31129940326
11562;31129940154
...

Next sample:
Using DBQuery with the following SQL statement on a MS 2008R2 or 2014 SQL database with around 110000 entries in the table.
The script consumes 572,2 MB of memory, which is also not released when the database connection is closed.
After the query a further dataprocessing is quite impossible, because the script slows extemly down.

The query needs 01:49:38h to run. Running the same statement in SQL Management Studio or sqlcmd it takes only 2-3 seconds.

Code: Select all

   Let>SQL=Select tImgCulIntImgIDPk,tImgCulIntPatIDFk FROM TImageRaw
   DBQuery>dbH,SQL,rsTImageRaw,numRecs,numFields

With other much more complex and nested queries I didn't have this problem. The difference is, that I do not run one select statement against the complete table, but run 100000 or more single queries one after the other. In this case the db connection isn't closed also until all the datasets are processed and the memory consumption and performance is not comparable indeed with running it in Management Studio or similar, but quite OK.

Any ideas?

[Marcus Tettmar]

With both those commands the data is read into memory. The data has been read into script variables. Those script variables will cease to exist when the macro terminates. So only when the macro terminates or if those variables are overwritten will the memory be released.

You're not seeing a memory leak. Just memory use.

[migro]

Hi Marcus,

thanks for the reply.

If I got it right the memory should be released when I set

Code: Select all

Let>CsvData=

Resp. when using DBQuery the return variable for the recordset?