A faster way to Separate

bsharp · Post by **bsharp** » Sun Mar 09, 2008 3:30 am

Example.

Let>filename=C:\post\kat6.apt
ReadFile>%filename%,APTData
Separate>APTData,CRLF,APT
Repeat>k
  Let>APTData=APT_%k%
  Let>k=k+1
Until>k>APT_Count

The "ReadFile" seems fast enough but the Separate function is a killer.
The file is only 1.2MB and around 17,000 lines.
I need to load every line into an array for processing. Is there a faster way to do it?

Bob Hansen · Post by **Bob Hansen** » Sun Mar 09, 2008 2:40 pm

Is this faster?

Separate>APTData,%CRLF%,APT

bsharp · Post by **bsharp** » Sun Mar 09, 2008 3:30 pm

For some reason after around two minutes I get an "out of memory" err with that Bob. But thanks anyway.

pgriffin · Post by **pgriffin** » Sun Mar 09, 2008 6:53 pm

bsharp,

I think you missed Bob's point. Your code with the Repeat> is useless. the Separate> function already places your lines into an array. In the example simply stop at the Separate line and your done.

Try it. Just address any member of the variable with APT_N where N is any number up to the value of APT_COUNT (which is the upper limit of the array).

bsharp · Post by **bsharp** » Sun Mar 09, 2008 8:28 pm

Yea I figured that one out. But the Separate function still seems to slow.

Waiting almost two minutes just seams to long for such a small file. I have around 4 days into this project and do not really want to recode the entire thing in C over something so trivial.[/img]

Me_again · Post by **Me_again** » Sun Mar 09, 2008 8:42 pm

I made a 17000 line file, it's about 1 meg, and the whole code in your first post runs in 20-30 seconds on a relatively slow laptop (uncompiled macro).

bsharp · Post by **bsharp** » Sun Mar 09, 2008 10:11 pm

I zipped up the test file and put it on my web server. If you could unzip it and post your results it would be great. I am running it on a 3.2 GHZ Core2 system with two 10K RPM hard drives?

Me_again · Post by **Me_again** » Sun Mar 09, 2008 10:36 pm

Working on it, I knew I should have saved that code!

Me_again · Post by **Me_again** » Sun Mar 09, 2008 10:48 pm

bsharp wrote:I zipped up the test file and put it on my web server. If you could unzip it and post your results it would be great. I am running it on a 3.2 GHZ Core2 system with two 10K RPM hard drives?

http://www.hellbentpcs.net/kat6.zip

35 seconds
CPU Type Mobile Intel Pentium M, 1600 MHz
Regular laptop drive 4200 rpm (not a fast one).
XP Pro
MS 9.1

bsharp · Post by **bsharp** » Mon Mar 10, 2008 2:09 am

It may be because I am running an older version of MS?
I am running 8.1 Pro.
I just started porting the code to VB6 witch has no problem with it. loads it along with 6 other files in around 2 seconds using this in VB6

Code: Select all

'Read the file into a string
Open FileName For Binary As #1
FileString = Input(LOF(1), #1)
Close
'now split the string into an Array
TheArray = Split(FileString, vbCrLf, -1)

Thanks for the help it was greatly appreciated

Me_again · Post by **Me_again** » Mon Mar 10, 2008 2:26 am

Very strange...

pgriffin · Post by **pgriffin** » Mon Mar 10, 2008 2:31 am

Why not download the eval of v10 (current version) and test again?

I think we would all be very interested in the result.

Post by **Marcus Tettmar** » Mon Mar 10, 2008 2:55 am

It's a bit unfair to compare speeds with a lower level, compiled language. Macro Scheduler is a high level automation tool with an interpreted script language. You'll not get the same performance as with C or even VB. You could use VBScript. Use FileSystemObject and Split.

jpuziano · Post by **jpuziano** » Mon Mar 10, 2008 6:49 am

pgriffin wrote:Why not download the eval of v10 (current version) and test again?

I think we would all be very interested in the result.

Hi bsharp,

I downloaded your file and ran the following against it using Macro Scheduler 10.022 Pro (latest version available):

Code: Select all

Let>aptfilename=C:\kat6.apt
ReadFile>%aptfilename%,APTData
Sec>StartSec
Min>StartMin
Separate>APTData,%CRLF%,APT
Sec>EndSec
Min>EndMin
MessageModal>%StartMin%%StartSec%,%EndMin%%EndSec%

Result: 5206,5525 = 3 min, 19 sec

This was on an older machine: XP SP2, Intel Pentium III processor, 450 MHz, 512 MB of RAM

Here's something interesting, I compiled it to an exe and ran it again and it was slower... didn't expect that.

Compiled Result: 0907,1251 = 3 min, 44 sec

It would be nice to see how fast VBScript could process this. If anyone does that, please post your results.

bsharp · Post by **bsharp** » Sat Mar 29, 2008 7:44 am

mtettmar wrote:It's a bit unfair to compare speeds with a lower level, compiled language. Macro Scheduler is a high level automation tool with an interpreted script language. You'll not get the same performance as with C or even VB. You could use VBScript. Use FileSystemObject and Split.

Because MS is such a powerful automation tool I use it as sort of a design sketch board for all kinds of ideas. Some ideas just outgrow it and just need to be done in a more effective solution. I have had Ideas that started in MS then grew into VB and then into C. I was not trying to say anything bad about MS as I think it is an awesome tool.