Heap / Stack Help

[Gavishti]

Can someone help me out on determining appropriate Heap and Stack settings? When I run tests on my large efs (1000+ lines) for a month of data on 1 minute charts it does fine. Quick turn-a-round. About a minute. When I go to the 6 month test it bogs down and takes a lot longer. Like a half hour, maybe a bit more. It seems that increasing the heap and stack settings would help, but I don't know enough about the memory model to make informed decisions. Just taking stabs in the dark at this point.

Currently heap and stack are set at 24576K / 64K. I have 1 GB of RAM so could increase without problem. The Task Manager bounces around the 160,000-190,000K area most of the time in these long tests. At the end it bounces up to about 240,000K when it starts plotting the results.

Any help is appreciated.

[gspiker]

Gavishti,

Unless you create array that can grow as the amount of data you iterate through gets larger, I don't think heap or stack will be an issue.

I have talked with a developer about this a long time ago, and he didn't think there would be an issue either.

I do know that there was a bug that when you ran out of heap, you would see odd errors. At that time I was told this would be fixed to print errors instead warning that the heap should be increased in size. I have never gone back and verified that this is what indeed happens.

I also know that the engine does garbage collection, so that may address some of your worries about memory usage.

Garth

[Gavishti]

Thanks Garth.

My arrays are all numerical. The two largest are 13 elements. Most are around 4 elements. Even considering that the builtin studies might use arrays and that some stuff might default to doubles I can't come up with more than 2K for all variables. It's probably less than half that. I've checked all the arrays and know that they are not overflowing.

The data is sitting in OS cache. All that I have been able to think of that can be a problem is how eSignal is handling the data. The big clue was the huge difference in processing time between 20 days and 120 days of data. So, I'm guessing that eSignal is trying to cache the data internally. Again guessing, I think that 120 days of 1 minute data is about 6 MB. That still doesn't account for the difference in performance.

Restricting to a day or two of data only takes 2-3 seconds to load. Real time is running at about 2 ms according to several tests on the Performance Monitor.

I don't know if any of this rambling helps anybody. But you can at least see that I've been thinking about the problem.

[gspiker]

Gavishti,

For grins, have you thought about doing any of the following:

1) determine if you are system CPU, Memory, Disk or Network bound during these longer operations (I would be surprised if it was network or disk, but I have been surprised by things like this before).

2) If CPU bound, which process (eSignal or the data manager) is using the bulk of the processor time.

3) If network bound, scan for error rate, collisions, number of hops and the latency of each hop, etc to make sure you don't have an networking problem someplace in your path to the data server.

The results of this could be telling. Then again, it may not be ;-)

G

[Gavishti]

Garth,

Thanks for the checklist.

WinSig is pegging my cpu (not the kernel - that peaks at maybe 15-20% and then drops off). The VM appears to be the problem. Over 8 million page faults. For some reason it is not using RAM. Maybe that is just the way XP works. At this point I don't know how to work around it.

I did try bumping up the Heap to 49152/64. Result was a crash. Took it down to 16384/32 and it took 40 minutes to process 120 days. No other applications were running. Over 600 MB of RAM was available. I changed the VM to "let XP figure it out" just before shutting my machine off last night. Will see if that helps.

I have write caching disabled. The performance doesn't seem worth the risk. If read caching is disabled I didn't do it intentionally. Never found an option for it. I am using a 7200rpm drive and have 50% free space. I defrag regularly. No network problems.