Multiple Core Experience

[Paulie]

Has anyone upgraded from a single core to a multiple core? How effective is Qcharts & Qlink/Excel at sharing the CPU load?

I'm running the Esignal 6.0 and of course love the better data, but it taxes my processor. It is a Pentium 4 @3.06G with 1MB L2 cache.

If I go to a dual core 3.0G do you think I'll get about twice the capability?

[tradertom]

I went from an Athlon single core to an Intel quad core and it helped. I didn't see four times the improvement however. From glancing at the processer load, apparently QC6 consumes all of one processor core when it freezes but instead of showing 100% CPU usage, it shows 25% now. The computer still drags a lot when this happens though.

It is still pretty slow on workspaces with lots of actively traded stock symbols on quote sheets which seems to be the big CPU draining factor. Workspaces with few active symbols but complex charts work fine. In fact, it is amazingly fast when I have no quote sheets. I suspect at this point that my problem is not CPU power but bottlenecked data feed (I need a bigger pipe into the house.)

Try this: take a workspace that is giving you problems and temporarily delete all the quote sheets (don't save it!). Do this during the first hour of the trading day, you are likely to see vast improvement in speed. I have found that QC6 works great with my charts and no quote sheets as long as I type in the symbol manually. I have 9 charts on a workspace that have lots of moving averages and studies (StochRSI, MACD, Directional ADX, AutoWave, Pivots)

All that said, I have faith that Jay, Mike, Alex, and company will get it figured out and we will all be amazed at the quality of the application once again!

Good luck,
T

[Paulie]

TraderTom:

I too have noticed a problem with QuoteSheets. I think it is the amount of data being sent, they send it all for each symbol, not just the stuff your displaying and probably each time a field changes.

Without QuoteSheets my processor is fine.

It sounds like to me that aside from QuoteSheets you think there a good upgrade.

Do you have Vista or XP?

I'd estimate QuoteSheets accounts for 50% of my load and 50% goes for everything else in Qcharts and my analytics in Excel/Qlink. This is on 6.0.

[Larry Marchman]

Hi T,

I don't have a multi-core box, and can't speak from actual experience here, so please bear with me.

From what I've read, and discussed with other multi-core users, my understanding is that one of your 4 processors fully pegged will show as 25% of your total %CPU. Two processors maxed would show as 50%, three 75%, and all four processors maxed would show as 100% total %CPU utilization.

What I heard you say in your report is that you're seeing a max of 25% for the QCharts process. If I heard you correctly, my concern goes to whether or not QCharts is taking advantage of multi-core technology. What I heard you say sure sounds like QCharts is confined to run in one processor rather than spreading itself across multiple processors. I would hope that you would be able to see more than a total of 25% max for the QCharts task/process, what with 25% being a total that's 1/4 of your total horsepower. I sure would like to be disabused of the notion that QCharts possibly isn't spreading itself equitably across processors. I hope there's something I'm missing here due to lack of hands-on actual experience with a multi-core box.

Regarding your comment on the %CPU incurred in servicing the quotesheet vs charts....
Yes indeedy, that's been my experience with 6.0 having rather heavily tested that notion in comparing 6.0 with eSignal 8 and 10. But that's not to say that there are not improvements that could be made to QCharts to more efficiently handle the GUI load. In 5.1, my GUI load is the vast vast majority of the load. But that's another subject.

Regarding the size of your pipe as possibly being problematic...
I actively monitor the data flow into and from my box moment to moment with DU Meter . It has a neat configurable small display that I run always-onatop. In my environment, when I peg my CPU with QCharts, I have just gobs and gobs of pipe capacity left over...on the order of 90% left over and available with my 7Megabit rated Cable service. When QCharts is pegged, it doesn't have the time to ask for more data than it's currently processing, that it's already in over it's head with. Kind of a self-limiting governor on data requests. I peg QCharts more easily and with less data demand than other folks due to differences in my horsepower vs their monster boxes. So, this type of measurement must be singular to a specific environment, a specific box and workspace. Bottom line is that if you have a good broadband service, I'm questioning whether or not you are running out of pipe capacity. Actually, I doubt it.

LAM

[tradertom]

Hi Larry,

I am not saying that QC does not utilize multiple processors efficiently; I haven't really done enough analysis to determine this. It is just an observation of processor utilization during periods of high activity (like equity market opening hour). I use various applications to determine this (Task Manager, Process Viewer, Process Explorer, etc). They show a different chart for each processor and when I watch these charts with QC running, I see one chart pegged at the top and the other three much lower. As I said, I have not done a thorough analysis... just an observation. I have attached a pic of the Task Manager Processor charts. Even here at 3:15am on a holiday morning there is a definite bias toward one processor with QC6

The quad core Intel is a faster machine than the single core Athlon, but the performance improvement in QCharts 6 was not as dramatic as I had hoped. Oh, I use XP Pro by the way. XP64 may be more efficient with multiple processor utilization but I was concerned with finding the right drivers for my hardware.

As far as my pipe goes... I have had issues with my ISP anyway; they are not as responsive as I'd like. I used to get ping times to the QC5 servers of 65 or so. Over time this has decreased to 140 or more. They have sent people out but no improvement in ping times. So whether or not my pipe capacity is an issue, I am considering changing broadband providers anyway. If I do this, I will let the group know any perceived change in response at that time.

Thanks,
T

[tradertom]

Image is attached here... ( I hope)

T

[Paulie]

I agree with LAM here on the pegging issue. (If pegging means the processor falling behind and kind of freezing.)

I watch the memory usage and if I start missing the cache to any great degree, I'll peg pretty soon there after. I've done real time programming and it seems to me that pegging occurs when the processor has too long an interupt queue and can't resolve which to handle, sort of a deer in the heedlights kind of event. There is a substantial overhead in switching between interrupts, so that overhead increases the processor inefficiency and the queue lengthens, eventually collapsing.

Thanks for the comments though. I'm getting the sense that it is worth upgrading. I think I'll have my son build me a new computer.

[ckelly]

Larry, Tom, Paulie -

Here's one other sample to add to your comparisons. I've got a dual core machine and ran a quick test with the same workspace in 5.1 & 6.0.

fyi,
Carol

[tradertom]

Carol,

If I remember correctly, don't you use Windows XP 64? I was thinking XP64 would be more efficient with distributing the work among multiple processors. Thanks for adding your input here. It is always appreciated!

All,

I will say that I am pretty ignorant on this subject from a technical perspective; I don't have any way to measure the processor cache loading (that I know of). I can only speak from my perception of speed and relative nimbleness. The quad core helped but it didn't seem to improve QC6 by a factor of 4.

Thanks,
T

[ckelly]

Tom,
Thanks for your comments I'm happy to help.

Yes, I am running WinXP-Pro x64 Edition. As for being more efficient, I found this info in the FAQ here:

"Q. Will Windows XP Professional X64 Edition support two dual core processors? Isn't that the same as four processors?

"A. Almost the same, but Windows is actually licensed per socket, not per core. So, even though Windows XP Professional x64 Edition is limited to only two processors, it will see the two dual core processors as two processors for licensing purposes, and four processors for running Windows."


I very much understand your frustration/concerns over the speed/nimbleness of QC - and know that Mike is making improvements here. Some of the testing we did last week showed significant improvements in stability and typical speed of populating charts. (I haven't had a chance to run a test on actual CPU improvements yet.) This next 6.0 Beta update should be coming out soon.

Carol

[Larry Marchman]

Tom....(and Carol below)
I'm still concerned about your initial 25% comments.
Can you get QCharts above 25% sustained as shown in the attached screenshot?
And if so, what do the processor graphs look like if substantially above 25% sustained.

One possible measurement test would be to...
1. Create QCharts 10% sustained cpu load and screenshot the 4 process graphs.
2. Add load giving a 20% sustained QCharts hit and screenshot the 4 process graphs.
3. Add more load giving 30% hit and capture.
I hope you get the drift.
It would be instructive to see the processor graphs at differing loads.
This would take preplanning and construction of workspaces, and/or symbols lists, and a defined set of actions to methodically and sequentially take. There are various ways to add more load. Open a higher load workspace, or importing new additional high vol syms into a previously empty quotesheets, and other techniques I'm sure.

Of course, there are considerations and complications to this type of testing. Like time of day the measurement is taken. A workspace that will give you 10% hit at 9:35 a.m. on a market day will not give you a 10% hit at Noon due to the market trade rate differences. And what was 20% one moment can change significantly a moment later as the market trade rate surges or wanes. And whether or not adding load is done via more quotesheet symbols, or by more charts, or a combo of either, and whether or not different threads are involved in servicing quotesheets and charts. And...well, it gets complicated with the caveats, the variables.

but bottom line, for now for me as a prospective buyer of a multi-core box is info on whether or not QCharts is spreading it's load around in a good fashion. Whether or not there's an inherent issue with a thread that's consuming most of the load and for all intents and purposes confining QCharts primarily to one processor. As I understand things, a program can have it's various threads spread to different processors, and a thread can be bounced from processor to processor by the operating system or by programatic assignment of affinity, but a singular thread itself can't be spread, broken into different processors. If this isn't correct, would appreciate some enlightenment here.

Carol...
Your screenshot seemed to indicate a load spread more so than Tom's.
Observing during market hours with a real streaming load would be interesting.
Can you get QCharts at, let's say, 60% sustained on your x64 box?
The same X%, then more %, then more% observations as detailed above would be interesting.

The two sets of observations, a 32bit quad-core and a dual-core 64bit could be instructive.

Again, I don't have a multi-core box to play with, to try this and that, so y'all please bear with my questions.
Thanks,

LAM

[tradertom]

Hey Larry,

Wouldn't Mike or Jay have some idea how QC manages multiple processors (based on the language and the compiler as well as source code?) I will do some data gathering, but it may be next week because I have to report for jury duty tomorrow... no idea what case (if any) I will have to possibly sit on. In any case (no pun intended), I will be away from the quad core machine as my home is 10 miles from the courthouse. If I get booted because I am unfit in one or both of the lawyers' opinions, I will let you know!

My plan, if you agree, would be to open my normal, loaded workspace around the open of the market and take some screen shots of CPU usage and then (probably have to close the unresponding program) open the same workspace with no quote sheets and repeat the screen shots; adding quote sheets until I am back to full complement of my original workspace. Should be interesting to see how this pans out. (Of course, if the charts are set up for me to make a trade, I might postpone the test.) In the mean time, maybe Jay or Mike can respond to the multiple processor/threading processes in the coding/compiler.

Thanks,
T

[tradertom]

Hey Paulie,

I am not really sure what factor of improvement I see from the single core Athlon to the quad core Intel. Besides processor, the chipset, memory, etc has such a huge influence on the performance. Selection of a motherboard is also crucial as I am sure you know. I bought an off-the-shelf solution and in retrospect I am sure I could have done better building my own. (I would probably have used XP64 too.) I wish I had more data on the best configuration hardware to assemble for running QCharts. I recall Bill (qcsoftwarehelp.com) did some work a few years ago with hardware but that was before QC6 and I am not sure how current it is.

It is important for me to have a responsive machine for running QCharts. I can run QC6 with a workspace having no quote sheets, but this is a workaround that I would rather not be forced into doing. If Mike could add a configuration switch that minimizes what data is displayed on quote sheets (net, net since open, last price is all I really care about if pushed to minimize) that would help me alot.

Anyway thanks for kicking off this thread, it is an interesting discussion for many folks here.

T

[Paulie]

Tom:

I'm of the same mind as you, responsive machine and have the same work around, no quote sheets. Too bad the quote sheets give you a quick feel for the market as a whole and your relative position.

You have pretty much convinced me that it is worth the upgrade. I think I'll have my son build me a new machine.

[tradertom]

Larry,

I am officially no longer on jury duty; case was settled out of court. See charts below. I notated them as best as I could but they give a range of scenarios that I thought were useful. Notable is the incredible drain on CPU usage when the mouse moves (2nd chart). Also notable is the bias toward one processor core of QCharts while it is simply idling (idling means I am not actively doing anything or moving cursor.)

My system: Intel Core 2 Quad Q6600 (K) 2.4 GHz (1066 MHz front side bus) with 3GB RAM. Intel P965 chipset on ASUS motherboard. MS Win XP Professional SP2. I run 4 monitors on this system and QC is spread across all four screens.

fwiw,
T