$spx data: gap problem?

[lorn]

hi there,

was wondering if there might be an issue with the opening/closing prices of the $spx from around late '99 to early 2001. there are hardly any gaps: open being <> close of previous day. i don't have any other trustworthy $spx data to compare it to, but in relation to the preceeding and following years it definitely raises an eyebrow. interestingly, and perhaps too coincidentally, the missing gaps start right after the doji problem, outlined in the thread below, ended:

http://forum.esignalcentral.com/showthread.php?s=&threadid=5118

weird huh? where do you guys (esignal) get your cash data from, particularly the $spx? i'm going to have to buy this data from somewhere else and i just want to make sure it's not the same source.

if anyone finds the data to be correct (corroborated with another reputable source), let me know.

sincerely,
lorn

[JayF]

I don't see any obvious errors or anomalies in the data. Could you provide some examples?



FYI, our data for the $SPX is sourced direct from the CBOE.

[lorn]

hi jay,

thanks for the quick response. like you said, it's not obvious when looking at the charts, but when you compare the gap frequency in relation to time you get a better picture of the possible problem.

below is a graph of the number of gaps, up or down, between the beginning of '96 to the present. it's not to scale, but it gives an idea of how there was definitely a stall at the end of '99 to mid '01.



the red flag for me was the beginning of the possible gap problem directly coincided with the end of the previous doji problem. since both problems deal with open/close issues and happen at or around the same time, it isn't too much of a stretch to think that this isn't just an anomaly.

was the previous doji issue a data corruption issue on the esignal side, or was that how the data was delivered from the cboe? from my experience, just because it comes from an exchange doesn't automatically imply quality. different exchanges could have vastly different protocols for quality control, ranging from just saving to disk to triple redundant crosschecking. at the end of the day, though, it's the data that tells the story.

lorn

[JayF]

The earlier problem was caused by some data corruption, and was corrected by importing data from another data source.

For an unknown reason, many of the Open prices are equal to the Close of the Previous Day within the date range you mentioned for $SPX. Most likely this was caused by a similar situation to the last problem. I checked a couple sources including the CBOE exchange, and none have the Open data available (or the Open is equal to the Prev. Close,) so we won't be able to address this issue at this time.

[lorn]

thanks jay, that's too bad and also a shame.

if anyone knows of a source whose data is accurate, please post it here. i'm going to check with csi.

lorn