Historical data & DDE export

[gabdem]

Hi everyone! My first post here.

I want to design my own mechanical trading systems but designing a reliable mechanical system in retail forex seems almost impossible. And here's why:

In order to design a reliable intraday trading system one would need, in my opinion, 3 things:

1) Good quality software (I use TradeStation, Metastock Pro, BioComp Profit, NeuroGenetic Optimizer, Trading Solutions and NeuroSolutions)

2) Historical intraday data

3) The ability to feed streaming data in the software in which the system was built.

The first step, buying the software, is easy.

Step 2, obtaining historical data, is not as easy. I strongly believe that in order to use the system in real time one would need data from the broker where the system will be traded. Using a system built on, for example, Dukascopy Data, on an fxcm account, is not a very good solution. If you write a system in EasyLanguage, test it on Dukascopy data, and then apply it in real time on your fxcm platform you will see that, at times, the difference between the two data feeds can be of 20 pips or even more. As a consequence your system can perform very good on Dukascopy datafeed, but not so good on live trading data from fxcm.

So I'm looking for a broker that offers historical intraday data (at least 1 year of 1 min data but preferably more).

The only broker I found is fxcm. Via fxcm's charts plugin it is possible to download intraday data starting in late 2001. However it is a nightmare to download the data because you can load only 300 bars per chart, then export to excel, load another 300 bars chart, export, and so on. I contacted several other brokers asking for historical data, but no one seems to offer hist data. I wonder why. I'm sure they have the data stored on their servers.

I also contacted eSignal and they said they currently offer only 2 months of hist data. Many members of this forum say they system trade using eSignal. My question to them is: how can you build a profitable system using only two months of data? I believe you can't. Curve fitting will make you achieve great results in backtesting but poor results in real trading.

Now let's assume you have historical data from your broker, you write a system in EasyLanguage, test it, and obtain outstanding results. It seems that you are ready to use your system in real trading. But then, you realize you can't use your system in real time because you can't export streaming data from your broker to Omega TradeStation!

It is exactly what happens at fxcm. They offer hist data but no dde streaming export. I asked fxcm and they said they are going to offer a dde export utility in the coming weeks for $40/month. By using this new utility I will be able to export data into Omega TradeStation (via MetaServer RT).

However, until this application is launched, there is no forex broker which offers both hist data and dde export. Metaquotes and MG offer dde export but they do not have hist data.

This is why I say it is currently almost impossible to design and then use an intraday trading system in retail forex. If you have data to test it on you do not have dde for applying it in realtime, if you have dde for applying it in realtime you do not have data to test it on.

So does anybody know a broker which offers both intraday hist data and dde export facilities?

Please give me your feedback on the issues above and have a great weekend!

[dhl]

I successfully use historical data (1 min resolution) in Tradestation to test all of my systems. I've used tradestation for the past 5 years in futures markets and I'm getting into forex. I too have wondered about the data feed differences. I'm almost positive the intraday data I have is from e-signal. I also heard reports that metatrader/metaquotes uses e-signal data (can't verify that but it seems to be the rumor). That much said, my plan if I ever start trading one of my systems, is to run a sort of experiment where I use my brokers data feed via DDE to run realtime in tradestation and then compare the results of one month on that data to what one month of my downloaded data would have been. If they're highly correlated than great time to make millions. If not then back to the drawing board. That's just my 2 cents on the subject.

[chaos]

Gabdem

I use MG Financial Group (forex-mg.com) as a data supplier via a DDE link for my EOD system. I also have their their live data feed hooked into my program, although I don't trade off it.

To get the DDE link, you must at least subscribe to their charting service. You also get it if you open an trading account. They offer data ranging from weekly to tick.

Don't know how many intraday bars are available, but their customer service is prompt.

Good luck,

chaos

[jpygbp]

Hi guys,

This question keeps coming up. I think we should compose a FAQ for some of the questions as people discover solutions. Anyway, here is what I have found over the past three years.

Historic Data: The oldest data that I have found is from www.is99.com/disktrading. It is not perfect and has a few holes in it but it works and can be purchased for about $15 USD per currency pair. It is available in several popular formats. I use it with Omega 2000i for historic testing of ideas.
Example: GBP A0-FX daily starting at 1975 and intraday starting near 05/1997.
The following shows how to import the data:

http://www.is99.com/disktrading/faq/
1. How to import Historical Data into MetaStock?
2. TS2000i does not recognize your FOREX data. How to fix this?
3. Adding FOREX symbols to the GlobalServer Portfolio; Importing FOREX historical data.


Real-Time Data: For 2000i, I use eSignal plus the tools at www.tssupport.com. UniServer is a utility with a unique function. It allows downloading the history data provided by eSignal directly into the GlobalServer. All you need to go is select the appropriate symbols in GlobalServer and historical data for those symbols will be downloaded directly to GlobalServer.

ESignal can be used real time to update your database but the holes have to be downloaded with a tool called Uniserver from tssupport. It is not perfect and has trouble occasionally. Sometimes my database process named “server” will go to 100% cpu utilization after a download.

Another problem is that eSignal sends bid and ask quotes and not trade quotes. Tssupport claims that they will have a fix for this by taking the bid+ask and dividing by two in the future. So you can either see the bid chart, the ask chart or the trade chart. The data from is99 is in trade format.

Omega 2000i is not sold in the US anymore, but you can find it on the net for sale from people getting rid of their old software. Beware of anyone claiming to give you pirated copies of software.

DDE:
http://www.tssupport.com/
MetaServer allows using TradeStartion2000i in realtime with datafeeds that are not supported originally. Presently it provides a steady data reception from the following feeds: DDE stream, Quote.com (QFeed), Tenfore WorkStation, QuoteSpeed, TalNet, Fainex.

Other vendors that I have used for realtime data and system design:

http://www.infotecnet.com/ The webtrader product gives delayed stock data and realtime futures and forex data. The programming language is rather cryptic but it works. It can give alerts, post up or down arrows, display text messages etc. The product will crash from time to time but is easy to use. It also has DDE export of prices.

http://www.aspenres.com/ This realtime vendor will actually write code for you as part of the support. The system can support multiple monitors, trading systems, has tic volume for forex etc.
They support either Comstock or Bridge for realtime data. Also a DDE platform.

[gabdem]

Thanks for your ideas!

dhl: I too have tried this. Building a system on hist data from http://www.is99.com/disktrading/ (as jpygbp said it is the oldest intraday data source available) and then testing it on a broker's live data feed.

It does not work. Some stops or limit orders will be hit on one data feed and not on the other. Also if you optimize your systems for hist data and then use them on another (live) data feed then the optimization will have no effect on this new data feed. You will only get similar results on eod data. But if you use 1 min data for your systems (I do too) you will almost certainly get different results on the two data feeds.

So back to the drawing board you say. Yes but if your broker doesn't provide hist data your drawing board will remain quite blank.

In february 2003 I opened a demo account at CMS and started downloading 1 min data from Visual Trading in order to see if my system will produce different results on CMS data. Now, a friend of mine who has a live trading account at CMS tells me that VT live trading data is different from their demo data. He sent me a file with CMS's live data and I compared it with the demo data I have. Sometimes the difference is 25 pips !?! So the demo data I downloaded daily starting in february is worthless.

chaos: MG Forex indeed offers dde. But you can download only 500 bars of hist data. The system I am designing uses 480000 bars of 1 min data (downloaded from FXCM). But as I said, currently FXCM does not offer dde. So I can't apply my system in real time.

jpygbp: I am using http://www.is99.com/disktrading/ and indeed it goes far back. But I compared their datafeed to fxcm and cms hist data and, at times, there are HUGE differences. So a system designed on DiskTrading data will have many chances of being unprofitable on fxcm data.

I considered eSignal. But one of their salespersons told me that they offer only 2 month of hist intraday data. Maybe you can tell me if this is true or not. 2 months of hist intraday data is not sufficient for designing a profitable system.

[bhale]

Download a free copy of metatrader. Set up a demo account. My copy only has EURDUSD, GBPUSD, USDJPY, and USDCHF. Go oto any of the time frames and rightclick. Choose save and you can save about 8,000 bars of data as comma sep values, which are ascii files that can be read by any software package. This can be used for backtesting. I have data files such as this if anybody wants them. I think the daily data I downloaded goes all the way back to the 80's. The software also has a history feature where you can do the same thing.

[gabdem]

bhale thanks for your suggestion.

I need at least 400000 - 500000 bars of 1 min historical data (1 - 2 years). 8000 1 min bars is very little.

[fk001]

I am a futures trader but no FX.

Just want to backtest some things.

Can somebody send me some historical data EURUSD in globalserver format.

Thanks in advance.

pankebo@pandora.be