Some source code for the previous post

This is the RNN and the data sets from the previous post about the RNN target data fitting.

https://www.dropbox.com/s/blv7zci9p37gvy4/eurusd4.tar.gz?dl=0

https://www.youtube.com/watch?v=ShdmErv5jvs

4dalulz
https://www.youtube.com/watch?v=ShdmErv5jvs

Fitting Pybrain's RNN prediction

After fiddling with some parameters in my original pybrain RNN, (such as the number of neurons, the training data set size and the normalization factor for the target data set) I've been able to produce RNN predictions which fit the shape of the target set, but not it's amplitude, or horizontal and vertical positions.

The following picture illustrates this problem. The blue signal is the RNN target set. The green signal is the raw RNN output for the training set input. The red signal is a crude attempt at fitting the green signal to the blue signal. It's merely a vertical displacement, together with an increase in amplitude, of the raw green signal.

Notice also how the red signal seems to be shifted to the right of the blue one. The following picture shows the prediction for 25 days after the last date in the training set. It has the same amplitude and vertical shifts as the previous picture, but I've added a left-wise horizontal shift of two days to this one.

I think I could get the red and blue signals pretty close with a simulated annealing algorithm which finds the best horizontal, vertical and amplitude shifts for the original raw green signal. I would have to use only the training set information for this, in order to make it useful for real-life prediction.

I think I'm getting pretty close to something useful here. I'll be posting some source code later.

First attempt at implementing recurrent neural network FOREX trend prediction

My first attempts at implementing the experiment from the russian paper with neurolab failed miserably. It just doesn't have all the options I needed. I've decided to switch to PyBrain (https://github.com/pybrain/pybrain/wiki/installation). Note that I've only worked with the EUR/USD trading rate.

Although the results were better, they were not satisfactory. The following plot shows how close the Elman RNN got to the normalized training set. After around 1500 epochs, PyBrain couldn't get any closer. Also, the quality seems to depend heavily on the initial random values of the net.

I did not follow the paper closely. An important step I missed was using the modified logistic function. I tried modifying it in the file named /usr/local/lib/python2.7/dist-packages/PyBrain-0.3.1-py2.7.egg/pybrain/tools/functions.py, but PyBrain's training function failed after the modification.

If you would like to try it yourself, you can find the source code and all the instructions here:

https://www.dropbox.com/s/3v63go1zlfux9l5/euro2.tar.gz?dl=0

All you have to do is install the dependencies and run a couple of shell scripts. The scripts will download the data directly from the european central bank, create and train the RNN, and generate the plots automatically.

My next steps will be to: 1. write a RNN builder and trainer in C from scratch, and 2. try to predict only return rate direction (positive or negative) with PyBrain.

Considering an ARIMA model

I came across a blog post and a paper describing methods for implementing an ARIMA model as a price forecaster. They both deal with the stock market, which is not  exactly the same as FOREX. But given the lack of ARIMA examples involving FOREX, I've decided to take a look at these:

http://programming-r-pro-bro.blogspot.mx/2013/04/forecasting-stock-returns-using-arima.html
http://www.hindawi.com/journals/jam/2014/614342/

From the blog post it seems that ARIMA doesn't have a very good resolution in it's predictions. The most it seems to be able to predict is a likely dynamic range for the future return rate. The paper shows how an ANN predicts fluctuations more closely. Still: this paper uses a FFNN, whereas the russian paper from one of my earlier posts uses a SRN.

ARIMA seems more limited than the SRN in it's predictive power. Therefor, I have decided to give priority to the SRN approcah using the newelm function from neurolab which I mentioned in an earlier post.

The ARIMA implementation, nevertheless, has inspired me to attempt a new rustic buy-and-hold algorithm. One which takes into account the density of price falls through time, and a likely dynamic range for it. I will be reporting on the results later on.

Kalman filter disappointment

I found two identical implementations for a univariate Kalman filter:

http://greg.czerniak.info/guides/kalman1/
http://bilgin.esme.org/BitsBytes/KalmanFilterforDummies.aspx

From the formulae for K_k and P_k, it seems that all this Kalman filter does is to damp the signal as K_k simply decreases with each iteration. Which is not very impressive.

I wasn't able to extract any "predictive" component from this implementation of the Kalman filter. I've decided to desist with this approach.

Using a Kalman filter to predict short-term currency price fluctuation

In the previous post I showed that price fluctuations have a gaussian distribution. This has a similarity with gaussian noise. Apparently, there are several ways to predict gaussian time series. One is described here:

http://www.gaussianprocess.org/gpml/chapters/RW.pdf

But it involves concepts which are too advanced and complicated for me at this moment. Besides: gaussian processes are not the same as gaussian noise. It seems that a gaussian process is formally defined as a process involving "multivariate normal distribution". Since the price fluctuations I'm studying have only one variable, I believe using gaussian process prediction methods would be overkill. I might be wrong, though.

Another method is one used to filter out gaussian noise. It's called "Kalman filtering". Kalman filtering seems useful, because it works by predicting gaussian noise in order to eliminate it. Because currency price fluctuations follow a gaussian distribution, I think the predictive component of the Kalman filter may be useful in predicting short-term future fluctuations in currency price.

The following seems like a good, comprehensive, introduction to Kalman filtering:

http://www.cs.unc.edu/~tracker/media/pdf/SIGGRAPH2001_CoursePack_08.pdf

So the basic idea is to treat FOREX price fluctuation as if it were gaussian noise, and try to predict it short-term with a Kalman filter.

I want to be able to predict short-term future price fluctuations because I discovered that sudden and large price decrements produce important losses when using my rustic buy-and-hold algorithm. These predictions may turn out to be useful in setting up an effective predictive stop-loss alarm for my rustic buy-and-hold algorithm.

Found normal distributions in price changes and price decrement frequencies

I've been measuring the distribution of negative fluctuations in forex market history for the EUR_USD. I discovered that most variations in price (from one instant to another, with a time base of 1 minute) are "small", when compared to larger ones. Sudden, large increments or decrements are scarce. The following plot shows the data from sorted (smallest to largest) price changes (both positive and negative) along many weeks (each week has a different color).

The plot shows a large amount of "small" values near the center (which is zero), and only a few large ones near the sides. This implies that a lapse in time with a high density of price decrements has more probability of containing large negative price fluctuations than a lapse with few price decrements.

To extract the frequency of decrements at any point in time, only the direction of the currency price is required. The following picture shows price decrements in red, and increments in green. Blank spaces represent unknown currency fluctuations in a given instant. What I try to find here is how many red squares there are within a given window. Then I shift that window to the right, and count the red squares again. I keep shifting and counting until I reach the end of the data set.

Interestingly, it turns out the set of frequencies (sampled by shifting the window from top-left to bottom-right) also have a normal distribution. This means there are more "small" frequencies than "big" ones in all consecutive windows of time.

Plotting the histogram for both data-sets showed an approximate normal (Gaussian) distribution.

This shows that the variations in price are mostly smooth, with occasional outbursts. That is: they are more likely to fluctuate within a given (relatively) small range at any given instant. There is a certain stability in currency price change.

First attempt at a rustic buy-and-hold forex bot

I've achieved detecting the beginning of tiny price jumps. Maybe if I tune the program's parameters I can make it find the beginning of larger jumps.

Here's the source code. It's a little rough, fyi:

 /*  
      Rustic buy-and-hold  
      Requires fine tuning. Perhaps make it adaptive.  
   
      Vicente Oscar Mier Vela  
      <vomv1988@gmail.com>  
   
      Example  
      $ cat samples_1407099600_1407531600_M1.dat | ./t6 1407099600 1407531600 60 200 20 20  
   
      Example of samples_START_END_TB.dat  
      $ cat samples_1407099600_1407531600_M1.dat | head  
      1407099600  
      1.34247  
      1.34359  
      1407099660  
      1.34285  
      1.34326  
      1407099720  
      1.34291  
      1.34331  
      1407099780  
   
      This one trys to link density of losses with absolute losses using  
      local average minimum / maximum algorithm.  
 */  
 #include <stdio.h>  
 #include <stdlib.h>  
 #include <math.h>  
   
 #define closeBid 0  
 #define closeAsk 1  
   
 int cmp(const void *x, const void *y);  
   
 int main(int argc, char *argv[]){  
      int startdate = atoi(argv[1]);  
      int enddate = atoi(argv[2]);  
      int timebase = atoi(argv[3]);  
   
      int nsamples = (enddate - startdate) / timebase;  
      int expdate;  
      int currdate;  
   
      double *samples[2];  
      samples[closeBid] = malloc(sizeof(double)*nsamples);  
      samples[closeAsk] = malloc(sizeof(double)*nsamples);  
   
      int i;  
      for(i = 0, expdate = startdate; i < nsamples; i ++, expdate += timebase){  
           scanf("%d", &currdate);  
 /*  
           The following assumes that the first date from the  
           dataset will always yield a value. That is: 1st date is  
           always "ticked".  
 */  
           if(currdate == expdate){  
                scanf("%lf", samples[closeBid] + i);  
                scanf("%lf", samples[closeAsk] + i);  
           } else {  
                while(currdate != expdate && i < nsamples){  
                     samples[closeBid][i] = samples[closeBid][i-1];  
                     samples[closeAsk][i] = samples[closeAsk][i-1];  
                     expdate += timebase;  
                     i++;  
                }  
                if(i < nsamples){  
                     scanf("%lf", samples[closeBid] + i);  
                     scanf("%lf", samples[closeAsk] + i);  
                }  
           }  
      }  
   
      int *diffs = (int *) malloc(sizeof(int) * nsamples);  
      diffs[0] = 0;  
      for(i = 1; i < nsamples; i ++)  
           if(samples[closeBid][i] - samples[closeBid][i - 1] < 0)  
                diffs[i] = -1;  
           else if(samples[closeBid][i] - samples[closeBid][i - 1] > 0)  
                diffs[i] = 1;  
           else  
                diffs[i] = 0;  
   
      int *diffs2 = (int *) malloc(sizeof(int) * nsamples);  
      int k;  
      int range = 10;  
      for(i = 0; i < range; i ++)  
           diffs2[i] = 0;  
      for(i = nsamples; i > nsamples - range; i --)  
           diffs2[i] = 0;  
   
      for(i = range; i < nsamples - range; i ++)  
           for(k = range * -1 ; k < 0 ; k ++){  
                if(diffs[i + k] == -1)  
                     diffs2[i] ++;  
           }  
   
      for(i = 0; i < nsamples; i ++)  
           printf("%d >> %d\n", diffs2[i], diffs[i]);  
   
      int j, t = atoi(argv[4]), mxs = atoi(argv[5]), mns = atoi(argv[6]);  
      double *s = (double *) malloc(sizeof(double) * t);  
      double *q;  
      double sum, loc_avg_min = 0, loc_avg_max = 0, bal = 0;  
      int flag = 0;  
      for(i = 0; i + t < nsamples; i ++){  
           q = samples[closeBid] + i;  
   
           for(j=0;j<t;j++)  
                s[j] = q[j];  
   
           qsort(s, t, sizeof(double), cmp);  
   
           for(j = 0, sum = 0; j < mns; sum += s[j], j ++);  
           loc_avg_min = sum / (double) mns;  
   
           for(j = t - 1, sum = 0; j >= t - mxs; sum += s[j], j --);  
           loc_avg_max = sum / (double) mxs;  
   
           if(flag == 0 && samples[closeBid][i + t - 1] <= loc_avg_min){  
                bal -= samples[closeAsk][i + t - 1];  
                flag = 1;  
           }  
   
           if(flag == 1 && samples[closeBid][i + t - 1] >= loc_avg_max){  
                bal += samples[closeBid][i + t - 1];  
                flag = 0;  
                printf("%f %d\n", bal, diffs2[i + t - 1]);  
           }  
      }  
   
      free(samples[closeBid]);  
      free(samples[closeAsk]);  
      free(s);  
      free(diffs);  
      free(diffs2);  
   
      return 0;  
 }  
   
 int cmp(const void *x, const void *y){  
      double xx = *(double*)x, yy = *(double*)y;  
      if (xx < yy) return -1;  
      if (xx > yy) return 1;  
      return 0;  
 }  

Idea for a very rustic buy-and-hold algorithm

From NN's to buy-and-hold. From the relatively complex, to the very simple.

In these last few days I've been imagining a new (?) algorithm for doing simple buy-and-hold trades. The key is in finding the most probable highest and lowest currency prices for a given point in time. It would involve something like taking a list of the last N prices, sorting it, and averaging the top M values to get some "local average maximum" and the last L to get a "local average minimum". Then, placing limit orders with those values. Come to think of it, it sounds a little like RSI's overbought and oversold indicators. But it's not quite it.

Pretty crude. But I'll do it while I work on the NN approach, and see what happens.

Second thoughts about tuning the MACD with a GA

Let's say that I find the most profitable MACD parameters for a certain month or year. This is no guarantee that the same parameters will be profitable next year or month. I think trying to find optimal MACD parameters for a particular dataset would be curve-fitting. I recently found a message board post about skepticism towards technical analysis. It's from a trusted source (the James Randi website [atheist/skeptic/critical thinker here BTW]). Do take a look at the references mentioned by the posters, though:

http://forums.randi.org/showthread.php?t=96372

That said, I've come across a paper which says neural networks do a good job predicting forex market trends (http://arxiv.org/pdf/cond-mat/0304469.pdf). It uses a neural network architecture which is a mix between an Elman and a Joran SRN. I believe it doesn't say what training algorithm they used to teach the network to predict market trends. In any case, I will probably be using RTRL, because it seems less resource-consuming than BPTT. I doubt my crappy computer can handle BPTT for the large amounts of data I plan to feed my SRN. Also, I would like to start with a pure Elman architecture, instead of the "Elman-Jordan" architecture suggested in the paper. I just don't have the expertise in NN's to copy the paper step by step.

I should mention I've never implemented a FFN, much less a SRN. After spending several days looking into the details of how FFN's and SRN's work, I've come up with a new TODO list:

1. Understand how FFN's work (check)
2. Understand how SRN's work (check)
3. Understand the backpropagation algorithm for FFN's
4. Understand the BPTT algorithm for SRN's
5. Understand the RTRL algorithm for SRN's

It would seem one can't understand RTRL without first understanding BPTT and the classic BP algorithm, as they are somehow part of RTRL. Fortunately, there are several youtube videos on all of them, as well as some pretty good online resources with worked examples:

https://www.youtube.com/watch?v=yecGyZFyfbQ

https://www.youtube.com/watch?v=hYenZlvBwr4

http://neuralnetworksanddeeplearning.com/chap1.html

From what I've gathered so far, the NN has to be a SRN because those are the ones useful for predicting time series. FFN's are useless for that. But that's all I know so far. I'm still grasping the BP algorithm, in order to implement it.

Also, I've found some interesting NN resources. The only library I've found (and liked) which includes an Elman network generator and trainer is "neurolab" for python:

http://code.google.com/p/neurolab/source/browse/trunk/example/newelm.py?r=26

I like the fact that it's in python because it would allow me to integrate it easily with a shell script. Still: I'd like to implement my own SRN in C. And also: I haven't found out whether neurolab's newelm uses BPTT or RTRL. I don't have enough knowledge about python to re-program the whole learning function for Elman, so I'm probably better off writing everything from scratch in C.

A final word about technical analysis though... I'm not sure if NN's fit into the "technical analysis" category. Maybe not ALL technical analysis is fake. In the Randi boards, they mostly mention the typical indicators: MACD, RSI, CCI, etc. Also, the mathematical analysis in the SRN paper from earlier goes pretty deep into statistics. I should be looking into that too. Grasping the statistics surrounding FOREX seems worthwhile.

So who knows... I guess I'll wait and see what my own experiments tell me.

MACD fitness function for the GA

I have finished programming the MACD fitness function in C. This function takes in the 3 usual parameters for the MACD and returns the balance after the last sell. This function will be useful in implementing the genetic algorithm (GA).

In the following source code, macdbal() is this fitness function. The program works by filling in all the missing candles from the data downloaded by the dl2.sh script from an earlier post. It then calculates the Simple Moving Averages used to obtain the MACD. Based on the MACD, macdbal() "buys" (subtracts the closing ask price from the balance) and "sells" (adds the closing bid price to the balance).

 /*  
      MACD fitness function  
      Vicente Oscar Mier Vela  
      <vomv1988@gmail.com>  
   
      Use the output of "Oanda 5000 candle limit bypasser" script  
      as input for this program.  
   
      Example:  
           $ ./dl2.sh "Aug 3 21:00:00 GMT 2014" "Aug 8 21:00:00 GMT 2014" M1 > 5KOUT  
           $ cat 5KOUT | ./gen2 1407099600 1407531600 60 17 91 99  
   
      The output from above should be:  
           Success rate 29.411765%  
           Final balance: 0.001420  
   
      The first two arguments of gen2.c are the dates used for dl2.sh in UNIX time format.  
   
      For example, use:  
           $ date -d "Aug 3 21:00:00 GMT 2014" +%s  
   
      to obtain  
           1407099600  
 */  
   
 #include <stdio.h>  
 #include <stdlib.h>  
   
 #define closeBid 0  
 #define closeAsk 1  
   
 double macdbal(int emalow, int emahigh, int emasignal, double *closeask, double *closebid, int length);  
   
 int main(int argc, char *argv[]){  
      int startdate = atoi(argv[1]);  
      int enddate = atoi(argv[2]);  
      int timebase = atoi(argv[3]);  
   
      int emalow = atoi(argv[4]);  
      int emahigh = atoi(argv[5]);  
      int emasignal = atoi(argv[6]);  
   
      int nsamples = (enddate - startdate) / timebase;  
      int expdate;  
      int currdate;  
   
      double *samples[2];  
      samples[closeBid] = malloc(sizeof(double)*nsamples);  
      samples[closeAsk] = malloc(sizeof(double)*nsamples);  
   
      int i;  
      for(i = 0, expdate = startdate; i < nsamples; i ++, expdate += timebase){  
           scanf("%d", &currdate);  
 /*  
           The following assumes that the first date from the  
           dataset will always yield a value. That is: 1st date is  
           always "ticked".  
 */  
           if(currdate == expdate){  
                scanf("%lf", samples[closeBid] + i);  
                scanf("%lf", samples[closeAsk] + i);  
           } else {  
                while(currdate != expdate && i < nsamples){  
                     samples[closeBid][i] = samples[closeBid][i-1];  
                     samples[closeAsk][i] = samples[closeAsk][i-1];  
                     expdate += timebase;  
                     i++;  
                }  
                if(i < nsamples){  
                     scanf("%lf", samples[closeBid] + i);  
                     scanf("%lf", samples[closeAsk] + i);  
                }  
           }  
      }  
   
      double x = macdbal(  
                emalow, emahigh, emasignal,  
                samples[closeAsk], samples[closeBid],  
                nsamples  
      );  
   
      printf("Final balance: %f\n",x);  
   
      free(samples[closeBid]);  
      free(samples[closeAsk]);  
   
      return 0;  
 }  
   
 double macdbal(int emalow, int emahigh, int emasignal, double *closeask, double *closebid, int length){  
      int i;  
      double *emalows = malloc(sizeof(double)*length);  
      double *emahighs = malloc(sizeof(double)*length);  
      double *emasignals = malloc(sizeof(double)*length);  
   
      for(i = 0; i < length; i ++){  
           scanf("%lf", closebid + i);  
           emalows[i] = emahighs[i] = emasignals[i] = 0;  
      }  
   
      double sum;  
      for(i = 0, sum = 0; i < emalow; i ++){  
           emalows[i] = closebid[0];  
           sum += closebid[i];  
      }  
   
      emalows[emalow - 1] = sum / ((double) emalow);  
      double lowmult = 2.0 / (((double) emalow) + 1.0);  
      emalows[emalow] = (closebid[emalow] - emalows[emalow - 1]) * lowmult + emalows[emalow - 1];  
   
      for(i = emalow + 1; i < length; i ++)  
           emalows[i] = (closebid[i] - emalows[i - 1]) * lowmult + emalows[i - 1];  
   
      for(i = 0, sum = 0; i < emahigh; i ++){  
           emahighs[i] = closebid[0];  
           sum += closebid[i];  
      }  
      emahighs[emahigh - 1] = sum / ((double) emahigh);  
      double highmult = 2.0 / (((double) emahigh) + 1.0);  
      emahighs[emahigh] = (closebid[emahigh] - emahighs[emahigh - 1]) * highmult + emahighs[emahigh - 1];  
   
      for(i = emahigh + 1; i < length; i ++)  
           emahighs[i] = (closebid[i] - emahighs[i - 1]) * highmult + emahighs[i - 1];  
   
      int signalstart = emahigh * 2;  
      for(i = 0; i < signalstart + emasignal; i ++)  
           emasignals[i] = 0;  
   
      for(i = signalstart, sum = 0; i < signalstart + emasignal; i ++)  
           sum += emalows[i] - emahighs[i];  
      emasignals[signalstart + emasignal - 1] = sum / ((double) emasignal);  
      double signalmult = 2.0 / (((double) emasignal) + 1.0);  
      emasignals[signalstart + emasignal] =  
           (  
                (emalows[signalstart + emasignal] - emahighs[signalstart + emasignal]) -  
                emasignals[signalstart + emasignal - 1]  
           ) * signalmult + emasignals[signalstart + emasignal - 1]  
      ;  
   
      for(i = signalstart + emasignal + 1; i < length; i ++)  
           emasignals[i] =  
                ((emalows[i] - emahighs[i]) - emasignals[i - 1]) * signalmult + emasignals[i - 1]  
           ;  
   
      double last = 0;  
      double balance = 0;  
      double prevbal = 0;  
      double lastsell = 0;  
      double total = 0;  
      double success = 0;  
      for(i = signalstart + emasignal; i < length; i ++){  
           if(emasignals[i] > 0 && last < 0){  
                last = emasignals[i];  
                prevbal = balance;  
                balance -= closeask[i];  
           }  
           if(emasignals[i] < 0){  
                if(last == 0){  
 /* memo to myself: get ready to buy before starting this loop */  
                     last = emasignals[i];  
                }  
                if(last > 0){  
                     last = emasignals[i];  
                     prevbal = balance;  
                     balance += closebid[i];  
                     if(lastsell < balance)  
                          success ++;  
                     total ++;  
                     lastsell = balance;  
                }  
           }  
      }  
   
 /* remove this for genetic algorithm */  
   
      printf("Success rate %f%\n",100*(success/total));  
   
      free(emalows);  
      free(emahighs);  
      free(emasignals);  
   
 /* return balance after last sell */  
      if(last > 0)  
           return prevbal;  
      else  
           return balance;  
   
 }  

Bypassing OANDA's 5000 candle limit

OANDA limits the amount of candles you can download using it's HTTP API. If you ever try to download more than 5000 candles, you get the following error message:

{ "code" : 36, "message" : "The value specified is not in the valid range: Resulting candle count is larger than maximum allowed: 5000", "moreInfo" : "http:\/\/developer.oanda.com\/docs\/v1\/troubleshooting\/#errors" }
This limit can be bypassed using the following script:
#!/bin/bash # OANDA 5000 candle limit bypasser # Vicente Oscar Mier Vela # <vomv1988@gmail.com> # Wed Aug 13 19:45:05 CDT 2014 # # Example: # # ./dl2.sh "Aug 3 21:00:00 GMT 2014" "Aug 8 21:00:00 GMT 2014" S5 # # This will output 3 element lists composed of # 1. A line containing the UNIX time of a particular currency price sample # 2. A line containing the closing bid price # 3. A line containing the closing ask price # # Remember that... # "No candles are published for intervals where there are no ticks. This will result # in gaps in between time periods." # http://developer.oanda.com/docs/v1/rates/#get-current-prices START=$1 START=`date -d "$START" +%s` END=$2 END=`date -d "$END" +%s` TIME=$3 case $TIME in S5 ) TIMES=5 ;; S10 ) TIMES=10 ;; S15 ) TIMES=15 ;; S30 ) TIMES=30 ;; M1 ) TIMES=60 ;; M2 ) TIMES=120 ;; M3 ) TIMES=180 ;; M5 ) TIMES=300 ;; M10 ) TIMES=600 ;; M15 ) TIMES=900 ;; M30 ) TIMES=1800 ;; H1 ) TIMES=3600 ;; H2 ) TIMES=7200 ;; H3 ) TIMES=10800 ;; H4 ) TIMES=14400 ;; esac LAPSE=`echo "$END-$START" | bc` CHUNKS=`echo "$LAPSE/(5000*$TIMES)" | bc` AUTH="Authorization: Bearer 12345678900987654321-abc34135acde13f13530" DFORM="X-Accept-Datetime-Format: UNIX" URL="https://api-fxpractice.oanda.com/v1/candles?instrument=EUR_USD&start=0&end=0&granularity=$TIME" DATESTART=$START >samples if test $CHUNKS -gt 0 ; then while test $CHUNKS -gt 0 ; do DATEEND=`echo "$DATESTART+(5000*$TIMES)" | bc` # echo START $DATESTART = `date -d "@$DATESTART"` # echo END $DATEEND = `date -d "@$DATEEND"` URLSAMPLE=`echo $URL | sed "s/start=0&end=0/start=$DATESTART\\&end=$DATEEND/"` curl -s -H "$AUTH" -H "$DFORM" -X GET $URLSAMPLE >> samples DATESTART=`echo "$DATEEND+$TIMES" | bc` CHUNKS=`echo "$CHUNKS-1" | bc` echo CHUNKS $CHUNKS done LASTCHUNKSIZE=`echo "$LAPSE%(5000*$TIMES)" | bc` DATESTART=`echo "$END-$LASTCHUNKSIZE+$TIMES" | bc` DATEEND=$END # echo START $DATESTART = `date -d "@$DATESTART"` # echo END $DATEEND = `date -d "@$DATEEND"` URLSAMPLE=`echo $URL | sed "s/start=0&end=0/start=$DATESTART\\&end=$DATEEND/"` curl -s -H "$AUTH" -H "$DFORM" -X GET $URLSAMPLE >> samples else DATESTART=$START DATEEND=$END # echo START $DATESTART = `date -d "@$DATESTART"` # echo END $DATEEND = `date -d "@$DATEEND"` URLSAMPLE=`echo $URL | sed "s/start=0&end=0/start=$DATESTART\\&end=$DATEEND/"` curl -s -H "$AUTH" -H "$DFORM" -X GET $URLSAMPLE >> samples fi cat samples | grep -e time -e closeBid -e closeAsk | sed 's/000000",/",/;s/"//;s/"//g;s/.*: //;s/,//'
I will pipe this script's output into a C program to produce the trading simulation I mentioned in an earlier post.

Starting with MACD using shell scripts and C

I've been interested in FOREX trading since I enrolled in college around 8 years ago, reading pieces of articles about it here and there. I am now a 26 year old electronics engineer with some graduate school experience in computer science. I think I've matured enough, mathematically and intellectually, to start playing with the FOREX market.

I admit I have no formal or previous experience with trading. This blog is not intended as a guide of any kind. Much less as trading advice. This blog is about the development of my FOREX learning curve, from the very beginning. The notes contained here are for my personal use, but I will share them in case anybody out there wants to collaborate, make suggestions, corrections, criticism, or simply to learn alongside myself.

That said, I'll start by sharing some of the experiences I've collected so far.

I've started by becoming familiar with the basic FOREX trading concepts. I believe I have a good enough grasp on pips, leverage and the MACD to start playing around with an OANDA test account.

Recenly I've run some experiments using the OANDA HTTP API, UNIX shell scripts and C programming. So far I haven't been able to turn any profit using my MACD schemes with the OANDA API. I blame a set of factors for this:

1. The input parameters I've used are based on only a few  trading simulations I ran using the EUR_USD price data of 3.5 days (5000 1-minute samples).
2.  My simulations do not entierly represent real trading. I must improve them.
3.  I am using MACD exclusively. I should try combining it with other methods, such as a 1-2-3 scheme and RSI.

I am aware that there exists software which already implements MACD and others. But I've refrained from using such things. Mostly because of my obsession with writing my own software, and with being in absolute control of what my software does. Still, I don't reject the idea of using pre-existing software. I think experimenting with pre-existing software may help me turning profit sooner. It may even help me improve my own software.

To solve the issues which have prevented me from turning a profit with my OANDA test account, I have formulated the following TODO list:

1. Writing a proper trading simulator.
2. Using such simulator to run a genetic algorithm which finds the optimal MACD parameters (genetic MACD tuning). These parameters include the typical 3 numeric values, and a fourth value representing the time frame for sampling closing bid price values.
3. Run my MACD shell script with the obtained values.
4. Copy this trading strategy: http://forex-strategies-revealed.com/simple/123-rsi-macd using only shell scripting and C programming, as to understand the nuts and bolts of MACD, 1-2-3 and RSI.
5. Copy the same trading strategy using the suggested pre-existing software packages.

Currently my source code is not ready for publishing. I will polish it and then share it on my github account. I will be reporting as I finish the elements in my TODO list.