Intraday Models

Problem Definition

The goal is to predict the price change of a security.​

At each point in the above graph, we want to predict the prices after 1 min.

Problem Definition

The following table shows the label for AAPL on 20170928.

Dataset

• Training Period: 20170901 to 20170915 (10 days)
• Testing Period: 20170918 to 20170922 (5 days)
• Numpy Array Shape: (2026106, 361) - before adding transformations
• Source securities: 'SPY', 'BND', 'XLK', 'XLU', 'VNQ', 'IEMG', 'AGG', 'VWO', 'XLF', 'EEM', 'QQQ', 'USO', 'GDX', 'IWM', 'VEA'
• Label mean: 0.0002, standard deviation: 0.0216 (This helps put the MAE values in context)

Features

• bid/ask/mid prices
• bid/ask sizes
• bid-ask spread
• bid/ask/mid moving averages over 1 min and 5 min
• bid/ask/mid trend over 1 min and 5 min
• volume/trades/updates over 1 min and 5 min

Normalization

• 0-1 normalization
• 0-1 normalization and division by mean

Linear Regression

• fit_intercept=False
• Unnormalized, 0-1 normalized and 0-1 mean normalized

LR Results

• Simple LR on all features does not work.

L2 penalized LR Results

• Trained using ElasticNet learning algorithm.
• All 360 features are selected.
• Out of Sample stdev is good.

Lasso Results

• Selects 11 features out of 360
• fit_intercept=False
• alphas: 1e-3, 1e-4, 1e-5

L1 + L2 penalties

• Selects 19 features out of 360
• fit_intercept=False
• alphas: 1e-3, 1e-4, 1e-5
• l1_ratio = 0.1 (l1/l2)

Pre-selecting features

• Using standard feature selectors by themselves do not seem to work.
• Using PCA to reduce the number of features does not seem to make much difference.

Using Less Data

• Penalized LR results are same as long as there are >= 2000 datapoints representing most of the time periods.
• In other words, reducing the number of datapoints by 512 is still okay.

Using Less Data

• Penalized LR results are same as long as there are >= 2000 datapoints representing most of the time periods.
• In other words, reducing the number of datapoints by 512 is still okay.

Ensemble Methods

• Checked RandomForest, AdaBoosting and GradientBoosting algorithms.
• They are better at preserving the stdev and generalize relatively better out of sample.

FNN

• Feed forward neural networks are relatively worse in terms of generalization.

Lessons (re-)learned

• Pure market data features have very limited predictive abilities. (They can be exploited a bit more by adding a regime component on top.)
• Learning algorithms and algorithm parameters that preserve the standard deviation scale are more important than small differences in MSE/MAE scores.
• Out of sample generalization is non-obvious and important.