Markov Chains (MC) models have fixed-sized context (order of the chain)
Fixed-size context == no context preservation for long term contextual information.
Information is passed via hidden states through time.
Although theoretically possible, RNNs are very bad at capturing long-term dependencies in data.
Can be improved by increasing hidden state size, but there's a limit to space complexity!