Do you remember those puzzles of the following form:

What is the next number in the series: 1, 4, 9, 13, ??

What you usually look for in this kind of puzzle is a rule that gives the next number based on the previous one or two numbers. In this case 9 is computed from 4 and 1 in the same way that 13 is computed from 9 and 4. Try to figure this one out.

View the answer in a pop up window.

One wit has said that when confronted with this kind of problem, he always answers "19."

How can this be?

It is always possible to interpolate a polynomial through the given numbers, plus the number 19. The polynomial that goes through the series 1, 4, 9, 13, 19 is
y = 9 - 18*x + 12.75*x2 - 3*x3 + 0.25*x4; where x = 1, 2, …

Plugging 1 through 4 into this equation generates the given series. If you plug 5 into this equation, you get 19.

This answer violates the assumption that you're looking for a rule that the next value is a linear combination of the previous two values. Put another way, the usual assumption is that you are looking for an autoregressive model of degree 2, denoted an AR(2) model.

What if we relaxed the assumption a different way?

Let's allow products of previous values in the series. In other words, we will allow multivariate polynomial autoregression (PAR) models. Try this series: 1, 2, 3, 11, ??

View the answer in a pop up window.

You can check that there is no possible AR(2) model that can describe this series, and in fact no type of linear model. (Although if you allow an ordinary polynomial, you can still answer 19!) This particular type of polynomial model includes an interaction term. The next one is a bit more difficult. It includes a quadratic term.

Can you guess the next number in this series?
1, 2, 4, 10, 30, ??

View the answer in a pop up window.

You can see that if you started considering more variables, including more lags, and allowed higher-degree exponents, the number of possible terms in the model increases greatly, even if there are few terms actually in the model. You would need a program like TaylorFit to discover the model that describes the data.