Grape Prices are Pretty Easy to Forecast

Hi all,

It’s been years since I posted anything here other than Grape Data Tool updates. I recently decided to grade some of my old, public forecasts. Perhaps the core question of my career has been: can we accurately forecast grape prices? By and large, the answer is a resounding yes. See below for my results data and an FAQ.

Can Winegrape Prices be Forecasted Accurately?

Very much so, but let’s give a little more detail here. Let’s first define the term ‘accurately.’ I’m going to say that any error rate below 13% is most certainly accurate. Those of you who forecast product sales data, for instance, would likely be thrilled to consistently hit that number.

For benchmarks, I am using the results of the Makridakis competitions. These are horse race forecasting competitions, where the world’s best forecasters compete to see who can most accurately forecast a bunch of time series data, without domain knowledge. Since the last of my public forecasts was from the 2018 data year (meaning the latest data at the time was 2018), I’ll use the M4 competition results, since it was run in 2018. For annual time series, the best results were – using my favorite measure of error, mean absolute percentage error – about 13% and went to one S. Smyl, head forecaster for Uber, using a hybrid statistical / machine learning model.

Here are all results from my public forecasts:

Most of these forecasts can be found on my blog, though some were made to the El Dorado Wine Grape Growers Association and one was given at the 2015 Vineyard Economics Seminar (that was my one public prediction on acreage figures.) Keep in mind the year of forecast is not the year the forecast was produced, but the last year of included data (generally the year before the forecast, since the Crush Report is released in spring).

As you can see, the mean average absolute percentage error, which I’ll just refer to here as error was about 5%, way below the 13% benchmark. So, yes, we can accurately forecast prices. I would also add that, were I to have included forecasts I was contracted to make, instead of just those I made publicly, the error rate would be lower.

How Far Out Can We Forecast Winegrape Prices?

I measured the correlation between time horizon and error rate at 0.078, which is quite low, meaning accuracy does not taper off too quickly for this sample. The one-year time horizons have an error rate of 4.43%, while the 4-year-out forecasts (only 3 in the sample) have an error rate of 2.83% and the one 5-year horizon data point had a 4.12% error. So, I think it’s safe to say that we can probably accurately forecast winegrape prices at least 5 years out. Again, if I included non-public data, the accuracy would be improved.

Is Winegrape Price Forecasting Becoming More Accurate?

The short answer is yes. Why do I say this? First, unless someone has new information for me, I feel safe assuming as a fact that I am the most accurate forecaster of winegrape prices and, therefore, can look at my own results to answer this question. Let me then give a concise history of my efforts to forecast winegrape prices.

Between 2009 and 2012, I worked on how to forecast winegrape prices using various simple models, multivariate regression analysis, exponential smoothing models and Box-Jenkins (ARIMA) models. As of 2008, I barely knew how to use Excel, but, between my previous job as a (successful) professional gambler, my pursuit of an MBA from Davis and a lot of elbow grease, I greatly increased my aptitude with Excel, statistical and financial modeling and forecasting.

In 2013, I decided to make a justified gamble and start making public forecasts. I figured this was the best way to get clients to pay for forecasts but knew that (a) I still had a lot to learn and (b) if my public forecasts didn’t pan out I was doomed. I was happy to see the results were more-or-less as I had expected.

Between 2013 and 2018, as a paid and practicing forecaster, I improved my skills. Within the sample above, the correlation between accuracy and years is about 0.21, which is small and far from statistically significant, but certainly positive.

By 2018, I made the decision to scale back my efforts to market myself. I had plenty of clients and wanted to increase my overall hourly earnings by reducing the amount of time I spent doing things that I wasn’t getting paid for. Since then, I’ve also added machine learning to my toolbox, as it has become more accessible to the public. My accuracy has improved further.

So, to tie things back to the question: yes, winegrape price forecasting is becoming more accurate.

Are Some Regions More Difficult to Forecast for than Others?

Yes. To some extent this seems a function of my own experience and domain knowledge. I’m very accurate for Sonoma, Napa and El Dorado, as you can see in the sample above. From experience, I can tell you that I’m also accurate for Mendocino, District 10 and statewide prices. Washington seems a bit more dubious, especially for longer time horizons, while Oregon is pretty easy to forecast, but not as accurately as California’s North Coast regions. Lake County’s results push the boundaries of what I consider to be accurate, which pains me, as it’s dear to my heart.

But Can these Be Done Quickly and Efficiently?

Yes. Accuracy increases with the amount of effort put in, but relatively high quality forecasts can still be produced quickly and cheaply. Typically, data set up take me less than an hour and then I can provide decent forecasts within an hour. On the other hand, for the highest quality forecasts – especially if you’re trying to forecast far out – I like to create much more complex models that are adaptive to new information. This takes about 40 hours for the first forecast and probably 10 hours to update each year.

What About Confidence Intervals?

Good question, hypothetical questioner! Yes, this question would stretch the definition of ‘frequent’ in the acronym FAQ, but it’s important. I actually don’t evaluate forecasting competency based on accuracy, which has a great deal to do with the dataset and the forecasted variable. Instead, I believe a good forecaster is able to understand how often he will be wrong by how much. For this reason, I generally give clients forecasts with a probability distribution attached (you can reference some of my public forecasts to understand what this looks like). This allows us to do significant risk analysis through scenario and sensitivity analysis and computerized simulations. This takes some more time, of course.

OK, but What Can I Use this for?

Here’s a list based on what I’ve seen so far:

· Improving financial forecasts and adding in break-even, scenario and sensitivity analysis, with probabilities attached;

· Securing and right sizing financing;

· Informing strategic questions by accurately evaluating risk-of-ruin;

· Informing contract strategy and negotiations;

· Improving income-based valuation methods;

· Optimizing vineyard investment strategies by improving income projections;

· Developing hedging strategies for a portfolio of vineyards by understanding the correlation between different regions’ and varieties’ income potential.

How Can I Do this Myself?

Well, I can’t teach you how 13 years of forecasting experience in a blog post, but some of this you can do yourself. First, track the results of forecasting approaches. Even if all you’re using are methods like rolling averages, CAGR, annual increases or the same-as-last-year assumption, testing them for accuracy will help you get better.

Excel also has options to add multivariate regression analysis and exponential smoothing capabilities. The latter is easy to use once you do it once or twice and can significantly improve your forecasts. The former is more complicated, but you can find materials to teach yourself how to use it.

Finally, work on combining some of these different methods. Hybrid forecasts are generally more accurate, in the long run, than single-method forecasts.