My previous job was doing operations research in a big logistic company. The problem of the day was making accurate forecasts. For example, the amount of work fluctuated significantly from day to day and we needed an accurate gauge of the manpower requirements for the next day to give us time to make adjustments.
While the people on the ground have been doing the forecast manually based on their years of experience, the management wanted us the techies to make the process scientific. Honestly, it was a lost cause to begin with. And the biggest reason for failure was that there was no accurate benchmarking of our forecasts compared to the manual prediction.
In real world situations, I don’t really think it is possible to make any reliable forecast. Yet, you see the economists and financial analysts telling you the dow jones will hit 12000, google’s stock price will hit $450. Mostly, we the silly people laps up their predictions but nobody bothers to check how good they were.
As an experiment, I tabled the GDP forecast by the Singapore Government Statistics Board. These were the official figures, not those dime a dozen ones given by analysts, as reported in the newspapers. So the combined wisdom of the Singapore Chief Statistician and his minions told us:
Jan 2005: 3% to 5%
Apr 2005: Low end of 3% to 5%
May 2005: 2.5% to 4.5%
Aug 2005: 3.5% to 4.5%
Nov 2005: 5%
Feb 2006: Accurate figure 6.4%
So despite the fact that they were continuously revising their figures as more data came in, they were still completely way off. One might argue that they chose to err on the side of caution but I honestly think that I might have done better by picking a figure at random. Random walk theory I guess.