The temperature in a large city such as Houston may vary be a couple degrees or more depending on the place of measurement. It is probably inappropriate to use as "actual temperature" the recorded temperature at an arbitrary location. It would be better to use the actual temperatures as recorded by each forecaster.
It would be interesting to know the extent to which these results for Houston TX are valid for another location.
We should consider measuring error in terms of squared difference from the actual temperature, rather than absolute value of the difference. This method gives more weight to the worst errors (we might consider a single day's error of 10 degrees to be worse than five days' errors of 2 degrees).
The word "consistent" has a specific statistical meaning. It is troubling that he uses the word "consistent" with reference to standard deviation. In addition, the author provides no comment on the interpretation of this "consistency". If a particular weather forecast is always exactly 30 degrees too high, the standard deviation of the error is 0, but this does not mean it is a good forecast.
It is impossible to tell if the ranking of the weather services is significant. Even when the design of a study is excellent, it is important to measure the statistical significance of any difference among observations. Based on the sample size, we could determine whether the observed differences among weather services are likely due to a true difference in accuracy, or to random fluctuations. It is a serious flaw that the author does not address this issue directly.
To his credit, the author does attempt to measure statistical significance by way of correlation, but I am not sure at this time if this is an acceptable substitute for the standard technique.
I should have noted in my post that I was more impressed by the idea of tracking forecast accuracy. I had trouble understanding much of the analysis myself and am impressed that you took the time to read and critique it.
2 comments:
There are some problems with this analysis.
The temperature in a large city such as Houston may vary be a couple degrees or more depending on the place of measurement. It is probably inappropriate to use as "actual temperature" the recorded temperature at an arbitrary location. It would be better to use the actual temperatures as recorded by each forecaster.
It would be interesting to know the extent to which these results for Houston TX are valid for another location.
We should consider measuring error in terms of squared difference from the actual temperature, rather than absolute value of the difference. This method gives more weight to the worst errors (we might consider a single day's error of 10 degrees to be worse than five days' errors of 2 degrees).
The word "consistent" has a specific statistical meaning. It is troubling that he uses the word "consistent" with reference to standard deviation. In addition, the author provides no comment on the interpretation of this "consistency". If a particular weather forecast is always exactly 30 degrees too high, the standard deviation of the error is 0, but this does not mean it is a good forecast.
It is impossible to tell if the ranking of the weather services is significant. Even when the design of a study is excellent, it is important to measure the statistical significance of any difference among observations. Based on the sample size, we could determine whether the observed differences among weather services are likely due to a true difference in accuracy, or to random fluctuations. It is a serious flaw that the author does not address this issue directly.
To his credit, the author does attempt to measure statistical significance by way of correlation, but I am not sure at this time if this is an acceptable substitute for the standard technique.
I do not recommend relying on these results.
I should have noted in my post that I was more impressed by the idea of tracking forecast accuracy. I had trouble understanding much of the analysis myself and am impressed that you took the time to read and critique it.
Bravo, and thanks for the analysis.
Post a Comment