Quote of the Week, July 19, 2007

Ten years ago, Astrophysicist John Nousek had this answer to Hyunsook Lee’s question “What is so special about chi square in astronomy?”:

The astronomer must also confront the problem that results need to be published and defended. If a statistical technique has not been widely applied in astronomy before, then there are additional burdens of convincing the journal referees and the community at large that the statistical methods are valid.

Certain techniques which are widespread in astronomy and seem to be accepted without any special justification are: linear and non-linear regression (Chi-Square analysis in general), Kolmogorov-Smirnov tests, and bootstraps. It also appears that if you find it in Numerical Recipes (Press etal. 1992) that it will be more likely to be accepted without comment.

…Note an insidious effect of this bias, astronomers will often choose to utilize a widely accepted statistical tool, even into regimes where the tool is known to be invalid, just to avoid the problem of developping or researching appropriate tools.

From pg 205, in “Discussion by John Nousek” (of Edward J. Wegman et. al., “Statistical Software, Siftware, and Astronomy”), in Statistical Challenges in Modern Astronomy II”, editors G. Jogesh Babu and Eric D. Feigelson, 1997, Springer-verlag, New York.

  1. hlee:

    Many thanks for the quote as well as the answer to my question. In a strict word, it is convenient to take the safe side of using chi-square, unless someone proves mathematically the statistical method is invalid to analyze such astrophysical data. The issue is that such fault finding is not the subject of astrophysics nor statistics. In addition, even if some statistical methods are proven to be more versatile than existing ones from Numerical Recipes mathematically, they may not be appealing to astronomers because those are hardly interpretable in astronomical language at their births and take time to be absorbed and mature. A unconventional statistical method introduced in an astronomical journal is a propaganda which is possible to become a belief later and those statistics you’ve mentioned are the examples. I hope we could see more new beliefs.

    07-20-2007, 1:21 am
  2. vlk:

    I am not sure what you mean by “propaganda”..?

    It is true that astronomers will push the envelope on an established statistical method simply to avoid arguing with referees. In many cases it is of little consequence — the majority of astronomers still operate in the highly conservative regime of “if it takes statistics to prove it, it probably isn’t true.”

    It does of course matter a great deal for marginal results, and astronomers will use the right tool for the job if convinced that it is worth it. Convincing them is not very easy, however.

    At the conclusion of the recent XGratings workshop (http://cxc.harvard.edu/xgratings07/), Randall Smith took down suggestions from the audience on what are the things X-ray spectroscopists should be looking at. Suggestions ranged from a more complete atomic line database to, yes, a desire to move away from using chi-square as the measure of quality of fit. (I was present, but I didn’t prompt it!)

    07-23-2007, 11:39 pm
Leave a comment