RC metasnark

I haven’t been watching RC very closely lately, but I stopped in and saw arrogance slipping back into the discourse. Also, I had trouble with the captcha thing, so in case I didn’t get through, I thought I should comment here.


So in the excellent and interesting article Antarctic warming is robust appears the following exchange:

jeff Id Says:
4 February 2009 at 11:21 PM
A link to my recent post requesting again that code be released.

[edit]

I believe your reconstruction is robust. Let me see the detail so I can agree in public.

[Response: What is there about the sentence, “The code, all of it, exactly as we used it, is right here,” that you don’t understand? Or are you asking for a step-by-step guide to Matlab? If so, you’re certainly welcome to enroll in one of my classes at the University of Washington.–eric]

I don’t know who Jeff is or whether he should be encouraged, but I find Eric’s response unsatisfactory. Here is my reply, slightly expanded.

Eric, you snark: ” What is there about the sentence, “The code, all of it, exactly as we used it, is right here,” that you don’t understand? “

I don’t understand how you think that could be true. You link to a nicely documented and from all appearances elegant library of matlab functions. Where are the data files? Where is the script that invoked those functions and plotted those graphs?

There is absolutely no substantive reason this should not be distributed along with the relevant publication. You shouldn’t be sniffing at people who want to replicate your work in toto. You should be posting a complete makefile that converts input data into output data. This is common practice in exploration seismology, thanks to the example of John Claerbout at Stanford University, and that in a field where there are sensible commercial reasons for confidentiality. A related effort to create tools to facilitate this pervasive reproducibility approach, called Madagascar, is being developed at U Texas and is 100% open source. A recent issue of Computers in Science and Engineering discussed reproducibility as a topic.

The paradoxical backwardness of science in regard to adopting the social lessons of high tech is well analyzed in this blog entry by Michael Nielsen. An open source and test-driven approach is far more productive than petty jealousy. “Publish the recipe, and build the restaurant.”


If you refuse to be helpful, don’t put up a pantomime please. I should be able to reproduce these results in five minutes, not after taking a course in Matlab. The course in Matlab should follow if I find the results interesting. RC again climbs on its high horse, doing none of us any good. You guys are the good guys. Please act like it.

mt
Update: Unsurprisingly Jeff is not one of what I’d call the good guys, but he is probably the genuine article, a serious person and an honest critic. Unsurprisingly he says pretty much what I said, though understandably angrier.

Ok, I’m madder than hell by now, my good Irish temper is taking over so this is what I posted.

You know, if you claim you archived the code, you could actually archive the code rather than a link to the functions which may have been used in the aforementioned code.

Over the last couple of days, I’ve seen Eric accuse people of everything short of fraud including Steve McIntyre who does everything in the open. In the meantime my reasonable requests for data and code have been cut from the threads repeatedly. This man is intentionally deceptive about his openness and he’s doing it pretty effectively because nobody at RC made it through the thread who had questions and because I was actually had to disagree with by smart people at CA who believed Eric actually had archived the code by his link.

Dr. Steig, put up or shut up. Enough bull about the posting all the code and the data (including the satellite data). I will call you out to no end and people will know the truth. If your paper is good, they will know – I promise. If not, I also promise.

——————–

Update, I got through. I did I really did. I sent this comment to Real Climate.

4 February 2009 at 11:21 PM

A link to my recent post requesting again that code be released.
[edit]
I believe your reconstruction is robust. Let me see the detail so I can agree in public.

[Response: What is there about the sentence, “The code, all of it, exactly as we used it, is right here,” that you don’t understand? Or are youasking for a step-by-step guide to Matlab? If so, you’re certainly welcome to enroll in one of my classes at the University of Washington.–eric]

I replied again like this.

I’d love to take your class, but I’m busy running a company.

You point to the RegEm manual for use. I was hoping you could show the setup and implementation used. i.e. The actual code you used. I’m sure you would agree, it is pretty important as far as understanding the result. Also the satellite data set.

If you present this as used and it works as advertised, you’ll find yourselves with a big pile of supporters instead of the ridiculous situation we have now. As I have repeatedly stated, I believe your result simply because warming is true everywhere else.

Jeff

———————–

Banned again Real Climate.

[edit — thanks for your support dhogaza, but I’m not allowing ‘jeff id’s’ rants to show up here, even if passed on to me by someone else–eric]

———————-

This stuff is so silly, you know it saps you to listen to it. Doggozza calling me all kinds of things, real climate not allowing quesitons, they want me to be a denier. It’s like they’re saying please Jeff please deny our work just don’t make us disclose the methods.

It’s sad really. Honestly I think it’s more ignorance than rudeness, but it is amazing. And given the stakes attached to the core credibility of the ideas, amazingly unfortunate.

Update: Tamino’s point in #92 is partly true. I find his argument that on the difficulty in publishing results such that they can easily be replicated unconvincing. One should always organize one’s work in that way as a matter of elementary sound practice, and it is commonly practiced elsewhere.

On the other hand, it is certainly true that the hacks want to find errors to use as weapons in their overdrawn attacks on the core conclusions of science. Coping with those attacks is indeed an important larger issue here. The advantage in this matter to eschewing openness is not clear, though. The failure to use new tools to enhance the efficacy of the scientific method dominates any convenience effect from having unqualified people poking at the code.

Any confidence that the scientific method will win in the end only works if the scientific method is sufficiently followed for its advantages to work. Not only that but we need it to work fast enough to keep us from that cliff we seem to be barreling toward. Remember that cliff? In principle science will out, maybe, but in practice it can take generations.

If standard practice in software engineering can accelerate the advantages of the scientific method and nonstandard practice (inattention to workflow as a component of work product integrity) can obscure those advantages, which choice should we take? Perhaps Gavin is right in suggesting that I overgeneralize when I say that scientists generally arrogantly refuse to learn from commercial practice. But those scientists who complain about ‘learning curves’ and lakc of ‘demonstrated benefits’ in the context of elementary best practices are not serving as counterexamples to the generalization.

Rather than being entirely argumentative here, allow me to commend to those willing to address this matter constructively the work of Greg Wilson of the University of Toronto: here is his excellent article called Where’s the Real Bottleneck in Scientific Computing? and his course on software carpentry is here. The relevant lesson for present purposes is automated builds.

Update: Woot! Gavin responds with a front page article on RC! On the other hand, no link back, for some reason.

Lots more commentary from Gavin in that article, much of which I find strange. My last word on this:

Reply to #89:

“[Response: My working directories are always a mess – full of dead ends, things that turned out to be irrelevent or that never made it into the paper, or are part of further ongoing projects. Some elements (such a one line unix processing) aren’t written down anywhere. Extracting exactly the part that corresponds to a single paper and documenting it so that it is clear what your conventions are (often unstated) is non-trivial. – gavin]”

Gavin, what you are describing here is what would be called, in any commercial or industrial setting, bad practice.

That is exactly the point I am trying to make. It’s not a point about openness, it’s about effectiveness. Good practice in any discipline evolves from long experience. The behavior you are describing is behavior every programmer occasionally does on quick projects. However, most of us know better than to defend such behavior on major work products.

It is considered bad practice with good reason. It takes a lot of effort to go back and replicate your own results from memory, but very little to maintain a script which can do all of it. If steps are expensive, you need to learn a tiny bit of rule-based logic, but that is hardly beyond the abilities of anybody doing scientific computations. The payoff is not just throwing the CA folks a bone to chew on. It’s a very important component of reasoning about computations, which are, after all, error-prone.

Basically, you are making it easier to make mistakes.

Why should pure science be held to a lower standard than applied science or commerce? Does climate science matter or doesn’t it?

Per #98,

So to make appropriate use of the code and paper (enough to enable us to DEMAND it all be available), we need a team…

The issue is not whether the CA people are competent to examine the process or not. They might or might not be.

The issue is that when Gavin claims that this is unreasonably difficult, he is making a claim that many readers already know, as a consequence of their own daily practice, to be false and indeed absurd. Indeed, these readers overlap strongly with the group of nonscientists most competent and most willing to evaluate scientific claims. This does the credibility of RC, and by further extension the whole of climate science, no good.

In any case, whether this is sound practice on the part of the scientist or not, whether it is responsible behavior on the part of the hobbyists or not, one can expect demands for replication on any observational climatology analysis. Observational climatology is not at fault for having the perhaps worst relationship with its interested public of any science ever. There really are, after all, some truly malign forces involved. But it’s nothing to celebrate, and it’s worth making some effort not to make it worse.

In summary: First, it is not true that maintaining end-to-end scripts is onerous. If large calculations are involved a series of scripts or a rule-based makefile may be practical, but these are easy skills to develop compared to the background needed to do science. Doing so in commercial and engineering settings is standard practice because it dramatically reduces error and increases the potential for objective tests.

Second, that some branches of science don’t do this is going to be perceived as an embarrassment. Defending the absence of a practice of end-to-end in-house repeatability is difficult, and coming from someone who has not spent much time thinking about it, likely to make silly claims.

Of course, as climatologists, we are barraged with silly claims, but in that we are not unique. We tend to lose patience with people making strident and erroneous claims about things they don’t understand. In this, we are not unique.

Strong programmers also tend to dismiss the opinions of those making strong claims they know to be untrue. Every technical mailing list has plenty of examples, some quite funny. (Strong programmers can be quiet clever in their putdowns.) Generally, if one is trying to convince others of the validity of one’s ideas, it pays to be modest and willing to learn about points where one may be less expert than the person one is trying to convince.

Advertisements