July 23, 2007

The State of the Art of Paper Reviewing…

Posted in Life at 10:31 pm by pmatos

Fortunately, we have great conditions here in the new ECS building in Southampton University. One of the nice things is definitely the coffee room where we meet sometimes and in some of these times we have very interesting discussions. One of the last discussion was about a subject which I’ve been about to write, but this discussion removed all the inertia I might have had to write about it.

Some of the people present were Noura Abbas, Florian Letombe, Jordi Planes, John Colley and myself (in no specific order) and everything started because Florian was really mad about some software he is working with because it was ill-programmed, ill-documented, and with just a few modification he was able to improve not only its readability but its performance. From his growl of horror after having to look at this piece of code, I grabbed this issue and generalized it…

Truth is, that some research software is very very good. Check for example, the MiniSAT SAT solver of the SAT4J SAT solver (which Daniel le Berre developed in Java and is fast… against all odds ;)). However, most of the software is developed for the sake of getting some publishable results and once the paper is published, that’s it. If you have the luck to get the source code you’ll have to spend an obscene amount of hours to understand it, if you don’t have the source code you need to gently ask for it. Or sometimes, you’ll only have the chance to get your hands on a binary in order to compare or reproduce the authors results. It seems most of this software is developed for that specific deadline and the results are forced to appear the night before the deadline so the it can be inserted into the paper and submitted. This should be unacceptable. If high standards of technical writing, results, ideas, concepts or techniques, are required from a published paper why are there no standards on the way the code is developed and after all on the way the results are obtained? The good solution would be to have a Code Committee to review the code  of, at least, some submitted papers to ascertain that its quality matches the ones in the paper and that the results are indeed reproducible. Unfortunately, I think this doesn’t happen due to the lack of manpower, to which I have no solution…

Another issue is the quality of papers themselves… How and why is it possible that some papers containing errors, which seem to come out of an horror movie, are published, some of them in well known conferences??? I guess the answer is that some reviewers are just sleeping while reviewing or don’t care about paper quality in the final proceedings. All of the people present in the meeting were able to know a paper like this. Noura mentioned the case of a paper that a colleague went through a set of equations to find in the end that they contained some errors. Other issues like this, some more flagrant were mentioned. Florian suggested a remarkable solution, which although simple would end these issues. In the final proceedings the reviewers of each paper should be included in the header of the footer of the paper pages, like (as Florian remarked) is done on some Mathematics Conferences. Now, I wonder, why is this not done in our Computer Science conferences? Why can’t we point the finger to those who do not care about published papers quality? Why can’t we applaud those who throughly review papers, accepting those that match conference quality and reject those that don’t?  Even if the previous suggestion of having a Code Committee would be hard to implement, this one is trivial and should put in practice today! It seemed to be common agreement that if one day we get to manage some kind of conference, this should be done. :)

Incredibly, although some reviewers just don’t care, science keeps going forward. Every day in major conferences, new techniques, algorithms, data structures, proofs, ideas, encodings etc are presented and is very very difficult to keep up with the latest advances. What this means is that there are incredibly hard working reviewers which are not only doing their own work but also trying to diminish the blow other lazy reviewers provoke on the overall quality of conference proceedings. To these hard working people, I and — I’m sure my friends — wish all the best and give them good round of applause, I’m sure they know who they are.

1 Comment »

  1. Carlos Serrao said,

    I don’t know for how long you are in the academic and scientific world, but I can say that I’ve been here for a while.
    From my experience what I can is that unfortunately, science has become more and more a business. I’m an author of several articles in conferences and in Journals, and I can tell you that they are not the best well written papers in the World, some have been accepted and others rejected (this is normal in science). The major problem with this is that when you read the comments from the reviewers, they are completely strange and most of the time denotes that the person who reviewed the paper is not at all an expert in the topic being discussed in the paper.
    I had also the opportunity to organize some conferences/workshops and to review some papers – it’s true that you can find everything. You can even find some guys from the Middle East and Africa, that submit the papers to conferences, in the hope that they be accepted, just to get a Visa to get out of the country, and afterwards they never appear or attend the conference. Pretty smart, hey?
    I also believe that in some conferences paper revision is just a light step for the paper to be accepted and published (and the quality is questionable for sure). If you are looking for quality conferences and for good paper revision, you should stick with IEEE or ACM ones.
    The case for Journal articles is completely different. The revision of papers can take up to a year, and even so the paper could be rejected at the end – and this is a multi-step process, with several interactions between the reviewers and the authors. Here revision, is much more serious and takes much more time.
    I also agree with you that the reviewers name should appear in the published paper, but only then. The authors should never know who is reviewing their papers, neither the reviewers should know the name of the authors during the revision process – that’s called blind revision.
    This is a complex issue… it would take a long time to discuss it. I’ll probably make a post on my blog, about this issue, using this comment as a start. That’s definitively a good topic.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: