Comment articles — are they worth it?
I have recently published a comment manuscript. This is the second time I’ve been involved in substantively critiquing published work in the literature, and the first I’ve taken this through the comment route (the other matter was dealt with via a primary research article).
This comment has now been published, and I am left reflecting on the process. I am unsure that this comment is going to be good ‘value for money’ in terms of time and energy invested. Ultimately I am left longing for a fairer system of checks and balances in the literature.
In case you don’t read further — we could have avoided this whole process if the data was released as an open data bundle (I’ll write on this soon!), and if the peer review process was refreshed.
The papers
To get these out of the way, if you interested in titanium alloys and high resolution electron backscatter diffraction (EBSD), you can read the articles following the links below. Otherwise scroll to the next section…
The original article:
The comment:
The response:
Why did I write the comment?
I reviewed the original manuscript in another journal. I found flaws. I sent them in anonymously, via peer review. My reviewer comments seem to have been ignored in this process, either by the editor of journal 1, or by the authors. The article was published, with minimal changes, in its new venue. I was alerted that the new manuscript was published through an article ‘citation alert’.
I wrote to the corresponding author and an author who I know in person. I received no response (this may have been a quirk in the email system — who knows?). I therefore contacted the Editor of Advanced Engineering Materials, and raised my comments. The Editor forwarded my comments to the authors’ and we had an exchange mediated by the editor. I found their explanation of their data analysis unsatisfying, and I expressed this to the Editor, who suggested that I write a comment.
At this point, less stubborn people may have left the matter rest. However, I remain convinced that the original manuscript is flawed, and I already spent time writing down my arguments in contacting the authors to address these issues. Fundamentally, I care about this manuscript as it uses a technique, high-resolution electron backscatter diffraction, which I’ve worked very hard on for several years. I feel that the technique is tremendously useful, but data obtained using this method has to be scrutinsed carefully before publication and industrial use (as you can have a ‘garbage in garbage out’ situation).
What do I hope to have achieved by the comment?
I want to highlight to the community how the results obtained within the original manuscript could have been obtained, and to highlight what checks users of the software tools could use to check whether their data is reasonable.
If you have a peek at my response, you’ll see that I try to articulate key issues that can arise when data is processed and to encourage readers to consider their data in light of materials physics, and not just what the computer spits out. At the end of the day, good science is more than just using a piece of software on an exciting problem, it is the holistic attempt to scope your findings within the context of the domain of the scientific problem.
Processes matter
The process of publishing this comment feels like it was flawed from the get-go.
The original authors’ were well within their rights to transfer the manuscript to another journal. We have all received reviews that we strongly disagree with. However I find it surprising that the discussion within the manuscript was not updated to reflect comments made in the first round of review.
For this paper, my comment originated as an email exchange, via the editor only, where issues were outlined. I felt that their response to my query was not substantive, and I was told that I could write a comment. Upon writing my comment (which is more tedious than an informal exchange, as I had to take my sketches and formalise them). Furthermore, I had to find figures that would substantiate my claims which could be assessed by a third party.
The comment was accepted without peer review, but it was seen by the authors in private. My comment was held in a private publication queue, before a response from the authors was issued. Due to a ‘quirk’ in the publication system from the journal, the response to my comment was issued in advance of my comment (see the publication dates above).
This was especially maddening, as I did not see their comment until I checked for my own. Furthermore, in an amusing twist this journal does not feel that it is reasonable for a response to a comment to cite the original comment within the text (so there is a fun effect that their response was originally published responding to a paper that did not even exist).
Why do I feel frustrated by the whole process?
In response to my comment, I feel that the authors have used qualitative words that are not rigorous and my two substantive critiques of their data and analysis remain. I maintain that I would be happy to run their data through my analysis code, and I would retract or amend my comment should my evaluation be incorrect.
Advice for future comments
Ask yourself, is it worth it? What do you hope to achieve? There are fewer and fewer comments, and therefore editorial teams are inexperienced at handling them. The outcome of a comment process is unlikely to leave you satisfied. There are alternative methods of resolving the issue, such as with a quiet phone call, a quick blog post, a hearty run, or even a primary article that demonstrably clarifies issues in the original work.
However, if you wish to pursue a comment, I recommend: (1) publish the comment in a journal that is separate to the journal of the original paper, this removes conflict from the publishing house from the process; (2) publish your comment to a preprint archive, as you send it to the editor for the final time; (3) be polite, professional and try to reason your arguments with care, such that a third party can evaluate the conflict and decide who to believe.
Ultimately, I have no idea of whether I have burned a friendship in this process, or what I will have gained. Truth be told, I do not have any confidence on reading how this episode will be viewed by my peers.
If you like this post, please hit the green heart and recommend it to others.
You can head over to Twitter to follow Dr Ben Britton as @BMatB, or keep up to date with the group’s work via @ExpMicroMech. We can also be found over at http://www.expmicromech.com
