Senior Scholars Journal Review Quality Survey

The inaugural 2018 survey asked authors to assess the quality and timeliness of reviews they had received in 2017 from the eight basket journals (excluding desk rejects). We received 237 responses, with most journals receiving 40-60 responses. MISQ received the most (92 responses), and JIT and JSIS the fewest (20 each).

The first analysis is the Net Promoter Score, which is a measure of loyalty to the journal. It asks respondents whether they would recommend the journal, using a 0-10 scale. Those responding 9 or 10 are “Promoters”, while those responding 0-6 are “Detractors”. The Net Promoter Score is the percent of Promoters minus the percent of Detractors, and thus is on a scale of -100 to +100.

The second analysis is a 2x2 graph of Review Timeliness by Review Quality (3 items, quality, fairness, helpfulness; alpha=.90), both of which were measured on 7-point scales with 4=Neutral.

Comments Explaining Review Ratings

We also asked respondents to provide comments explaining the reasons for the ratings they gave. Most respondents did not provide any comments, but a total of 80 individuals provided comments to support their ratings -- either praising the review process or suggesting areas for improvement. Sometimes these comments were interrelated (i.e., a good SE helped make a good editorial review process) but we picked only one primary reason from each comment for each review mentioned. Below are tables indicating the positive and negative reasons for a rating, the number of times that type of reason was mentioned, and an example or two.


Reasons for Positive Review Experiences


# Mentions


Good editorial review process


“It was all good in the main”;

Good review


“In general, my experience in the past year was that the reviews I received were of very high quality across the journals that I submitted to.” “Very helpful and unbiased reviews.”

Developmental review


“The reviews were very constructive. It was clear that reviewers and the editor were looking for opportunities to improve the work rather than reasons to reject.”

Good team/team member


Good SE; Good AE; Good Reviewer; “We received a revise and resubmit decision promptly from the senior editor. She identified a number of areas which should be addressed before the paper was sent out for review, indicating we should resubmit the paper with her comments and our response attached. This has been extremely helpful and has improved the quality of the paper, hopefully reduced the review cycle, and the burden on reviewers. Although it is a big ask of senior editors, it is certainly a process that gives a good experience for authors.”



“Reviews were timely.”





Reasons for Negative Review Experiences


# Mentions


Took too long


“… the paper was with the AE after receiving the reviews for over 9 months. In the end the SE intervened after multiple inquiries.” “My colleagues and I waited eleven months to receive reviews from XXX on a paper we submitted (this was an initial submission). We contacted the XXX editorial office after eight months, but doing so appeared to have no effect on the timeliness of the review process.”

Decision not justified


“There was a difference between the reviews and the SE reports. Two of the three SE reports, were very dismissive and to quite some part ignoring the reviewers' comments and recommendations, sounding as if reasons were constructed to reject the papers.” ”Reviews were good, but decision seemed arbitrary and not supported by the reviews.”

Lack of expertise of team member (s)


“From the reviews it was clear that the reviewers did not have the right expertise” “the SE in each case went considerably beyond his or her expertise and/or overruled the much more knowledgeable reviewers.” “Review team was did not quite understand the research methodology” “Although we asked for AEs with background in our research domain [two different journals] (on two different papers) did not assign an AE with relevant background or knowledge. I find this review process non-scientific.”

Poor review


“Obviously, the reviewers spent little time to prepare their reviews (e.g., cited papers which obviously do not address the problem context of the submitted paper).” “most of the time the reviews are biased, unworkable, and ego-driven”

Poor editorial review process


“The completely opaque editorial process of XXX is a joke.” “This was by far the worst experience I have ever had. First, I suspect that the SE knew very little about the subject area. Second, he took almost a year to submit his report. Third, he didn't care to respond to our mail seeking clarifications on the revisions that had been recommended. Fourth, and most appalling of all, he sent us the reviewers' comments but not their final recommendation.”

Review not developmental


“In many cases, reviewers mention what the paper is lacking but don't say anything about how to fix the shortcomings.” “Some of the comments are not constructive or the directions are not clear”

Random aspects of review process


“The peer review experience in general continues to be based on luck of the draw.” “So much depends on who you get as an AE/SE”

Bad team member(s)


“The AEs for both journals could have been more helpful in helping me to navigate the comments and make sense out of competing recommendations by the reviewers, when they occurred.”



“The paper was judged out of scope, and I totally disagree. Our paper was about the IT profession and the professionals.”



“Extremely difficult to promote new ideas.”





Connect With Us :