Some Thoughts On Peer Review In The Global Internet Context

Transcription

1 Some Thoughts On Peer Review In The Global Internet Context Narayanan Komerath Daniel Guggenheim School of Aerospace Engineering Georgia Institute of Technology Atlanta IMETI 2010, Orlando, FL June 2010

2 What is a scholarly or peer reviewed journal? Scholarly and professional journals feature articles written by researchers and practitioners in a particular subject area. The authors often have particular specialties. Peer groups of researchers, scholars and professionals within a specific discipline are the audience for scholarly literature. Well-accepted indicator of quality scholarship. Process by which an author's peers read a paper submitted for publication. A number of recognized researchers in the field will evaluate a manuscript and recommend its publication, revision, or rejection. Articles accepted for publication through a peer review process implicitly meet the discipline's expected standards of expertise. Articles selected by an editor or board. Standards of scholarship in such journals are often equal or comparable to those of peer-reviewed publications, although this is not always the case.

3 From West Virginia University Library What is a Peer-Reviewed Journal?..Sends articles submitted by authors to a group of experts in the field for review before publication. The review process helps to ensure that research published in the journal is of high quality and contributes new information to the field. Characteristics of Peer-Reviewed Journals: Purpose is to share the results of original research with the rest of the scholarly world. Usually have a plain cover with lengthy and in-depth articles that contain charts, graphs, and tables...minimal use of photographs and color. Any advertising is geared towards the subject discipline. Articles are written by researchers, scientists or scholars in the field. Authors cite their sources in footnotes or bibliographies, which are often extensive. Uses the technical terminology appropriate to the discipline and assumes that the reader will have a similar scholarly background.

4 peerreview.gif

5 paradigm/spring_2008/img/peer_review.jpg

6

7

8 Reviewer s S.O.P. According to n.k. 1. Read the paper back and forth at least 3 times. Relate conclusions to what is shown. Put away for a couple of days and mull on the paper. 2. Consider originality. Search for related prior work if necessary. 3. Follow author s train of thought and check for reasonableness. 4. Read carefully and note questions 5. Compose review and pose questions 6. Develop summary recommendations and ratings 7. Get the review back to the editor in a timely manner. 8. If requested to review again, do so quickly. A good peer review should engage the brains of the authors and the editors. Takes time to do it right, but your professional standards are at stake. How would you like someone like yourself as reviewer when you are the author?

9 My Background 32 years, full-time researcher/ academic tenure track/ tenured professor in a major technological institution - Combustion diagnostics / propulsion experiments - Turbulent combustion/ fluid dynamics - DSP / Optical and other diagnostics development - Experimental aerodynamics / wind tunnel operations / vortex flows / combat aircraft / helicopters - Guiding 15+ engineering PhDs. - Institutional peer review panels. - Proposal and paper reviewer for national and international organizations ~ 250 publications, ~90 of them peer-reviewed. ~ 14 years internet / interdisciplinary / digital library / resource building & maintenance ~ 14 years in formally multidisciplinary endeavors ~ 12 years in Strategic / International / Security affairs ~ 8 years in social sciences interactions in community vs. academics

10 Some Questions From the Cross-disciplinary Perspective Prevailing Definition Journals feature articles written by researchers and practitioners in a particular subject area. The authors often have particular specialties. Peer groups of researchers, scholars and professionals within a specific discipline are the audience for scholarly literature. Well-accepted indicator of quality scholarship. Process by which author's peers read a paper submitted A number of recognized researchers in the field will evaluate.. and recommend publication, revision, or rejection. Articles accepted for publication implicitly meet the discipline's expected standards of expertise. If Articles are selected by an editor or board: Standards of scholarship in such journals are..not always comparable to those of peer-reviewed publications Questions/Criticisms Narrow? Are credentials relevant? Excuse for incompetent reviewing? Tactic to keep out disruptive ideas? Modify to accommodate crossdisciplinary endeavor? Circular metric? Do they really? CAN they? Are researchers scholars? Circular logic? What is the absent process here? Are Editors and the Board of a journal experts? Researchers? Scholars?

11 Killing (or sneaking in) a Paper is Easy if the Editor is Asleep (or Worse) The dual role of quality control and motivation encounters conflicts of interest, especially when there are underlying competitive aspects and financial pressures. Delay Dismissive rejection, or glowing acceptance Unusually harsh or mild comments Demand for additional work Delay following rebuttal Re-review to think up new excuses for rejection

12 So why have Peer Review? Thoughtful review and comments from interested and knowledgeable persons is more relevant than ever to motivating original thought, disciplined research and progress Help set in context of prior work Help author improve thinking, calculation, presentation and writing: Ask questions that may not have occurred to author Catch errors Provide opportunity for author to rebut/revise/explain Increase utility to other readers through an open discussion of important issues. What I have deleted is Accept only High-Quality Papers

13 Observations on rejecting papers Some process for rejection is essential, if only to encourage other authors. High rejection rate is a low-quality metric of high quality. Professional organization s responsibility to not publish nonsense is outweighed by the risk of rejecting innovative work. There are better ways..

14 Not possible in traditional model, to eliminate conflicts of interest. Reviewers are anonymous, and do not have to defend their (in)actions Editors can delay publication Willing reviewers are hard to find Competent reviewers are harder to find and are usually competitors

15 Evolving technology and habits of researchers, and the attendant fragmentation of time that many experts face, have driven traditional peer reviewed paper journals to the verge of irrelevance. Adversarial / arrogant attitude (IP is author s!) Poor value addition by sloppy reviewing Cost ( paper charges ) Delay in review and publication Poor visibility: Libraries cannot afford to subscribe to all, or most, journals Misses the Google Searcher Audience Citations will be more on papers that are accessible! Present generation does not follow linear or hierarchial methods of knowledge organization If u can t b found on Google screens 1-3, u don t exist

16 Casual adoption of technology in review systems poses its own problems, especially when used in reviewing innovative work Review Panels typically support the status quo (priceless quote from an experienced authority from this conference!) Adding technical convenience does not improve the amount or quality of time and thought devoted to review. dibujo _peer_review_not_always_useful_for_editors.jpg

17 Academic promotion/tenure processes are relying more on prestige and percentile rankings in a field metrics along with number counts of publications Fragmentation & proliferation of journals inhibits cross-disciplinary innovation. I reject this route as being destructive of original thought and cross-disciplinary knowledge dissemination. uploads/2008/06/peer-review-tears.png

18 Professor Komerath is one of the Top 3 Experts in the World in measuring pressure-velocity correlations in turbulent premixed flames using cross-spectral analysis of response-compensated Pitot microphone probes with ceramic tips and a coherence metric (Great, but who else would want to join that club?) Interim solution adopted by specialist communities to the lack of competent expert reviewers in their narrow fields: Creation of ever tighter peer circles, leading to fragmentation of knowledge and the inability of any library to maintain a comprehensive journal base. Thus the most prestigious publications may be so exclusive that few can even find them.

19 Scholarly publication and academic freedom vs. responsible community behavior Egregious examples of abuses in religion academia. Refusal of the Academy to publish well-informed rebuttals and exposures from lay practitioners destroys academy s credibility. Journals feature articles written by researchers and practitioners in a particular subject area Peer groups of researchers, scholars and professionals within a specific discipline are the audience for scholarly literature. Lesson: Scholarly communities that do not welcome and facilitate well-reasoned disagreement from lay persons, are not worthy of respect

20 Technological capabilities and evolving trends in capturing knowledge from prior work Internet Search Engines Google Books and Google Scholar Delivery to personal communication devices (Iphone, Ipad) Global access Moderated Discussion forum Twitter

21 Open, two-stage peer-review From Nature (2006) doi /nature04988 Journal of Atmospheric Chemistry and Physics Stage 1: Rapid pre-screening (access review) and immediate publication as discussion paper on website. Public interactive discussion for 8 weeks, with comments of designated reviewers & others are published alongside the discussion paper. Reviewers sign or remain anonymous, but comments by other scientists must be signed. Stage 2: traditional manuscript revision and peer review. If accepted, final papers are published in the main journal. Every discussion paper and interactive comment remains permanently archived and individually citable.

22 Peer review model for cross-disciplinary innovation Model Rationale Web-based journal with periodical volume and date identifiers Submitted on-line in specified format Assigned to 3 anonymous reviewers by editors after editorial review. Article posted on-line for comments (BLOG model). Author can respond. Anonymous review comments sent to author only. Author has fixed period to modify or withdraw article Where disagreement remains, both the anonymous comment and the author s rebuttal are published. Article plus discussion becomes archival record. Further comments published as new. Archival, with swift global search enabled For possible hardcopy publication Editor can reject. Review assignment all over the world No censorship. Comments moderated. Encourages reviewers to help author improve article without penalty Main quality control: Archival reader sees countering opinions and author s rebuttal Captures discussion and allows reader to form an informed opinion Provides closure in a reasonable time.

23 Preliminary empirical observations Social-media forum to motivate experts to participate: Examples: 1. Strategic affairs discussion forum 2. Space Solar Power forum 3. Discussions on major news items (Gulf of Mexico oil leak) Most engineering journals are not likely to generate many comments from readers. Getting authors to submit articles, requires recognition of quality control processes and archival reliability of journal Strong moderation is required Author ego is a problem with many authors unused to social media Notes: J. Atmospheric Science reports excellent impact factor. Deters submission of low-quality papers: low rejection rate

24 Conclusions Purpose of peer review should be carefully re-emphasized Abuses cannot be avoided with traditional paper journal review system Swift & unobstructed publication essential. Open but moderated, interactive discussion provides good quality Archived reviews/rebuttal is a good experiment Motivating participation requires thought and resources ACS.gif

25 ACKNOWLEDGEMENTS The work reported in this paper was made possible by resources being developed for the EXTROVERT cross-disciplinary learning project under NASA Grant NNX09AF67G S01. Mr. Anthony Springer is the Technical Monitor. docinthemachine.com/ wordpress/wp-content/ uploads/2007/02/ gspz0625.jpg