Author: hickstro

This afternoon, partially as a way to procrastinate from my own writing and partially because I was genuinely interested in the invitation, I participated in an “open review” of Remi Kalir and Antero Garcia‘s forthcoming manuscript, Annotation. Their open review process will continue through August 23, 2019, so jump in! They request that commentary adhere to the following, all good advice for any scholarly dialogue:

Civil. We can disagree. And when we do so, let’s also respect one another.

Constructive. Share what you know. And build upon ideas that are relevant and informative.

Curious. Ask honest questions and listen openly to responses.

Creative. Model generative dialogue. Have fun. Contribute to and learn from the process.

Having read hundreds of academic articles in the past 20 years, as well as offering blind peer review for dozens more, as well as blind reviews of probably two dozen academic books, I thought that this would be interesting. (And, again, I was procrastinating on my own writing, so an engaging intellectual task that can carry me away and still feel like I am getting work done is always welcome). Here are a few things I learned while reviewing their book which, again, you, too, can contribute to through August 23rd.

My Stance as a Reviewer

When I offer peer review to academic articles and books, I am typically using the “track changes” and commentary features in Word or, in some instances, by offering comments and edits on a PDF (my favorite tool for doing that is the iOS app Good Reader). I typically frame these comments as direct suggestions to the author(s) of the article/manuscript I am reading, and I engage in a professional, yet conversational tone.

With my review of Annotation today, I think that I maintained some of that approach, yet I knew that my comments would be captured, in perpetuity, in Kalir and Garcia’s public version of the document. While I didn’t hold back with questions and concerns, I did realize that I changed my tone. Whereas I would try to be explicitly clear in comments and questions (perhaps even providing examples of what I was aiming for with unclear writing) in blind review, I didn’t want that to be part of the public record.

For instance, in the example below, I offered a comment that could spark further dialogue amongst others reading the text, pushing toward some broader implications for teaching and learning. At other points, I was replying to the comments already made by others, and I would specifically say something like “I agree” or “Along these lines.” Also, at points, I directly wrote to Kalir and Garcia in ways that I could do so with colleagues I know, and would be comfortable saying in front of a group of others.

My Commenting Style in an Open Setting

Yet, still, it felt strange. In the first few chapters, there were some other annotations/ors, yet they fell away. Even those that remained were offering suggestions for links, not the generative kinds of peer review that (I hope) I have always aimed to offer in the peer reviews that I complete. For instance, I would describe problems and ask questions like:

• I may simply not be reading this right, but making the comparison of submitting an expense report in relation to the openly annotated future just didn’t ring for me here. Sorry, but perhaps you could find a different example?
• This is an interesting example, but I don’t know that it fully draws out all the ideas that you mentioned above related to “shifting social norms, changing financial and organizational incentives, and evolving scholarly practices.” Perhaps you could reorganize around — and particularly elaborate upon — these three ideas in relation to SciBot?
• This is an important, if technical, point, and deserves some elaboration. Why is it important that some are built into the browser, whereas others stand alone. And, for that matter, why have you not mentioned OneNote, Evernote, Google Keep, or SimpleNote anywhere in the text, and especially here before you launch into the important questions you pose below?

By the end of the process — which took me just as long as any other book review — I began to wonder/wander, leading me to other directions.

Reflecting While Reviewing

Of course, during a normal review, the kinds of internal dialogue that I have with myself may make it into the first draft of my comments, but I usually do some editing before a final draft heads off to the editor. Here, I figured that Kalir and Garcia’s invitation to be civil, constructive, curious, and creative would welcome some of these thoughts.

As I went through the process, and saw fewer and fewer reviewers in subsequent chapters, I got discouraged. While this is no fault of the authors, and I know that they have extensively shared their open manuscript, welcoming reviews, it does make me worry a bit about the hive mind, and whether the power of collaboration and collective intelligence is, perhaps, not as powerful as we might hope. A few of my musings, especially as they relate to why scholars may choose not to participate in an open review:

• This [vision of social annotation and scholarship] is aspirational, and I appreciate it. Yet, I think that you can elaborate more on what actual changes would need to happen to make it a reality. Be specific, and talk about faculty workloads, department/college T&P requirements, and the ways in which “open” is still perceived as subpar.
• And, yet, there still seems to be reluctance, or at least lack of widespread acceptance [of open review]. For instance, in your attempts to make this manuscript open and accessible (which I applaud), I am still wondering how many total scholars will participate. Even for those of us who saw the invitation to begin with, a gentle nudge was in order for us to participate. And, in the end, I don’t know that my review of this manuscript will “count” on par with doing a review for an established journal or publisher when (and if) I include it in my promotion materials. Of course, for me at least, this doesn’t matter as much as it would to a junior faculty member who needs to decide whether to spend a few hours trying to write her own work, or to participate in a “normal” editorial review board/process as a blind reviewer for an established press/journal. Both of those actions are rewarded in the academy. As much as I respect Remi and Antero (and that’s why I am doing this annotated review), the simple fact of the matter is that I am doing this because I care, not because it will “count.” These are part of the material reality of academe, and I don’t know how we will change that, even with open annotation and peer review. At the end, there is only so much time in the day…
• So, I have held off until now, but I have to ask… and only partially in a cynical manner… Like the tree falling in the forest, does an annotation really make a sound (ripple, impact, effect, etc)? That is, I appreciate your utopian vision, yet I wonder if you might want to reign it in a bit here. Sorry… not trying to pop the bubble, especially after nearly two hours of reviewing and annotating your manuscript, but I am just being realistic. The first few chapters had a few annotators. Now, here at the end, it is just me. And you two, as the authors. Are we really connected to a “robust information infrastructure?” Or, are the three of us walking alone in the woods?

In the end, I appreciate the opportunity to do this review, and to pause here to reflect on the process. I struggle both with how to structure class discussions in digital spaces as well as how to be a social scholar, so reading Kalir and Garcia’s manuscript was serving many more purposes for me than merely procrastinating on my writing. I am hopeful that the ideas I have offered to them (and those who might continue to annotate over the next month) are helpful. And, of course, I will continue to think about practices of annotation in my own scholarship and teaching.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.