Peer Review in Public

This afternoon, partially as a way to procrastinate from my own writing and partially because I was genuinely interested in the invitation, I participated in an “open review” of Remi Kalir and Antero Garcia‘s forthcoming manuscript, Annotation. Their open review process will continue through August 23, 2019, so jump in! They request that commentary adhere to the following, all good advice for any scholarly dialogue:

Civil. We can disagree. And when we do so, let’s also respect one another.

Constructive. Share what you know. And build upon ideas that are relevant and informative.

Curious. Ask honest questions and listen openly to responses.

Creative. Model generative dialogue. Have fun. Contribute to and learn from the process.

Having read hundreds of academic articles in the past 20 years, as well as offering blind peer review for dozens more, as well as blind reviews of probably two dozen academic books, I thought that this would be interesting. (And, again, I was procrastinating on my own writing, so an engaging intellectual task that can carry me away and still feel like I am getting work done is always welcome). Here are a few things I learned while reviewing their book which, again, you, too, can contribute to through August 23rd.

My Stance as a Reviewer

When I offer peer review to academic articles and books, I am typically using the “track changes” and commentary features in Word or, in some instances, by offering comments and edits on a PDF (my favorite tool for doing that is the iOS app Good Reader). I typically frame these comments as direct suggestions to the author(s) of the article/manuscript I am reading, and I engage in a professional, yet conversational tone.

With my review of Annotation today, I think that I maintained some of that approach, yet I knew that my comments would be captured, in perpetuity, in Kalir and Garcia’s public version of the document. While I didn’t hold back with questions and concerns, I did realize that I changed my tone. Whereas I would try to be explicitly clear in comments and questions (perhaps even providing examples of what I was aiming for with unclear writing) in blind review, I didn’t want that to be part of the public record.

For instance, in the example below, I offered a comment that could spark further dialogue amongst others reading the text, pushing toward some broader implications for teaching and learning. At other points, I was replying to the comments already made by others, and I would specifically say something like “I agree” or “Along these lines.” Also, at points, I directly wrote to Kalir and Garcia in ways that I could do so with colleagues I know, and would be comfortable saying in front of a group of others.

Screenshot of Specific Comment on

My Commenting Style in an Open Setting

Yet, still, it felt strange. In the first few chapters, there were some other annotations/ors, yet they fell away. Even those that remained were offering suggestions for links, not the generative kinds of peer review that (I hope) I have always aimed to offer in the peer reviews that I complete. For instance, I would describe problems and ask questions like:

  • I may simply not be reading this right, but making the comparison of submitting an expense report in relation to the openly annotated future just didn’t ring for me here. Sorry, but perhaps you could find a different example?
  • This is an interesting example, but I don’t know that it fully draws out all the ideas that you mentioned above related to “shifting social norms, changing financial and organizational incentives, and evolving scholarly practices.” Perhaps you could reorganize around — and particularly elaborate upon — these three ideas in relation to SciBot?
  • This is an important, if technical, point, and deserves some elaboration. Why is it important that some are built into the browser, whereas others stand alone. And, for that matter, why have you not mentioned OneNote, Evernote, Google Keep, or SimpleNote anywhere in the text, and especially here before you launch into the important questions you pose below?

By the end of the process — which took me just as long as any other book review — I began to wonder/wander, leading me to other directions.

Reflecting While Reviewing

Of course, during a normal review, the kinds of internal dialogue that I have with myself may make it into the first draft of my comments, but I usually do some editing before a final draft heads off to the editor. Here, I figured that Kalir and Garcia’s invitation to be civil, constructive, curious, and creative would welcome some of these thoughts.

As I went through the process, and saw fewer and fewer reviewers in subsequent chapters, I got discouraged. While this is no fault of the authors, and I know that they have extensively shared their open manuscript, welcoming reviews, it does make me worry a bit about the hive mind, and whether the power of collaboration and collective intelligence is, perhaps, not as powerful as we might hope. A few of my musings, especially as they relate to why scholars may choose not to participate in an open review:

  • This [vision of social annotation and scholarship] is aspirational, and I appreciate it. Yet, I think that you can elaborate more on what actual changes would need to happen to make it a reality. Be specific, and talk about faculty workloads, department/college T&P requirements, and the ways in which “open” is still perceived as subpar.
  • And, yet, there still seems to be reluctance, or at least lack of widespread acceptance [of open review]. For instance, in your attempts to make this manuscript open and accessible (which I applaud), I am still wondering how many total scholars will participate. Even for those of us who saw the invitation to begin with, a gentle nudge was in order for us to participate. And, in the end, I don’t know that my review of this manuscript will “count” on par with doing a review for an established journal or publisher when (and if) I include it in my promotion materials. Of course, for me at least, this doesn’t matter as much as it would to a junior faculty member who needs to decide whether to spend a few hours trying to write her own work, or to participate in a “normal” editorial review board/process as a blind reviewer for an established press/journal. Both of those actions are rewarded in the academy. As much as I respect Remi and Antero (and that’s why I am doing this annotated review), the simple fact of the matter is that I am doing this because I care, not because it will “count.” These are part of the material reality of academe, and I don’t know how we will change that, even with open annotation and peer review. At the end, there is only so much time in the day…
  • So, I have held off until now, but I have to ask… and only partially in a cynical manner… Like the tree falling in the forest, does an annotation really make a sound (ripple, impact, effect, etc)? That is, I appreciate your utopian vision, yet I wonder if you might want to reign it in a bit here. Sorry… not trying to pop the bubble, especially after nearly two hours of reviewing and annotating your manuscript, but I am just being realistic. The first few chapters had a few annotators. Now, here at the end, it is just me. And you two, as the authors. Are we really connected to a “robust information infrastructure?” Or, are the three of us walking alone in the woods?

In the end, I appreciate the opportunity to do this review, and to pause here to reflect on the process. I struggle both with how to structure class discussions in digital spaces as well as how to be a social scholar, so reading Kalir and Garcia’s manuscript was serving many more purposes for me than merely procrastinating on my writing. I am hopeful that the ideas I have offered to them (and those who might continue to annotate over the next month) are helpful. And, of course, I will continue to think about practices of annotation in my own scholarship and teaching.

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Prepping for KQED Summer Bootcamp on “Developing and Assessing Digital Writers”

Camp KQED Teach 2018 LogoThis week, KQED’s Bootcamp on “Developing & Assessing Digital Writers” kicks off, with the overarching idea that

[b]logging can help develop your students’ digital writing by combining traditional writing (text) with a digital platform (sharing online), along with the opportunity to incorporate other forms of media-making.

My role for the bootcamp will be to provide a brief, asynchronous presentation called “Rethink the Link.” And, in working with KQED’s Jordan Stewart-Rozema to prepare my session, I’ve been (re)thinking (over) a number of ideas.

In short, I want to help teachers consider when, why, and how we invite students to create hyperlinks in their digital writing, in addition to considering the typical questions of where, what, or to whom they will be linking.

To that end, I’ve been gathering up a few resources, beginning with Vannevar Bush’s essay “As We May Think” and his original conception of the memex as

a future device … in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

We will return to Tiffany and Bud Hunt’s essay from 20 years ago in English Journal, “New Voices: Linkin’ (B)Logs: A New Literacy of Hyperlinks” and explore M-W’s definition of “link.”

From the perspective of “link” as a verb, we will think about what a writer does by including a link, considering the kinds of reaction(s) she might want from her readers. As a noun, we will consider how the connection to other ideas serves the writer by invoking the broader academic conversation.

If you are interested in thinking about linking — and blogging more broadly — then there is still time to sign up. See you in the KQED Bootcamp community!

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

#MichEd Chat – 4-11-18 at 8:00 PM EST


#MICHED CHAT 4/11/18

Wednesday, April 11th, 8-9pm EST

The idea of a professional learning network has existed for quite some time, built on some of the foundational work related to “situated learning” and “communities of practice” developed by Jean Lave and Etienne Wenger in the 1990s.

With the emergence of Web 2.0, Stephen Downes described “learning networks in practice” in a 2007 paper, arguing that “The idea behind the personal learning environment is that the management of learning migrates from the institution to the learner.”

Combined with the 2006 emergence of Twitter, a new idea had taken form, and educators began using hashtags to start a variety of ed chats, including our own #MichEd which was inaugurated Nov 7, 2012.

Chat Questions

This week, we reflect on our own experiences being a part of the #MichEd network and, more broadly, what it means for each of us to develop our own PLN. We will be joined by students from CMU’s Doctorate in Educational Technology, and the chat will be hosted by Troy Hicks. During the chat we will consider:

  1. What motivates you, personally, to create and maintain a PLN?
  2. How do PLNs change with time, for you personally and across the network? Think about #michED and who was there at the start, who has joined, who has left (or is less active) and WHY?
  3. How do we keep our networks diverse in thought? We don’t want them to be echo chambers for our ideas, but to be constructive spaces for dialogue. How can we achieve that goal?
  4. Besides sharing great resources, what can a PLN teach us about how to be an educator? How does participating in a PLN become part of your professional persona?
  5. OK, let’s get specific. What, exactly, can we learn from PLNs? Along with soft skills of collaboration and sharing resources, what other digital or pedagogical skills can we learn?
  6. Finally, what’s next for PLNs? How can we nurture and sustain them? How can we invite new voices? What should a group of doctoral students studying educational technology be thinking about?