In the wake of the Hoffman Report (if you missed it, it details how the American Psychological Association was...well...a touch complicit in torture. Not good), there was a lot of interesting discussion among social psychologists about what exactly to do. Not in terms of whether we should be facilitating torture (I'll go out on a limb and say that we don't like that). But rather about how it could/should alter our publication practices.
Thing is, the flagship journal of social and personality psychology, the Journal of Personality and Social Psychology, is an APA journal. Do we social psychologists really want to be affiliated closely with APA in light of the Hoffman report? Sanjay Srivastava had a really nice blog post detailing a lot of the intricacies of SPSP's relationship with APA. And there was a nice discussion over on the ISCON facebook page (tip of the cap to Alexander Etz for teaching me how to link that).
Some raised the question of whether one approach would be to boycott JPSP and turn to other journals less tainted by the stench of torture complicity. Seems like an okay idea, on the surface. But it does raise some complications. As I mentioned, JPSP is our flagship journal. People take it very seriously. A first author publication in JPSP is a Good Thing to have when applying for jobs. It can help shift a candidate from the "no" pile to the "maybe" pile. On a search committee last year, I saw this in action, as at least one committee member consistently emphasized JPSP-ladenness as a clearly desirable quality (aside for folks fretting over the need for JPSPs on the market: Jazmin Brown-Iannuzzi is the newest addition to UK social psych, and were were lucky to hire her. Irrelevant: she hasn't published in or submitted to JPSP. I hear Jay Van Bavel is also doing well, despite his lack of JPSPs). 'Tis not a bottleneck through which I've yet swum, but I imagine JPSP publications are also lubricants at tenure and promotion meetings. And I remember in graduate school hearing how JPSP papers were the real deal when it came to social psych research. Psychological Science was the style, JPSP the substance. Psych Science has sizzle, JPSP steak. You get the point.
So, if people decide to boycott JPSP, what do we do for a flagship journal? Create a JPSP2? Pick another journal and decide to send our best work there? I mulled over this question for a while:
Should I boycott JPSP?
Thinking about it, I realized that I effectively have been boycotting JPSP for a while now. I have published one article there, and--with the parent's pride that lets one overlook a few warts--it's a good one (where else can you read the phrase "phlegm soaked, flaky skinned protagonist"). The review process wasn't that bad (by JPSP standards...you'll hear some pretty gnarly tales of papers getting rejected after the third revision and whatnot). I think we had one or two manageable revisions. Not too painful. And the paper's done well.
Despite having nothing but fairly positive personal experiences with the journal, I haven't submitted a paper to JPSP in more than five years. Since then, whenever I've had a cool set of findings, I've just never been tempted to go to JPSP with it, even though I am confident that several of the papers would've had a great chance at acceptance there. What gives?
Well, for one thing, JPSP is slow. My paper with a fairly painless review process still took 403 days to go from initial submission to acceptance. A mere 136 days later, and the paper finished its copyediting and proofing and appeared in print. 539 days from submission to publication. Roughly a quarter of my graduate career, waiting on one paper (not even counting the time it took me to actually science that science). Unfortunately, that's not a crazy long time lag at JPSP.
So, when I had cool "JPSP-esque" data, I'd weigh two factors:
- People take JPSP papers super seriously
- It takes foooooorrrreeeevvvver
So, for a few years, I didn't really consciously boycott JPSP. I just didn't really ever find myself tempted to submit there. Maybe I had an overblown sense of the novelty of what I was doing, and was worried about being scooped (I now prefer the term "pre-replicated"). Maybe I'm just lazy or impatient. Who knows? I've been called worse.
Does JPSP walk the walk?
Now, I really do get the appeal of having heuristics like "JPSP = real science." We've gotta have some way to judge the quality of papers, researchers, and the like. So if JPSP takes a long time because science is hard and the long review process really does ensure that accepted papers are the cream of the field's crop, that's good!
Is that the case, though? Are JPSP papers truly the best? Maybe not.
Credit to Fraley and Vazire
Fraley and Vazire looked into the empirical quality of social psych journal (based on sample sizes), and found that JPSP isn't really backing up its high impact factor with high empirical quality. Check out this figure taken from their paper.
Well that isn't good. But it's also consistent with Uli Schimmack's new analyses about the replicability of findings (I think that "replicability" is a slight misnomer, but Uli can name his tool whatever he wants to) across different journals. Of 26 journals included, both of the social psychology sections of JPSP were near the bottom.
And, another black eye for the journal, there was the Reproducibility Project. If you haven't heard of this project yet, I commend you on your off-the-grid vacation to somewhere remote and peaceful. Based on Altmetrics, this is one of the 100 most talked about scientific papers on any topic of the 4 million+ evaluated. In about a month. The Reproducibility Project was a three year project where a huge team of researchers attempted to replicate published studies from three journals, including JPSP. And, most of the studies did not replicate. This led to lots of press about how psychology isn't a real science (what a novel complaint!). And how did JPSP fare? Depending on the metric used to gauge replication success, it looks like around 25% of the social psychology studies published in JPSP actually replicated successfully. Now, there are plenty of caveats around this (can't generalize a ton from 31 studies, etc). But no matter how you dice it, that doesn't look good.
We take JPSP papers super seriously. Realistically, careers have been altered based on publication (or not) in this journal. Yet, despite the time it takes for JPSP to vet its papers, it doesn't necessarily look like the individual studies published in JPSP--on the whole--deserve to be put on a pedestal. But I could definitely be wrong about that.
Do we need a JPSP?
It's nice to have heuristics about which journals publish the good stuff. Heuristics save time. And when you've got a stack of 100 dossiers, you gotta whittle things down somehow. I get that. So it would be lovely to have a Go To journal. If people decide to boycott JPSP over Hoffman, do we need a JPSP2?
Personally, I'd love a shift away from being a subfield with one dominant journal. Other psychology fields have 2-3 "top" journals. Social psychology seems a bit more boom (JPSP) or bust. With a plurality of "top-ish" journals, I think we'd see more competition for journals to speed up publication decisions, evaluate for things like adequate power and generalizable samples (not to mention actual behavioral measures).
And another benefit-that-doesn't-seem-like-a-benefit-at-first is that it might help break the "does this person have publications in Journal X" heuristic. Losing the heuristic would kind of suck. Heuristics save time and effort. Oftentimes they work well. So not having the heuristic would be hard. But at the end of the day, thinking harder and judging papers on their contents rather than journal covers isn't necessarily a bad thing. I'll freely confess that I used that heuristic and others when on a search committee, and didn't have a ton of time to really dig into dossiers and pubs until very late in the game. But maybe that's just because I still had heuristics to fall back on.
So, am I boycotting JPSP?
I'm the unripped guy in pink. Garland Hurt is the ripped guy. Photo credit: Becca Hsu
Not really. This isn't some public declaration that I'll never publish there ever again. But de facto, yes, I guess I am boycotting JPSP and have been for about half of my academic life. Maybe one day I'll be tempted to dip my toes back into those slow moving waters (and sometimes I love slow moving waters! Photo attached!). But right now I'm still a little hung up on the whole "torture, time lag, power, replicability, overuse of heuristics" thing. These things take time. Until then, I'm shopping around. Luckily, there are plenty of fish in that sea of journals.
Oh yeah, I always have caveats.
Despite all appearances, I'm most definitely NOT writing this to rag on JPSP. As I said, my experience there was more-or-less positive. And I have the utmost respect for everyone who edits, publishes, and reviews there (I review there a fair bit lately). Truly, many of my favorite papers are published in JPSP. Today, I was just pondering why I never seem to submit there. To be perfectly clear, I'm not saying it's a bad journal, or worse than other social psych journals. I just don't think it's scientifically enough better than any of our other options to justify 1) the impact factor, 2) the seriousness with which it is taken, or 3) the time lag. It's not an outlier on anything other than people's subjective impressions of it, and maybe its pace. So if it's not an outlier on objective quality, why not both shop around when submitting and also treat it like any other journal when evaluating?
And I also find it interesting that plenty of journals get trashed on social media. These are things I've learned from the TwitterFaceBlogsNets: PNAS and Psych Science are tabloids, Science is a useless glamour journal, Nature is for nerds, PLoS is pay-to-play, etc. But (torture aside) JPSP gets a bit more of a pass. Yet in terms of some objective things (sample sizes, replication), JPSP leaves a lot to be desired--as do the other social psych outlets. Thus, I really don't like its heuristic value. We have a lot of journals to choose from. Lately, I had great experiences at both JEP: General (gulp: APA...but at least the editorial process went smoothly!) and SPPS (OMG: look at the incoming editors and mission statements...love it!). PLoS was fun (x2). I'm a young scientist, why do I need journalistic monogamy for my best work?
So let's all publish lots of great stuff in lots of great journals. And make all the journals better. And speed up peer review. And save the whales.