Things to read while the simulator runs (higher ed. edition)
1.
Reforming Science: Methodological and Cultural Reforms by Arturo Casadevall and Ferric C. Fang
An opinion piece published by the American Society for Microbiology (with a popular summary here, titled Has Modern Science Become Dysfunctional?) about the increasingly perverse incentives in modern science. Despite the implication of the title, the authors are not arguing that science is failing to produce new or useful knowledge, merely that the social system of rewards discourages the type of curiosity-driven investigation that drove the dazzling scientific discoveries of the 20th century. On this point, I am highly sympathetic.
To be successful, today's scientists must often be self-promoting entrepreneurs whose work is driven not only by curiosity but by personal ambition, political concerns, and quests for funding.
They go on to argue that this hyper-competitive system does not, in fact, produce the best possible science because scientists choose projects based on their likelihood of increasing their payoffs within these narrow domains (more grant dollars, more high profile publications, etc.), rather than on their likelihood to produce genuine scientific progress. In short, scientific progress only sometimes aligns with ambition, politics or funding priorities, but behavior almost always well, eventually.
This line resonated particularly strongly. When my non-academic friends ask me how I like being a professor, I often describe my first two years in exactly these terms: it's like running a startup company (nonstop fund raising, intense competition, enormous pressure) with all "temp" labor (students have to be trained, and then they leave after a few years). There are certainly joys associated with the experience, but it is a highly stressful one, and I am certainly cognizant of the reward structures currently in place.
2.
Psychology's Bold Initiative, by Siri Carpenter
Continuing the above theme, Carpenter writes about particular issues within Psychology. The problem is known as the "file-drawer problem" in which negative results tend not to be published. Combine that with noisy results from small sample sized experiments and you have a tendency for statistical flukes to be published in high profile journals as if they were facts. Carpenter describes an interesting community-based effort called the PsychFileDawer to push back against this pattern. The idea is to provide a venue for studies that only try to replicate existing claims, rather than focus on novel results in experimental psychology. Carpenter's coverage is both thoughtful and encouraging.
That being said, experimental psychology seems better positioned to succeed with something like this than, say, complex systems. "Complex systems" as a field (or even network science, if you prefer that) is so broad that well defined communities of dedicated, objective researchers have not really coalesced around specific sets of questions, a feature that seems necessary for there to be any incentive to check the results of published studies.
3.
Our Secret Nonacademic Histories, by Alexandra M. Lord, and
For Science Ph.D.'s, There Is No One True Path, by Jon Bardin.
Both in the Chronicle of Higher Education.
These pieces cover parts of the larger discussion about the crisis in higher education. The first is more about the humanities (which seem to have a greater disdain for non-academic careers than the sciences?), while the second focuses more on the benefits of getting a science PhD, in terms of intellectual sophistication, problem solving, project completion, etc., even if your career trajectory takes you outside of science. I'm highly sympathetic to the latter idea, and indeed, my impression is that many of the most exciting job opportunities at technology companies really demand the kind of training that only a PhD in a scientific or technical field can give you.
Update 6 April: Regarding the point in #2 about complex systems, perhaps I was too hasty. What I'd meant to suggest was that having a community of researchers all interested in answering the same basic questions seems like a sufficient condition for science to be productive of genuinely new knowledge. In other words, the best way to make forward progress is to have many critical eyes all examining the problem from multiple, and some redundant, angles, publishing both new and repeated results via peer review [1]. But, this statement seems not to be true.
Cancer research can hardly be said to have a dearth of researchers, and yet a new editorial written by a former head of research at the big biotech firm Amgen and an oncology professor at the University of Texas argues that the majority of 'landmark' (their term) studies in oncology, many of which were published in the top journals and many of which spawned entire sub-disciplines with hundreds of followup papers, cannot be reproduced.
First, we know that a paper being peer reviewed and then published does not imply that its results are correct. Thus, that 47 out of 53 results could not be reproduced is not by itself worrying. But, what makes it a striking statement is that these results were chosen for testing because they are viewed as very important or influential and that many of them did generate ample followup studies. That is, something seems to have interfered with the self-corrective ideal of the scientific community that scientists are taught in graduate school, even in a field as big as cancer research.
Derek Lowe provides some nice commentary on the article, and points a popular press story that includes some additional comments by Begley, the former Amgen researcher. The important point is the Begley's position at Amgen provided him with the resources necessary to actually check the results of many of the studies [2]. Here's the money quote from the popular piece:
Part way through his project to reproduce promising studies, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies.
"We went through the paper line by line, figure by figure," said Begley. "I explained that we re-did their experiment 50 times and never got their result. He said they'd done it six times and got this result once, but put it in the paper because it made the best story. It's very disillusioning."
"The best story." Since when has science been about the best story? Well, since always. The problem, I think, is not so much the story telling, but rather the emphasis on the story at the expense of the scientific, i.e., objectively verifiable, knowledge. It's not clear to me who in the peer-review pipeline, at the funding agencies or in the scientific community at large should be responsible for discouraging such a misplaced emphasis, but it does seem to be a problem, not just in young and diffuse fields like complex systems (or complex networks), but also in big and established fields like cancer research.
-----
[1] Being published after peer review is supposed to mean that there are no obviously big mistakes. But, in practice, peer review is more of an aspirational statement, and passing it does not necessarily imply that there are no mistakes, even big ones, or obvious ones. In short, peer review is a human process, complete with genuine altruism, social pressures, "bad hair days," weird incentives and occasional skullduggery. The hope is that it works, on average, over the long run. Like democracy is for forms of government, peer review may be the worst way to vet scientific research, except for all the others.
[2] Can you imagine the NIH funding such a big effort to check the results of so many studies? I cannot.
Posted on April 01, 2012 in Things to Read | permalink | Comments (1)
Visualizing the ocean and the wind
Last week while I was in Germany at the DPG 2012, I noticed some buzz about a cool visualization of wind flows over the United States. The web animation, put together by Fernanda Viégas and Martin Wattenberg (who lead Google's "Big Picture" visualization research group) pulls surface wind speed and direction forecast data directly from the big NOAA's National Digital Forecast Database. While it's not direct measurements, I expect that the National Weather Service forecasts are pretty close to what actually happens. These wind patterns are a mesmerizing example of fluid turbulence.
Another beautiful example of global turbulence comes from a recent global visualization of the world's ocean's surface currents (here's a popular description), put together by NASA using three years of satellite and other data. These empirical measurements were used to parameterize a detailed ocean dynamics simulation called MIT General Circulation Model. The output of the model is a visualization that looks a lot like van Gogh's Starry Night, with whorls and vortices abounding.
One cool thing is the scale of the coherent flows. We've all seen pictures of vortices in fluids, but typically these are somewhere between a few millimeters to a few meters in size. But, like a truly turbulent system, ocean currents exhibit structure at all scales, and that means vortices up to hundreds of miles across, in addition to all the small scale structure we normally think about. These are so big that you wouldn't know you were in a whirlpool because the curvature of the flow is so gentle that it would just feel like a regular current. Other cool things include the vortex shedding around South Africa, which persist well out into the Atlantic Ocean, thousands of miles away.
The wind visualization uses almost real-time model-based forecasts, but the ocean visualization is reconstructed from historical data. It would be especially cool if the latter could also be done in near real time. I can't think of a practical benefit for it (well, maybe container ships or pirates would like to know), but it would be cool.
Posted on March 30, 2012 in Pleasant Diversions | permalink | Comments (2)
Oops, I tweeted again
After some peer pressure from friends, I've signed up for twitter. This will be a purely professional account, focusing on science and research. If you're into that kind of thing, you can follow me @aaronclauset.
Posted on March 19, 2012 in Self Referential | permalink | Comments (0)
Milestone
Today is a milestone. About a year ago, I blogged about the meteoric rate that my paper with Cosma Shalizi and Mark Newman, on power-law distributions in empirical data, was collecting citations. On that day, the paper had just crossed 500 Google Scholar citations and I used that milestone as an excuse to ask when it might cross the mind-boggling 1000 citations. [1] Since we know a thing or two about citation counts, I decide to apply a little model-based statistical forecasting to come up with a principled guess.
This produced a probability distribution of answers, with the modal crossing date among all the bootstrap models being 21 April 2012 (the 90% bootstrap confidence intervals were 11 Jan. 2012 to 29 Nov. 2013, but the bulk of the distribution is centered on Spring 2012). And, to my surprise and great amusement, 15 March 2012 was the actual crossing date, only a month off from the prediction. Here's what the forecasts from a year ago looked like, along with the actual citation data overlaid. I've also marked where on the forecast distribution the actual prediction landed.
In the new citation data, we again see strange drops and jumps in the citation count. These are presumably from the Google Scholar team tinkering with their algorithms. In fact, the crossing today was caused by the sudden appearance of 44 new citations in the past 5 days, which is high above the normal accumulation rate. But, this may have been a change to the algorithm that restored the citations misplaced in the large drop that occurred in late 2011, it seems reasonable to treat this as a real event. Either way, the closeness of the true crossing data to the forecasted one is a little eerie.
So, there you have it. A milestone. Huzzah. Perhaps I'll buy a mug to commemorate the event.
-----
[1] It is worth saying that the popularity of this paper has been both pleasantly surprising, and gratifying, and I am immensely grateful for what great collaborators Cosma and Mark were on the paper.
Posted on March 15, 2012 in Self Referential | permalink | Comments (4)
Friends for the win!
Individuals often compete for personal status, for jobs, for mates, and groups of people, whether formalized as an organization or not, often compete for glory, for dominance, for financial rewards. Although the most visible form of human competition today is probably professional sports, competition via computer games is an increasingly common form of entertainment for regular people [1]. And, most of us play these games socially, playing with or against our friends, but sometimes playing with strangers.
Last year, with Winter Mason, I started a project aimed at understanding the dynamics of complex competitions where decisions are made largely in real-time under large uncertainties at both the player and the game level [2]. The idea was to investigate whether there are general patterns in the way humans compete in these environments, how well we could explain those patterns in terms of exogenous effects like player skill versus endogenous effects due to the rules of the game or the game environment, and whether we could build better tools for either predicting the outcome of competitions or for designing better games overall.
This is all very high minded, but the starting point was much more mundane: the rich and detailed data that Bungie made available for their blockbuster MMOFPS [3] Halo: Reach. It is hard to describe just how Big this data is. When I blogged about this project last summer, Reach players had already produced a staggering 700,000,000 competitions. The number now stands at 958,887,052 (and counting) [4]. Through a web API, Bungie let us download statistics about each and every game of Reach ever played.
These data provided the raw behavioral information about what happens inside the 4-on-4 or 8-on-8, etc. player-versus-player competitions (as well as data on the various player-versus-environment game types). But they didn't tell us much about who was playing. To gather this information, we launched an anonymous web survey in which we asked Reach players to tell us a little about themselves, how they play Halo and who they play it with. The goal was to get real data from real players so that we could understand the role that friendships play in determining success by both the individual and the team in these complex competitive environments.
What we found was cool and surprising in several ways. Friendships, it turns out, are extremely important in shaping not only the performance of individuals, but also their teams. Friendships also shape the way we play the game.
Before diving into the friendships stuff, let's start with some cute results from the survey. First, we had 1191 unique individuals represented in the survey. The distribution of reported ages looks like this:
Unsurprisingly, there's a large population of college-aged players, and the median age was 20 years old [5]. This contrasts with the statistics for MMORPGs, where the plurality of players are in their 30s. In our sample, only about 13% of players reported being 30 or older, so Reach is largely played by younger adults. Another interesting point is that the average number of hours per week spent playing video games of all kinds by our participants was 23.3 (3.3 hours per day). This might seem high to non-gamers, but it is slightly lower than the 25.9 reported for MMORPG players and the 27.5 reported in 2007 by the industry association for all gamers [6]. The point is that our survey participants were not unusually serious gamers [7].
By looking at the game histories of each of our participants, we did discover several interesting age-related patterns. First, unlike the stereotype described so vividly in Gus Mastrapa's Wired Magazine article "21st-Century Shooters Are No Country for Old Men," older players are, in fact, better at the game (kills per game) than younger players, and this is especially true for the team-oriented players. Here's the figure:
The difference in the number of kills between the age groups is not large, but it is definitely an increase. Also, we define "older players" as being at least 24 years old (the oldest 30% of the population), which may not be what everyone thinks of as being "old". Second, in Reach, it's possible to make an "own goal" by killing a player on your own team. If this happens, it counts as a penalty against the team and may result in the offending player being booted from the competition. What we found is that younger players (at most 17) do this anti-social act much more often than older (18 or older) players:
Age does seem to correlate with the preferred style of playing, with younger players (slightly) favoring the "lone wolf" style. This supports one popular perception about younger players, but it turns out that younger players are not actually better at this role than older players (see the previous figure). That all being said, most players (almost 80%) prefer team-oriented roles. That is, Reach players seem to be strongly motivated by the social aspects of the game.
In fact, players seem to structure their activities within the game around opportunities to play with friends. Using fairly simple heuristics like looking at the length of "runs" of two players playing together, we can fairly easily recover the ground-truth labels on friendship we collected. That is, we asked our survey respondents to tell us who among all the other players they played games with were their friends. Accurately guessing these friendship labels turns out not to be a hard task when you have access to the game history alone.
Given that we can identify the friends, we can now ask whether playing with them changes a person's behavior in the game or changes their success. The answers are yes and yes. Two places we see a strong friendship effect are again with the betrayals, and with "assists," where two players cooperate to score a point.
It turn out that friendship matters a lot and encourages strong pro-social (cooperative) behavior within a competition. As the number of friends on a given team increase, the number of "assists," where two players cooperate to score a point, increases while the number betrayals decreases. And, these are large effects, with the assist rate increasing by almost 50% and the betrayal rate decreasing by almost 25% between a team of all friends and a team of all strangers [8].
Friendship also has positive effects on both the performance of individual players and the team overall. That is, friends who play together tend to play better together than when they play on their own, even if they play with other good players. This shows up both in the net number of points scored by a player (above and beyond what you'd expect based on skill alone) and the probability of winning, both of which increase with the proportion of friends on the team.
What's important about this "friends for the win" effect is that it appears despite Reach's best efforts to eliminate it. That is, when Reach assembles a new competition from the pool of currently online players, it explicitly tries to balance the teams so that they have equal skill levels. From a game designer perspective, balance is important because a mismatch might lead to a frustrating user experience: a fun competition is a close match. But, the algorithm Reach uses [9] does not control for the synergistic effect that comes from playing with friends, an effect that we see clearly in the data.
It is not really surprising that teams composed of "friends" [10] do better than teams composed of strangers. Friends have likely spent considerable time practicing together and thus may be able to effectively anticipate or adapt to each others' actions or strategies without an explicit need for verbal (and thus time consuming) communication or coordination. Friends may be able to more efficiently divide up multi-person tasks by falling into familiar, and pre-determined and practiced, roles. And, these benefits are precisely what sports teams and military units are aiming to reap when they train together. What's nice is that we see these effects appear even in a virtual environment like Halo, suggesting that they may be fairly universal, and not merely limited to the traditional domains like sports and war, where practicing together has a long tradition.
There's more, of course, but these were some results that seemed particularly interesting. If you'd like to read the rest, there's an arxiv version of the paper available [12].
-----
[1] The entertainment software association claims that in 2011, 72% of American households play computer or video games. It's not clear exactly what they count as "playing" a game (probably something like "did you do it a non-zero number of times over all of 2011"), but it's certainly a very common form of entertainment today. The report I linked to above is filled with made-for-media factoids and you can absorb the entire 13 page document in about 30 seconds of skimming.
[2] This contrasts with classic game theory where generally much more of the game structure is known to the players and decision-making is typically not so highly constrained. That being said, there are some interesting extensions of game theory to similar domains.
[3] World of Warcraft, a social online RPG-style computer game played by millions, is called a massively multiplayer online role playing game, or MMORPG. But Halo: Reach, a social console FPS-style game played by millions, should probably be called a massively multiplayer first person shooter, or MMOFPS.
[4] To give you a sense of the raw popularity of this game, and how quickly its popularity faded, the first 130,000,000 competitions were generated within the first 2 weeks after the game was released on 14 September 2010. That rate of 10M games per day then gradually declined to about 2M games per day by 6 months later. That is an immense amount of Halo.
[5] You'll notice the anomalously large spike at 18. This is almost surely due to under-18 year olds misreporting their age in order to bypass the IRB-required parental consent step in the survey. But, the left-tail of the distribution does not look badly distorted and a large number of under-18s did successfully participate despite the extra consent step.
[6] The MMORPG number is likely fairly accurate. The industry association number may not be.
[7] But they were certainly unusually skilled at Reach relative to the typical player. This is not surprising given that we advertised the study through Halo community forums, where folks with a serious emotional investment in the game tend to hang out.
[8] The fact that the betrayal rate does not go to zero suggests that friendship only goes so far toward encouraging purely pro-social behavior.
[9] It uses the TrueSkill algorithm, which by design assumes that the skill of a team is the sum of the skills of the individual team members.
[10] As a caveat, it's true that we have not been precise about what exactly we mean by friendship here. We did not tell our survey respondents exactly what "friendship" meant, but instead allowed them to decide for themselves who was and wasn't an "online" or "offline" friend. Respondents did use the distinct labels, so they do mean something. That being said, it is possible, even plausible, that people labeled as "online friends" were, in fact, simply familiar individuals with whom they have practice a great deal, rather than some stronger notion. Or, it could indicate a stronger bond. It's not clear.
[11] Winter Mason and Aaron Clauset, "Friends FTW! Friendship and competition in Halo: Reach." Preprint, arxiv:1203.2268 (2012).
Posted on March 12, 2012 in Scientifically Speaking | permalink | Comments (3)
Reinventing academia
Attention conservation notice: This post is about a talk in the Denver/Boulder area.
Jon Wilkins, a long-time colleague of mine at the Santa Fe Institute, and all around great guy, will be giving a talk at the Colorado School of Mines next week about his experience and efforts to create and maintain an independent scholarly institute, one that provides a virtual home for researchers unaffiliated with any existing research or higher education institution. The Ronin Institute is a true Internet-era research institute, having no physical location, only an electronic one.
"The Ronin Institute, or how to reinvent academia"
by Dr. Jon Wilkins, Ronin Institute
4:30 P.M., Tuesday, February 28, 2012
Alderson Hall Room 151
Abstract: After more than 10 years of working in traditional research institutions (Harvard University and the Santa Fe Institute), Dr. Wilkins founded the Ronin Institute with the objective to create an organization that can help to connect and support scholars who, by choice or by chance, do not have an affiliation with a university or other research institutes. In this lecture, Dr. Wilkins will share his motivation to found the institute, his long term vision, and how the Ronin Institute fits in the current academic ecosystem.
About the Speaker: Dr. Wilkins is an external professor at the Santa Fe institute and founder of the Ronin Institute. He received an A.B. degree in Physics from Harvard College in 1993, an M.S. degree in Biochemistry from the University of Wisconsin in 1998 and a Ph.D. in Biophysics from Harvard University in 2002. His interests are in evolutionary theory, broadly defined. His prior work has focused on coalescent theory and genomic imprinting. His current research has continued in those areas, and has expanded into areas like human language and demographic history, altruism, cultural evolution, and statistical inference.
Posted on February 22, 2012 in Simply Academic | permalink | Comments (2)
The cost of knowledge
Did you know that Congress is considering prohibiting the free dissemination of knowledge produced by federal research dollars? That's what the Research Works Act would do. The bill is backed by companies like Elsevier, who profit mainly from the volunteer labor of scientists (who produce the research and who vet the research for quality), and thus have a vested interest in preventing the free exchange of knowledge [1,2], or at least in extracting rents from everyone involved in it.
Other commercial publishers may not be as bad as Elsevier, but there are serious problems with them as well. Computers have arguably reduced the valued added by commercial publishers because they allow scientists to do themselves many of the tasks that publishers used to perform (like typesetting, spell checking, etc.), and they have virtually eliminated the cost of distribution and storage. Prof. Michael Eisen, writing in the New York Times [3], laid out the case for why open access publishing is not only realistic, but also morally responsible. To be honest, I am deeply sympathetic with these arguments and am reminded of them whenever I try to access journals when I'm off campus.
More recently, the Fields Medalist Tim Gowers [4] has started a petition to let working scientists declare their opposition to the Research Works Act [5] by promising to boycott Elsevier. (See also his explanation of why he's doing this.)
You can help by declaring that you (1) won't publish with them, (2) wont' referee for them, and/or (3) won't do editorial work for them. Please consider signing up, and also encouraging your colleagues to do the same:
Finally, John Baez has some thoughtful analysis of the problem, its origin and some potential solutions on his blog.
Tip to Slashdot.
Update 27 February: Elsevier has dropped support for the Research Works Act, and has written a letter to the mathematics community. The claim they will now reduce the overall cost they impose on the mathematics community, but in fact, this is merely a cynical sop because mathematics is a tiny part of Elsevier's portfolio.
-----
[1] Elsevier used to make some money from the military arms trade, but partly due to a furor raised by scientists, it eventually cut its ties to the international arms fairs in 2008. Given Elsevier's history, it seems unlikely that they would have made this choice without the public pressure the furor generated.
[2] Elsevier is perhaps the worst offender in the private scientific publishing industry. Their journals (even the crappy ones) typically cost significantly more than other private or non-profit publishers, they've even been caught taking money from the pharmaceutical industry in exchange for creating fake medical journals in which to publish fake research, and a few of their journals have been implicated in more academic types of fraud.
[3] Michael Eisen is one of the founders of the highly regarded Public Library of Science (PLoS), an open-access scientific publishing group. PLoS's founding story is relevant: it is well respected in the scientific community because many of its original journals were started by the members of journal editorial boards for Elsevier, who resigned en masse in protest over Elsevier's odious practices.
[4] The Fields Medal is a bit like a Nobel Prize for Mathematics.
[5] The bill's name really is a lovely example of Orwellian double speak.
Posted on January 27, 2012 in Simply Academic | permalink | Comments (3)
