University of Illinois computer scientist Kalev Leetaru is working to create an entirely new field of human behavior forecasting. Culturnomics 2.0, as he calls it, combines network analysis with other methods of text and data mining, to look for connections and patterns in human behavior. He says his models would have been able to predict the Arab Spring and even pinpoint Osama bin Laden’s compound in Abbottabad within a 200-kilometer radius. Medill National Security Zone spoke with Leetaru about his work, and his thoughts on the future of using news and social media to forecast social and political events.
Medill National Security Zone: You’re written about being able to locate Osama bin Laden’s location based on mining news reports; how were you able to do that?
Leetaru: I did it actually very simply. I just grabbed all the news coverage that mentioned bin Laden, grabbed out all the geographic references in all those articles, made a map of them all. In each article if two cities are mentioned together I make a connection between them and what you end up with is a map that effectively connects the whole world, all roads lead to a 200-kilometer radius around Abbottabad.
Now, Abbottabad I think was mentioned only once in the entire news media mentioning bin Laden. The city he captured in was not mentioned in the news or alongside of him, but all the other references if you look at all the geography ever associated with him in the news media prior to his capture you end up with this pretty amazing map of the entire world revolving around this small area.
One of the purposes of this paper is not to come forward and say, “Hey, I’ve invented the magic silver bullet that solves all questions.” It’s more to say kind there’s a whole world here we’ve never explored. It’s almost like an ocean, we spent decades focusing on the surface, but there’s this whole world underneath when you start looking at tone and space.
If we look at the business literature, they’re pouring out papers about how Twitter predicts the stock market and blogs predict movie sales. In the economic literature, we’re seeing a ton of that and they’re all finding the same thing, that this seems to work. But, there hasn’t really been much of this in the social sciences world.
So the question was, “If the tone of news media can predict stock movement, can it predict a country revolution or could it predict where someone is located?” The answer seems to be, from this paper, that there seems to definitely be some promise here. I would be the first to say that it’s not a guarantee, but it seems to be that there is a lot of interesting potential here that really hasn’t been explored in the past.
MNSZ: Where do you think would be the best applications for this work going forward?
L: Obviously there’s a lot of national security implications, because again we’re never going to get to the point where we can say “riot at 5:00 next Thursday at the following street corner,” that’s too far in the future. I think what we can do and what this work strongly suggests is that we can get to the point where we can say, “Hey, there’s something emerging over here that bears taking another look at,” or “In Egypt the mood is really darkening there, this is something that maybe we need to take a look at.”
Or when something happens. So in the first week of February we’re trying to figure out is Mubarak going to go or not. And obviously that has huge implications on policy to be able to use data like this to be able to say, “Given that he’s the most negative, the whole world has turned against him, even if he wants to claim the power he has lost international legitimacy so he really doesn’t have a choice but to leave.” To have this type of data to be able to look at things in a new way, I think from a national security standpoint has obvious ramifications.
MNSZ: What’s the role for social media in this field?
L: I think from a journalism and media perspective we’re still just beginning to understand how emerging media, how social media plays into this communication landscape. There’s been some really high profile studies coming out showing how when something breaks halfway across the world how does that make its way back to the United States, how does it hop from outlet to outlet and if it crosses into social media, what role does social media play in this landscape.
So there is a lot of thought, there’s been some big studies coming out, but we’re just starting to understand that landscape. This overall approach of thinking about news as a historian, historians the way they approach information is they look at what are the connections between people, places and things, how are they arrayed and how is that changing in time and space and in particular views and posturing, how are these characters interacting with each other or viewing each other. These are all critical things that historians look at, but you really haven’t seen that applied to news before.
MNSZ: In terms of your research now, are you looking forward at any specific regions or events?
L: The idea right now actually is to look forward across the entire world. The main emphasis right now is in two areas. One is to make this work in real time. Two is to go at a higher resolution. Obviously, right now the forecasting was done at the country level, but as you saw from the bin Laden example and those country maps the real interesting stuff happens when you go down to the level of individual actors or terror groups or cities.
So the real focus is this all comes out of this network of about 100 trillion relationships, what happens when you leverage that? So in this case I used that to define interesting patterns, then went and used a simplified method to do the paper that others could follow. But, what happens if you stayed at that higher resolution? It’s vastly more computationally complex. But, what would happen if you stayed at that level, could you get even more interesting findings out of this? I’m specifically interested in the entire world, because I think that’s the most critical.
If you take nothing away from the recent Arab Spring the thing that most people should take away is the fact that the things always happen where we’re not looking. There weren’t too many analysts that were watching Egypt waiting for revolution late last year and then it just kind of happened.
MNSZ: How far in advance do you think those warnings might have been based on what you looked at?
L: My real focus is short term, high accuracy and very short term. So there have been lots of people that have claimed to be able to predict things 10 and 20 years out. I really don’t think that you can make those types of forecasts. In my view, your most accurate forecasts will be days to weeks to months.