Twitter predicts the stock market (18 October 2010). Image: physics arXiv blog.
Two big Cyberpunk movie franchises from the turn of the century, The Terminator and The Matrix, depended on a cataclysmic moment when computer networks turned sentient. There is no danger of that yet, although there are signs of networks taking on an organic momentum of their own. The Internet may reflect the confluence of human thought and action in such a way as to make it representative of the collective senses of society. In the minds of some, like society itself, the Internet is almost a living creature. This idea lay at the heart of the excellent Darren Aronofsky movie, Pi. The concept clearly drives Google's
seminal research into Artificial Intelligence.
Recent research along these lines by
Johan Bollen and colleagues at Indiana University shows that
Twitter is exhibiting the characteristics of an organic entity. This result was reported last October at the
Technology Review:
"An analysis of almost 10 million tweets from 2008 shows how they can be used to predict stock market movements up to 6 days in advance." This is because
Twitter is apparently a good index of the emotional state of the
Twitterverse at large and that emotional state
can be gauged:
Numerous studies show that stock market prices are not random and this implies that they ought to be predictable. The question is how to do it consistently. Today, Johan Bollen at Indiana University and a couple of pals say they've found just such a predictor buried in the seemingly mindless stream of words that emanates from the Twitterverse. For some time now, researchers have attempted to extract useful information from this firehose. One idea is that the stream of thought is representative of the mental state of humankind at any instant. Various groups have devised algorithms to analyse this datastream hoping to use it to take the temperature of various human states.
One algorithm, called the Google-Profile of Mood States (GPOMS), records the level of six states: happiness, kindness, alertness, sureness, vitality and calmness.
The question that Bollen and co ask is whether any of these states correlates with stock market prices. After all, they say, it is not entirely beyond credence that the rise and fall of stock market prices is influenced by the public mood. So these guys took 9.7 million tweets posted by 2.7 million tweeters between March and December 2008 and looked for correlations between the GPOMS indices and whether Dow Jones Industrial Average rose of fell each day. Their extraordinary conclusion is that there really is a correlation between the Dow Jones Industrial Average and one of the GPOMS indices--calmness. In fact, the calmness index appears to be a good predictor of whether the Dow Jones Industrial Average goes up or down between 2 and 6 days later. "We find an accuracy of 87.6% in predicting the daily up and down changes in the closing values of the Dow Jones Industrial Average," say Bollen and co[.] That's an incredible result--that a Twitter mood can predict the stock market--but the figures appear to point that way. ...
But there are at least two good reasons to suspect that this result may not be all it seems. The first is the lack of plausible mechanism: how could the Twitter mood measured by the calmness index actually affect the Dow Jones Industrial Average up to six days later? Nobody knows.
The second is that the Twitter feeds Bollen and co-used were not just from the US but from around the globe. Although it's probably a fair assumption that a good proportion of these tweeters were based in the US in 2008, there's no way of knowing what proportion. By this reckoning, tweeters in Timbuktu somehow help predict the Dow Jones Industrial Average.
Bollen's publication is listed at Cornell University Library
here. The possibility that
Twitter could be used to predict the stock market caused a
fair amount of buzz. But it really has much broader implications. Bollen and his co-authors speak of the collective emotional state of society, as assessed by evident patterns of behaviour
on the Internet:
Behavioral economics tells us that emotions can profoundly affect individual behavior and decision-making. Does this also apply to societies at large, i.e., can societies experience mood states that affect their collective decision making? By extension is the public mood correlated or even predictive of economic indicators? ... We analyze the text content of daily Twitter feeds by two mood tracking tools, namely OpinionFinder that measures positive vs. negative mood and Google-Profile of Mood States (GPOMS) that measures mood in terms of 6 dimensions (Calm, Alert, Sure, Vital, Kind, and Happy).
Separate research at Princeton has examined
how ideas proliferate (Hat tip:
Erekalert via
Lee Hamilton's blog). Again, the researchers' focus is identifying the next big idea before it happens. In other words, it is not just a question of viewing the Internet as a system in which ideas hatch and grow. This vision extrapolates those patterns so that
the Internet becomes a tool of prognostication:
Princeton computer scientists have developed a new way of tracing the origins and spread of ideas, a technique that could make it easier to gauge the influence of notable scholarly papers, buzz-generating news stories and other information sources.
The method relies on computer algorithms to analyze how language morphs over time within a group of documents -- whether they are research papers on quantum physics or blog posts about politics -- and to determine which documents were the most influential.
"The point is being able to manage the explosion of information made possible by computers and the Internet," said
David Blei, an assistant professor of computer science at Princeton and the lead researcher on the project. "We're trying to make sense of how concepts move around. Maybe you want to know who coined a certain term like 'quark,' or search old news stories to find out where the first 1960s antiwar protest took place."
Blei said the new search technique might one day be used by historians, political scientists and other scholars
to study how ideas arise and spread. ... Blei said their model was not meant as a replacement for citation counts but as an alternative method for measuring influence that might be extended to finding influential news stories, websites, and legal and historical documents.
"We are also exploring the idea that you can find patterns in how language changes over time," he said.
"Once you've identified the shapes of those patterns, you might be able to recognize something important as it develops, to predict the next big idea before it's gotten big."
This desire to find the 'big picture' by following the paths of information coursing through our culture reappears in the study of
'transmedia.' Transmedia, according to
Hukilau is: "interactive storytelling across multiple media and platforms. The story actually cuts across, loops between and re-enforces the different strands that tie together the different platform exploitations.
In other words, each component part is relevant to a whole – the bigger picture."
The suspicion that there is a process embedded within the Intenet that makes it an entity unto itself fuels Millennial conspiracy theories. At no other time in the history of the world could you have so many innocuous bits of data juxtaposed, seemingly granting them meaning. The resultant mentality responds well to stories that associate different agents in the global economy, granting that association a malevolent aspect. In late 2010, a typical example appeared in
Pravda (via
Before It's News), which reported the fact that
Monsanto recently purchased Blackwater (now called Xe). Monsanto has developed an "intelligence wing" that conducts industrial intelligence operations and also keeps an eye on the biotech giant's many critics.
Without a doubt, a team-up between Monsanto and Blackwater is fairly alarming. However, another aspect to this story is our basic, knee-jerk response to it. Our automatic credulousness is a major problem. We want to believe in deep dark secrets everywhere, as helter skelter blobs of information suddenly end up next to each other, seemingly creating a whole new dimension of menace. Where are the sources for some of these brooding news bytes? Who manufactures these news items - and to what end? There is always another wizard working the levers behind another velvet curtain. To be clear: it is part of the nature of the Information Revolution to mash data together mechanically without any rhyme or reason. Yet in these mash-ups, we inevitably perceive sense and order where there really isn't any.
Informatics specialists' research into a 'bigger picture' inside the Internet is the sober aspect of this credulousness. This research retains the assumption that the Internet can be tamed and harnessed. These researchers accept the fundamental premise that there is a demi-god in the machine, an internal order, that can be comprehended and released, Phoenix-like, and hatch a golden egg called the
Singularity.
The Singularity is perhaps the most concrete expression of this type of magical thinking. It is a view of the Tech boom that also reveals generational rifts. In general, the core proponents of the Singularity - the point at which technology in our society exponentially transforms us and our societies beyond all recognition - are Baby Boomers. They betray the same rapturous idealism when speaking about the potential of the Information Revolution, the same iconoclastic convictions, that they manufactured forty years ago in their collective youth.
I09's top editor, Gen Xer
Annalee Newitz, pours cold water on it:
I don't believe in the Singularity for the same reason I don't believe in Heaven. Once I met a Singularity zealot who claimed that eating potato chips after the Singularity would induce sublime ecstasy. Our senses would be so heightened that we could completely focus our whole attention on the ultimate chippiness of the chip. For him, the Singularity was just like Sunday school Heaven, full of turbo versions of everything we love down here on Earth. But instead of an all-powerful God zotting angel puppies into existence for our pleasure, we would be using the supposed tools of the Singularity like nanotech and A.I. to conjure up the tastiest junk food ever.
That is not a vision of social progress; it is, in fact, a complete failure to imagine how technology might change society in the future. Though it's easy to parody the poor guy who talked about potato chips after the Singularity, his faith is emblematic of Singulatarian beliefs. Many scientifically-minded people believe the Singularity is a time in the future when human civilization will be completely transformed by technologies, specifically A.I. and machines that can control matter at an atomic level (for a full definition of what I mean by the Singularity, read my backgrounder on it). The problem with this idea is that it's a completely unrealistic view of how technology changes everyday life.
Case in point: Penicillin. Discovered because of advances in biology, and refined through advances in biotechnology, this drug cured many diseases that had been killing people for centuries. It was in every sense of the term a Singularity-level technology. It extended life, and revolutionized medical treatment. And yet in the long term, it wound up leaving us just as vulnerable to disease. Bacteria mutated, creating nastier infections than we've ever seen before. Now we're turning to pro-biotics rather than anti-biotics; we're investigating gene therapies to surmount the troubles we've created by massively deploying penicillin and its derivatives.
That is how Singularity-level technologies work in real life. They solve dire problems, sure. They save lives. But they also create problems we'd never imagined - problems that might have been inconceivable before that Singularity tech was invented.
Newitz is correct: there is no proof that globalized, overlapping computer systems and algorithms will produce some ultimate rationalized process. The Internet offers something else: cultural entropy, disintegration and degradation. For example, look at how the Internet shreds time and consumes personal lives. I like to imagine someone from the 18th century alighting in our society, and witnessing millions of people, staring for hours into grey boxes and little black gadgets. To the time traveller, it would be a picture of hell.
That is not to say that we will descend into chaos because of the Tech Revolution. Neither good nor bad outcomes are certain. The assumption that we are all on a road up into the light of progress colours researchers' inquiries into the internal mysteries of the Internet. It would be just as easy to argue and prove that we are heading toward fractured social oblivion and total tech-driven war - as it would be to say that sites like
Twitter can explain the collective emotions of online users and predict the performance of the stock market. Moreover, this research comes out during the worst recession since the 1930s.
In short, algorithms may correlate blobs of information in a way that suggests causal connections between them. That is a
logical fallacy. Some predictions garnered from deep study of online structures may produce results that make sense. But it is still magical thinking. The notion that the Internet can predict the future and unlock our ultimate mysteries is an illusion, which will sometimes work, and sometimes will not.
Posts
