Nothing ventured, nothing gained

Data mining the world for anti-U.S. sentiments

The new software would allow much more rapid and comprehensive monitoring of the global news media, as the Homeland Security Department and, perhaps, intelligence agencies look “to identify common patterns from numerous sources of information which might be indicative of potential threats to the nation,”” a statement by the department said.

<snip>

The approach, called natural language processing, has been under development for decades. It is widely used to summarize basic facts in a text or to create abridged versions of articles.

But interpreting and rating expressions of opinion, without making too many errors, has been much more challenging, said Professor Cardie and Janyce M. Wiebe, an associate professor of computer science at the University of Pittsburgh. Their system would include a confidence rating for each “opinion” that it evaluates and would allow an official to refer quickly to the actual text that the computer indicates contains an intense anti- American statement.

Let’s hope that something like M-x spook on Emacs doesn’t render the $2.4 million system ineffective