Now, however, the 'other' open source, known as Open Source Intelligence (OSINT), is gaining ground at national levels and within commercial and non-profit organisations.
Steele, whom Microtimes - a US technology magazine for small and medium sized businesses - called the "father of the open source intelligence", is regarded as one of the authorities on integrating open source information with classified information.
He has begun a global campaign to establish generic intelligence standards that can unite what he calls "the seven tribes of intelligence": national, military, law enforcement, business, academic, non-governmental and media, and the cultural tribes of religions, clans, and citizens.
Steele's theme is that intelligence is about actionable answers. It is not necessarily about text. "It is not about a room full of three-ring binders put together by MBAs who can't even compile an executive summary," he observed.
Steele is the author of The New Craft of Intelligence: Personal, Public, & Political(2000), and was one of the editors of Peacekeeping Intelligence: Emerging Concepts for the Future. Collaborating with Steele as co-editors were Ben de Jong and Wies Platje, both well-known intelligence experts.
Steele makes a strong case for a sea change in how corporate and government intelligence is assembled and analysed. The principal driver for the "new craft of intelligence", according to Steele, is that threats are proliferating.
Simultaneously budgets are coming under pressure. Open source intelligence is fast becoming a key area of interest in the US, Canada, the European Union, Asia, and - to the surprise of some - the lesser developed countries in Latin America, Africa, and Central Asia, where satellites are unaffordable and smart labour is inexpensive.
Companies are acquiring information from countries that have declassified certain types of information. The data ranges from satellite imagery to traditional cartographic maps and hard copy documents.
Other organisations are 'repurposing' open source software and creating finely tuned online intelligence engines.
The financial result is that an intelligence officer using open source tools can do the work of several traditionalists in military intelligence. This simple fact translates to greater use of open source and a shift in work for the career intelligence officer.
Steele points to East View Cartographic, founded and managed by Kent Lee out of Minnesota. East View began as an intermediary for gaining access to Russian grey literature, but quickly became the world's best acquisition and exploitation channel for both Russian and other third-party cartographic information at the 1:50,000 level that militaries need to be effective.
A former Marine Corps officer and intelligence professional within the US government, Steele said: "The reality is that the US taxpayer is paying $35bn (£21.1bn) a year for a dysfunctional archipelago of intelligence and counterintelligence agencies.
"The CIA's own accounting suggests that open sources which it does buy, at a cost of one per cent of the total national budget, are actually providing 40 per cent of the useful intelligence in some areas, such as economics. Open sources provide 95 per cent of the intelligence."
With economics like this, it is easy to see why open source is emerging as an area of keen interest. The volume of information is staggering but the payoff from pure technology is elusive.
"The emphasis in government has been to spend on very complex information technology for collection, then to spend almost nothing on information technology for processing," explained Steele.
"These agencies rely on relatively young and inexpensive people to try to make sense of the flood of information that cannot be processed automatically.
"In North America and Europe, I estimate that we process less than 10 per cent of our images, less than six per cent of our intercepted signals, and almost nothing on the more than 100 countries that comprise what the intelligence community calls the 'rest of the world'.
"Technology has failed those who need it the most: the human analysts and the decision makers we are all supposed to be supporting."
According to Steele, commercial organisations struggle with similar challenges. "Most business intelligence is nothing more than expensive software applied to internally generated sales and production data. Most corporations - like most intelligence agencies - do not have a structured process for 'casting a wide net'," he said.
The technologies that matter are those that pick up the weak signals. One example was substitution of steel by plastic that the French and US steel industries both missed in the early 1990s.
The Rich Site Summary technology has enormous promise, and it is virtually unknown outside a relatively small group of experts such as my colleagues at the other OSS (Reston, Virginia).'
In Europe, Steele points to InfoSphere AB in Sweden, run by Mats Bjore, the founder of the Swedish military open source intelligence unit, who went on to manage McKinsey's Nordic knowledge management operations before starting his own enterprise.
Bjore was decorated by Sweden for his innovation and global reach in establishing open source collection and processing and analysis capabilities.
Steele draws some clear distinctions about open source information. "It is important to understand that intelligence is about answers and supporting decisions or actions, it is not about secrets," he stated.
In Steele's view "technology is a means to an end". The trajectory of open source intelligence is to implement a form of information sharing.
"We are starting to explore ways to share intelligence. The idea is that partners cooperate to create what I call 'information coalitions' which can be either persistent or ad hoc.
"For Indonesia, to take one example, OSS recently organised a sharing arrangement among three governments and several private parties to reduce cost and improve understanding of certain events in that country.
"This is a promising avenue that blends shared services, classified and open source information, and XML and standards-compliant database technology."
The biggest difference between Steele's approach and that of others, apart from his command of open sources in more than 29 languages, is his emphasis on multi-party arrangements.
"Governments and corporations waste a great deal of time and money pretending to have unique information-sharing relationships with 20 to 40 parties at one time, each in their own channel," he explained.
"We break all those barriers down, and create a 'cone' of access where participating parties share the general risk and environmental information at the bottom, and then decreasing numbers have access to the more expensive and rarer insights that they are funding at the top of the cone - the strategic cream."
From a budgeting point of view, Steele sees a change in the way available resources are allocated for intelligence activities.
"On balance the trend is to consider spending spend a quarter of our money on secret human and technical collection, a quarter on open source collection, a quarter on advanced information processing, and a quarter on expert human analysis that is not multi-cultural," he said.
With the proliferation of new tools, technology seems to be making great strides. Established firms such as SAS, Cognos, SPSS and newcomers such as Databeacon are revolutionising data manipulation. Steele has a different view.
"The software industry is not ready for prime time yet. Most software companies follow the model of arcane constructs where less than 20 per cent of the functionality can be used by any normal person," he said.
"Software needs to be intuitive and learnable on the fly, and it has to come with robust data capture. Most software today requires that the end-user enter the data.
"I am reminded of the new military rations, which are much lighter than the wonderful old spiced beef, pineapples in juice, pound cake, and so on. They just have to be mixed with water. What they don't tell you is that you have to carry the water.
"Analytic software is a long way from being useful because we have not established data capture and meta-tagging standards."
What's ahead? Steele sees another wave of change in information. "Secret bureaucracies, and corporate bureaucracies that refuse to focus on knowledge and process are history," he insisted.
"As Howard Rheingold notes in his book Umlaut, there is going to be a change in the balance of power.
"Elitist top-down secretive hierarchies are going to be overthrown by multicultural collaborative teams inspired and organised from the bottom up. The long view will defeat the short view, and openness will defeat secrecy."
Stephen Arnold is president of Arnold Information Technology.
www.oss.net
www.eastview.com
www.infosphere.se
Computeractive
Back Issues
Reader comments