Intelligent Enterprise

The New York Times publishes exceptional visualizations. A couple this week stand out: All of Inflation's Little Parts, graphing the average American's spending by category, and a map of the human "diseasome" that supports the article, Redefining Disease, Genes and All.

What distinguishes these visualizations — the first is a form of treemap, a "space-constrained visualization of hierarchies," and the second a network-connectivity diagram — is their success at communicating relationships along multiple data dimensions.

Cheers to Microsoft Labs for their release of Search Commands, an Office 2007 add-in that "helps you find commands, options, wizards, and galleries in... Word, Excel, and PowerPoint." The embedded Guided Help calls it "a useful complement to the usual method of browsing for commands by clicking tabs on the Ribbon."

I'm all for a way to work around Office ribbons, a set of interface elements introduced in Office 2007 that I characterized last September as "visually unbalanced." Ribbons degrade Office usability. I wrote in September that "they force extra clicking around for routine work and make it hard to find less frequently used functions." Microsoft is now, essentially, pleading guilty. Search Commands' Guided Help, in addition to calling the awkward process of "browsing for commands by clicking tabs" a "usual method," says Search Commands is "especially useful for finding commands that you use less often."

Adoption of text analytics has accelerated in the years I've followed the topic, with growth in expected and unexpected directions both. It wasn't hard to foresee extension of leading data mining workbenches to text, but I'll admit I had thought BI vendors would be much quicker to build handling of "unstructured information" into their technology stacks. (This lag has created partnering opportunities for a number of BI-focused text analytics companies. Business Objects' acquisition of Inxight and SAS's of Teragram show that the BI big guns are closing the gap.) And I didn't anticipate the nature of the solutions, other than semantically enhanced search and expansion in the legal, tax & regulatory (LTR) sector, that would be responsible for the greatest market growth. I'm referring to sectors such as media & publishing and applications including competitive intelligence and Voice of the Customer analytics supporting CRM, product management, and marketing.

BI Scorecard author and IE blogger Cindi Howson writes that she "gasped" on hearing Jim Davis of SAS talk about the commoditization of BI. Yet I'm with Davis — BI softwareis a commodity — the technology, that is, and not BI as a whole. That "as a whole" includes extensions wrapped around the commoditized technology core, extensions that build-out common-place core BI into solutions, extensions that adapt the technology — that package it in suite or application or embedded form — and link it to information sources and presentation and, nowadays, decision management.

I wrote about BI's information angle in an earlier BI-market appraisal. I didn’t bother to take on BI technology as a commodity in that earlier article because the technology’s commodity status was almost self-evident to me. But to take it on now, I’ll cite what I wrote in a look at database management software. Condensing a bit:

So far as that functional core is concerned, 1) there's broad consensus on definitions, 2) basic interface standards are well established, 3) leading products are interchangeable, 4) vendors compete on extended or niche capabilities, and 5) vendors don't compete on software price. I'd further add that with a commodity there is no barrier to user entry-level adoption and there's similarly only a low entry barrier for a would-be technology provider.

This description fits BI technology. It fits reporting and OLAP and ETL from the leading BI firms. It explains why the innovation is outside these core areas, differentiated by (claiming) improvement on the commodity core.

Prediction markets are mathematically based but human powered, a tool for turning collective human insights into forecasts. The approach enables individuals to bet on ideas or events. Twists such as anonymous participation, restriction to experts, and pre-screening for character traits are designed to reduce bias and boost accuracy.

The New York Times reports that "companies use prediction markets to funnel ideas from the work force" (Betting to Improve the Odds, April 9, 2008). Steve Lohr's article provides illuminating examples from companies including Best Buy, InterContinental Hotels, and Hewlett-Packard. Lohr also notes, "for years, public prediction markets have been used for politics, where buyers and sellers bet on which candidate will win a particular race." It’s the kind of article that everyone interested in decision sciences should read, and then follow-up on to understand not only the mechanics of the techniques but also their limitations.

Intelligent Enterprise and TechWeb are not the New York Times, so we get to publish some of the news that's seemingly Not Fit to Print:

Data warehousing powerhouse Teradata has acquired BI/DW solutions provider Claraview.

The news is there on Claraview's Web site — "Claraview is a division of Teradata Corporation (www.teradata.com), the world's largest company solely focused on raising intelligence through data warehousing and enterprise analytics" — but I don't have a clue why Teradata hasn't seen fit to announce the acquisition. There are a few hints of the take-over out on the Web, and an individual in a position to know told me the deal is (or was) "an open secret."

SAS has announced their take-over of text-analytics vendor Teragram. The companies' joint press release states that "the acquisition will enhance SAS's own robust text mining and analytical BI offerings and extend them to enterprise and mobile search." The release cites Teragram's natural language processing (NLP), categorization, and enterprise search technologies, but it leaves much of back-story untold. Here's my take on positioning, technology, and solution considerations that likely motivated what seems like a smart move for both companies.

The essence of the announcement is captured in SAS CEO Jim Goodnight's statement, "Teragram's technologies augment, strengthen, and extend SAS's ability to combine structured and unstructured data – not only in our text mining solution but embedded across the entire SAS Enterprise Intelligence Platform – to drive better answers faster." SAS chief text mining strategist Manya Mayes added in e-mail to me, "Teragram has a broad solution that includes enterprise search and mobile BI" that "complements SAS Text Miner."

I learned of the March 5 death of computer scientist Joseph Weizenbaum from a posting on the CPSR e-mail list. CPSR, Computer Professionals for Social Responsibility, recognized Weizenbaum with their Norbert Wiener Award in 1988. But he was of course best known for creating Eliza, a mid-'60s computer program that conducted natural-language conversations, notably mimicking a psychotherapist's interview with a patient. Eliza was named for the character in George Bernard Shaw's Pygmalion (familiar as the source of the musical My Fair Lady), someone who was similarly taught to impersonate something she was not. Weizenbaum wrote in his 1976 book Computer Power and Human Reason, "Eliza created the most remarkable illusion of having understood in the minds of the many people who conversed with it."

Greenplum recently released a new version of its BI optimized DBMS, Greenplum 3 (G3). The software is based on the PostgreSQL open-source database system; proprietary extensions add support for parallel loading and query and other scalability and reliabilty features.

But with G3, Greenplum appears to be moving ever farther from the company's open-source roots. Specifically, Greenplum sponsors Bizgres, an open source, BI-optimized but non-MPP DBMS, downloadable as an April 2006 0.9 version at bizgres.org or via Greenplum's site. Later versions are available in the code repository but no later version has been packaged as a General Availability release. Note that Greenplum does, separately, contribute to core PostgreSQL development. Frankly, I wonder if/why that contribution isn't the focus of the company's open-source involvement.

Just as every presidential candidate this cycle is the candidate of Change, it seems that all the DBMS vendors offer the preferred data-warehouse appliance solution. That's the message I heard from appliance panelists at today's TDWI Washington DC chapter meeting. For a couple of them it was a real stretch, which in one case wasn't a bad thing. The net take-away is that we are seeing Change in the DBMS world, even if for the politicians that word is still only a promise.

TDWI-DC's panel consisted of Doug Cardin from IBM, Victoria Eastwood from Infobright, Phil Francisco of Netezza, Foster Hinshaw of Dataupia, and Rita Sallam of Oracle. Now my definition of DW appliance is a packaging of processor, storage, operating system, and DBMS that is optimized for data warehousing. A scalability model is essential. And only one of the represented companies hits the mark: Netezza, with an asterisk for IBM.

The saying "There are three kinds of lies: lies, damned lies, and statistics" is attributed to Benjamin Disraeli, and it's nicely illustrated in a couple of Intelligent Enterprise reality-check photos from this week's TDWI conference.

Check out the TDWI Image Gallery photos posted by Intelligent Enterprise Editor-in-Chief Doug Henschen. Executive Summit attendees used the "dots method" to identify Important BI Technologies and Biggest BI Challenges. You get simple histograms showing what's hot and what's not. And you get clear illustrations of "selection bias": not TDWI's fault, but an effect to keep in mind when you assess formal and informal research findings.

My last blog article relayed key points about e-discovery and potential knowledge-discovery (KDD) applications in the legal sector that were reinforced by my participation in the recent LegalTech conference. A LegalTech exhibitor I spoke to mentioned his company's discussions with IBM, so I dropped IBM text-technologies researcher Aaron Brown a note to learn his company's side of the story.

Aaron is program director, Content Discovery and Search, IBM Information Management Software. His thoughts on legal-sector KDD were very much in line with mine. He graciously gave me permission to share his response, which I'll post verbatim —

"Discovery" is a legal process whereby parties to a lawsuit request and provide documents and information that may be pertinent in litigation. "Discovery" also describes an analytics goal that has nothing to do with the court system: extraction of useful information — data, facts, and rules, which together constitute knowledge — from databases and textual sources.

I had expected the December 2006 federal rules amendments on discovery of electronically stored information — "discovery" here in the legal sense — to open new vistas for application of knowledge-discovery technologies: data mining, machine learning, visualization, and the like. The reasoning is simple. Corporations must now retain vast volumes of electronic records including e-mail and information from enterprise operational systems. To comply with e-discovery mandates, they must be able to "produce" records in response to discovery processes, and that means metadata-management, classification, search, and similar systems.

Curt Monash shares my disdain for PowerPoint: not the software per se but rather the rigid communication dysstyle (="dysfunctional style") it encourages. Seeming solutions such as pecha-kucha — "a simple set of rules to presentations: exactly 20 slides displayed for 20 seconds each" — are seductive. You pick up the pace and limit the text that can appear on a slide, which seem like pluses. Per Daniel H. Pink's description in Wired, you "say what you need to say... and then sit the hell down." On the other hand, you’re still locked in that rigid PowerPoint sequence. Just think of that stereotypical image of the American tourist abroad: If foreigners don't understand English, speak slower and louder and maybe they'll get it. It doesn't work of course. Similarly, faster, simpler presentations aren't necessarily better presentations.

The same principle applies to communicating analytical results.

I’m a fan of network visualizations, by which I mean display of interconnectedness mined from disparate sources. The subject matter could be just about anything: witness the collection of projects at Manuel Lima’s VisualComplexity site. Social networks inferred from on-line media prove particularly interesting, the sort of stuff you’ll find in static form at Jeffrey Heer’s and Danah Boyd’s vizster site and dynamically in Linkinfluence’s Map of the Political Blogosphere, which I wrote about last month. Silobreaker (as an on-line application) takes these efforts a big step further.

Silobreaker visualizations add huge value to the company’s underlying news-aggregation service. They classify nodes by type. If you hover your mouse cursor over a node, you can explore its connectedness, and if you hover over the node’s text, you can learn more about that node, whether it represents a person, company, key phrase, or other type of entity. Hover over a link (edge) and you’ll see “documents indicating a relationship.” Naturally, you can double-click on a node to remake the visualization. Please, visit the site yourself and explore… and do some searches on terms that interest you. Then try the 360° Search, which aggregates content related to the search topic and displays retrieved and analyzed information via a variety of widgets.

Prof. David Farber has issued an interesting challenge:

Endlessly people talk about the Next Generation Internet. In fact the term has been used so badly that it is meaningless. I need a name for the Internet-like network we will need when we are faced with end to end optical communications at hundreds of gigabits; multi-core computers (large number) and other now-research technologies.

The best analysis of the motivations and implications of Microsoft's bid for Fast Search & Transfer (FAST) is Stephen Arnold's, yet his in-depth look, along with the rest of the reporting I've read, has nothing to say about some of the most interesting technology that Microsoft will acquire. Sure, given about FAST that "Our Business is Enterprise Search," and given that enterprise-search mojo and customers are what Microsoft seeks to acquire. There's much more to FAST, however: a fresh approach to data warehousing, a search-integrated BI dashboard, and an ad-delivery platform, that last being where the real search money is to be made.

ParAccel has won well-deserved attention in recent months, including Intelligent Enterprise recognition as a Company to Watch. They're a start-up that boasts an all-star cast of executives, positioned in a hot category, namely column-store DBMSes that are optimized for analytics. There's irony, however, in their market positioning. It's not that column stores, most notably SybaseIQ, have been around for decades. It's that ParAccel chose to explain their product with an application, analysis of U.S. Census data, that is essentially owned by a competing column-store system, SuperSTAR from Space-Time Research.

I have personal history here: I designed the U.S. Census Bureau's Census 2000 tabulation system, working on subcontract to IBM. Back in 1998, I wrapped up the selection of SuperSTAR over competing options. We chose SuperSTAR for superior performance and ease of use. I then led the development team that created a system that supported both ad-hoc queries and the production of hundreds of billions of statistical tables for subsequent publication via the Census Bureau's American FactFinder Web site.

The $1,000,000 Netflix Prize competition has produced interesting results, even if no winner, 15 months in. Some of those results are a bit surprising; others we should have expected but didn't anticipate. So while participants haven't yet bettered the accuracy of Netflix' Cinematch recommendation algorithm by 10%, the threshold to win the $1 million prize, we can still take away lessons about predictive-analytics fundamentals.

I recently checked on competition status after receiving a note from Alex Lupu, VP Marketing USA for Scio Systems; Alex has been keeping me apprised of his company's progress toward launch of property-lease abstracting and analysis tools. Like Alex I'm into text analytics, and I liked his take that "intelligent communication between customer and the [Netflix suggestion] system" could provide an alternative route to better recommendations. Alex sees analysis of "'open questions' that allow customer to write a sentence or two" about movies as potentially beneficial in complementing traditional, pure-numbers predictive modeling. Alex says "assuming the customer is a static entity seems wrong to me, thus looking at databases only is not of much help."

It has been a year since Intelligent Enterprise magazine went on-line only. The last print issue, dated January 2007, came out last December. I thought I would miss the paper edition but now I see that, from a writer's point of view, the overhead of a print run, particularly for an IT publication, is a greater liability than may be justified by the extra value delivered.

I was afraid that writing for Web distribution would diminish my authority as an analytics-industry observer. After all, the expense of producing and distributing printed magazines says that someone, even if only the publisher, thinks that the content justifies the cost. Sure, items on the Web are more findable and, simultaneously, timely and long-lived. But for an established author, the threat of losing the distinction conferred by paper counterbalances greater reach and timeliness. That threat hasn't been realized.

I wrote last week about a set of New York Times campaign visualizations that caught my eye. They met my "good" criteria: data-appropriate, designed to communicate rather than (merely) show off. The good is often contrasted with the bad and the ugly. Let's check out examples and then look at a TIBCO-Spotfire demonstration site.

The Bad: A Map of the Political Blogosphere from Linkinfluence, a company that "engineers mapping, monitoring, and analytics solutions for the social web." I read about the site in Matthew Hurst's Data Mining blog. Matthew writes, "I believe that they have put plenty of effort in to the design of the data visualization and the overall look and feel to really make the site stand apart from others in this space."

I do admire a nice visualization, one whose composition suits the nature of the underlying data, one designed to communicate rather than as a means of showing off technology. Given these criteria, the New York Times delivered twice last Sunday with a pair of visualizations that nicely distill presidential-campaign themes and dynamics from what was otherwise a mighty big pile of words: debate transcripts. The Times's visualizations are useful in another way. They exemplify good design, especially when contrasted with other technology-first visualizations on similar topics.

Facebook is good for something (beyond wasting time)! It brought me to a BI 3.0 discussion thread started by Darren Cunningham, prompted by his LucidEra colleague Ken Rudin's blog entry, What's in Store for Business Intelligence in 2008.

Ken is perceptive. His five predictions are:

1. SaaS BI will continue to gain market traction.
2. BI innovation will be led by newer vendors.
3. There will be a shift away from tools towards prebuilt analytic applications.
4. Applications that integrate data and improve processes across transactional systems will drive the next wave of SaaS.
5. A new breed of BI channel partners will emerge.

Do follow the link and read the full blog article, and then consider my BI 3.0 comment addressed to Darren, that LucidEra does interesting enough work, but Ken Rudin's hot-in-2008 list is mighty solipsistic. Is there really nothing (significant) in store for BI in 2008 that isn't touched on by LucidEra offerings?

Given the strengths of open source database management systems (DBMSes), open source seems like a natural platform for data warehousing. We've seen a number of success stories over the last few years, Travelocity, O'Reilly, FTD, and Frontier Airlines among them, but the roster of case studies is mighty thin. But I've only recently (re-)started looking — in the last couple of years, on the open-source front I've covered mostly BI (e.g., May, March) — and I hope to find many more for a report I am planning on open source (based) data warehousing (OSDW).

I'm getting pretty tired of the agonizing whether recent market events and trends mean the end of business intelligence as we know it.

Some of my fellow pundits are scrutinizing vendor consolidations and they're studying the impact of the emergence of new analytical approaches and application-delivery methods. Consolidation will mean a refocusing of product development for acquired vendors, that's all, not "the end of BI as a separate application" as Ephraim Schwartz, for instance, sees it. Schwartz acknowledges authoritative views that are contrary to his — "As Howard Dresner, principal at Dresner Advisory Services, says, 'For every vendor that is acquired, there are 20 emerging companies offering new approaches, technologies, and business models.'" — but dismisses them and slights the impact that Software as a Service, SaaS, has already had on BI.

IBM buys Cognos. SAP buys Business Objects. I agree with Neil Raden: "One big yawn."

Neil's take:

[Hyperion, Business Objects, and Cognos] have made substantial progress in refurbishing their products for a completely new world, but refurbishing only goes so far. The architectures can't really cut it. They need to scale, they need to be intelligent, they need to react in real-time when necessary, unattended when appropriate. They need to live on the Web.

Products like that are coming from start-ups.

Neil's analysis is spot on but calls for elaboration. Neil sees outmoded product (and process and business?) architectures as impeding innovation, for the established BI vendors and implicitly for the organizations that rely on their tools. But there's more to the picture than scalability, "intelligence" (whatever that is), real-time reaction, autonomicity, and webification. The more is imagination: openness to, and the ability to deliver, new ways of analyzing data and using analytical findings.

In a year when the Net is abuzz about Web 2.0, Tableau Server, out this week, qualifies as Web 0.2. But don’t get me wrong. Web 2.0 is about social media and collaboration, user-driven integration, on-demand access, and a first level of semantic search and discovery. On the back-end, Web 2.0 is about network-accessible services that enable all that stuff. Tableau Software’s first foray onto the Web is a modest step when considered in light of Web 2.0 agendas, and also in light of the very high expectations created by the company’s stand-alone Tableau Desktop application. It is not, however, a failure. Rather it shows caution, implicit care to get it (collaborative Web computing) right and not overextend and underdeliver.

It's disconcerting to live in a world where security can be seen as delivering competitive advantage, yet that's the idea behind Unisys's Enterprise Security initiative. But after all, the company's Trusted Enterprise Model only extends the security selling point that is a marketing mainstay for financial institutions and that has been adopted or embraced by IT vendors, sometimes far too slowly, with the rise of network computing.

I promised to follow an earlier article that looked at database management systems as a commodity technology with a similar assessment of business intelligence. In drafting the promised article, however, I realized that I couldn't limit my evaluation to the software side of BI.

BI is complex. It is simultaneously software, transformational work practices, and business information. Admittedly, I am going far beyond IE Editor in Chief Doug Henschen's take on BI, but consider: What value is reporting or OLAP or data mining — software — that doesn't tap all data that contributes to relevant business insights — information — that can help you restructure, realign, or optimize business operations — practices? To understand if BI is a commodity technology, we need to examine all three, complementary aspects of business intelligence: software, information, and practices.

Let's start with information, with BI sources and BI results.

A quick notice to let everyone know that the nomination period for the 2008 Jolt Awards is now open.

I'm a Jolt Awards judge, my second year, while this is the 18th go-around for the awards. The main sponsor is Dr. Dobb's, like Intelligent Enterprise a CMP computing magazine (or is IE a business magazine focusing on computing or is IE a portal rather than a magazine?)

The Jolt Awards target various aspects of software development practice: development environments, management and coding tools and utilities, libraries, books, developer networks, and lots more.

Prof. Jorge Cardoso of the University of Madeira, Portugal, has written a very interesting paper titled "The Semantic Web Vision: Where are We?" Cardoso surveyed over 600 academic and industry researchers in December 2006. He published his findings in the September-October 2007 issue of IEEE Intelligent Systems. They include that "mainstream adoption is still five to ten years away."

Cardoso defines the Semantic Web as "a machine-readable World Wide Web" and he notes "a significant evolution of standards as improvements and innovations allow the delivery of more complex, more sophisticated, and more far-reaching semantic applications." (Bill Inmon, please note.)

Is there anything to add to an item that was the rage of the political media a month back, the misuse of one of our favorite miscommunication tools, PowerPoint, by U.S. military leadership? Check out Gen. David Petraeus' September 10, 2007 slideshow explaining and justifying the drawdown of U.S. troops inserted into Iraq in the recent "surge." The bloggers (e.g., Yglesias, Benen, Drum) found particularly notable an egregiously amateurish slide that related simply that in nine months, troop levels would be back to the levels of nine months previous.

Is database management a mature, commoditized software technology? It depends whom you ask. Start with IE Editor-in-Chief Doug Henschen, who wanted my take last July after he and I each wrote on Oracle 11g. My answer: definitely!, which no doubt confirmed Doug in his intention to later write that Oracle President Charles "Phillips's higher calling was to dispel the idea that database management systems have been commoditized in a mature market."

OK, I admit that the president of Oracle speaks with more authority than I do although he perhaps speaks with more bias as well. I don't have a 47.1% share of the RDBMS market to protect.

Nor is Microsoft's 17.4% RDBMS market share my responsibility. Contrast Phillips with Microsoft Technical Fellow David Campbell, quoted in an interview published in the September issue of Database Trends and Applications magazine. DBTA asked Campbell about factors distinguishing Microsoft SQL Server and its primary competitors. Campbell's response: "For 99 percent of your information-management needs, you can get the job done with SQL Server, DB2, or Oracle." Campbell's conclusion was that "you want to choose the one where your ongoing cost of operations is lowest."

OpenOffice.org has reached a significant anniversary. Earlier this month, OO passed the five-year mark as the only office software on my laptop computers, first installed when I bought a Windows 2000 machine in 2002, reinstalled a couple of months ago on a replacement laptop running Windows Vista and Ubuntu Linux. With open-source Apache Tomcat, Cygwin, Firefox, MySQL, Python, R, and Thunderbird to keep OO company, there's been no looking back. Instead, given diverse project-health indicators such as the release of IBM's new Lotus Symphony, the reported assignment of 35 programmers to the project by IBM, and the continued evolution of the NeoOffice native version for MacOS X, I'm looking forward to my next five years of OpenOffice.org.

Complex Event Processing (CEP) seemed like a no-brainer for broad-market acceptance when I first wrote about a key constituent technology a couple of years back. Relational data warehouses and conventional analytics have not kept up with the explosive growth in real-time data volumes and the perceived demand for real-time analytics. CEP promised to fill the gap: technology developed for the extreme high-volume, low-latency processing demands of capital-market algorithmic trading and communications networks, compatible with emerging service-oriented architectures, applicable to a broad spectrum of security, logistics, and click-stream challenges. Further, CEP is supported by a vibrant, diverse community of academic and industrial researchers. IBM and Oracle and other established companies are doing very significant work, and the field has spawned half-a-dozen start-ups. Yet two years on, CEP is still struggling for market definition outside of capital markets.

I'm grateful to Actuate for giving me an preview look at BIRT Exchange, a new community site set to launch next Monday, September 24. Like the sponsoring company, the new site straddles the commercial open and closed source worlds. It will surely benefit BIRT Java programmers whether they use the open-source Eclipse version of BIRT or the closed source Actuate version. But make no mistake: Actuate's motives remain staunchly commercial and the company will retain tight control over BIRT development.

Cognos's planned takeover of performance-management (PM) and OLAP vendor Applix was eminently predictable. BI companies have been hungry for PM capabilities; think of the Actuate-Performancesoft, Business Objects-Cartesis, Oracle-Hyperion, and SAP-OutlookSoft deals. Cognos's step -- preceded by their 2003 Adaytum acquisition -- leaves Information Builders and Microstrategy as the only major BI players without deep, in-house PM solutions and Longview as one of the last remaining independent PM vendors, albeit noting their alliance with Information Builders and others.

I've been puzzling out a technique, used by sites that machine-aggregate content, that may boost pages' Google rankings. The aggregators stuff their pages with (Google) ads and contextually similar - albeit just similar enough - content. All that pseudo-content surely moves them up the Google rankings. How else to explain the success of the bottom-feeders who exploit others' content in order to sell ads?

Search has many limits but the price is right and the alternative, reliance on traditional interfaces and human-structured information, is increasingly perceived as unacceptable. Yet there's still much to be said for old-school information-retrieval methods. Witness a new, search-free IT-market research tool, TechNavio, just launched by Infiniti Research.

The appetite for Search continues to grow rapidly, and Fast Search and Transfer (FAST) has been one of the most aggressive players in the market. I've covered FAST's move to provide a contextual advertising alternative. Another company initiative boldly claims to revolutionize business intelligence. Yet FAST's recent difficulties, which appear to involve technical missteps and not just operational issues, should make us rethink the limits of search, particularly when it comes to extravagant claims about search-BI.

I received Gartner e-mail this week marketing their up-coming open-source summit. The message contains gems that illuminate Gartner's perspective on open source and the larger IT world.

I characterized Gartner as the oracle of IT establishment and looked at their summit plans in a blog entry last week. Analysts will explain the heretofore anti-establishment open-source movement, albeit without the help of representatives of the communities that lend open source its power and vibrancy. Gartner's theme -- a quite valid one -- seems to be that establishment IT needs to come to grips with open source, and of course that Gartner is the organization that can show the way. They claim to be good at it.

Yes, Oracle 11g is a blockbuster release, sure to maintain the company's dominant market position.

There couldn't be a greater contrast between the views on semantics of Web creator Tim Berners-Lee and of data warehousing figure Bill Inmon.

Gartner has announced an Open Source Summit for this coming fall. The summit will bring together, on the one hand, an analyst firm known for authoritative pronouncements on all things IT, and on the other, a disruptive model for software development that is, at its core, anti-authoritarian. The term that comes to mind is cognitive dissonance. How will the Gartner summit bridge two conflicting world views?

As Curt Monash reports in his Text Technologies blog, Voice of the Customer was a central theme at this year’s Text Analytics Summit. The aim is to stay on top of reputation, quality, and product-design issues by crunching blog- and message-board text, call-center notes and e-mail, and free-text survey responses. (Some vendors call these activities "Enterprise Feedback Management.") Yet VOC and the analytical approach it typifies are only half the overall text-analytics picture. Text analytics still delivers very high value in traditional, non-VOC application domains such as life sciences and intelligence, areas where vendors still derive the major part of their revenues.

The relationship between search and text analytics was a recurring topic at this week's Text Analytics Summit in Boston. The one supports information retrieval and the other just about anything else automated you can do with a document set, from knowledge extraction to automated classification and processing: complementary functions that rely on similar technical underpinnings. Ramana Rao, who has a wonderful ability to clarify, put it this way: "Google's white box makes everything seem so simple," but "we got to simplicity without handing the complexity of reality." It's text analytics, of course, that will equip search to handle complexity.

Retired Vice Admiral Lowell Jacoby, former director of the Defense Intelligence Agency, currently an executive at a Washington DC IT solutions provider, gives exactly the keynote presentation one would expect. He gives a keynote whose essence I've heard before: IT is a information sponge that can clean up some nasty, real-world spills. I heard this theme in 2003 when Richard Perle, former assistant secretary of defense, spoke at a Capital Hill program on data mining. It was the rationale for DARPA's ill-fated-but-resurgent Total Information Awareness program. It bespeaks an attitude that would apply IT on a massive scale in a rear-guard attempt to contain a political situation gone horribly wrong - we have to do something, right? - with not a moment's thought given to alternative paths.

Buying Inxight is a smart move for Business Objects. Business Objects has strong analytics and is well positioned in the BI marketplace. Folding the capability to extract information from text into their technology stack is a natural next step for the company.

This acquisition affirms the text-BI/integrated-analytics strategy being pursued by other vendors, notably Attensity, Clarabridge, Intelligent Results, SAS and SPSS. It also follows the precedent of data-integrator Informatica's late 2006 purchase of text-analytics vendor Itemfield. While the Itemfield deal closed less than six months ago, a demo I saw at last week's TDWI World Conference showed that Informatica has made quick work of seamless integration of text-extraction into their flagship PowerCenter ETL product. Given that Business Objects has been an original equipment manufacturing (OEM) licensee of Inxight technology for several years – which provides at least one reason they didn't go after fellow French company TEMIS – I expect similarly quick inclusion of Inxight text analytics into Business Objects' Data Integrator product.

An Accelovation briefing earlier this week was doubly helpful in affirming my take on the maturity of the text-analytics market and in showing me that I might be doing my job wrong.

Regarding the first point: Despite an hour-long presentation by company CEO Jonathan Spier and VP of Products Jens Tellefsen, I don't have have a clue how the company's "business insight discovery technology" works. That's because the company is pitching solutions to business analysts and not to IT geeks like me, a sure sign that the underlying technologies are stable and capable, and Spier and Tellefsen steered our conversation away from tech talk.

That a vendor can center its message on what rather than how is a hallmark of a maturing market.

I profited from my recent Rome visit to learn more about an aggressive open-source business intelligence (OSBI) contender, SpagoBI, positioned to go head-to-head with leading OSBI rivals. I've known about SpagoBI for a couple of years; the software is produced by Rome-based systems integrator Engineering Ingegneria Informatica. It uses some of the same components as software from Pentaho and JasperSoft, packaged however in a framework that Technical Director Gabriele Ruffatti asserts is more flexible and extensible than those of Engineering’s OSBI rivals.

While in Rome last week to teach a class on "Open Source for the Enterprise," I had the pleasure of getting together with Roberto Galoppini, who consults and writes a very perceptive blog on commercial open source. Check out, for instance, this useful table classifying licensing and revenue models of companies commercializing open source software (OSS).

Roberto thinks a lot about open-source business models, and he took issue with an over-simplification in my course materials. We agree that a third column is missing from a table I had prepared contrasting Open and Closed approaches, this table –

I had the privilege of chairing last week's European Text Analytics Summit in Amsterdam. The event was very enjoyable, in no small part because of the diversity of attendee backgrounds and roles. I've never attended any other computing event (outside the summit series) that mixes scientists, police investigators, and media-company product managers with technologists. While I can't say I learned anything completely new (to me), quite a few points surfaced that are worthy of note. I'll report some of them, grouped under the headings user stories, market, and technology.

Enterprise search vendor FAST is poised to strike a blow for SNaaS – Software NOT as a Service. With a planned April 30 software release, FAST plans to alter the Web's money equation, which to date has been service mediated. The FAST AdMomentum platform – provided for installation by online publishers – is designed to shift control of delivery of contextual advertising from third-party service providers. Company CEO John M. Lervik claims that by adopting a SNaaS model, media companies, retailers, and telecommunications service providers will be able "to maintain control of their revenue, serving their advertisers and audiences more effectively," something "difficult to do with third-party platforms."

Ronen Feldman last year posed a grand challenge problem for text mining: to create "systems that will be able to pass standard reading comprehension tests such as SAT, GRE, GMAT etc."

Feldman is one of the great authorities of the field, a computer science professor, author, and co-founder of text-analytics vendor ClearForest. No one is more qualified to suggest text mining's research agenda than he. Indeed, the aim of Feldman and his 2006 SIGKDD co-authors proposing Data Mining Grand Challenges goes far beyond research. It is to "get researchers, press, funding agencies, venture capitalists, and public interested, greatly stimulate research, and produce dramatic advances in science and technology." This is a worthy vision and goal.

Confusion and controversy about Open Source licensing did not start with current Free Software Foundation efforts to revise the GNU General Public License (GPL). Nor will emergence of an acceptable GPL V3 – or of a revised Lesser GPL or Affero GPL (thanks Dana Blankenhorn) – make OS licensing much less problematical for enterprise users. Concerns are both alleviated and complicated by a profusion of options that range from GPL's communitarianism to the Common Public License's collaborative focus to BSD's laissez-faire liberality. The variety of schemes in use creates opportunity: witness, for instance, Apache's magnificent munificence. But one must also take care to avoid bait-and-switch, pretend Open Source licenses that promise freedom in both common senses, liberty and price, but ultimately deliver neither.

I spent a pleasant and illuminating 90 minutes recently with Justin Langseth, president and co-founder of Clarabridge. Clarabridge sells text-mining software designed to integrate with business-intelligence tools. The company's solutions target both established text-analytics markets, such as life sciences, law enforcement and intelligence, as well as rapidly growing segments: marketing, CRM, reputation management and the like. But boosting Clarabridge is not my job, and, at least for those 90 minutes, it wasn't Justin's either. We did talk about the company's latest software release, but the bulk of our conversation, the helpful and illuminating part, was about the changing market landscape.

InfoWorld has announced that their April 2 issue will be the last to appear in print. The magazine – the computing trade rag I'd most want to write for if I weren't part of the Intelligent Enterprise family – follows in IE's footsteps in going on-line only; IE's last print issue appeared in January. Like IE, InfoWorld cites the Web as "a more efficient delivery mechanism" and they also cite advertisers' desire for "more immediate gratification and measureable results than print can afford them."

Yet there's another important factor to on-line delivery that InfoWorld does not explore: reader preferences.

"How do companies with such a trivial product get such [extensive] press?"

I had written in a recent blog entry about the claims and the coverage garnered by a BI software as a service (SaaS) company. Their products may be quite nice – their architecture and positioning seem sounds – but their grandiose self-depiction overstates their impact on the overall BI market.

The person who sent me this question founded a company that creates BI solutions using open-source software, a rival of the company I wrote about. His question was half serious, half rhethorical – he surely had his own answer in mind – but it's worth a moment's thought. Consider the following as two minutes of PR 101 from someone on the receiving end of many press releases.

The launch of LucidEra, an "on-demand, reporting and analysis solution that focuses on simplicity," has generated quite a bit of attention. The company has been put forward as the poster child for software-as-a-service (SaaS) BI, a pacesetter for an emerging BI revolution. My take? The company appears to have spun the impact of a modest, narrowly focused (and perhaps quite nice) solution far out of proportion to its real import.

LucidEra is designed to offer services via an integrated, hosted, Web-accessed BI stack. Their software platform covers data-source integration, ETL, reporting and OLAP functions. Sales Vice President Tina Babbi explained to me, "We sell solutions, not the BI platform. The first solution is a forecast-to-billing solution that analyzes opportunities in [Salesforce.com] through the billing cycle in the finance system."

Salesforce.com is a hosted customer-relationship-management (CRM) system. Fair enough; sounds like a solid foundation for a business. Babbi wrote in e-mail to me that "more applications are planned. Unfortunately we are not ready to share that roadmap at this time."

Open-source business intelligence has changed nothing, yet it is making all the difference in the world.

What conclusion other than the first can you draw from analyst indifference and the yawns of established BI and enterprise application vendors? Six years after the launch of the open-source Mondrian OLAP server – widely used in the open-source (OS) world – there's no trace of open-source BI (OSBI) anywhere in Gartner's 2007 BI magic quadrant analysis. Yet Mondrian is the analytical core of at least four OSBI suites – JasperAnalysis, OpenI, Pentaho, and SpagoBI – and the packagers claim to be "selling" software like hotcakes. Clearly the explanation for the seeming contradiction is that we have two or even three different BI markets at work.

Observations I drew from this week's LinuxWorld OpenSolutions Summit are that (1) location does matter, in both physical and market space and (2) some people have a strange notion of what constititues an IT solution. Regarding market space – namely how to go about creating some – the interesting news at the summit was the annoucement of a new Open Solutions Alliance. But I'll get to that after first explaining my point on strange notions.

I agree with Gartner's assessment that "SAS offers the most comprehensive BI platform in the industry" with unmatched advanced analytics. It has been twenty years since I first programmed with SAS. I've invested thousands of hours in the company's products. I want the company to do well. And fortunately my experience over the years and my on-going monitoring of the broad BI market has rewarded my confidence.

Yet Gartner also reports that SAS BI software is perceived as lacking usability. I agree with that assessment, too, and I've welcomed SAS efforts to counter it. It has been a chronic limitation that SAS functions are incompletely exposed through the graphical interfaces and that the variety of powerful analytical and presentation modules are not well integrated. I'd like to know that those situations have changed.

I’ve been writing and speaking and consulting on text analytics for years. This work led to a recent call from Philip Russom, an analyst at the Data Warehousing Institute, late of Forrester, Giga, Hurwitz, and Intelligent Enterprise. Philip invited me to contribute an expert comment – my take on “text analytics” in six sentences or fewer – for a forthcoming TDWI report on BI search and text analytics.

I failed. I took eight sentences – we’ll see if Philip cuts them down – and I thought I’d share the lot with you.

The intersection of search and business intelligence has gained lots of attention in recent months. But are companies actually implementing search-BI solutions? Or is search-BI mostly talk, as yet unworthy of serious consideration by organizations with real-world problems to solve?

E-mail from a friend seeking advice suggested these questions.

Christine works for a prominent, expensive BI-DW consultancy. She's helping a client with information access problems. Her client regularly generates large numbers of reports that "sit within a data warehouse system, run off Business Objects/Cognos against one source or another, or more commonly are created manually by copying/pasting or rekeying data into a spreadsheet/Word doc/Powerpoint/Web site." The client envisions "something like the Amazon site that allows search, and when someone selects a report it lists other reports, noting 'people who looked at this report also looked at... and a way for users to rate the report, etc.'"

Yes, even folks who rekey data into spreadsheets are allowed to dream.

Andy Astor, CEO of EnterpriseDB, stated in November, "Our offering is
unique among open source databases." That's blatant, open deception; a cynical attempt to exploit the open-source label. Even Microsoft's shared source is closer to open than EnterpriseDB's licensing terms!

How does EnterpriseDB see it? CEO Astor, in an April 2006 interview, said that "the extensions that we've done [to the open source PostgreSQL that EnterpriseDB is based on] -- the Oracle compatibility for example -- is code that we share with customers who subscribe to our service." Paying customers -- and only paying customers -- can see and modify the code but "just can't redistribute it." That's open source?!

The January 10 New York Times ran an intriguing article, "Computers Join Actors in Hybrids On Screen". It describes a new James Cameron film, "Avatar." The movie's alien characters will be designed by computer but played by human actors.

The Times reports that "their bodies will be filmed using the latest evolution of motion-capture technology -- markers placed on the actor and tracked by a camera -- while the facial expressions will be tracked by tiny cameras on headsets that will record their performances to insert them into a virtual world."