Intelligent Enterprise

Search isn't that great anymore. For one thing, it's become so commercial that it's really more like an ad search engine. SEO programs game the big ones to the point that you have to go to page 20 before you find something that isn't trying to sell what you're looking for. I want the Scotty Effect for myself (see my previous post). Why can't I ask a search engine questions and get sent to exactly the places with the answers, not 10,000 hits? Why can't the search engines help me assemble the information I need?

Tom Davenport suggests that the competitive playing field for businesses is analytics. I think we'd all be a lot better off if we could do some analytics for ourselves. What do you think? Here are some things I wonder about:

Do you remember the movie "Star Trek IV," when the crew needs to go back to the 20th century to find two hump back whales? When that movie was released, twenty-five years ago, we were already building pricing models with DSS software, we already had SAS to build models and do statistical work and we could write reports in FOCUS or any number of other tools. Compared to the things we can do today, this may seem primitive, but how different is it really?

Consider that the density of hard drives in the same period has increased five orders of magnitude, CPU speed even more so and the cost per unit of storage or MIP has fallen off the table. With that kind of improvement, a new BMW today would go from 0 to 60 mph in 0.00008 seconds, have a top speed of 15 million miles per hour and would burn gas at a rate of 2 million miles per gallon. Oh, and it would cost about 30 cents to buy. I haven't figured out the lease yet.

What's the Deal With Server Virtualization? Over the past few years, I've been suggesting that people stop managing from scarcity. What that means is that the cost of hardware has fallen so sharply that we should reevaluate our methodologies and designs that sacrifice function for resource efficiency. In data warehousing, we still create summarized versions of detailed data in order to avoid costly queries from eating up the big data warehouse, depriving people of the analytical content of the whole picture. In other cases, people are restricted to certain times of the day, or certain subsets of data or governors are placed on queries. With the cost of computing power dropping four or five orders of magnitude since data warehouses were invented, is this really necessary?

It's no mystery why diamonds are often referred to as "ice." Sure, they look like ice, all clear and faceted. But have you ever touched one and noticed it was cooler than you thought it would be? You probably thought it was just the power of suggestion, but it really is cooler. The reason is that a diamond's stiff crystalline structure actually shields the atoms from heat vibrations. So what does this have to do with information technology? A lot.

There has been a lot written about the suitability of relational databases in the ever-expanding Web world of text and pictures and video, even in Rajan Chandras' latest blog. Relational is given a lot of credit because of its staying power and incumbency, which is often confused with universal usefulness. But if you step back and think about it, there is nothing special about a relational database and in the world as we see it evolving, the physical structure, and even location of data, no longer matters. What made relational special was not the database, it was SQL itself.

Perhaps you noticed that I renamed this blog "Competing on Decisions" from "Addicted to BI." Now I'll admit that the latter had a certain anti-chic appeal with its allusion to substance abuse, but frankly, I'm recovering from my BI addiction, so it's time to move on. I've also come (slowly it seems) to the conclusion that informing people (BI) is part of an incomplete cycle. If we as a company make an investment in BI, it isn't the informing of people that matters, it's what happens next. The decisions.

Last year, James Taylor and I wrote a book, "Smart (Enough) Systems," to make the case for creating "decision services" in order to automate certain kinds of high-volume, low-latency decisions. Dubbed EDM for enterprise decision management, we laid out the sort of reference architecture for getting this done, which included predictive modeling, business rules engines, some form of either process automation, or at least a smooth handoff to operational systems and back end analytics to both evaluate the quality to the decisions and manage some form of adaptive control of the decision models (in other words, test new models and compare to the results of the existing ones).

It wasn't so long ago that if you were considering a data warehouse, your choices for a relational database platform were limited to Oracle, IBM, Teradata or Microsoft. In fact, I often wondered where all the choices went. Fifteen years ago, that list would have included lots of other choices. There were standalone database vendors like Sybase (their transactional database that is now Sybase ASE; Sybase IQ was not out in circulation yet), Informix and Red Brick. Most of the hardware vendors had their own offerings, too. We built some pretty good data warehouses for the times with Tandem, but there were also offerings from Digital, HP, Pyramid and probably a few others I've forgotten.

Database choices are now back, and then some.

There were lots of provocative questions and comments on my previous two posts ("Technology is Not the Driver of BI Adoption" and "BI and Technology: Part II"), so I thought I'd just batch all my responses together.

Thanks to all of you who responded to my last post with thoughtful comments. Rather than respond to each in a Lincoln-Douglas style debate, as Kurt Schlegel suggested, let me shift the discussion a little. Instead of arguing that technology alone can't move BI along, I'd rather explore the issue of what can.

To be effective, BI has to focus on simplicity of operation to achieve pervasiveness in the organization and beyond it. The model is the Consumer Web, which provides only the necessary presentation to perform the tasks at hand, and relies on open standards and loosely coupled services to perform the functions, which can be reconfigured dynamically. In the same way the users of the Consumer Web are willing to pay little or nothing directly (except for purchases), the cost of BI has to drop drastically from expensive, front-loaded perpetual licenses to pay-as-you-go on demand schemes.

I'm having some problems with a March 20, 2008 article titled "Gartner: Emerging Technologies Will Help Drive Mainstream BI Adoption." This has been the Holy Grail of BI vendors for over a decade — to increase the number of "seats" using their products, widely reported to be about 20 percent of an organization but clearly much less than that. What troubles me the most about this article, or rather, about Gartner's analysis, is the supposition that new technology is going to crack this old chestnut. It won't. There are only two pieces of enterprise analytical software (broadly speaking) that ever gained currency in organizations in the past two decades — Excel and Google. Wouldn't it be a good idea to understand why?

If you ever spend time as an administrator or even an editor on Wikipedia, you find that your initial enthusiasm for the concept wanes pretty quickly. I thought Wikipedia was a forum for interested people to present their knowledge in an open and influence-free environment, to be vetted by like-minded, optimistic people. As it turns out, it became a dumping ground for every crackpot, agenda, vendetta and misinformation-broker on the planet, which in turn, spurned a dizzying collection of Wikipedia policies and a subculture of enforcers of the policies. Obviously, there was a need to enforce these policies to eliminate all but the most carefully crafted articles, free of conflict-of-interest, lies, libel, etc., but the net result is that discussion of policy far exceeds discussion of substance today.

Harvard Business Review recently hosted a two-day conference in Miami called Think!Analytics (not to be confused with the firm with the same name sans the exclamation point) featuring Tom Davenport and Jeanne Harris, co-authors of the current best-seller, Competing on Analytics. Some of you may remember that I was pretty tough on Tom when the article of the same name came out in the Harvard Business Review in January, 2006, but we've since mended our fences; so much so that I went to Miami to join the festivities and Tom has agreed to keynote the Enterprise Decision Summit in October which James Taylor and I are co-chairing.

In "Measuring and Managing Performance in Organizations," Dorset House Publishing, 1996, Robert Austin made a very clear case that performance measurement often leads, paradoxically, to distortion and dysfunction instead of improvement. According to Austin — and I agree with him, having witnessed this phenomenon firsthand more than once — measuring an indicator of a performance (since we usually can’t indicate the actual performance itself), raises the risk of making things worse. How can that be?

Okay, I waited for the last shoe to drop. Now that IBM plans to gobble up Cognos, leaving only Microstrategy as an independent BI pure-play, and a much smaller one than Hyperion, Business Objects or Cognos, I'm ready to offer my opinion of the whole thing.

Yawn.

Who cares? All it means is that Business Intelligence software as we know it is a mature technology that finally got some attention from well-heeled giants like Oracle, SAP and IBM. Don't expect them to take these platforms and rocket them into the stratosphere. All they are after are the impressive customer lists and what we used to call in the commercial property and casualty business, gross line underwriting. One-stop-shop. Give me the whole deal.

Okay, so no one has used the term "MIS" (Management Information Systems) in years, but there was no way to work "IT" into the title with misinformation. I always used to laugh, by the way, when before the current CHIEF craze (CEO, CFO, CIO, CMO, CTO, etc.), the head of IT was often called the MIS Manager. The qualification for the job was, appropriately, being skilled in MIS management.

CIO Insight's October, 2007 report "How Valuable Is Business Intelligence to the Enterprise?" is another example of so-called research that makes no sense. The most curious aspect of this survey was that the respondents were all IT people. For my money, if you want to know how BI is doing, you should ask the people who use it (or don't use it).

Business Objects Labs is continually releasing new prototypes that can be downloaded and evaluated. You have to applaud this sort of approach, and some of these widgets are interesting and, as prototypes, indicate that there is a lot of creative thinking going on there. I think that Business Objects "gets it" sometimes, but then at other times, they have me scratching my head. They claim on the Labs Web site to deliver on, "the vision of 'ambient business intelligence (BI)' by allowing end users to access lightweight, secure, and personalized BI widgets at all times."

That isn't ambient BI, that's mobile BI. Ambient BI, according to my definition in January, 2006, should advise and drive businesses with embedded analytics, real-time decision tools and vastly improved capabilities for people and unattended processes in every corner of the organization, and beyond it. I suppose it's encouraging that Business Objects is investing in Web 2.0 capabilities, but their product offering as a whole feels a little like a strip mall — lots of things to offer, but no coherent thread running through it.

You can't swing a dead cat by the tail in this industry and not hit a story about exploding data volumes, service-oriented architecture (SOA), pervasive/operational BI and software-as-a-service (SaaS). Moore's Law is supposed to handle that first one, but can it really? And what about the others? Are we really ready for them?

Moore's Law, it's true, has driven the cost of computer hardware down, relatively speaking, but in one area, hot storage, there is a monster under the bed. While the density of disk drives has doubled every eighteen months or so, and the price per megabyte (really, of gigabyte now) has followed a similar pattern, there is a component of disk drives that is not electronic and doesn't follow the same trend. The actual data transfer rates have not been improving at a rate anywhere close to the increases in drive capacities. So we may have bigger, cheaper drives, but it doesn't mean that we can read or write more data any faster, or, at least not at the dizzying rate of increase of CPU's, memory and disk drive platters. That's the first problem.

Five years ago, I bought a covered wagon and put my family and whatever belongings we had into it and headed across the prairie. Well, figuratively speaking, anyway. I shut down my data warehousing/BI systems integration business and re-invented myself as a consultant to software companies. It wasn't really that bold of a move. I was tired of the constant travel and I had a fair amount of residual goodwill with the vendors so that I was able to pick up some work right away.

Along the way, I've learned a lot about how software companies work. And now, after five years, I'm something of a presumptive expert. From time to time, CEO's and founders and even venture capitalists ask me for my opinion about how to do this or that, and I always respond the same way: "I've never earned a dime running a software company, why would you ask me?" And it's true. I have the utmost respect for the entrepreneurs and managers who build something from nothing and have the attention span to attend to the details. I can't do that. I can help them in other ways, but the creation and nurturing is up to them. I'm not a company guy.

There is a lot of conflicting information about the term “Operational BI.” We need some research to sort out the jargon and propose a clear definition for the term (I’m willing). All of the following are being positioned as Operational BI (not a complete list):

• Data warehousing of operational data for reporting, with or without integration
• Replication of operational data for reporting
• Direct reporting from operational systems
• Federated reporting from operational systems
• Process Intelligence
• Inline integration
• Sensing applications
• Decision services
• Real-time BI

There are plenty of software moguls living in Santa Barbara, but after twenty years of living here, I haven't met very many. Perhaps it's because, for the first fifteen years, I wasn't here very much, or maybe it's just because I'm not a software mogul myself. So isn't it ironic that I had a very nice lunch a couple of days ago, in Santa Barbara, with one of the few software company CEO's from Santa Barbara I do know, John Patton, CEO of Sight Software. Ironic because he now lives in Hanover, NH.

I spent a couple days at the Data Governance Conference in San Francisco last week, not as a speaker, but strictly as a listener. It was nice of Davida Berger and Tony Shaw to invite me, and of course, it's always great to see friends and colleagues, but what I really wanted to get out of the three days was a deeper understanding of this concept of governance, what it means and how it works. I admit, I cringe at the word "governance," because it reeks of IT control and restrictions, something we've clearly had enough of for the past few decades. This, combined with the government's renewed interest in everything, seems like a perfect medium for the control freaks and compartmentalizers to push their agendas, creating more bureaucracy and less data democracy (though that is another term that makes me cringe, but let's save that for another time).

Some of you may have already seen a presentation floating around with a lot of WOW statistics about China and India and technology. If not, you can view it here: http://www.glumbert.com/media/shift

We're told that China has more honor students than the US has students, has the largest English-speaking population in the world, that Nintendo spends more money on basic research each year than the US spends on research into education and that a college freshman studying a technical topic will be learning things that are obsolete before he/she graduates.

I hope that the debate between Davenport and McAfee about Enterprise 2.0 was more enlightening than the excerpt I read, because I'm left with the impression that neither one of them gets it, and I'm pretty sure that isn't true. Web 2.0 is driving the way companies are doing business. For proof, look no further than the fact that VC money has virtually dried up for enterprise software over the past five years. The only true innovation going on now is at the edge of the Consumer Web.

I read with interest Tom Davenport's article, "Humans and Black Boxes" in the June, 2007 issue of BIReview. He raises the issue about whether humans are required in the analytics process anymore, given the offerings of vendors of unattended data mining tools. After all, with all of the hardware and bandwidth at our disposal, shouldn't systems be smart enough yet to swim around in the data and come up with predictive models that are more accurate than we mere humans can? Of course, Davenport doesn't believe that, and neither do I.

I don't typically watch C-Span, but I was watching last week because my wife was giving testimony before a Senate Committee. There were two panels, each with four panelists, and they included an Assistant Surgeon General, the head of the FDA Enforcement Unit, a senior person from the NIH who is in charge of the Women's Health Initiative, the largest (and most costly) clinical study of hormone replacement therapy ever, a Harvard professor of medicine and the head the Endocrine Society. How my wife managed to get to go last and get the last word is a mystery (though it's de rigueur around here), but that isn't what caught my attention. Here is what did.

These are extended responses to comments made in the original blog "Who Defines BI?"

Cliff Longman, CTO at Kalido, commented: "I think of BI as the car and data warehousing as the engine…Data warehouses should represent the historical view (and the "what if?" views as well if it is a business requirement) of data that a business relies on to judge its performance."

Cliff, we're pretty much in agreement. I think what we have is a problem of semantics (what a coincidence). We need to separate the data warehouse from data warehousing, which I think you did. The data warehouse is a repository of re-used information with historical context. Its use, going forward, will be diminished to some uncertain degree by advances in technology. It will not go away, at least not anytime soon. I have no quarrel with the data warehouse as a data source for reporting and analysis, but not as THE source.

I was more than a little surprised when I read the article "Think Critically When Applying Best Practices," by Bob Becker and Ralph Kimball. Unless I misread it, they have come around and defined BI as the total process, including data warehousing. This is something that the other prominent data warehousing guru's did a few years ago when, fearing they would miss the boat of the suddenly hot BI market, declared their IT-oriented data warehouse environment as BI. The fallacy in this is that the people who use BI were always conspicuously absent from the diagrams and descriptions of the data warehouse. Their architecture blueprints depicted "users" (and keep in mind that there are only two industries that call their customers users) as little stick figures crushed under the weight of their elegant, multi-colored architectures, or through demeaning models with names such as "Farmers."

Other analysts are preoccupied with the Oracle/Hyperion deal, but I'm not. There are so many important things to keep on top of that the financial doings of the industry are only secondary interests for me. After all, Hyperion itself is composed of three major companies – Arbor, Hyperion Software and Brio, some of which made major acquisitions of their own, including Brio's purchase of Scribe (SQR). This is just another step in an endless process. Some worry about consolidation of the market. I don't. There are dozens of companies waiting in the wings to become the next Hyperion.

Cindi Howson asked in a recent blog, "Given the perceived value of predictive analytics, why does it seem to have had such lack luster success to date? Like most things, I suspect the answer is part cultural and part technological."

I couldn't agree more. Predictive modeling isn't a crystal ball, and despite the efforts of Business Objects, Hyperion, Microstrategy and SAS to get predictive modeling into mainstream BI tools, there are a lot of other reasons for its lack of success. Knowledge Discovery is something that is best left to the experts, those with PhDs, years of experience or both.