Metrics Semantics and Methodolgy - Devil is in the details.

The goal of any KPI/dashboard initiative should be more than identifying the key metrics but also rationalizing and socializing the semantics and calculation methodologies

There is always talk about having a single version of the truth as part of BI/Dashboard initiatives but the key point that is often forgotten is that it is merely a "version" and in most businesses there are always multiple truths.
A simple example is how businesses measure headcount. There are multiple ways to measure it, census approach (based on the number of noses), another way to do it is to look at is as FTE's (Full time equivalents) if your company has a large number of part time employees. Both valid approaches to measure the total number of employees. So semantic understanding of the metric also are critical when looking at metrics

Once the meaning has been decided, looking at the calculation methodology upfront is often an overlooked aspect. Pulling the thread on the above example. The heuristic calculation would be to count the number of current employees in your employee file. Simple enough except if you are going through layoffs or there is a manager who has several associates on leave of absence so the methodology has to be further refined maybe to only count those that are "active" or not on some kind of pay continuation benefit.

Educating the users of the semantics and metholody of the KPI is as important a part of a business performance management initiative but in my experience is often overlooked until the dashboard is built and then the conversations begin leading to multiple rework loops for all involved

IT Quality

I think a lot of organizations struggle to measure the quality of the IT organization. IT systems are installed at most businesses to improve the quality of transactional business processes, but I have rarely seen effective systems to improve the quality of IT processes that support those business processes.
A lot of times IT organizations react to poor quality by adding bureacratic governanance processes that require multiple levels of reviews and approvals only hoping to improve quality but without any measurable way of gauging if it actually does. I would be curious to see if there are any organziations that handle this effectively.
The one measure that I think may be an effective gauge of IT quality at least from an execution perspective would be to measure the number of defect free releases in a production environment. My suspicion is that very few IT organizations (and probably software companies) that get a move to production right on a consistent basis without a rework loop. Now you complement that with a measure of cycle time that can show the elapsed time from when a customer/client requests a change to the time it is operational and we may have the beginnings of a set of KPIs that could be used to measure the effectiveness of an IT organization

Data Validation for Business Intelligence

Recently, there have been several exciting products released in the BI space around data visualization and optimization to handle sophisticated information challenges. However, I have been surprised that there are no software products or technologies to help with something as fundamental as data validation for data warehouses. Most of the projects that I have encountered end up spending an inordinate amount of time at the tail end in trying to get the data to tie in the data warehouse. There is usually a subset of transactions missing, that were not thoroughly understood as relevant to a particular calculation and must be replicated from the source ERP system to the data warehouse.

There are some products that help with data quality, and information profiling but they fall short in semantically walking the lienage of the metric that you may see on one report to a metric that you may see on another.
We can argue that the root cause of the issue is related to multiple versions of the truth, however, I belive that multiple versions will always exist because some events are relevant to certain decisions vs. others. eg. If I am trying to understand customer sales, I am primarily interested in product sales vs. for financial reporting shipping and handling charges will usually be included as part of a sales number. Unfortunately, human beings has neither the patience nor the presence to use all their words so when there is a conversation going on, both those numbers may be reported and referred to as sales.

In my 12 years in this domain, this has been an ongoing issue, how can we "tie" the data in the data warehouse to an existing "known" source. It is irrelevant whether the known source is right or wrong, the information has to link to the source system for it to gain acceptance in the data warehouse.

I also find it fascinating that most of the business people can look at a number and state that it doesn't tie, even though they have no idea how that number is calculated in the first place.

The reason this problem has not been solved because understanding the context and semantics are very important and most technologies stil fall short in trying to "understand" relevance. Maybe this is a search problem after all and the new semantic web will help us conquer this fundamental BI problem at some point in the near future.

Map Reduce and Apache Hadoo

I learned about some intriguing new technology recently that drives the data backbone for Yahoo and Google. Highly parallel distributed computing, based on functional programming techniques of recursion.

http://labs.google.com/papers/mapreduce.html

There is also an open source version.
http://wiki.apache.org/hadoop/HadoopMapReduce

I have always been intrigued about the advances in crawlers for unstructure data such as web content, but nothing exists to index and crawl all the structure data that exists within corporate databases or spreadsheets and link it semantically for search and retrieval.
It would be pretty powerful to have an engine that can "search" for all the versions of sales in the south and present it to a user. It nothing else it would drive visibility into the data profliferation and characterize the one version of the truth problem that exists within most corporations.

It has brought to life a topic I spent hours contemplating as part of my masters thesis that I have recently woefully realized has atrophied as I moved on from tackling technical challenges to people and organizational challenges. It might be time to reignite thoughts about the problem and see what else has changed over the last 9 years.

Web Analytics

Good blog http://www.kaushik.net/avinash/

Pricing Analysis

Sometimes looking at the data can drive a culture change in an organization that is even tougher to implement than the analytics tool itself. If an organization practices cost buildup or cost plus pricing, transforming it to a value based or list discounted price is like moving the titanic.
The nature of such organizations is generally decentralized, where sales and sourcing are controlled by the same function or leadership. All the cost productivity that is gained by the organization is passed on to the customer in the name of getting additional volume without regards to aggregate margin. So take for example a product that used to cost $100 and sold for $120. The sourcing manager busts his butt to get 50% cost productivity but in cost buildup pricining a 20% markup maintened so now costs $50 and is sold for $60. The result is a 20% markup but the net margin dollars actually go down because a 50% reduction in price does not necessarily translate to a 100% increase in volume that would be required to maintain margin. This is why cost buildup pricing is not practiced by most progressive organizations.
So what happens if you start looking at the information and realize that you want to change this practice. You have to first change the organization structure in order to enable the digestion of the information and align accountability.
How has your organization responded in this transformation?

Ran across this neat concept on using swarm intelligence / genetic algorithms to make decisions based on imperfect information. A much more eloquent description is available on http://www.technologyreview.com/read_article.aspx?id=17397&ch=infotech
or straight to the source
http://www.icosystem.com/hunch.htm

I still believe that there will be a point where computing will create an answer to the source of "human intelligence". Just hope it isn't in my lifetime