Big Data – Small Problem! Is the data good?

For those of us who have been in the industry long enough – we know that IT is a hype driven field. New fads and fashions come and go… some residual effect and benefit does remain but it is soon overridden by the next hype. Nothing wrong with this per se.

However, there is a disturbing and predictable pattern. Every hype is presented as the next new thing… everyone is supposed to jump on the bandwagon immediately, get enamored with the cutting edge and reap immediate benefits and gain the “first mover” advantage. Sounds nice. But does not deliver the promised value… in most cases!

The problem is that just because something new, genuinely useful and has been marketed well, does not mean we lose track of what is currently happening. Everyone would want you to be at the cutting edge… but there is gap between where you are and where your hype driven aspirational destination is.

In the process what happens is really sad – for the technology vendors and the customers. It is called a lose – lose situation. Most assume that data is already available in a usable and easy to analyze format – which is absolutely wrong. Usually it is available but in the worst possible format. People spend most of their time cleaning it. What is the point?

This article the beginning of a new series about data. It will cover – data concepts, data gathering, data clean up, effective analytics and finally the psychology of analytics.

Continue reading