×

'Just Enough Manifesto: Don’t Drink the Big Data Kool-Aid', by Mikko Kotila, CEO & Founder, STATSIT

Data has gotten heads spinning. It’s too many in kind and too much in quantity. Data is becoming a big problem. More data means more cost, and more inputs lead to more complexity. Complexity, in turn, leads to more cost.

The cause and effect of this relationship is relatively clear and predictable. However, the relationship between between data and value is not at all clear.

While we focus on soap-peddling innovation, the Big Data cartel has a bigger agenda. While consumer goods companies want consumers to buy more goods, armed with glossy brochures and catchy whitepaper titles, the IT companies want us to buy more IT. They call it Big Data.

The World of Abundant Data vs The World of Big Data
The way we store and process data has a direct impact on the efficiency of the way buyers and sellers are able to trade. Market efficiency is the sum of the efficiency of all trades.

Putting things to perspective, the difference between storing something for a day or storing the same thing for a millisecond is nine hundred thousand times in storage cost. The more you store and process, the more you add cost. This seems to be the first force in data business.

A big part of the big data Kool-Aid is the idea of storing a lot and processing a lot. If we as an industry follow such a philopsophy, then as a result we will create an equally inefficient market. The world is filled with examples of market inefficiency. For example, mass production of food consumes more energy than it produces.

Generally speaking, we seem to end up with markets where a few very large players have dominant positions and everyone else is struggling. Economies of scale is not really a solution to underlying inefficiency, it just hides it better. Basically, the bigger the data (scale) is, the harder it becomes to change anything. This is another notable force in data business. The bigger the data gets, the more married you are with that particular data and data variety.

Scale is how we compensate for diminishing returns. Which was fine in the world where data was scarce. Big Data, as a concept, seems to be founded on the principles of scarcity. Yet the world we live in is one of infinitely growing data.

Due to this underlying condition, you wouldn’t want to use lossless data-hoarding or brute-force processing for an ad tech solution, any more than you would want to use a waterfall method for building a website. It seems that when it comes to the storing and processing of data, we are ridden with conflict of interest, convention and gut feelings. It seems that we’ve had our share of the Kool-Aid and are craving for more.

Effective Solutions Create Effective Markets
If you were running a paper factory, you wouldn’t want any more raw material input than you can effectively transform in to paper. Nor would you want to produce more paper than you can get out from the loading bay. Under no circumstance would you want to invest in new storage space so you would have the raw materials there just in case. The ad tech industry can learn a lot from just-in-time manufacturing.

If we have an optimisation algorithm to allocate client spend based on performance inputs, then we have to have another (at least equally powerful) algorithm that allocates our own resources to store and process data based on performance inputs on the return that particular data is delivering.

The client’s ROI is connected directly with two things. One is the decision we make in regards the client's campaign. Another is the decision we make on the use of data that leads to the decision in regards to the client's campaign. That seems to be the basic principle of data business.

Regardless of industry, the seller’s interest will remain the creation of recurring, perpetually-growing profits. When it comes to Big Data, the seller’s interest is just-in-case, but ours, as an industry, is just-enough.