David-J-Hand

Hand Writing: Data, Data, Everywhere, But Let’s Just Stop And Think

Hand Writing: Data, Data, Everywhere, But Let’s Just Stop And Think

Surely nobody who has the slightest awareness of what’s going on in the world can be unaware of the phrase ‘big data’. Almost every day the newspapers and television make reference to it, and it’s ubiquitous on the web. In November, a Google search for the phrase ‘big data’ yielded 1.8 billion hits. Google Trends shows that the rate of searches for the phrase is now about ten times what it was at the start of 2011.

The phrase defies an exact definition: one can define it in absolute terms (so many gigabytes, petabytes, etc) or in relative terms (relative to your computational resources), and in other ways. The obvious way for data to be big is by having many units (e.g., stars in an astronomical database), but it could also be big in terms of the number of variables (e.g., genomic data), the number of times something is observed (e.g., high frequency financial data), or by virtue of its complexity (e.g., the number of potential interactions in a social network).

Data-Scientists_Infographic

However one defines it, the point about ‘big data’ is the implied promise—of wonderful discoveries concealed within the huge mass, if only one can tease them out. That this is exactly the same promise that data mining made some twenty years ago is no accident. To a large extent, ‘big data’ is merely a media rebranding of ‘data mining’ (and of ‘business analytics’ in commercial contexts), and the media coining of the phrase ‘big data’ goes some way towards explaining the suddenness of the rise in interest.

Broadly speaking, there are two kinds of use of big data. One merely involves searching, sorting, matching, concatenating, and so on. So, for example, we get directions from Google maps, we learn how far away the next bus is, and we find a shop stocking the item we want. But the other use, and my personal feeling is there are more problems of this kind, involves inference. That is, we don’t actually want to know about the data we have but about data we might have had or might have in the future. What will happen tomorrow? Which medicine will make us better? What is the true value of some attribute? What would have happened had things been different? While computational tools are the keys to the first kind of problem, statistical tools are the keys to the second.

If big data is another take on data mining (looking at it from the resources end, rather than the tool end) then perhaps we can learn from the data mining experience. We might suspect, for example, that interesting and valuable discoveries will be few and far between, that many discoveries will turn out to be uninteresting, or obvious, or already well-known, and that most will be explainable by data errors. For example, big data sets are often accumulated as a side-effect of some other process—calculating how much to charge for a basket of supermarket purchases, deciding what prescription is appropriate for each patient, marking the exams of individual students—so we must be wary of issues such as selection bias. Statisticians are very aware of such things, but others are not.

As far as errors are concerned, a critical thing about big data is that the computer is a necessary intermediary: the only way you can look at the data is via plots, models, and diagnostics. You cannot examine a massive data set point by point. If data themselves are one step in a mapping from the phenomenon being studied, then looking at those data through the window of the computer is yet another step. No wonder errors and misunderstandings creep in.

Moreover, while there is no doubt that big data opens up new possibilities for discovery, that does not mean that ‘small data’ are redundant. Indeed, I might conjecture an informal theorem: the number of data sets of size n is inversely related to n. There will be vastly more small data sets than big ones, so we should expect proportionately more discoveries to emerge from small data sets.

Neither must we forget that data and information are not the same: it is possible to be data rich but information poor. The manure heap theorem is of relevance here. This mistaken theorem says that the probability of finding a gold coin in a heap of manure tends to 1 as the size of the heap tends towards infinity. Several times, after I’ve given talks about the potential of big data (stressing the need for effective tools, and describing the pitfalls outlined above), I have had people, typically from the commercial world, approach me to say that they’ve employed researchers to study their massive data sets, but to no avail: no useful information has been found.

Finally, the bottom line: to have any hope of extracting anything useful from big data, and to overcome the pitfalls outlined above, effective inferential skills are vital. That is, at the heart of extracting value from big data lies statistics.

David-J-HandBy David J Hand

David Hand is Senior Research Investigator and Emeritus Professor of Mathematics at Imperial College, London, and Chief Scientific Advisor to Winton Capital Management. He is a Fellow of the British Academy, and a recipient of the Guy Medal of the Royal Statistical Society. He has served (twice) as President of the Royal Statistical Society, and is on the Board of the UK Statistics Authority. He has published 300 scientific papers and 25 books.

Original post can be seen in the Institute of Mathematical Statistics Bulletin, January/February 2014bulletin.imstat.org

CloudTweaks

Established in 2009, CloudTweaks is recognized as one of the leading authorities in cloud connected technology information and services.

We embrace and instill thought leadership insights, relevant and timely news related stories, unbiased benchmark reporting as well as technology related infographics and comics.

Contact us for more information on how to get involved in our content related services and placement programs.

CONTRIBUTORS

Data as a Service

Data as a Service: 5 Strategies to Transition How You Access Data

Data as a Service Information wants to be free — at least that’s the saying. And like any good saying, ...
Will 2018 Be the Year Augmented Reality Moves Outside ‘Pokémon Go’?

Will 2018 Be the Year Augmented Reality Moves Outside ‘Pokémon Go’?

2018 Augmented Reality If you’ve never heard of “Pokémon Go” — or at least never had the concept explained to ...
Waiting for IT? Rise Up Citizen Coders! Cloud Now Eliminates the Need for Developers

Waiting for IT? Rise Up Citizen Coders! Cloud Now Eliminates the Need for Developers

SaaS Application Development All we seem to hear about is the shortage of developers, people who can code. Times are ...
AI – The Present in the Making

AI – The Present in the Making

I attended the Huawei European Innovation Day recently, and was enthralled by how the new technology is giving rise to industrial revolutions ...
“God Does Not Play Dice with the Universe” – Albert Einstein. Oh Really? Welcome to Quantum Cloud Computing

“God Does Not Play Dice with the Universe” – Albert Einstein. Oh Really? Welcome to Quantum Cloud Computing

Welcome to Quantum Cloud Computing It is real but makes no sense to our everyday experience. And, it is now ...
Technology and the Future of Cognitive Computing

Technology and the Future of Cognitive Computing

Cognitive Computing Recently popularized by IBM's highly intelligent Watson supercomputer, which competed on the hit game show Jeopardy, cognitive computing ...
How The CFAA Ruling Affects Individuals And Password-Sharing

How The CFAA Ruling Affects Individuals And Password-Sharing

Individuals and Password-Sharing With the 1980s came the explosion of computing. In 1980, the Commodore ushered in the advent of home ...
Cross-Site Scripting - Why Is It A Serious Security Threat For Big Data Applications?

Cross-Site Scripting – Why Is It A Serious Security Threat For Big Data Applications?

Security Threat And Big Data Applications IBM, Amazon, Google, Yahoo, Microsoft - and the list goes on. All these leading ...

NEWS

OVH Announces New Hosted Private Cloud Offerings for US Market

OVH Announces New Hosted Private Cloud Offerings for US Market

OVH delivers next-generation services for hosted private cloud, disaster recovery, and hybridity leveraging industry-leading solutions RESTON, VA--(Marketwired - Nov 20, ...
HPE CEO Whitman's surprise exit stumps Wall Street

HPE CEO Whitman’s surprise exit stumps Wall Street

(Reuters) - Shares of Hewlett Packard Enterprise Co (HPE.N) fell 6 percent on Wednesday after Chief Executive Officer Meg Whitman’s ...
Cloud Security Alliance Issues New Code of Conduct for GDPR Compliance

Cloud Security Alliance Issues New Code of Conduct for GDPR Compliance

EDINBURGH, Scotland, Nov. 21, 2017 /PRNewswire-USNewswire/ -- The Cloud Security Alliance (CSA), the world's leading organization dedicated to defining and raising awareness of best practices ...

SPONSORS

Ransomware's Great Lessons

Ransomware’s Great Lessons

Ransomware The vision is chilling. It's another busy day. An employee arrives and logs on to the network only to ...
The Skill & Training Mandates of Big Data

The Skill & Training Mandates of Big Data

Big Data Mandates For some years a dearth of data scientists and analysts has caused concern, with McKinsey expecting a ...
How Printers Help Hackers Hide In Plain Sight

How Printers Help Hackers Hide In Plain Sight

Printers and Hackers Spies and thieves often do their best work by hiding in plain sight. No one suspects the ...