The Lighter Side Of The Cloud – Company Profits
The Lighter Side Of The Cloud – Data Storms
The Lighter Side Of The Cloud – The Migration Strategy
Hand Writing: Data, Data, Everywhere, But Let’s Just Stop And Think

Hand Writing: Data, Data, Everywhere, But Let’s Just Stop And Think

Hand Writing: Data, Data, Everywhere, But Let’s Just Stop And Think

Surely nobody who has the slightest awareness of what’s going on in the world can be unaware of the phrase ‘big data’. Almost every day the newspapers and television make reference to it, and it’s ubiquitous on the web. In November, a Google search for the phrase ‘big data’ yielded 1.8 billion hits. Google Trends shows that the rate of searches for the phrase is now about ten times what it was at the start of 2011.

The phrase defies an exact definition: one can define it in absolute terms (so many gigabytes, petabytes, etc) or in relative terms (relative to your computational resources), and in other ways. The obvious way for data to be big is by having many units (e.g., stars in an astronomical database), but it could also be big in terms of the number of variables (e.g., genomic data), the number of times something is observed (e.g., high frequency financial data), or by virtue of its complexity (e.g., the number of potential interactions in a social network).

Data-Scientists_Infographic

However one defines it, the point about ‘big data’ is the implied promise—of wonderful discoveries concealed within the huge mass, if only one can tease them out. That this is exactly the same promise that data mining made some twenty years ago is no accident. To a large extent, ‘big data’ is merely a media rebranding of ‘data mining’ (and of ‘business analytics’ in commercial contexts), and the media coining of the phrase ‘big data’ goes some way towards explaining the suddenness of the rise in interest.

Broadly speaking, there are two kinds of use of big data. One merely involves searching, sorting, matching, concatenating, and so on. So, for example, we get directions from Google maps, we learn how far away the next bus is, and we find a shop stocking the item we want. But the other use, and my personal feeling is there are more problems of this kind, involves inference. That is, we don’t actually want to know about the data we have but about data we might have had or might have in the future. What will happen tomorrow? Which medicine will make us better? What is the true value of some attribute? What would have happened had things been different? While computational tools are the keys to the first kind of problem, statistical tools are the keys to the second.

If big data is another take on data mining (looking at it from the resources end, rather than the tool end) then perhaps we can learn from the data mining experience. We might suspect, for example, that interesting and valuable discoveries will be few and far between, that many discoveries will turn out to be uninteresting, or obvious, or already well-known, and that most will be explainable by data errors. For example, big data sets are often accumulated as a side-effect of some other process—calculating how much to charge for a basket of supermarket purchases, deciding what prescription is appropriate for each patient, marking the exams of individual students—so we must be wary of issues such as selection bias. Statisticians are very aware of such things, but others are not.

As far as errors are concerned, a critical thing about big data is that the computer is a necessary intermediary: the only way you can look at the data is via plots, models, and diagnostics. You cannot examine a massive data set point by point. If data themselves are one step in a mapping from the phenomenon being studied, then looking at those data through the window of the computer is yet another step. No wonder errors and misunderstandings creep in.

Moreover, while there is no doubt that big data opens up new possibilities for discovery, that does not mean that ‘small data’ are redundant. Indeed, I might conjecture an informal theorem: the number of data sets of size n is inversely related to n. There will be vastly more small data sets than big ones, so we should expect proportionately more discoveries to emerge from small data sets.

Neither must we forget that data and information are not the same: it is possible to be data rich but information poor. The manure heap theorem is of relevance here. This mistaken theorem says that the probability of finding a gold coin in a heap of manure tends to 1 as the size of the heap tends towards infinity. Several times, after I’ve given talks about the potential of big data (stressing the need for effective tools, and describing the pitfalls outlined above), I have had people, typically from the commercial world, approach me to say that they’ve employed researchers to study their massive data sets, but to no avail: no useful information has been found.

Finally, the bottom line: to have any hope of extracting anything useful from big data, and to overcome the pitfalls outlined above, effective inferential skills are vital. That is, at the heart of extracting value from big data lies statistics.

David-J-HandBy David J Hand

David Hand is Senior Research Investigator and Emeritus Professor of Mathematics at Imperial College, London, and Chief Scientific Advisor to Winton Capital Management. He is a Fellow of the British Academy, and a recipient of the Guy Medal of the Royal Statistical Society. He has served (twice) as President of the Royal Statistical Society, and is on the Board of the UK Statistics Authority. He has published 300 scientific papers and 25 books.

Original post can be seen in the Institute of Mathematical Statistics Bulletin, January/February 2014bulletin.imstat.org

Follow Us!

CloudTweaks

Established in 2009, CloudTweaks.com is recognized as one of the leading authorities in cloud computing information. Most of the excellent CloudTweaks articles are provided by our own paid writers, with a small percentage provided by guest authors from around the globe, including CEOs, CIOs, Technology bloggers and Cloud enthusiasts. Our goal is to continue to build a growing community offering the best in-depth articles, interviews, event listings, whitepapers, infographics and much more...
Follow Us!

Sorry, comments are closed for this post.

Support

Recent

Beyond Gaming: Three Practical Applications For Oculus Rift

Beyond Gaming: Three Practical Applications For Oculus Rift

Three Practical Applications For Oculus Rift  Since the announcement of the Oculus Rift in 2012 gamers and game developers alike have been frenzied trying to both get their hands on the unit or build their own proprietary VR machine. The VR gold rush has since lead to the announcement of Project Morpheus from Sony and…

The Lighter Side Of The Cloud – Due Diligence

The Lighter Side Of The Cloud – Due Diligence

By David Fletcher Please support our comics by sharing, licensing or visiting our cloud sponsors (Below). Your support goes a long way in allowing us to continue to produce our lighthearted comics each week.   About Latest Posts Follow Us!CloudTweaksEstablished in 2009, CloudTweaks.com is recognized as one of the leading authorities in cloud computing information.…

Popular

5 Ways The Internet of Things Will Drive Cloud Growth

5 Ways The Internet of Things Will Drive Cloud Growth

5 Ways The Internet of Things Will Drive Cloud Growth The Internet of Things is the latest term to describe the interconnectivity of all our devices and home appliances. The goal of the internet of things is to create universal applications that are connected to all of the lights, TVs, door locks, air conditioning, and…

Using Big Data To Make Cities Smarter

Using Big Data To Make Cities Smarter

Using Big Data To Make Cities Smarter The city of the future is impeccably documented. Sensors are used to measure air quality, traffic patterns, and crowd movement. Emerging neighborhoods are quickly recognized, public safety threats are found via social networks, and emergencies are dealt with quicklier. Crowdsourcing reduces commuting times, provides people with better transportation…

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

What Cloud Technology Has Allowed Us to Do Better The cloud has made our working lives easier, with everything from virtually unlimited email storage to access-from-anywhere enterprise resource planning (ERP) systems. It’s no wonder the 2013 cloud computing research IDG survey revealed at least 84 percent of the companies surveyed run at least one cloud-based application.…

5 Ways CIOs Can Tackle Cloud Fears

5 Ways CIOs Can Tackle Cloud Fears

5 Ways CIOs Can Tackle Cloud Fears  CIOs are tired of hearing about cloud computing concerns. They’ve spent years reading about how cloud resources are subject to risks, and wonder – what can they do to help people trust the cloud?  The truth is that despite being a hot issue for years, the topic of…

12 Promising Business Intelligence (BI) Services For Your Company

12 Promising Business Intelligence (BI) Services For Your Company

12 Promising Business Intelligence (BI) Services Business Intelligence (BI) services have recently seen an explosion of innovation and choices for business owners and entrepreneurs. So many choices, in fact, that many companies aren’t sure which business intelligence company to use. To help offer you a solution, we’ve compiled a list of 12 Business Intelligence companies…

Sponsored Posts

How CYOD Helps Make Hybrid Cloud More Productive And More Secure

How CYOD Helps Make Hybrid Cloud More Productive And More Secure

How CYOD Helps Make Hybrid Cloud More Productive and More Secure Recent analysis by IDG/DELL has found that companies are expecting “increased IT efficiencies” and “lowering total cost of ownership (TCO) with optimized application delivery” as they move toward hybrid cloud and private cloud deployment. In short, they expect things to work better and cost…

Cloud Logo Sponsors

hp Logo CityCloud-PoweredByOpenstack-Bluesquare_logo_100x100-01
cisco_logo_100x100 vmware citrix100
Site 24x7


Contributor Spotlight

Established in 2009, CloudTweaks is recognized as one of the leading influencers in cloud computing, big data and internet of things (IoT) information. Our goal is to continue to build our growing information portal, by providing the best in-depth articles, interviews, event listings, whitepapers, infographics and much more.

Branded Content Programs

Advertising