Big Data is a catch-all term that encompasses wildly diverse information sources. Perhaps because of this, it has captured the imagination of scientists and cultural critics for years. Notably, many of these 'big' data sets are generated by intelligent machines, particularly the Internet of Things — which is deeply in conversation with itself, all the time. Most of all, Big Data analysis is transforming the consumer marketplace, and its success in forecasting how consumers behave continues reshape marketing and retail strategies. However, Big Data is moving beyond its commercial roots and is evolving fast.
As 2015 nears its close all information-handling professions are cogitating about the impact — and the potentia l— of Big Data. For example, new forms of data analysis are revolutionising how humanities scholars perform their work, and they are joining forces with data scientists. Newly minted doctoral students in fields such as English literature and art history are adding data analysis to their fundamental skill set. The excitement created by data science is driving a creative wave of exploration not only in the humanities, but in every field.
Yet there is another trend that is underway alongside the general excitement. Professionals of every stripe are also assessing their comfort level with automated decision-making that originates with smart systems. Many white collar workers are concerned that crucial tasks they perform might be automated in the not-so-distant future. The Economist has commented on this trend, but the editors also make the important distinction that knowledge workers who perform 'non-routine' tasks are faring quite well even as Big Data proliferates (see 'Automation Angst,' The Economist, August 15, 2015, p. 68). This is a significant observation, as it makes it clear that the impact of Big Data and machine learning is unpredictable and may have its greatest effects in surprising locations.
This widespread concern has given rise to a number of important questions about Big Data. What is the proper balance between human data analysis and machine learning? To what extent can analytical procedures be automated in the first place? Can the 'full-speed-ahead' mentality of data-driven marketing and sales strategies be allowed to change how we do science, or perform other high-value knowledge work? These questions are appearing in a number of fields, ranging from law and medicine to journalism and the humanities.
Big Data is without question a powerful force that is reshaping the world of work, professional practice in many fields, and society as a whole. But it has also spawned what may be an equally powerful and opposing force, as algorithms compete with human brain power. In fact, many data scientists themselves are deeply involved in reassessing how to deploy Big Data responsibly, and how to ensure that the data remain accountable to people.
Given the importance of understanding how artificial intelligence, Big Data and human analysis work together, it is crucial for information professionals to monitor trends in Big Data's growth. Current research and debate imply that human analysts are not quite ready to give up their primary role in reviewing data, even when datasets become truly massive.
Although I am a strong proponent of data science in all its forms, I am looking forward to addressing what I regard as Big Data’s next challenge at Internet Librarian International: the proper balance of human and machine-generated analysis.
Terence Huwe will be speaking about 'Big Data's Next Challenge' in Session C101 at Internet Librarian International (for more information about the conference, see the website.)
Image by Justin Grimes via Flickr (Creative Commons).