Data Science and interpretability I’ve been involved in industrial applications of machine learning, analytics and what is generally referred to as ‘data science’ for about 5 years now. My experience in academia also probably extends that number.
We’ve seen remarkable advances in that time, and a greater appreciation of the value of data in industry – now we live in a world where practically every business process – whether it is a supply chain process, a marketing process or a selling an insurance product has as part of that value chain the collection of data. With newer sensors it’s getting even more ubiquitous.
This has led to very large data sets and also very diverse data sets. Together with the immense computing power that is now accessible via cloud computing we saw the rise of what we’ll call ‘machine learning’.
Historically accuracy has been more important than interpretability, and this has resulted in black box systems that can classify the world better than before – but we can’t explain them. With the rise of newer regulatory regimes such as GDPR and changing consumer preferences – the ‘computer says so’ answer will be unsatisfactory.
There are a set of techniques that allow you to both handle small data sets, explain what your model is predicting and encode in domain knowledge. The solution is a statistical technique called Bayesian inference. Read more from peadarcoyle.wordpress.com…
thumbnail courtesy of wordpress.com