Art exhibition “Waterfall of Meaning” by Google PAIR displayed at the Barbican Curve Gallery. (Tristan Fewings/Getty Images for Barbican Centre) The rise of deep learning has been defined by a shift away from transparent and understandable human-written code towards sealed black boxes whose creators have little understanding of how or even why they yield the results they do. Concerns over bias, brittleness and flawed representations have led to growing interest in the area of “explainable AI” in which frameworks help interrogate a model’s internal workings to shed light on precisely what it has learned about the world and help its developers nudge it towards a fairer and more faithful internal representation.
As companies like Google roll out a growing stable of explainable AI tools like its What-If Tool , perhaps a more transparent and understandable deep learning future can help address the limitations that have slowed the field’s deployment.
Since the dawn of the computing revolution, the underlying programming that guided those mechanical thinking machines was provided by humans through transparent and visible instruction sets. While the complexity of today’s software can yield myriad interaction effects that yield behaviors outside the scope of a programmer’s intent, at the end of the day it is a human that fully expresses and understands the choices their software is designed to make and can modify it to address changing conditions.
In contrast, deep learning systems outsource the codification process, handing the design process off to algorithms that return a sealed black box that, from the outside, meets the design requirements, but whose internal workings are largely unknown. Such systems can unwittingly encode unexpected bias, since even the most carefully curated data can still encode traces of demographic and other traits that can be inferred from combinations of other variables. They are also extremely brittle, since the contours of their encoded worldview are not visible, meaning a system that performs with human-like fluency can abruptly and unexpectedly degrade into gibberish with a single changed word that it had incorrectly learned as a key variable. The internal world representations learned by systems can run afoul of even the most carefully curated data, as machines learn the predictive power of spurious artifacts of their training data their human creators look past.
These limitations are especially problematic in regulated industries like medicine and finance, in which errors must be understandable, explainable and directly addressable.
As deep learning has matured sufficiently to find widespread adoption in industry and as developers require increasingly greater understanding of their creations in order to pioneer new advances, the AI community has begun investing heavily in explainable AI as a way to render their black boxes transparent.
Google has been an early leader in emphasizing interpretability and how practitioners can build more understandable, representative and resilient AI solutions. Last year the company unveiled its What-If Tool , which offers a range of interactive visualizations and guided explorations of a TensorFlow model, allowing developers to explore how their model interpreted its training data and how subtle changes to a given input would change its classification, yielding insights into the model’s robustness.
Last month Google took its What-If Tool a step further , allowing it to be run on models deployed on the company’s AI Platform , making it even easier for developers to peer under the hood of their creations.
Offering integration with AI Platform, Colab and local Jupyter notebooks and supporting TensorFlow, XGBoost and Scikit Learn models, Google’s exploratory tool lends considerable quantitative insight to the deep learning development process.
Looking to the future, it is likely that tools like the What-If Tool will become increasingly common components of deep learning workflows and likely increasingly automated, with machines sifting through models searching out learning gaps, unexpected inferences, zones of fragility and other weaknesses. One could even imagine inline systems that evaluate every prediction, systematically identifying clusters of inputs for which the model’s predictions are especially uncertain and perhaps even automatically retraining the model for those inputs. Such systems would be especially useful in addressing “input drift” that today typically requires manual intervention.
In the end, as deep learning increasingly infuses into the enterprise, it is likely that today’s black boxes will give way to the transparency and understandability that has defined the computing era from its inception.