Explainable AI refers to the ability to interpret model outcomes in a way that is easily understood by human beings. We explore why this matters, and discuss in detail tools that help shine light inside the AI "black box" -- we wish to not just understand feature importance at the population level, but to actually quantify feature importance on a per-outcome basis.