DESCRIPTION: AI is increasingly used to make important decisions in our lives like predicting the onset of a disease, the suitability of a job candidate, the likelihood of a criminal to reoffend, or the next president of the United States. In such cases, having AI model outcomes that are explainable, or interpretable, is often just as important as having model outcomes that are accurate. In this episode, Xyonix Data Scientists engage in a robust discussion where we explore model interpretability and discuss why you should care.

Want to dig in more? Some of our related articles:

DOWNLOAD: https://podcast.xyonix.com/1785161/8578918?t=0

AUTOMATED TRANSCRIPT:

DEEP: Welcome to your AI injection, the podcast where we discuss state-of-the-art techniques and artificial intelligence with a focus on how these capabilities are used to transform organizations making them more efficient, impactful and successful.

DEEP: All right, on this episode, we're going to be talking about explainable AI, how to look inside the Black Box bill, you want to kick us off. We've got our just, we've got a couple of folks here that are xyonix vets, Bill,

BILL: Constantine,

DEEP: and carsten Tusk, so Bill, you want to give us a quick background and, and maybe just a few words, and why you think folks? Are asking for explainable AI.

BILL: Yeah, I don't know about you guys. It seems like you know, in our work over the decades with machine learning or data science or whatever they want to call us at the time, a lot of it involved, you know, can we generate models to do things that humans could do? You know, and we figured out pretty soon with the advance of fast computers and all that yet. Well, these neural networks and such can do a really good job and being able to predict things very, very accurately people. Sort of went hog-wild with that attitude. That notion. But you know, and many, many circumstances with the you know, people that we work with, on my life being accurate is not is not enough, you have to be able to explain what's going on, you know, behind the scenes. And that's particularly. So when you think about applications of AI, for example in the medical industry like you know if you have a model that supposed to predict when someone has cancer you know a doctor is not going to act on that just because the model is accurate. He needs to know why it is that model. No, thanks think so. So it's kind of went from, I think being like, wow, these things are really cool. They can predict things very, very actually, sometimes better than humans to be like, well, you know, we need to know kind of why the models are doing those, if not, for, you know, kind of just trying to understand what's going on for legal reasons, right? And for practical grounds of. Yeah. So,

DEEP: well, well, well will dig into that and, and kind of And kind of tease it out. So, Carson, you want to jump in here? Maybe like give us a quick background on yourself. And what do you think of? You know, when you hear explainable a I like what, what are those terms mean to you? And like what kind of stuff jumps into your mind?

CARSTEN: Yeah, sure. So I'm Karsten not to AI. That's my background. No, I'm a computer scientist and I've been working with machine learning and AI for the last 20 years. Pretty much well explainable AI. It's kind of like what? Bill explained already, right? So you're not just interested in the results but you're interested in what in your input, let the model to decide a certain way. In other words, you know, if something is detected as a cat in an image, then which sections for example, of the input image, make them bottle think that this is a cat. Yeah, I think we'll put a pretty well. We want to know in certain cases by models, make their decisions to mostly to put a judgment call on it. See whether or not it's reasonable, right? And that the model is doesn't detect the pictures a cat because there's a red fire hydrant in the background in all pictures with cats also had red fire hydrants. It's kind of like the short of the story.

DEEP: Yeah, I mean, you know, I'm just thinking about it from like a business Vantage, you know, like, A lot of problems that that I've seen, it's like there's, you know, there's there's like kind of two parts of it. There's like a model trying to predict something, you know, whether it's it could be a like, you know, predicting whether a customer is going to churn or not or whether they're you know, going to be like a loyal customer or it could be you know something like how many patients are going to show up in a hospital particular time but usually like businesses that that I see you know, they've got, they've got The thing that they want to forecast, but then they also have parts of the business that are softer, like, maybe marketing or sales where there's kind of like a de-facto Playbook that they're that they're running where where they might be. So, for example, of marketing, firm might be trying to figure out how to run ads against a Persona of a particular kind of class of user and leveraging machine learning for not just telling you, like, who do you need to reach out to and Now, but also like, you know why, I mean, you know, I've seen that at least with, you know, a lot of the, a lot of the customers that we've got here at xyonix. That's been something that's really resonated that kind of a thing, like digging in.

BILL: Actually, Carson says something super important in, not to not to dismiss what you just said, but yeah, no, please. He said, you know, you talking about say you're dealing with images and, you know, you just happen to have a fire hydrant in the background, which, you know, maybe is more has to do with dogs. Doing well with documents and say or something. But, you know, part of this exploration is to is telling a bit more insight into your data itself, you know. And maybe if there's you know, in this case some biases and the data like the biases. Well, this thing only really works when there's a fire happening around which is not, you know, it's not generalize

DEEP: desired.

BILL: Yeah. It's not desirable. I mean, you know how and you can imagine a scenario for example, like, you know, sort of the typical applications, some like this is, you have ai that's being used. Be able to say whether somebody gets a loan or not you know. Yeah and if it's behind the scenes, you know, so you're a model is really good at predicting when somebody's going to miss payments. And that's kind of, you know, your kind of judging it based on lots of features for these people and so forth. Your model comes up, somebody very accurate. You have to be able to explain that to the customer. And if the model says, hey you know one of the reason the main reasons is this guy is in the age. You know this is this person's age which turns out that could be actually illegal, right? You know, so from a lending standpoint so whether it's like the red fire hydrant, or you know,

CARSTEN: some aspect like age,

BILL: it's very important in sort of those kind of aspects, understand your data more. And what your data is telling me--never, has by biases that maybe our are something you want to avoid.

DEEP: Yeah, I think that's a really good point. Like and then a question is like what do you what do you do about that? Let's take that scenario a little bit like I had won a similar or one where It was in a predictive policing scenario, where, you know, there were cities kind of large Metro cities that were trying to figure out how to predict where to Route police resources. So if you know like let's say you're you know you haven't yet released. This thing you're playing around, you find out that the model is honing in on these very problematic. You know, kind of demographic variables, like you're talking about, then what the questions like what do you do about it? You know, like, do you strip those variables from the data set, like what? Do you actually you know, like how do you how do you handle some scenarios like that?

CARSTEN: I think I think bias is even more complicated and expandability, right? Because the model could be right model. Could because your data set is biased, and there is actually a majority of, let's say negative cases in a certain demographic and you definitely don't want to use that because it's it's strongly biased against the demographic. But as far as the model is concerned, mathematically speaking, it's completely correct, right. And Yeah, you can remove the variable, you can try to like take that variable and balance your data set so that you know all the cases you're looking at the equally distributed according to that variable tough, tough cookie.

BILL: Yeah, I think my point, my point is explainable AI doesn't solve that problem, but it alerts you to the problem, right? Yeah. And and as you know, that's it, your question is the tricky one. Especially, you know, nowadays was a lot of controversy around this issue. And, and what do you do about it? I DM for example, has is a company that has a product that is sort of geared towards. It's called AI Fair. A fair AI or something kind of an initiative to try to provide non-biased data and non-biased modeling. Yeah. Well we'll probably do

DEEP: a whole episode on just that. Yeah, at some point soon, you know, I mean, just to kind of like, we'll have to move on and second. But I mean, with those policing apps, one of the things that I found interesting was that That they actually kind of limited the data down just to, you know, a lat-long coordinate. So, you knew where the crime took place, the type of crime that it was at a time stamp. And with just, so there were none of these other variables were present. And then, they just use these three and you still had, you know, the models performed quite well. And then if you then sort of look at the potential biases, as if the bills were there, you know, the bias is actually still there. You can't actually take it out and so it's a problem from, you know, like an explainable. Like, how do you communicate that to the public, you know, when a lot of times folks just don't trust anything going on in an algorithm. But in reality, it's like saying, you know, I don't know. So, let's so. So, there's a new technique bill that you've been pretty jazzed about, and I have been talking about these Shap values and we started using them on some projects here key. Yeah. Maybe like touch me a little bit about like what did you do? You know pre-shop for explain ability and like what did Shep bring to the table? That's you know, a little bit

BILL: different, you know. Yeah. So the course of action is you get a bunch of training. You get a bunch of data, you divide it up at the test and you divided up into sections, you can say 70%, goes to the training, your model. And then you have this holdout portion we used for test and you know, you choose the model does your, whether it's random forest or somewhere. And that's working to train up your model and then you spit out some results. And then the customer wants to ask you. By the way, I just, I gave you like a thousand features, you know, the age and city population, blah, blah, blah. We just want to know like you, like you said before deep, you know, our marketing department be really interested in figuring out. What are the most important features out of this set that, you know, the models using? And so a lot of times these models that have they'll have they'll have feature importance functions that you can just call and say, you know, what did you think was important in making your predictions but it's working on a bulk basis, number one. And number two, if you switch models, you know, if you go from say a tree based model to neural network or whatever, You know, you're going to get maybe a completely different but valid view of what features are important, so you can get total, you know, the marketer is asking the same question, right? The market is saying what features are important,

CARSTEN: and if the worst filter is, if you don't have to switch models, all right? You can see you could, you can train the same randomize model it. So you can train three, different three models, and each one will give you a different predictor importance, and sometimes you see that sometimes it's great. It makes sense, because it agrees with you on a human level and sometimes Looking at it and you're like well that doesn't make any sense. Yeah. Now it's the same model and it's just you know, three different training runs with different initialization and you get three different particular importance is. So

BILL: yeah, that's it. That's a so I always I think I think, you know, Carson we had a conversation long time ago, you called it like the black magic or something. You know, the like the black art of trying to figure out what you know these models are actually thinking. So personally I never felt super comfortable in answering that question knowing those. You know, the variabilities vagaries that are associated with coming up with what features are important. So, okay, jumping ahead. There's the package out there in the world called shap and it stands for shapely additive prediction and we can talk about that. But basically cutting to the chase, this package is one step forward towards kind of shining a light on the so-called Black Box by by letting us know what features were important for a given model. All a Tapper decision level instead of this entire training. Yes, that level. And so that speaks directly to explain ability if we go back to the analogy of you know you're using this to say whether somebody's going to loan or not you know, and let's say they're leveraging in a bank and it comes back negative, you know? It says yeah I don't think you can trust this guy, you know, for give him alone. Now you got to explain it to them and you can say, well, here's the reasons why well, it shows, you know, something simple like You know, looks like you missed a couple payments in 2016, that kind of was a big blemish, you know, you haven't been with the bank very long. So that's kind of goes against you a little bit and maybe, you know, as a person, a real person, interpreting the results, you can have a conversation with that individual now and say, you know what about this? And it's like, well, you know, it's going through a bad time, you know, maybe they have a minute, some human element there. So, you know, I don't think the shared values are you know, like the end of the story but the, the beginning of a good direction that so we can talk about things on a per decision level instead of a gross level

DEEP: You're listening to your AI injection brought to you by xyonix.com. That's XY. O ni x.com check out our website for more content or if you need help injecting AI into your organization's. Well, let's let's cut a dig in there. Like, when we talk about, like, what did we? What do we learn? When we talk about it on a gross level? As let's say, as the person trying to like improve model performance and then what do we learn on the specific level? Maybe not the communicating to customer Vantage, but like again as the person trying to improve performance of the model, like what's the difference between those two vantage's and how might, you know, one sort of in Inform model Improvement

CARSTEN: the third quest to what degree can you actually trust them?

BILL: Yeah, yeah,

DEEP: yeah. Because it's very tempting to just download some, you know, you know some python package and run it through

BILL: and and get a great plot. The question is like, yeah, how do you, how do you, how do you gain some confidence in this stuff? So I kind of wish I could say car said I'd love you to jump in on this but for me, you know, I've worked in in scenarios where you had Thousands and thousands of features and, you know, you definitely want to trim things down in a way that again, it has to do with explaining and not just the customers, but to your boss or amongst yourselves, why is it that our model is? What is it thinking in these, in this sense, if you can if you if you put aside, the fact that there is its variability and you sort of trust with the what what things are saying in bulk? You can sort of say, you know, we're dealing with a situation here where it looks like 10 of these 10.

CARSTEN: I

BILL: was in features of the things that are really moving this model, so maybe we'll save a ton of money, for example, by not collecting, all these other. This other data, you know, why are we wasting our time?

DEEP: Or the flipping or the flip side, which is like, hey, this thing that's really important is there any way to get more about

BILL: it? You know, is there a way to me like you've run

DEEP: some surveys or like extends Russians into some existing

BILL: surveys or whatever? Totally and in what you just said that you feed that back into the loop and shoot. Maybe your accuracy goes up like, well, we're going down the right path, but Carson.

CARSTEN: So let's go back to that. What I said earlier, where I have a decision tree and I trained three different decision trees on the same data set with different initialization and they do a feature importance and I get like completely different hierarchies of feature importance, for least three trees, how do you prevent that? Or what does your Chef model are they trained on or are they analyzing a given trade model? Do they train their own models? How they produced And what sort of stability is in those models, right? So, because it could be that I'm looking at a certain particular importance, but it doesn't apply to my model at all. So the more data set related or more model related,

BILL: so can a jumping ahead in the conversation and shared values are sort of advertisers being able to be applied to any model, the computational burden can be quite high and so some of them have customized routines for tree based models, customize routines for neural network, based models and then they have kind of like this generalized version that you can use for, literally any model you want to

CARSTEN: seem to it today. Retreaded retrain your model or do they work? The trend

BILL: model they work with it, the train model, but substituting subsidy Baseline features for other feet. So we're kind of getting into the details, but the I wanted to make a comment about what you just said, because it's totally true. Like decision trees are known to be super fickle beasts. Like a small change in the training data will change a tree. May be completely and therefore change what features are important to it. But the try to offset that effect with Things like random Forest where you grow hundreds of trees or thousands of trees and you're taking random samples of the data both in terms of the features that you use and how many observations are feeding that thing. And so there you're kind of getting something that's probably a bit more, a lot, more generalizable than a decision tree. So, my first, my first, I agree with you about the decision trees but there are maybe some solutions towards something that like that using rainforest, but they shot the Sabbath. I'll use our like again one of the selling points is is there not Hold on to one particular model when I've actually used

CARSTEN: right. But I mean just just decision to use as an example. Right. It's not it's not that particular model it's about the the instability of feature important no matter what model you're using right. Neural networks have the same thing, train a neural network, three times. And well depending on where the gradient descent, when you have a different network and you have different features that are important.

BILL: So as a practical measure, what I have done is I have tried multiple models, not trusting just one with the shat values and looking for a common set of features that that Chef thinks are important. And you know, so in other words, if I run something through random forest and I get one to you, I run it. You know, again, you know, you looking for variability, within one model that also across different types of models,

CARSTEN: right? That's pretty interesting. Kind of variability that you experience there. How how, how much did the sets of features differ from each other?

BILL: You know, with the random Forest stuff it tended to for the for the top. The amount, of course this is all very much the asterisk asterisk on this conversation. It's sort of always. This is all week data-driven stuff, right? And you can't talk about this in a general sense of to talk about in terms of given the data that I was dealing with at the time. What do I experience? Variants and I found that, you know, like the top you get these as a stack rank list and the top like five or so. We're pretty consistent within a given like the random forest model and then if we switch to something else, like maybe instead of using like, we used some sort of gradient boosted or some sort of weak learner tree. Based model, I found in this particular case that there are quite a few which was nice. Features that it said were important that were common with the random forest model and it tended to be those top features when you get down to the start of the minutiae features that don't really seem, then I start to trust it a lot less. And in. So I mean, I guess my takeaway Carson was II, totally agree with you. That I have this sense of like, you can't really trust his stuff, but when you start seeing consistency, both within a model and across models, you know, maybe there's something more to this. I didn't go, I didn't have the time to go into step into a neural network model at that point say, but that would have been a great experiment to run. So

CARSTEN: yeah I mean at least it gives you an idea and you get some confidence. The same candidates show up again and again

BILL: and dude, I'm going to I'm going to say this. There's that and then the other thing was and deep talked about it earlier was quite a few. These features were very intuitive. It's like. And then when we went back to the customer with you that they're like, oh yeah, actually there was a couple of them. I didn't quite understand, we're back to the customers. Like look, this is a kind of consistent message, are we hearing from the shop values? What about this? And they say oh yeah I can you know that's what Only see that.

DEEP: That's a really important Point Bill. Like a lot of times, you know, when you're interacting with somebody who really understands, you know the data is context, you know, as the data scientist, we don't always know that but sometimes but you know in our case we're usually working with a customer understands our data. Well to the extent that you learn something from you know, about your about your variables, whether it's chaperone, other techniques, when you get that validation from the customer, they come back in like, oh no, that makes a lot of sense. And here's why. And then they end up it. End up a conversation. We're all of a sudden the customers talking about, you know. Well there's you know, there's this these other four or five variables that may be as the days since we didn't know what they were. They were just a bunch of cryptic values, but all of a sudden they're like, yeah and they start talking about their business logic or their business rules. That sort of ultimately described give you a lot of insight into what's going on. I've just found that conversation to be super valuable because it usually

BILL: helps us understand like you know what other data do? We need to go after? Sure. And get Yeah, I totally agree. It really, I mean it's it doesn't it feel like I mean for me it felt like a new conversation that I couldn't have had a decade ago, you know, like like with any confidence at all you know, so

DEEP: well. So that's that's that's something that I'm not I'm not. So there's a difference between like, you know, like a decade ago, you could still have taken a model and you could still do things like you know, withhold a variable Run it look at your predictions and then stack rank, let's say the variables based on their omissions impact to, you know, to efficacy. Yeah. You know, you could do stuff like that. You could do that with combinations of variables and you could have an overall picture of, you know, maybe not perfect but you would you'd be able to understand like you know certain variables and maybe combination of variables that were really important. Like that's where I think shapes different and I don't I don't fully understand us and maybe you can kind of Speak to the Press, understand it, or how you speculate, it works. But like make like what? When you have the ability to say something at a particular, You know, customer or prediction or inference level, but what's what's going on? That's different than this. Let's say, this this variable or mission strategy.

CARSTEN: But also we, before we move to that, this was before a Fluss by track of thought, the whole explain ability. It's also a philosophical question, right? You said earlier like it needs to be intuitive and needs to make sense and sometimes it doesn't. Right. For example, the decision tree is a model, you can always explain completely. But whether or not the decision, it makes you can you can I can tell Exactly why it made that decision because I just traced down the tree but it may or may not make sense. And so there's like this element where it's really important that not only, can you explain why the model decides. It also needs to make sense to the human Observer before they reject it, or accept the model, right?

BILL: Yeah, I agree with you. I almost think it's like it is philosophical, but it's building trust in what this thing is telling you with your customers or with, you know you understanding Going on under the hood, it's a trick. It is a true and people think of, you know, I know that depending on what type of exact you are. You might think about a models performance on a global scale, but like, customer service doesn't they want to know about Bob, like, you know, and most a lot of times these models are used on a very personal level. They're trained with a giant swath of a population, but they're usually applied down to an individual level. So, knowing something about an individual decision is super

DEEP: Organ

BILL: and that in that world,

CARSTEN: right? It goes back to what we talked about earlier the bias. Right? Let's say you have like because the model learns was in the in your data. If your data is not good, your model will not be good and you might get lucky and you might spot it. If you look at predictor importance, you see most successful surgery surgeries are those were the surgeon wears a red bandana. Yeah. Now, if I see that, I would certainly question that model and I would say, what the hell would question right region, but the red problem, but it can be much more Subtle, the the same bias could be present as a gender or age bias as you can see. Well, it's in people over 60 years, but it's wrong. And the reason why the model thought that is because in your particular data set, that is actually true, but in reality isn't and so some of those buyers are easier to spot. Some are harder to spot. Yeah. Sometimes predictor importance can give us a clue. Sometimes it can

BILL: I think you raise a really good point. Like I felt really good about getting that feedback that yeah these things make sense to us. You're right. They don't necessarily need to make sense. I almost feel like again you set up almost like a trust issue with those people to have a broader conversation about look, here's something that doesn't make sense and you know, why is it? Maybe we need to study this, a lot more to figure out why that we agree on these top three things but there's this fourth one, you know, that is doesn't really seem to make sense to us. Intuitively? Why is that in there? And then you start to look at the data, you start to look at maybe There's a problem with the data. Maybe there's an error in recording Etc.

CARSTEN: And it's even worse. When when you have to situation a deep mentioned earlier where you are, not the subject matter expert. You are just the data scientists. You got these, like 300 features. You have no clue what some of them are. Yeah. Because then you can, you can analyze this but you are not in a position to make any judgment about what you find. I don't know if variable X makes sense, or doesn't make sense. And

BILL: so this particular situation man, that happened with me because I didn't understand there was a name before Me that I thought I might understand that I didn't really know what it meant and and so it led to naturally to a much better conversation about what this actually means and actually where there might be a problem with that data and maybe that's why or maybe that's why. So you get this feedback loop with this type of analysis deep that your may be alluding to earlier that maybe that you couldn't have done before because you're starting to you starting to look at the atoms instead of the Impound. You know

DEEP: it's not just about this variable is predictive at X rate or not it's about well in these particular you know, value ranges or whatever. Well I think it's

BILL: super important because right before you're talking about like on average over this yes thousand blah blah blah, this is what happened. Well, so then you asked me at the shop values. One of the things that it's kind of neat about this qualitatively. The shot values are measured, Feature important set of her decision level. We've said that over and over again but they're always doing it relative to some reference point and typically that would be like the average Joe in your dataset. Like we ran in an article that we wrote. We did a mod models to predict whether somebody is going to survive that the Titanic or not. Yeah I don't make it so what what the shot values give you is like, okay, we have all these people in the training dead and say the survival rate was like whether going to live or die, that's a the survive. Oh wait, I don't remember what it was, but it was like, maybe, you know, 25% and now you're looking at a particular passenger, you know, Joe and the Chevalier was say, you know, given this feature, like, for example, given that he got a first class ticket that feature alone, tend to migrate him from a 25% chance to like a 40% chance of surviving. Yeah. So it gives you a relative score to some reference in this. In this case, By default is the average Joe. But also you can say things like well he wasn't. He was an older guy. Like he was above 65 years old which decreases the chance that. Yeah. So the added a part of shapley additive is the fact you're basically saying okay from some reference point I'm moving right? Or left, depending on what they felt the influence

CARSTEN: was

DEEP: here listening to your AI injection brought to you by xyonix. Dot-com. That's XY. O. Ni x.com check out our website for more content or if you need help injecting AI into your organization's. when I read that, you know, that article of yours, one of the things that really struck struck me was, I've never really Like it's not normally, you don't get that much. That deep of an intuition about who lives and who survives, you know, usually you might focus in you build your model, but when you read that article it was pretty clear. Like, you started to get a really crisp picture about, okay. You know, if you're female and you're young, and you're rich and you had a title and those types of things on the, on the Titanic, you lives like, you know, and I think it's easy when you're when you're just hiding behind up, you know. Floating point value between zero and one of your prediction that you don't really get into that kind of those darker spaces as to what's going on. That was one of the things that I found really, really cool about this technique and this approach,

BILL: you know what, I wanted the thing about that was is that I think, you know, you got to be careful because you could go about getting a result from Chap and then making up a story line that aligns with that no matter what. But I will say, personally, I didn't have to work that hard Hard to see that if you were rich and you got a first class ticket, you tended to survive when you were, you know, like things that made a lot of sense from a human perspective. Yeah. You know. And so that was to me having kind of used it for the first time. I was like, well maybe maybe there is a little bit of something going on with this stuff. So it's kind of interesting. So cool. Do we want to do? We want to talk about like shape? Like sharply like it just may be the man. Maybe it's just a side note. Yeah shap shap. Lee was this mathematician dude? Who anointed shapley? Yeah, ended up winning a Nobel Prize for his work economics and he was involved with Game Theory stuff and like his work was centered around the idea that you have a bunch of people that contribute to a coalition and they went there. Um, sort of overall, gain for that, for that work. Yeah. How do you divide the spoils fairly amongst the participants? So, like, it's great to think about, like a basketball game like, say, you know, we three participate in the three have three tournament over in Spokane or something. And, you know, we play a bunch of games and then, dude, we win. You know, we get, we take home the $10,000 prize. Now, it's like, how do we divide the 10,000 bucks, fairly? I mean, we could divide it by 3, but You know, so one way to look at, it would be like, well, how about just? Let's just say, whoever scores the most points. Uh-huh. And that would be one way, but that's a very naive thing because we know in basketball, there's a lot to do, you know, winning with defense. So maybe deep, you're like the dominant man, under the hoop and your SWAT and stuff away all the time and that's me right? Or maybe, maybe I'm putting this out there. Maybe Carson, he has a soothing effect on us. So when he's when he's in the game, We just are much more common. We're hitting trays, like no problem, you know, so forth. So you can imagine. There's there's all of these totally sounds like me. Yeah, exactly.

DEEP: Yeah. I'm dominating the center with more with. No, it, no ACL. And,

BILL: and I'm just going to put it out there that I probably could get not the line. Share. That's probably right. But you understand the problem is that you can imagine this for any sort of Coalition type of game. You know, what is fair and getting your share of the spoiled. So shapley basically to cut to the chase came up. With a mathematical formulation for how this is, how one might go about doing this sort of exercise and do it in a way that's fair. And so, yeah, without delving into the theory, anything but that was kind of his contribution and it's very interesting because it you know it's also like a you can imagine. And in a three-on-three situation we need all three of us. Imagine that will hold basketball team, obviously some players, sit out certain games and so forth. So So, and when players are introduced and do game might matter like, you know, deep view winter because you're the D Man and so forth. So to cut to the chase, what goes on, behind the scenes with these computations is a lot of these sort of mathematical experiments that are run almost like simulating these games and different different times when players are different features. In our case are put into the game to serve things out and then for a given features contribution, it's sort of a It's all these sort of permutations and

DEEP: does that basically what's happening in the it is in the shape value extraction is that they're like kind of manipulating the variable presence or values or ranges of values that are present. Yeah. And based on that, there's this kind of evolving understanding of their

BILL: roles kind of like yeah, you take conjunction with

DEEP: other variables.

BILL: Like if let's say, we are all passengers on the Titanic. We would take and we want to figure out what bill We'll survive, you know. Yeah, thumbs up. You know, you take my features. And then what in a what you're doing is you're taking say one of my features like bill is so old, and then you're filling the rest of the features in with the sort of the normal drill Baseline features and and then you add. So you're sort of introducing a bill is that the certain age and then build ticket first class taken, so forth. And then you re run that experiment and then you You change the order, and then you take sort of the average of all these things. Anyway, it gets quite complicated, but not that you're sort of filling in the blank with some sort of reference Average Joe value. And so it's always relative to something like that. Now

DEEP: that's super helpful because then that gives you an idea of like what aspect of Bill is resulting in this

BILL: classification. Yeah. And mean that so and a piece by piece. Yeah. What what the person that made the shack package did it was we sort of took this economic theory and translated into the machine learning world and came up with some You know, computational goodness, to speed the calculations. And yeah, it's pretty interesting, pretty powerful stuff. But I think when I saw some videos of him talking about the work, it's like, you know, how do we explain what's unique about Bob relative to the average? Joe, and that that's something that's very powerful. Maybe you are, they may be are there Average Joe that maybe your. Yeah, something about a huge unique.

DEEP: I want to throw this question out there for both of you guys. Like, do you like, what is the Roll if at all of explain ability and shop for unstructured problems. So we've been talking a lot about Titanic. We've got a structured data set, you got named aged eight, you know, birthday wealth, status, Etc. But what about in like unstructured cases where you just got a blurb of tax or you've got, you know, somebody or you got some imagery like you know, what kind of a role does, this sort of explain ability have in that context and how do you go about maybe leveraging it?

BILL: I'll let Carson speak after me. I I've seen it used like he mentioned there are certain aspects of the images that tend to like certain portions of like an ear or something that tend to to be highlighted by shat values in terms of like detection, Died. I seen it used in. That case is as examples, I don't have any experience in it personally but I could see where you know, you might when you speaking of sort of like images or something, you can see where certain portions of images tend to be the thing that's causing something to trigger for a classification exercise or not.

CARSTEN: Yeah, Soso but unstructured data you mean things like audio or video or imagery?

BILL: Yeah,

CARSTEN: yeah I don't I don't think you can apply it because I don't think that the game theoretical variational approach. Is that make sense? Where you compute? Like, you know, the distributions that bill was talking about and then vary them slightly or replace the average Joe and swap certain pieces of it out. I don't think you can apply that there. And so people came up with other techniques. For example for for object recognition. You have these, these great chem class activation mapping approaches where they basically based on the gradients and a certain classes with activate it in your classification model traced back which which features of the convolutional layers were mainly responsible and contributed strongly to that decision. And then you get these weird heat maps that kind of like highlight you know, imagine you're detecting hey something is a cat, we have a cat sitting on a Yeah. On a desk you'll see then hide the heat map highlighting the cat in your like great. That is correct. So that helps, but it from our experience, we have we have worked with in some surgery video context where we were like trying to detect instruments or trying to detect certain certain events happening in these videos. It doesn't always make sense it. Sometimes it highlights the instrument that you're tracking and sometimes it highlights something completely different in the image. And then this

DEEP: family's interesting. I remember seeing you put together one of these with some cancerous lesion detection. Yeah. Like for an endoscope and and in the in a for those haven't looked inside of an Endless Sea of the, you know, you got these, like, jellyfish-like creatures in there and a lot of times, yeah, it like honed in on the, on the clear lesion but a lot of times it's like honing in on other Reich, random parts and you're looking at it and you're kind of scratching your heads.

CARSTEN: Exactly, exactly. So I feel like you're diving into this. Unknown section of the model where there was a distribution, your data that led to the model believing that these sections were important and maybe in the context of what you have showed in this training data they were but in the overhanging global context that we as human know about the problem space it doesn't make sense and so you always run into those issues and I feel like those are the same issues that you run into with structured data as well. When sometimes the variable pops up and you're like, I don't know why I chose that makes no sense to me. I think it's the traces down to the domain of the data that the model has been trained on compared to you and your background knowledge and reasoning as a human. Because in the end, this these models, don't reason they're still pattern recognizers. And so, all they know is what the patterns, if they can derive from the data, you're showing them.

DEEP: Yeah. Yeah. It's this part of like, trying to Like map back what you're seeing to what the model seeing and how the models interpreting it. I mean that that's almost always the source of like really fascinating conversations. Like I remember I was I think was a city of Chicago had built a system to predict. I thought I'd want to say was like illicit cigarette sellers like whose bootlegging cigarettes and and they had like all this data you know, about Merchants, you know, and The and one of the features that was like, you know, the most predictive of who was bootlegging. Cigarettes was the presence of males and name in the store title. And everyone is like what like what is that like you know, it Ken's Market or whatever and and then you know like it triggered all this kind of Investigation that's digging around and eventually you know like it kind of Makes sense, right? Like you think about it, like if you don't have the presence of a that, then you're talking about a 7-Eleven or you're talking at, you know, which has like a corporate structure like corporate, you know, like controls in place where as it turns out like the first, you know, the first name, presents the mail. First name presence, was indicative of a small shopkeeper. So it's like a small shop. There's no 7-Eleven Corporate board to worry about, you can kind of do whatever. And like, you know, hey here's some It's like under the table and so it just kind of got me thinking that explain ability isn't always like, I mean, there's like, there's like you're sitting at in some, you know, you accidentally stumbled into some like, you know, something outside of your department or field like you wind up in the English Lit, Department, talking about some obscure thing from the 17th century have no idea what everyone's saying, but it's perfectly well explained. But like, you just, there's like some background knowledge, guys. This has been an awesome conversation. Thanks so much. I'm going to just leave it with one last question for the two of you. If there's if if our listeners want to find out more, you know, about explain ability like what are either some good articles, some good things to search for, like Search terms or some, some, some good stuff to look for?

BILL: So, I'm gonna, I'm gonna, we wrote a couple of articles that I think are good starting points, because we got a

DEEP: lot of paper at a lot of things. I like

BILL: it. Next.com, XY, o ni x.com and or in your favorite search engine. And look up shap sha p-values and xyonix are shap values and Titanic and you're going to hit a couple of articles there and then and then there's the maintainer of the shape stuff. Scott Lundberg is his name. He's now at Microsoft research I believe or Microsoft land somewhere. You could you could look up some information on him as well.

DEEP: And Carson, you mentioned the grad cam stuff. Is there any anything that our folks could look up to get closer to some of those techniques you were talking about?

CARSTEN: Yeah, Google is absolutely great. Just Google crack cam. You'll get rights of information is the same thing for explain ability and I just Google explain religion Ai and you can read four days or just, you know, be more specific look for it on arxiv.org if you're more interested in scientific papers, but there's plenty of information out.

DEEP: Alright guys, thanks a ton till next time.

BILL: Next time. Alright care

DEEP: that is all for this episode. I'm Deep Dhillon, your host saying check back soon for your next AI injection. In the meantime if you need help injecting AI into your business, reach out to us at xyonix.com that's x-y-o-n-i-x.com, whether it's text audio video or other business data. Do we help all kinds of organizations like yours automatically find and operationalize transformative.