Serious question, a guy at a Chicago meetup said never trust a model that can't show its work, and he was spot on | Algorithmic Fairness Debate - Free To Speak
Did he explain what made him say that? I get the feeling though. I've been burned by models that just spit out answers with zero way to check them, and you end up trusting stuff you shouldn't. When a model shows its steps you can actually catch where it went wrong instead of just hoping it's right. That guy sounds like he learned it the hard way.