I know what you're saying. 55mph downhill on pavement or a 30mph in a crit on 700x23, no problem. Cyclocross, loved it. But put me on dirt, or worse for me, loose gravel, with fat tires and my fear factor sets in. Probably because I had a big motor, but couldn't develop the bike handling skills. With a soft tail and I always felt less in control than with a hart tail. Doesn't make sense, but I always felt the MTB would be how I got hurt.
(edit, weird thoughts in the early morning) MTB's are kinda like fishing stillwaters for me. I'd rather be on a road bike or cx, just as I prefer river fishing, but hey, it's close enough to still be a lot of fun.