We have an 04 Sienna and It's been on some adventures over the years. Ours is a 2WD, though.
It's hard to beat the practicality and room. We've taken several long trips and always run overloaded. We never travel light. Kids, dog, camping gear, coolers, food, more food, beach toys, Yakima car-topper (the really big one), and much more crap. We've also run it at 80MPH through Montana and Wyoming. That was not fun being as overloaded as we were, but the van handled it without a hiccup. My bung hole is still a little puckered, though.
I think an AWD Sienna would be an awesome stealth camping vehicle for all the reasons mentioned in the posts above. My only worry would be the lack of ground clearance. It won't quite go where a Sprinter will.
Plus, it has that uncool "I've pretty much given up on life" vibe, which could let you travel under the radar.