Expectations and such aside, I can't figure out what the EV Mode button does that "stealth mode" doesn't do better. With or without the button, I can't even back out of my driveway without the engine turning on.
The button makes it easier to hold maximum EV. I use it on ramps in parking structures where I have a reason to not want the ICE to come on but want full EV. Using the button means you can push the pedal further than normal and stay in EV. I find it makes the HSI a little "stickier" around the mid mark but it also lets you push past that and keep the ICE off. Sure, go too far and the EV Mode flicks off, which makes sense. Going past the mid mark doesn't give you extra EV power, so if you exceed it by more than a bit, clearly you intend for extra power and the car obliges. All the EV mode button is about is just telling the car to cut you a little extra slack to go a little past the mid mark while keeping the ICE off. No more, no less. I find it useful..
Whatever info I've seen from Toyota only makes reference to leaving your neighborhood or a parking lot quietly. I think, keeping that in mind, that's about all your going to get out of it.