Unless you have a plug-in, this should apply. The North American Gen 2 2004-2009 did not have an EV button, some other markets did. The EV button can be useful if you need to move a Prius within a parking lot, or you need to get home silently from perhaps one block away. In general, however, it will get poorer mileage over all than just letting the computers decide when to use EV.
It's typically used in the following situations (not an exhaustive list but at least it gives you an idea) Moving your car a short distance (e.g. from one carpark space to another carpark space) Moving inside an enclosed area (e.g. underground carpark) You know you're going to travel down a steep mountain and you want to use a little bit of battery so that you have more space to recharge when you travel down the hill (delays the use of your brake pads if you can use the electric motor to charge the battery. Once the battery is full, the car will revert to the brake pads to slow the car down). Note that excessive or poor use of the EV Drive Mode button can increase fuel consumption because the car's computer likes to keep the battery charge at 6 bars on the display (that's about 60% real state of charge). If you drop it below, the engine will come on to recharge the battery so you may end up using more fuel in the end.