I have to answer "very little difference" because I actually haven't measured it. I think driving at highway speed with windows down may actually use more fuel than having the windows up and the AC on. In the Gen3 the ECO mode operates the AC in a more efficient manner and still keeps things comfortable.
The higher you set it the less energy it will use, I usually set mine at 77 or 78. Of course when you set it that high you may need to reset the temperature on cool mornings or it puts out to much heat. I wish it had two settings one for heat and one for cooling as the one in my house does.
I think air does have a negative impact on short trips. Part of this relates to my soc being lower when I start up. But, in the texas sun, I would not drive without ac. It would be nice to have that dual setting along with max fan speed would be nice. I can't imagine it would really cost toyota much to implement it, but smarter air conditioning should be a new tech addition. My car is confused when I leave the garage, but I quickly need air conditioning. It would just be nice to have.
Air Conditioning Versus Open Windows Air Conditioning Versus Open Windows : Driving Economically Air Conditioning: Does It Really Burn Gas?