Rear Brakes - Weird Symptoms

Hello everyone,
I have a 2011 Prius. Yesterday I noticed a strong brake smell coming from the driver side rear wheel. Today I pulled the wheel off. The brake pads were worn to the metal closest to the center of the hub but on the outer area had 1/4" left. The rotors showed rust on the outer part. The caliper piston and pins moved/slide perfectly. Using the ebrake showed good movement on the mechanism but I don't know how to test that properly. Also note...I basically never use the ebrake so that is not the reason for the smell. After reassembling the smell didn't appear again after a few 15 minute drives.

I'm not really use what to make of this. Normally I would change the caliper but these symptoms don't make sense to me and I don't want to waste time/money. Figured I would ask here before putting on new breaks/rotors and potentially messing them up.

Any thoughts/ideas would be apprecited.