If the little red light (on the FOB itself) doesn't illuminate brightly or at all when you push the button, it's time. Photos and the change procedure are documented in the User-Guide. .
Another clue is the car beeping at you for no apparent reason. It beeps when it's on and loses contact with the fob.
Yep, my 06 Prius smart key stopped working. Replaced battery and now it works. The spare smart key little red LED comes on when I used the key, so I replaced the battery on the spare smart key as well. Thanks for the smart key assembly instructions Prius Chat!