Coasting downhill, in neutral, just cleans the rust off the brake disks which don’t get much use otherwise. I’m not worried about wearing out the brakes! Also, it’s very easy to shift into D from neutral so I don’t consider it a big safety concern. My hill drops about 500’, not thousands of feet.
I've found that turning the HVAC fan 'off' also has a tendency to let the car go into EV mode when the engine is cold.
Likely, the best thing you can do to prevent this from happening is to (1) make sure every device that prompts an automatic ICE start up (example: using cabin heat at outside temps under 14 degrees F) is turned off and (2) get the battery warm prior to driving. You can get the battery warm prior to start by parking in a heated garage or, if parked outside, charge the battery for at least an hour prior to driving.
It also makes the brakes HOT! Your short hill doesn't make them too hot, but there are plenty of taller hills that will. The Pikes Peak road in Colorado is serious enough to have a mandatory brake inspection station where the attendant measures brake temperature and orders vehicles with excessive temps to park in a 30-minute cooling off zone. And these are people who descended in Drive, not Neutral. I witnessed a Florida pickup, stinking to high heaven, ordered to park there. Easy with modern automatic transmissions, yes, especially with Prius-type 'transmissions' where it is all done with electric motor controls, no mechanical gear shifts. But difficult to impossible on old-era manual transmissions, especially on heavier vehicles. That is why it was outlawed generations ago.
If the next drive starts with a downhill segment, DON'T fill the battery all the way full. Going any significant distance downhill on a full battery will spin up the ICE for engine braking, requiring the ICE to also be warmed up. So when charging at the top of a hill, stop the charging before the battery is full, leave the EV range several miles short of its full range.