Does anyone have issues with hands free system when calling over VoLTE

So I have 2015 Prius 4 with the entertainment package and in past month or two I started having weird issues when someone called me I couldn't hear the person and similarly the person also couldn't hear me. After about 10 seconds the phones just hang up by themselves. When I initiate the call, everything works fine.

I investigated it more and noticed that it is tied to VoLTE (Voice over LTE, also called HD Voice), and this issue only happens when both phones have it enabled. I read more about it, and looks like it provides much better sound quality, and to work the Bluetooth device also needs to support it, but if it doesn't then it would downgrade quality to the standard one.

Does anyone else experienced this issue? I can't be the only one.