Change Volume while Listening to Youtube (Audio)?

I have some podcasts etc. that I listen to as background via Youtube. I don't want to project the video to the infotainment center, but I would like to listen to the audio. That works fine, so long as I hit "play" on my cellphone, but if I ever change the volume in the Prius it will pause Youtube and start playing Spotify (or whatever other media app it has selected). Any way to change this?

Using Android Auto.