The Yamaha won't work for 2 channel. I had an old Pioneer VSX1014, a decent receiver for HT, but thin and hollow sounding for 2 channel. Just like you describe the Yamaha.
I had an old 15 watt Kenwood receiver that sound miles better in the stereo set up. You'll need a different amp/receiver for your 2 channel listening, I'm afraid.