This is likely due to a poor seal in the ear.
When no music is playing, your voice should sound louder and ambient noise should sound quieter, very similar to wearing a pair of ear plugs.
Creating a tighter seal is simple. Insert the ear tip into the ear canal and then gently pull it back. This adjustment eliminates virtually all external noise so you can play your music at a lower volume, for a longer period of time, without experiencing ear fatigue.