A novel approach for realizing a microwave photonic image rejection mixer is presented in this paper. It is based on using a widely tunable narrow-band optical notch filter implemented by stimulated Brillouin scattering (SBS) to remove the image signal. The image rejection mixer has a simple structure as it only requires a single integrated dual-polarization modulator for both frequency conversion and pump wave generation for the SBS process. The absence of electrical components enables the image rejection mixer to be operated over a wide frequency range with a high image rejection ratio. A technique is also proposed to suppress the noise generated by the SBS process in the image rejection mixer. Experimental results demonstrate a high and constant image rejection ratio of over 32 dB for the image rejection mixer operating over an RF signal frequency range of 11.8-19.6 GHz, and around-5 dB conversion efficiency.