Image Sampler Channels

I'm not sure that I have a complete understanding of how the various image sampler channels work. I am trying to create an image that will distort a surface on the brightness channel (as below) but not on the red green or blue channels. But even though the image I'm using has no color, it's behaving exactly the same on those four channels. Can anyone explain to me how the sampling is processed?

Thanks!