Yes color matters. But the color changes frequently and is relative to light, water clarity, current, bait, and possibly many other factors.
It is also true that changing color back and forth is helpful. But this, I mean will keep trying different color baits until I get a consistent bite. Then sometimes when the bite stops for a while, I'll throw another color and immediately catch a couple. And then I'll switch back and forth with these two colors and always catch more than those around me. I am pretty consistent in my presentation and retrieve so the difference in color helps get their attention again.