Ive been out there and experienced the same thing, but I've also been in the boat that was wearing em out too. There could be a lot of different reasons like a little underwater structure that the fish were holding too, also I have seen times where I wasn't catching anything using just minnows on a gold hook (thats bout all I drift with...I'll throw a jig while I'm doing it, but as far as drifting goes minnows don't come in but one color ) and the ppl that were catching had jigs tipped with minnows...It's frustrating, but I guess that's why they call it fishing and not catching.