I always try larger baits first especially in stained or muddy water. In my experience, I have caught large and small fish on large baits. But, I let the fish tell me what they want. If I don't get a lot of bites I will start downsizing. Also, if there is a bunch of small baitfish in the area I am fishing or if I think the fish are in a negative mood like after a really hard front then I will start going smaller. Sometimes in crystal clear water where the fish can see the bait real good I will also use smaller baits. In conclusion, there are no hard and fast rules. Every time on the water it is trial and error until you dial in to what the fish are wanting. That is what makes fishing so much fun.