1st scenerio- If I'm catching smaller fish consistantly, usually I can move to another location using the same techniques and find bigger fish. You may be in a school that is simply holding small fish.
2nd scenerio- You said you were on the lake early if so it may not have been the wind at all, just the window of opportunity narrowed. Or if I am spider rigging, especially now with the weather patterns being irratic, they may want the bait to be almost motionless. When the wind pics up, the baits maybe moving a little or a lot more than they are wanting. Try finding a wind break and fishing the same way. Also what J White suggested is also a great option.
This only my opinion, but nothing you can say will change my mind. That makes it a FACT.
Today is a Blessed Day and a Prosperous Day