Are you using any kind of scent? Juiced up baits panfish scent works GREAT in situations like this. The fish inhale and hold on longer allowing for a good hook set.
The scent comes in a spray or gel so it really stays on for a long time.
Also hook size compared to the size of the bait. You need to make sure you have enough hook exposed so it stays in the fish better.
Hope this helps?