I think you would be doing the sunfish population a favor by taking a few of the out. You would never miss them. They lay every full moon for several months & the little fish are probally getting eaten by fish that you are putting back. So the population, by taking some, is not affected at all. Might help the growth size.(my opinion only)