Sounds more like bluegill than anything else. Are you using a collared jig or one with no collar or retainer? If you are using a collar then try a small dab of super glue. I've never had a problem with anything other than bluegill with pulling down my plastics. Never had any fish completely remove it though. You've got one fine bait that you are using. Can't beat BG Baby Shads.