Like I said ... it probably won't hurt.
Some people rub their jigs in the slime of the fish they catch ... cheapest "scent" going
I smoke, handle gas & oil products, and suntan lotion ... and probably transfer some of that to the baits I use. Hasn't seemed to have much of a negative effect, but there's really no way to know for sure ... since you can't go back in time and fish the same place/way, minus those chemical influences. But, like I said ... that's just me. Other's results may differ.
... cp