I use neoprene fishing gloves. They keep your hands warm even when wet, you can stick your whole hand in the water and your hand stays warm. There is a hole to stick your thumb and index finger through when needed.
Here is the pair I use: Bass Pro Shops Neoprene Fishing Gloves | Bass Pro Shops: The Best Hunting, Fishing, Camping & Outdoor Gear