If you keep them outside- you need some kind of cheap security light to shine on them at night. Otherwise, they will take a stroll as you have discovered.
The light on at night pointed at your worm pile is an old trick that alot of the georgia boys use to keep their worms in place.