I'm more of the mind that a bouy is more of a reference point than an exact mark of where structure is. I can accurately cast and fish 6 feet to the left of a bouy just as easy as 6 inches to the left.
I'm also of the opinion that dropping a 2 lb lead weight through the middle of a school of crappie is a great way to scare them off, at least some of them.