If you are planning on fishing at Gifford Pinchot State Park (in Lewisberry), Hawghead Bait and tackle is right across the street from the rangers station on Rt 177 (Rossville Rd) open 7-7, 7 days a week.
If you are planning on fishing around Newville - closest place that I know of is Green Mtn General Store on Rt 34 south of Mt Holly Springs about 3 miles, - which isn't really close to Newville, about 25 miles away.
You can also always catch your own if you have a net (4'x4' seine net), but the creek minnows usually don't last too long in the warmer lakes around here. You might also want to try Little Buffalo State park near Newport - there is a place up there on Rt 34 as well - also near the juniata river, good small mouth bass.
Both Pinchot and Little Buffalo have crappies, closest lakes to newville - bigger lakes are around York and Hanover (Marburg, Longarm, Redman/Williams), or Huntington (Raystown)