Well that was dumb. While I was posting the question, it occurred to me that I should have put another rivet in there and see if it would force the other one out. It did! The scoring on the rivet shaft was keeping it from falling out as it usually does. Don't know why i didn't think of this 4 hours ago.
I tried driving it out with a nail the same size but that didn't work. I probably need to go to bed now, before i break something.