Justin, if you feel a need to do the Arthur Fonzarelli thing every now and then with the starter switch then maybe you have a dicky starter switch.
I'm no rocket scientist but it might be a good place to start. Sounds like it's gonna let you down when you least expect it.
There's nothing worse than trying to push start your Harley. There's always some Ducati wanker watching, waiting for a photo op.
Roo