Why don't you have a little fun with him?
Tell him you asked around and the only reports you ever got of it happening was when someone was really dogging the car causing a little flex around the windshield causing the mirror to pop off. Reiterate to him that it ONLY happens when someone is really beating the **** out of the car.
He's having fun with your car while you're away. You may as well have a little fun with him too!
+1
It requires a special Viper glue to hold due to the torque it must embrace. The Viper glue goes for around $200 an ounce. An ounce might work, but most people need 2 to be sure it doesn't fall off in the future.
And to stay on topic - I've had this happen with a car that was around 10 years old. I was young and didn't know better, was borrowing 1 of my parent's vehicles, and freaked out. I wasn't exactly sober, and when trying to fix it I just used elmer's glue. I also didn't think to take the mirror off the bracket, and the mirror made it too heavy for the glue to dry correctly. It ended up falling off and my mother brought it to her normal mechanic that she trusted, and he only charged her $150 to glue it back on... such a nice guy. That was when I first learned about shady mechanics.
If you are to fix it yourself, get automotive glue designed for this purpose, and I haven't really looked at the rear view mirror mount, but if it comes apart, take it apart and glue the mount on then put the mirror back on when it is dry the next day.