I only have the soft top. Admittedly it ain't fun taking it on or off but it's not that bad.
It takes 5 seconds to unclip the rear, it takes about ten seconds to unhinge the front levers, 5 seconds to get out of the car and another 5 seconds to lift the roof off the car. Now you've spent 25 seconds to get the roof off the car. If you're at home when you do this and don't want to bring the roof with you (a choice you don't have with the hard top), then you're done. That is usually what I do, I've only folded mine up maybe two times to put it in the trunk. I have NOS in the trunk and still have room for the roof when folded so space isn't a problem.
Now if you really want to have it with you, it takes about 30 seconds per side to unscrew (at least on the old style like mine) the pins. Another 10 seconds to fold in the supports and maybe another 30 seconds (if you're anal) to roll the pieces into a tube. Then let's say another 20 seconds to open the trunk, prop up the support, grab the roof and lie it in the trunk. Another 8 seconds to relock the trunk support and close it. Let's see, that puts you at 2 minutes and 8 seconds to get it in the trunk.
Reassembly is just everything in reverse and takes the same amount of time.
The problem is, and I'm sure this is what your friend is referring to when he says its a pain, it's 2 minutes and 33 seconds in total that you could be driving instead!!