No you don't have to but I do anyway since it is so easy. It really helps when you get to the back spark plugs and the back tubes on the headers. That is always the first thing I remove.
By leaving the driver side loose you went up even with the motor, it actually helps to pull the motor sideways to get the header out and back in. Good luck