My experience has been that with the window out, it seems more 'windy' on normal roads/speeds, but less 'windy' on the expressway/higher speeds. I do like the look better without it.
Overall it wasn't worth taking in and out, so I just leave it in (Michigan weather).