Actually I ran some tests comparing turn time hosting with graphics in test mode.
Athlon 1.8G 1G Ram running Linux
Early Era, all nations Large random map.
By turn 10-12, regular turn generation time was up around 1 minute.
I copied the save and generated the turn 5 times using the GUI and 5 times in text mode. The text mode had a slight advantage, but nothing significant.
Now if your video card is already thrashing just drawing the background animation, then that may bring the whole system to a crawl, but it doesn't look like there's anything specific to updating the graphics slowing turns down.
One unanswered question here, I will answer:
If you press the "f" key (for "fast-forward", natch) during battle, everything moves a lot faster. It's nice to speed up all those arcing projectiles and spells.