It can be done without GUI however (and Ive fought this long and hard) it does require that some graphic libraries be loaded. Basically, install it and try it in text mode but you might have to have xwindows loaded even if you dont use it. I havent tested that lately but thats how it was with Dom2.
I host ALL games in text mode. Its one of the things I love about Dominions. The simplest way is to type
dom3 -ST
which tells dom to run server mode, text mode. It will prompt you with questions about port number, era, map, game name, etc etc. Then you can work up to running in in nice nohup background mode with all those answers given in a script.
Here is a link to a thread on the subject (Im afraid it starts out too involved then gets simpler at the end)
Linux Admining 
Newbie Question re: Hosting
Linear Mode
