I don't know much about debian, but dmd itself doesn't need X, nor should phobos. I compile and run D programs on headless web servers every day. If you can't ignore the dependencies with the apt, downloading the dmd zip file should be just about as easy and it doesn't pull anything else with it.