Building the dedicated server uses files such as code/qcommon/unzip.c, code/null/null_client.c and some others, and these files include client.h
This is a problem because it means that building the dedicated server requires the speex header files, despite the fact that as far as I can tell the server doesn't require any of the libraries or codecs.
In most cases, these includes just need one or two of the header files included in client.h, rather than that file itself - in some cases, the header files aren't needed at all. I'm going to attach a patch that sorts these issues in a second.

It was difficult for me to test this properly since I can't get voip working *anyway* (probably my hardware or openal or something), but I don't see how it could break anything.