Although FileMaker recommends a dedicated server to run their product, buying a second server is completely unnecessary if you're just serving FileMaker databases on port 5003, and not using any of the Web Publishing features. Nonetheless, FileMaker gives you no way to disable its use of ports 80 and 443, and in fact, the FileMaker Server installer will not even run without you allowing it to first disable other services on these ports.
These kinds of programming decisions are puzzling to me, but luckily, there is a way to forcibly change the web ports for FileMaker Server, and in such a way that you can still serve your FileMaker databases on port 5003, while leaving ports 80 and 443 free to be used by other web and secure web-based protocols and services.
As always, be sure to backup your server first! Then, all you need to do is take a look at the following two files on your FM Server:
And then open them up in your favorite editor (I personally love TextWrangler), and you will see a line in each that lists the port that the service listens on. I changed "80" to "81" (in the first file) and "443" to "444" (in the second file). I restarted my server, and everything worked fine, both in FileMaker server, and in the other server apps that wanted to use ports 80 and 443.
Be sure to keep backups of the files that you modify, as I would guess that when performing certain FileMaker Server admin functions, or when updating the FM Server version, the customized files you create will get overwritten. And obviously, this is not a supported FileMaker workaround, so your milage may vary. That said, I hope this is helpful!