As usual - I failed to write this down when it happened before, and every time I encounter this issue I fail to remember how to solve it.
The issue - you install a new sharepoint machine and create a web application with a custom host header and then try to browse to it from the server itself using the host header (which is correctly configured in the DNS and host files and all) and you get prompted for a username\password three times and then get thrown out.
This happened with sharepoint 2003 as well, and with windows 2003 server and has the same reason behind it - a loopback check on the server. In 2008 however it threw me several times when I created a web application, and browsed to it and it worked and then I had to recreate it and it suddenly stopped working from the server, while other machines could browse to it with no issues. arrrggghhh!
So, if you have a web application with a custom host header, and if you browse to it from a client machine you can view the sites, but from the server you get prompted for password and nothing you enter works...try the solutions described in this KB article (KB896861). I know the title refers only to IIS 6, and I know the article mentions windows XP and windows 2003 - trust me, it solves this issue for windows 2008 just as well.