I have a server which runs this site, and a few others. It was working just fine, but since I live in a household with multiple LANs and network connections, I wanted the server to be multi-homed so that it could be used to share to DNLA devices and other computers.
I picked up a PCIe 10/100/1000 network card for it, and installed it. The server runs Linux (of course), so the nic was ready to go as soon as the machine was turned on. I configured the router on the new nic's network for static dhcp, just like my own, but on a different subnet. I put the server in a "closet", which is actually a space under the stairs which has been fully wired for this purpose. If you're curious, the server is a Dell Poweredge 1900, and I have the front facing the rear so that the fans are blowing out in to the open air. To the left of the server you can see the power outlets (still have to add a UPS), light switch, and phone line. Directly below the power outlets is my modem (black) and to the right of that is my phone ATA (silver and white). Mounted on the vertical beam, if you couldn't guess, is my wireless router. My office is on the other side of the insulated wall, so the signal is excellent.
Once I got it all set up, I tested the website, and it seemed fine. I then tried it through a proxy, and ... nothing. It wasn't serving pages! I cursed to myself, and looked over apache's settings. I am using name-based hosting, so apache is listening on all ip addresses (but only port 80), as seen by the directive <VirtualHost *:80> in my sites' configuration files. I tried changing those apache config files to only serve pages on the original nic, but I still couldn't get pages outside of my LAN. I fiddled with my modem, which seemed to have a faulty firewall. I put my router in my modem's DMZ. I bridged my modem and authenticated with my router. I even spend a few hours trying to upgrade the firmware on my modem with BootP and TFTP, which I had to use because the standard upgrade requires Windows. Bah. I got so frustrated that I went to bed shortly after midnight, which for those who know me is exceptionally early.
The next day, my brother-in-law picked up a new modem for me during his lunch (thanks!!). I tried swapping out my router for another one that I had lying around to confirm that that wasn't the issue. I got too lazy to switch it back, and the slow internet speeds that I had got me nice and frustrated again. I decided that I would go out and purchase a new router (wireless N, 10/100/1000), new gigabit cable, and a new nic for my desktop. I got back, and with the help of my 4-year-old niece, got it configured properly. I checked my sites again, and ... ?! (interrobang standing in for expletives)
I was at a loss. I'm fairly experienced, but I couldn't figure out why apache wouldn't serve to the internet, but with a hosts file entry on LAN, pages appeared like normal. Off to IRC. If you don't know what IRC is, it's one of the oldest forms of internet communication, and as far as I know, the first that allowed real-time chat. It started in 1988. It is now used as a media for tech support communications. I joined ##networking on Freenode, and asked my question. I got a response from someone using "tripflex" as their nickname (mine is Seven_Six_Two), and we went over all of my settings. The last thing to look at was my kernel routing tables. AHA! Or Eureka! if that's how you roll. There seemed to be something wrong. Everything looked correct except that there were 2 entries for default routes, one for each subnet, when there can be only one. (Enter Connor; cue lightning!) It seems that when the network cards are set up using dhcp, they are automatically assigned default routes. Default routes are used to tell the router where to send packets for any address that isn't on the local network. With the way my routing table was configured, any requests for my website were being sent out on the highest-priority (lowest "Metric" value) default route, which at the time was the new nic's network. The wrong network.
The fix was much simpler than purchasing a new modem and router. I just needed to add an entry in to /etc/network/interfaces that set a static ip address and netmask for the new network card. Once I did that, and restarted my network, SHAZAM! (and a touch of Bippity Boppity Boo!) It started working! That you are reading this is proof, even though this site isn't pudding.
Thanks for reading, and feel free to discuss below!