7pm in Tokyo - Server Fun

POSTED BY DANNY CHOO On Mon 2009/03/02 15:47 JST in Japan
Sorry folks for server going MIA. Just one line of code was enough to bring down the server - spent pretty much all day trying to figure out the problem. Initially thought it was hackers and was looking for security holes.
I launched the Japanese News Items functionality yesterday and when doing so I wrote a line of code that should not be there which caused some news items to go missing which then sent the server bonkers...
Everything was fine after uploading that file and doing QA but after the usual wave of user activity, things started to break which was something I could have not envisioned. Need a service to simulate user tidal waves.
This was the cause of some of your news items going missing too.
Am working as hard as I can on making the dannychoo.com service faster and have a goal of getting the top page to load within 3 seconds - recently got some some new hardware to help me do that.
For those who are wondering, dannychoo.com is not hosted on my Mac Mini's! only some of the static images are hosted here. At the moment images are being served from img.dannychoo.com:3331 but will be on img.dannychoo.com tomorrow - I know some of your firewall settings at home or work may block port 3331.
For the tech savvy, use the following command in the terminal to see where http is getting stuck - this command saved my booty.
for i in `ps -elf |grep http|awk '{print 4}'|sort|uniq\`; do ls -la /proc/i/cwd ; done|awk '{print $11}'|sort|uniq -c |sort -nr
Would like to apologize once again and thank you for all your support as always - love you all! *tears*