I wanted to take a second to post about the issues that have been on the site recently. Let me summarize this post in just one sentence: All issues with the ads/images/database weirdness should be fixed.
So what was the actual issue?
We had an issue on our server where it was complaining about disk space. However, we were only using about 50% of the disk space so this required some investigation. After a few minutes, we discovered that we had actually ran out of inodes (which of course is on a different screen than the actual disk space). I'll spare the technical details but suffice it to say, we had a lot of very small files that were taking up inode entries. Remove files, inodes available, all is well again.
How did you not know?
We have a few key items that we are monitoring that I receive a direct email alarm if it happens. Things like the site going down, running out of disk space, database not responding, etc. Keeping track of inodes was not among that list. But it is something we know to look for in the future. The good part is that we learned something from this, and as long as we walk away with a little more knowledge then that helps everyone.
What was affected?
Anything that involved writing something to the disk was affected. The database was only partially affected. The biggest part that was affected was the view check and image uploading. So it is possible that views came in, and not properly checked or accounted for. Images could not be upload either. Even though we do not store images on our server directly, we have to put them there temporarily to send them elsewhere. Since that temporary write failed, the entire process fails.
Like I said before, so long as something was learned from the process then we can all gain from it. All should be back to normal now. And I'll be checking those inode reports more often now :)
Image Credit » Created and Owned by Persona Paper