OK, I know your not made of money, but from what you said…
“…I figure it has to be either the motherboard, cpu or RAM”
Wouldn't it be cheaper to run some tests to see which is the problem and FIX the problem? RAM is a simple test, same for CPU, board..little more time but possibly worth it. Question, is your CPU at the max for that board or can you still go higher? How much RAM are you running now? Cost-wise I would go for the fix. But if your just in the mood to build a system then remember, you get what you pay for and can very well end up right where you are now. So here is some advice for the system your building (keeping in mind you DO know computers)…
Memory – Go with the Corsair, it works better than the Kingston (better, high quality chips), they are better on their RMA's (should something go wrong with it) and it's cheaper suprisingly! I tried dealing with Kingston before and had a hell of a time with them. Slow _ RMA's. And I haven't had too much luck with ther RAM holding up to any major abuse. Guess it's all on how you will be using your system. FYI – $160 get's you the same RAM as Fubar and I and thats next day delivery with tax from Frys.com.
Hard drive – Why an 80 gig? Bag your lunch for a couple weeks and get a 320gig for $100. You won't have to add another hard drive later or deal with not having enough space. Plus all the “high speed” drives or “RAID” labeled drives aren't worth the money unless your screen name is FATAL1TY! (look it up if you don't understand).
DVD – Scrap the Sony and get an HP DVD80i like I got for $40 (it's black, only problem…maybe). It's a dual layer DVD RAM drive that also has Lighscribe should you want to use it. Save a couple bucks here and put it towards your graphics card. HP's don;t have trouble like other drives writing to different brand medias. They pretty much will write/read anything.
Graphics – BFG, and go with the 7600 GT OC or the 7900 GS OC. They both have mail in rebates right now which drop the price by $30 or $50. I have used their RMA and they are fast. Plus they will ship you an RMA card without you sending yours in first. Takes about 3 days to get your replacment! FYI, they are about $95 -$170.
Bottom line, get something that is going to last and hold up with a decent warranty and tech support!
As for your case, just spend the $50 and get another one. If you decide later to find out what was wrong with your old system at least you'll be able to use it. Cases arent expensive anymore. You can get a decent tower cheap.
FYI – from shopping around, I built my system for $2,100 and thats with shipping and tax. Take out the little extra stuff (lcd screen wire wrapping, cables) and it was done for about $1,750. Let's see DELL do that!
Good luck on your build! Let me know if you need any help!