Don't bother maxing out the RAM, but you will need a better processor for encoding. The new Mac Mini will work just fine for what you want, but it's going to cost you an arm and a leg.
I built a PC to replace our Mac Pro, and while setting up and configuring things, plus a few hours of troubleshooting, was a real pain, it has now worked flawlessly for about a year and a half. I've been in this exact same situation (multi-camera stream of a church service).
