I can't really say as I only use modul8 on a MacBook Pro 2.4 Ghz with 4 GB RAM. When I use three outputs its through a triple head and for what I do it performs fine. Then again, what do you want to do? Have loads of full res layers of video playing with freeframe effects? Or play minimal style with sound reactive stuff? Do you need to play at full res?
The only thing I can say from your specs is get more RAM