Ah, no encrypted DMG support -- that's the answer, thanks! I must have missed that when perusing their FAQ.
Unfortunately that makes it pretty useless for almost any DMG you download. I was able to get it to mount a homemade DMG, though. Maybe I'll revert back to MD7 and the Windows 7 workaround for now. I'll also have to ask Mediafour why they removed the feature; wonder if it's a licensing issue.
Anyhow, problem solved! Thanks again for saving me the trouble of going through another fresh install to rule out VMware as the culprit.