Did some quick search on Bugzilla and came up with the following:
Mozilla allows external protocols as discussed in:
http://bugzilla.mozilla.org/show_bug.cgi?id=167475
They seem to blacklist the following external protocol handlers:
(patch http://bugzilla.mozilla.org/attachment.cgi?id=102263&action=view)
hcp, vbscript, javascript, ms-help, vnd.ms.radio
A simple solution would be to add the shell protocol to this list.
Personally I think a secure blacklist is hard to maintain as new
dangerous external protocols could be invented by third-parties leaving
Mozilla vulnerable again.