[Full-disclosure] High performance exception/traceback reporting system

["Cal Leeming [Simplicity Media Ltd]" <cal.leeming@xxxxxxxxxxxxxxxxxxxxxxxx>]

Hey all,

For the last two years, I've been meaning to write a reporting server which
allows webapps to post their exception tracebacks, which are then viewable
from a centralized location. After having Thunderbird corrupt my mailbox due
to over 250 thousand debug emails, this project has now been given a bit
more priority ;)

The current prototype stores basic exception information (the file path,
line number, exception type, exception value, originating webapp, node
hostname etc) in the database, and the traceback details are then
serialized, dumped into a file, and the path to that file stored against the
row. A web interface then allows you to browse through these exceptions
(currently via Django admin), and view them using the same prettified
exception page which it shows for actual exceptions. This prettified page
also shows the variables within each frame in the stack, which is very
handy!

>From a developers point of view, this makes life extremely easy, because all
your webapps report to a single place, you can do sphinx searches, alerts,
custom reports etc, and it looks pretty lol.

The entire thing is going to be open source, and will eventually be a
one-click install with a set up page etc.

Here are some of the features I am planning on adding, but if anyone has any
suggestions as to what they would like to see in this, please feel free to
mention them!

   - Tracebacks can be sent to the server primarily via POST request, but
   custom plugins will allow it to pull in via other means (such as mail
   attachments)
   - Alerts can be given different classifications (for example, you could
   configure specific nodes, webapps, or exception types to alert you via
   BulkSMS)
   - Prettified traceback page should initially support Python/PHP, other
   languages can be added as and when.
   - Basic authentication / IP restrictions for the admin login
   - Authentication support for when the tracebacks are POST'd to the server
   - Tar source should pre-package a lightweight nginx/uwsgi/python
   environment, so it is self sufficient (this will need to be security
   maintained obviously).
   - A nice, pretty, easy to use interface, because this just makes people
   feel all nice and warm inside ^_^

I don't want to go as far as to say that it should be used to collect
error_log outputs, I think that would be going a bit too far, the main
reason for having a system like this is simply due to the sheer amount of
information usually contained within a traceback dump, and the Django
prettifier makes it so much easier to debug with!

Thoughts/criticisms welcome!

Cal

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/