jsd me -d $HOME/yourgraphicsdir -p xv[If you don't have a parent jsd in daemon mode, leave out the "me". See the manpage.]
It's got everything my roommate asked for, and more. :)
Note: this needs a fairly recent joystick driver, included in linux 2.2.x.
Typical usage, in your rc.local:
jsd -d /mp3_dir -p mp3_player
jsd needs read access to your joystick device (ie. /dev/js0). To use the command unix-domain socket ability, jsd needs to be able to open a socket in /var/run. You can run it as a non-root user if that user has access to your joystick device (ie. /dev/js0), and if you don't care about the command socket. If you start jsd as root, it will open the /dev/js0 and the command socket as root, and then drop its perimissions to either the joystick user (if such a user exists) or the nobody user. If you really want it to retain root, you can use the -r option.
The permissions on the command socket are 622, unless a joystick group exists or the -g option is used, in which case permissions are 620.
jsd does use any temp files, create any output files, or open any internet-domain sockets.