Bought some awesome 1mF capacitors. Time to test them out.
Pressing the button switches on the power and lights the led. The capacitors are also charged. Release the button to cut primary power, the capacitors will slowly empty and the LED will fade. Vary the size of the capacitor to increase/decrease the stored charge and time delay.
See also: Make Presents: The Capacitor