Open regedit, navigate to HKEY_CLASSES_ROOT\HTTP\DefaultIcon.
In the right pane you'll see
(Default) Reg_EXP_SZ %SystemRoot%\system32\url.dll,0
doubleclick on (Default)
in the window that comes up type the path to the icon you want to use
D:\icons\coolfavicon.ico for example
click OK and close regedit.
You may have to refresh your icon cache. Either using TweakUI or in display properties-appearence-advanced change the icon size to 33 pixel, click ok and apply, go back and change it to 32 pixel again.