I've noticed the same thing. When I want the task manager under normal operation, it works fine. But if there's a hung program, it sometimes takes up to a full minute to come up or doesn't come up at all.
The solution I've found that works for me is to right-click an empty area of the Taskbar and pick Task Manager from the menu. That seems to do the trick 95% of the time. This trick works in all version of Windows, btw.
-JG