Those lists are compiled from what is in the HKEY_CLASSES_ROOT section of the registry (you can view this by running regedit.exe), specifically, what is in the shell folder of the file type you are right clicking on. For example, if you right click on a folder, the Folder key under HKEY_CLASSES_ROOT is one of the places the program looks to find out what to put in the context menu. I think * is another key that is consulted. If you look at the shell folder of these keys, you will see the items on the context menu.
I would assume that Windows Explorer will be faster than the other file managers that you are using, because the other file managers have to indirectly use the processes that Windows Explorer uses directly to make the context menu.
As to what is slowing it down, I don't know. Maybe you have a lot of items in the HKEY_CLASSES_ROOT section of the registry (I have a lot of items but mine is not slow), maybe there is something messed up in the shell folders, maybe you installed a program which is putting itself on the context menu but doing so in a slow way, or maybe the registry is corrupt in that part of the registry.