I would only remove the ones that you know you will never need.  That will save you hassles later.  I don't think that all of the keyboard layouts are loaded at startup.  If they are, it only takes a split second, so removing them will not noticeably make Windows load faster.  I would more remove them for the space they take up in the registry, but then again that is still minimal.
   Reinstalling Windows should get them back.  Or Microsoft has the Microsoft Keyboard Layout Creator that may help recreate some.  Or the easiest thing would be to keep a backup of what you removed and restore it if you need deleted keyboard layouts.

When I first installed I went crazy downloading and installing all the pretty fonts, themes, and other extras unnecessary to my composition of poems or the research that helps put paper and bread on the table. So  by ridding the system of all but the few, three, themes that I really like, the fonts that I don't need (you made that easy), and all of the other things that I don't use often enough. I figure that if I don't use something at least once a week, I don't need it. The keyboards may be a small difference, but added to all the other small things may be significant. For example I had some of the small programs from Gibson Research, like Shoot The Messenger, SocketLock, UnPlug n' Pray, etc. I also had Windows Worms Doors Cleaner which covered the same areas. I deleted all of the Gibson ones except for IDServe and SocketLock which are unique on my system. I want to strip XP down to what I actually need to run the system, protect it, and run the programs that I need to do my work and play. I removed all the games except Solitaire and Pinball; I am not all serious no play. I set the system for Best Performance. I don't need or want eye candy. So each little deletion will add up in the end.