Some other program may be accessing your drive, e.g, an indexing program, an automatic defrag program.
Removing the USB drive will usually not cause problems in XP (I don't know about Vista). But if you are using Windows 2000, you may lose data. My understanding is that Windows 2K writes the data to the drive when you click 'Safely Remove', but Win XP writes the data immediately.
You could test this by writing a small text file to the drive, then remove it without using SafelyRemove. If the file is on your drive when you re-insert it, it is safe to remove it, although if there is another program indexing it, you may get some error messages. If you lose the file, you may have to reboot to be able to remove it.