Fear was cast on an attackable NPC to allow a player to avoid combat with that NPC. Fear had two entirely different effects, depending on the state of the target NPC:
NPC in combat
When Fear was cast on an NPC engaged in combat with the casting player, the NPC would retreat from combat, and stop being aggressive toward the casting player. This was often used as a move of desperation for players overwhelmed by the target NPC.
NPC not in combat
When Fear was cast on NPCs not engaged in combat, the spell would make the NPC "disappear." Essentially, the NPC would appear to log off, just as a player would. No bones would be left behind, and minutes later, the NPC would respawn as normal. While it is unknown whether this effect was a bug or not, many players used the spell as a joke on other players who were trying to train combat, casting fear on others' training targets. Other players used it to steal kills, much to the annoyance of others.