The nue is one of the more bizarre Japanese mythological creatures. It is a chimera that makes no sense at all: monkey's head, tanuki's body, tiger's legs, snake for a tail. Oh, and the voice of a nue. Before you say "Well, duh," though, let me point out that what was meant by this was "voice of the bird known as nue since ancient times in Japan, probably White's thrush."

This is already getting confusing. Let's rewind. The nue appears in the Kojiki, specifically in the courting song of Ōkuninushi (a.k.a. Yachihoko no kami):

... 我が立たせれば　青山に　鵼は鳴きぬ　さ野つ鳥　雉はとよむ　庭つ鳥　鶏は鳴く　心痛くも　鳴くなる鳥か　この鳥も　打ち止めこせね　いしたふや　天馳使　事の　語言も　是をば

... While I am standing [here], the nuye sings upon the green mountain, and [the voice of] the true bird of the moor, the pheasant, resounds; the bird of the yard, the cock, crows. Oh! the pity that [the] birds should sing! Oh! these birds! Would that I could beat them till they were sick! Oh! swiftly-flying heaven-racing messenger, the tradition of the thing, too, this!" (Basil Hall Chamberlain's 1919 translation)

Here, the nue (nuye being the old form) is just a regular bird like any other, representing for the mountains. But when we encounter the same bird in the Man'yōshū (example), it's associated with melancholy and grief. Nuedori, literally "nue bird," is actually a pillow word for cheerful concepts such as unrequited love and weeping in one's soul (uranakeru).

By the Heian period, the voice of the nue had become an omen of ill import, the sort of thing you needed to hire a consultant to deal with. ("Which is the most auspicious direction in which my retinue and I can flee?") Things got even worse for the poor nue when someone noticed that it was described in Chinese texts as a 怪鳥, literally "monstrous/uncanny/eerie bird," although this really just meant "nocturnal bird" in Chinese.

So, by the Kamakura period when the monster-nue made its first appearance in the Heike monogatari, it totally makes sense for it to have the voice of a bird which is by now associated with supernatural doom and woe.

The remaining mystery is how that monster came to be called a nue too. It is not actually given this name in the Heike monogatari. It isn't given a name at all. A. L. Sadler's 1928 English translation, the one now available from Tuttle, includes the sentence "The Nue they put into a boat and set it adrift," but the original for this is "Sate kono henge no mono o ba, utsuhobune ni irete nagasarekeru to zo kikoeshi", i.e., "They put the monster into a boat [made of a hollowed-out log] and set it adrift" + "lo have I heard" storytelling cruft. Helen Craig McCullough's 1988 translation for Stanford, incidentally, renders the same sentence as "The monster was put in a dugout and shipped downstream."

Now, the text does follow the monster story up with another nue-killing anecdote, and in this second story the word nue is used to describe the victim — but this nue is specifically described as a 怪鳥, a bird-monster, and not the weird chimera of the first story which has become the standard nue-monster of today.

So, if the monster was originally nameless, at what point did people start calling it a "nue"? Yamaguchi Nakami 山口仲美, from whose most readable book on birdsong onomatopoeia through the ages, Chin-chin chidori no naku koe wa (ちんちん千鳥のなく声は, "Chin chin, the plover's cry" — it's a song), I obtained most of the above, believes that it happened quite early in the monster's career. She observes, for example, that in the Genpei Jōsuiki (a sort of director's cut of the Heike monogatari), the name had already stuck:

大国の養由は、雲上の雁を落し、我朝の頼政は深夜の鵺を射る
In China, Yang Youji shot a goose down from above the clouds; in our land, Minamoto no Yorimasa shoots a nue in the middle of the night.

And why did this happen? Looks like the author merged the chimera-nue and the monster-bird-nue anecdote mentioned above, creating a super-story featuring a super-chimera with the body of a chimera and the name of a nue. Yamaguchi also suggests that the monster was dubbed "nue" because, well, what else are you going to call it? Everyone knows what a monkey or a tiger is. Not everybody knows what a nue is, except that it's scary and flies.

Anyway, the result is that today, a nue is a monster, with the bird a distant, oft-forgotten secondary meaning. There's even a derivative word, nueteki ("nueic"), used to describe things that are mysterious or unknowable.