あな醜賢しらをすと酒飲まぬ人をよく見ば猿にかも似む

oh, how frightful!
they put on airs
and do not drink
look closely and
they seem like monkeys

I thought I was misunderstanding this at first, but I was forced to conclude that yes, in this poem, it is the sober people who are being compared to animals.

(I dug this out of the Manyoushuu Search Engine I linked to earlier in the week.)