The Anti-Defamation League has labeled Pepe the Frog as a hate symbol because white supremacists on the internet have made bigoted versions of Pepe... not as rare as they ought to be. From the ADL website:
In recent years, with the growth of the "alt right" segment of the white supremacist movement, a segment that draws some of its support from some of the above-mentioned Internet sites, the number of "alt right" Pepe memes has grown, a tendency exacerbated by the controversial and contentious 2016 presidential election. Though Pepe memes have many defenders, not least the character's creator, Matt Furie, who has called the alt right appropriation of the meme merely a "phase," the use of racist and bigoted versions of Pepe memes seems to be increasing, not decreasing.
via Anti-Defamation League