Rare souls will give you rarer characters, and are guaranteed only to be at least a 3* toon. There is no guarantee you'll see the latest Eddie from one of these souls, although many times they do become available through rare souls. You may also get 4 and 5 star characters from these souls, but at a lower chance than getting 3*.
When you open them, red eyes means you got something above the base value of 3*. This can lead to excitement, often followed by disappointment when it's only a 4*. Keep in mind, even 3* characters can lead to some decent sacrifices.
Gathering these is more difficult than common souls, but the rewards are much higher. You will get some from Skull Quests, LoL, Missions, Events, opening common souls (via fragments), and Sacrifice. While not usually thought of as a source for more souls, sacrifice typically drops higher level souls in greater numbers than any other place in the game.
Rare souls come in several variations, so we'll cover them all.