Today, quite by accident, I discovered a plugin for Rhythmbox called RhythmArty: it lets you browse music by album art and it looks really cool!
You can download it from the RhythmArty page at SourceForge. Download it and use your package installer to unzip and install it to Rhythmbox.
Once installed, (in Rhythmbox) go to Edit > Plugins and check the RhythmArty check box.
It may take a while to download the album art, but it's worth the wait. You can toggle the browser window from the new toolbar button.