How do I remove the RSS feed (the one on the bottom right)?
Try going to your admin control panel
Home -> Administration Center -> News and Newsletters -> Settings
Uncheck enable rss/xml feeds
I couldn't find the News and Newsletters?
It's there, Admin > Main > News and newsletters > Settings
Or use this direct link
Domain.com/index.php?action=admin;area=news;sa=settings
Edit
Just a little trick: you have a search box in Admin panel where you can type and search for a setting, in this case type RSS and it will bring you directly to the page where you can enable or disable the rss feeds.
Will it improve server performance (use less CPU) if I disable the rss/xml feeds?
Probably very little, unless there are many subscribers. In that case you may want to keep it though?
I'd say almost nothing.