To delete a Mail Collector or Connection in version 3, click on the little waste-bin icon at the left of the green bar on the Mail Collector or Connection settings page.

We have made this more obvious in version 4, by including a bigger “Delete” button in the top-right.