Is it a federal offense to throw away someone else's mail?

Yes, it is against Federal Law to dispose of someone else's mail. The postal inspector is correct in telling you to write "return to sender" or "wrong address" on the mail.