As backup devices, USB flash drives and hard disks are by no means perfect; but they give the most bang for the buck today, and they're fast, too, which is why you should be using them regularly.
Just a small tidbit when you're backing up: if you're backing up a large file, say, an ISO, to a FAT/VFAT-formatted USB drive (which is most likely the case) be aware that there is a 4GB file size limit. This is a limitation of the FAT file system. Thus, you'll need to break you up into smaller chunks.
Neither ZIP nor TAR.GZ or any of the other default compression formats on Ubuntu can do this, however. You'll need to install p7zip-full or rar. The Create Archive... dialog that appears when you right click on a file should recognize these formats once they're installed.