We had in mind BackupExec System Recovery 2010 for SBS. Can it be used in this situation ?
1) Must the BESR be installed on a Windows Server not running on the ESXi box in order to be able to backup the VM's, or can it do it from within the VM ?
2) Can the tape backup be used if connected directly to the ESXi server ?
You can install BESR on the guest OS VM. You can point the image to a different VM over the network, will work just fine.
For the tape options I normally prefer to do it on a physical server although I have seen customers do a passthrough and connect a tape drive directly to a VM. The performance is much less then a regular tape drive.
Another solution may be to use a simple workstation (physical) and large sata drive with a network share. Connect the VS160 to the workstation to the workstation and then you can use ntbackup to backup the images as flat files.
If a restore is needed the nearline images are on a share and you can then use the tape for offsite backups.
To restore a BESR image you would use VMware vConverter vs Symantec process, it is faster and much easier to use and know about BESR images natively.
Hope this helps