When you come to set a reference, you will get an enormous list of possible libraries and dlls. Some will already be referenced (ticked). There is no point adding references until needed.
The references only apply to a particular project (e.g. a particular Excel workbook, Word document or template, Access database), so if you acquire someone else's working document, the appropriate reference will already be set in it.