To do it without vba, it would ahve to use the INDIRECT function.
Whereas = A2 takes the contents of A2, =INDIRECT(A2) takes the contents of the cell or range referred to in cell A2. So you would ahve to build of string of text that represents the filename, sheet reference and cell reference. So in your example:
A B
1 1002 =INDIRECT("'C:\Business\[invoice " & A1 & ".xls]Sheet1'!$B$4")
2 1003 =INDIRECT("'C:\Business\[invoice " & A2 & ".xls]Sheet1'!$B$4")
In this example the reference to A1 and A2 will change as the formula is copied down but the reference to B4 will not, even if you take away the $ signs, because it is within the quotes so recognised as part of the text. You can split each component of the string to a variable in a referenced cell if so required, so long as the whole syntax generates a valid cell reference. Try it without the INDIRECT first and you will then see the cell reference that the formula is generating, then enclose it within the INDIRECT brackets.
As mentioned earlier, INDIRECT linking to another workbook only works while the source workbook is open, once the source workbook is closed the result stays valid until an event forces calculation in the destination workbook. At that point it will turn to an error.
The result of your attempts was trying to look for a file called "Invoice A3.xls" which it was unable to find so gave you the dialog box to give the option of finding it manually.
Cheers
Rob H