When you use a form as a subform, it does not become a part of the Forms! collection, so in the macro expression:
"[CustomerDetail]![CustomerDetailID]=[TransactionDetail]![Cust_DetailID]",
[CustomerDetail]! does not refer to an open form (it's a subform) so you get prompted to enter for a value.
I hate macros with a passion (!) so in the CustomerDetail Subform I replaced the macro with an Event procedure:
Code:
Private Sub cmdTransction_Click()
DoCmd.OpenForm "transactionDetail", , , "Cust_DetailID = " & Me!CustomerDetailID
End Sub
and it worked fine.
Notice the use of Me! in the DoCmd - Me! will always refer to the form where it is used, even when it is a subform.