IS NULL works in domain aggregate function WHERE argument, no matter where the function is used. The IS NULL is within quotes and processed by the WHERE argument which follows SQL syntax.
Note use of continuation character (_) and & use when breaking a string. Ampersand could be at beginning of following line if you prefer. Be sure not to lose spaces when breaking strings.
Code:
ChkoutCopies = DCount("*", "tblBorrowedBooks", _
"[fkBookId]= Forms!frmBorrowingBooks!cntBorrowedBooksSubForm.Form!txtfkBookId " & _
"AND DateReturned Is Null AND DateBorrowed Is Not Null")
If code is behind subform, instead of embedding full control path, could concatenate.
Code:
ChkoutCopies = DCount("*", "tblBorrowedBooks", _
"[fkBookId]= " & Me.txtfkBookId & " AND DateReturned Is Null AND DateBorrowed Is Not Null")