You say textbox is txnsearch but then your code uses txtsearch and declares txnsearch as a variable.
The txnsearch variable can never be Null. Only Variant type variable can hold Null. Code will have runtime error if the DLookup returns Null. But if txtsearch is null, then certainly the DLookup will return Null.
Don't put reference to textbox within quote marks.
Code:
Private Sub Command5_Click()
If IsNull(DLookup("Txn_number", "tbl_fxmain", "Txn_number= " & Me.txtsearch)) Then
MsgBox (" Please enter Transaction Number "), vbOKOnly
Me.txtsearch.SetFocus
Else
DoCmd.OpenForm "frm_user_edit", , , "[Txn_number]=" & Me.txtsearch
End If
End Sub
However, use a LimitToList combobox instead of textbox and users can only enter valid txn number and DLookup would not be necessary.
Suggest you give button more meaningful name than Command5.