You can use the properties which count and reference the selected items in the listbox control. I put this code in the On_Click event of a listbox, and it does just what you are looking for, including un-selecting the item which was just selected:
Code:
Private Sub List0_Click()
Dim SelectedItem As Integer
If List0.ItemsSelected.Count > 3 Then
MsgBox "you cannot select more than three items"
'
' reverse the selection
'
SelectedItem = List0.ListIndex
List0.Selected(SelectedItem) = False
End If
End Sub
Change List0 to the name of your listbox, change the message to what you need, and it should be fine.