In order to do this, you'd need to disable the default IPSec policy,
See the "Disable or Enable the Built-in IPSec Policy" section in this article:
You'd then need to make a policy from the specific IPs you want to be able to contact the firewall via IPSec, and make that policy to firebox.
*Note that this policy also governs site to site VPNs, and IPSec (IKEv1) mobile VPN, so you will need to account for this in your policies.
WatchGuard Customer Support