System Center Operations Manager 2007 R2 uses mutual authentication to communicate with agents. This can be done using Kerberos v5 or certificates. In case of same domain as that of SCOM server or if the two domains have a two way forest trust we can use Kerberos. But if you want to monitor machines in workgroup or in non-trusted/one way trusted/*external trusted domain we need certificates because Certificates will allow mutual authentication.
*External trust uses NTLM authentication.
If SCOM server is in domain A and you want to monitor machines in untrusted domain B, use certificates along with a gateway server. But this time you don’t need to install certificates on all machines in domain B. Simply install Gateway Server in domain B and have certificates installed on SCOM management server of domain A and Gateway Server of domain B. Within Domain B, Kerberos is security mechanism between agents and Gateway server and certificates are used for mutual authentication between the Gateway server and SCOM server. See Diagram below
Attached is a link to an install guide which I have composed that will walk you through the installation steps.
NOTE:-It assumes that you have got a Root Certificate Authority installed in your environment