Automating Let’s Encrypt certificates lifecycle for Unified Access Gateway’s Horizon and Workspace ONE services
The UAG is very important part of Workspace ONE and Horizon platforms, delivering secure access to services like virtual apps and desktops (including Horizon 8 and Horizon Cloud on Azure), Content Gateway, Tunnel and Secure Email Gateway.
If you deploy Unified Access Gateway in production, you probably put it behind a virtual IP (VIP) that will load balance the service, offload SSL and bridge with UAG. There are several use cases (PoC, Pilot, Lab, Demo, UAT etc.) where you will have your UAG exposed directly to the Internet. There are also scenarios where you just don’t want to or can’t offload SSL on the load balancer and the traffic is passed unaltered to the UAGs – client certificate authentication is one of this scenarios as it can’t work across the SSL bridge.
Production or not, you still need a trusted, public SSL certificate on the front. In many cases it is convenient to use a Wildcard or UCC certificate with many subject alternate names – and this is very expensive. Last year I’ve paid around 300 dollars for such certificate. This year when it came to renewal, I’ve decided there are better ways to spend this money, especially when you can have it for free!
Introducing Let’s Encrypt and ACME protocol for certificate lifecycle management
Let’s Encrypt is a free, automated, and open certificate authority (CA) provided by the non-profit Internet Security Research Group (ISRG). In a very quick summary, Let’s Encrypt provides the following:
– Single hostname, SAN with up to 100 hostnames and Wildcard certificates
– Certificates trusted by almost all modern browsers, operating systems and applications thanks to cross-signature from IdenTrust
– Certificates valid for 90 days
– Based on Automated Certificate Management Protocol for requests and renewals
– Automated installation of requested certificates in operating systems, web servers, load balancers and other services
– Possibility to save all the flavors of certificate formats as files (PEM, PFX etc)
– Has clients for Windows (I recommend Posh-ACME) and Linux/UNIX (for instance Certbot)
– Provides secure and automated domain ownership verification by HTTP-01 or DNS-01 challenge – easy to integrate with your DNS provider REST API.
– Certificate expiration notices over email and automatic renewals with scripts
Logical workflow of the automation
The scripts presented below are intended to automate the request, install/replace and renewal operations of the certificates on Unified Access Gateways and Horizon Connection Servers. They are designed to run on a Connection Server. In my example they request a wildcard certificate for the domain.com, perform a DNS-01 challenge against GoDaddy DNS server with REST API, put the certificate on the UAG through the REST API interface on port 9443, install the certificate in the Local Machine store on the Connection Server, replace add the ‘vdm’ Friendly Name (during renewals they rename the old certificate’s Friendly Name) and restarts the Horizon Connection Server service to pick up new cert.
At the moment, the scripts work only against Connection Servers they are run from. It is possible to install the certificate on additional Connection Servers remotely with some modifications to the code. Here is a code snippet to give you an idea on how to perform it. It still requires a mechanism to put a PFX file on the file share.
$conservers= connserver2.domain.com, connserver3.domain.com
ForEach($conserverin$conservers){
Enter-PSSession-ComputerName$conserver
#Commands below this point will execute remotely
Import-PFXCertificate-CertStoreLocationCert:\LocalMachine\My-FilePath\\server\path\to\cert.pfx
Exit-PSSession
}
You can use the above approach with ForEach loop also to make the script work with multiple UAGs.
Prerequisites
Install Posh-ACME from the PowerShell gallery:Install-Module -Name Posh-ACME
Generate GoDaddy API Key and Secret:
1. Login to https://developer.godaddy.com/keys
2. Click “Create New API Key”
3. Change the envrionment to “Production”
4. Save the Key and Secret immediatly (Secret will not be displayed again once you close this window).
Create a dedicated admin account for REST API calls on UAG.
Request.ps1
This script is intended to run manually once on the Connection Server during first request of the certificate. Redirect the output to a log file for reference and potential troubleshooting:
.\MyScript.ps1 > certreq.log
<#
.DESCRIPTION
Script requests certificates from Letsencrypt with automatic DNS-01 challange and installs them on UAG and Connection Server
.NOTES
Author: Roch Norwa <a href="https://digitalworkspace.blog/">https://digitalworkspace.blog</a> @rochnorwa <a href="https://linkedin.com/rochnorwa">https://linkedin.com/rochnorwa</a>
#>
#go back to C:\ just in case the location i set to cert:
Set-LocationC:
#information to be used in the cert request. use comma seperated domains for UCC/SAN certificate.
$domain= "*.domain.com","domain.com"
$uag_host= "uag.domain.com"
# information for letsncrypt POSH-ACME powershell script
$psMod= "Posh-ACME"
$dnsPlugin= "GoDaddy"
$pArgs= @{GDKey="9ZtodG7jWNC_84teqTR1231U3tpRV4PGKJ"; GDSecret="3TPk3B123rCyCAXhKEadQa"}
$email= "certadmin@domain.com"
# letsencrypt server address. Use LE_STAGE to test, if everything ok switch to LE_PROD.
Set-PAServerLE_STAGE
# Request the certificate
New-PACertificate$domain-AcceptTOS-DnsPlugin$dnsPlugin-PluginArgs$pArgs-Contact$email-Verbose-Force
# transform PEM key and certificate into single text lines
$PACert= Get-PACertificate
$key_multiline= [IO.File]::ReadAllText($PACert.KeyFile)
$key_oneline= $key_multiline.Replace("`n",'\n')
$cert_multiline= [IO.File]::ReadAllText($PACert.FullChainFile)
$cert_oneline= $cert_multiline.Replace("`n",'\n')
#replace cert on UAG with REST API call
#provide UAG credential and info for API access in the format username:password encoded with base64 <a href="https://www.base64encode.org/">https://www.base64encode.org/</a>
$uagcred= "YWRWJDWDJDdsd13YXJlMSE="
$API_Settings= @{
Headers = @{ "Authorization"= "Basic $uagcred"}
Method = "PUT"
Body = $json_uag
ContentType = "application/json"
}
#Create json string and make api call. Refer to <a href="https://docs.vmware.com/en/Unified-Access-Gateway/2.8/com.vmware.access-point-28-deploy-config/GUID-EDC244DD-07AB-4841-A893-84ADF8D59838.html">https://docs.vmware.com/en/Unified-Access-Gateway/2.8/com.vmware.access-point-28-deploy-config/GUID-EDC244DD-07AB-4841-A893-84ADF8D59838.html</a>
$json_uag= '{"privateKeyPem":"'+ $key_oneline+ '","certChainPem":"'+ $cert_oneline+ '"}'
$API_Endpoint= "<a href="https:">https://</a>"+ $uag_host+ ":9443/rest/v1/config/certs/ssl"
Invoke-RestMethod$API_Endpoint@API_Settings
#now lets replace the certificate on Connection Server. First we install the certificate in the local machine store
Install-PACertificate
#we will check if there are an existing certificates with a friendly name of "vdm"and rename it to "replaced" with the current date for the record
Set-Locationcert:\LocalMachine\My
$old_certs= Get-Childitem|Where-Object {$_.FriendlyName -eq'vdm'}
ForEach($old_certin$old_certs){
$old_cert.FriendlyName = "Replaced $(Get-Date -format 'u')"
write-host$old_cert.Thumbprint
}
#now we will set the friendly name of this certificate to "vdm"
$cert_thumbprint_dirty= Get-PACertificate|flThumbprint
$cert_thumbprint= Out-String-InputObject$cert_thumbprint_dirty-Width100
$cert= gci $cert_thumbprint.Substring(17)
$cert.FriendlyName = “vdm”
#and the last thing is to restart Hotizon Connection Server service to pickup the new certificate
Restart-Service-Namewsbroker
Set-Location C:
Renew.ps1
This script should be scheduled to run every 60-80 days (start automatic renewal at least 10 days prior to expiry of the current certificate to give yourself time for manual renewal in case of failure). This should be run under the account that was used to start the Request.ps1 script on the same Connection Server. Redirect the output to a log file for reference and potential troubleshooting:.\MyScript.ps1 > output.txt
<#
.DESCRIPTION
Script requests certificates from Letsencrypt with automatic DNS-01 challange and installs them on UAG and Connection Server
.NOTES
Author: Roch Norwa <a href="https://digitalworkspace.blog/">https://digitalworkspace.blog</a> @rochnorwa <a href="https://linkedin.com/rochnorwa">https://linkedin.com/rochnorwa</a>
#>
#go back to C:\ just in case the location i set to cert:
Set-LocationC:
#information to be used in the cert request. use comma seperated domains for UCC/SAN certificate.
$domain= "*.domain.com","domain.com"
$uag_host= "uag.domain.com"
# information for letsncrypt POSH-ACME powershell script
$psMod= "Posh-ACME"
$dnsPlugin= "GoDaddy"
$pArgs= @{GDKey="9ZtodG7jWNC_84teqTR1231U3tpRV4PGKJ"; GDSecret="3TPk3B123rCyCAXhKEadQa"}
$email= "certadmin@domain.com"
# letsencrypt server address. Use LE_STAGE to test, if everything ok switch to LE_PROD.
Set-PAServerLE_STAGE
# Request the certificate
#New-PACertificate $domain -AcceptTOS -DnsPlugin $dnsPlugin -PluginArgs $pArgs -Contact $email -Verbose -Force
# Renew the certificate. Keep the above New-PACertificate hashed out!
# -AllOrders means all certificates requested by a specific account (email address). -Force will renew even if the certificate is not at the renewal period (30 days before expiration)
Submit-Renewal-AllOrders-Verbose-Force
# transform PEM key and certificate into single text lines
$PACert= Get-PACertificate
$key_multiline= [IO.File]::ReadAllText($PACert.KeyFile)
$key_oneline= $key_multiline.Replace("`n",'\n')
$cert_multiline= [IO.File]::ReadAllText($PACert.FullChainFile)
$cert_oneline= $cert_multiline.Replace("`n",'\n')
#replace cert on UAG with REST API call
#provide UAG credential and info for API access in the format username:password encoded with base64 <a href="https://www.base64encode.org/">https://www.base64encode.org/</a>
$uagcred= "YWRWJDWDJDdsd13YXJlMSE="
$API_Settings= @{
Headers = @{ "Authorization"= "Basic $uagcred"}
Method = "PUT"
Body = $json_uag
ContentType = "application/json"
}
#Create json string and make api call. Refer to <a href="https://docs.vmware.com/en/Unified-Access-Gateway/2.8/com.vmware.access-point-28-deploy-config/GUID-EDC244DD-07AB-4841-A893-84ADF8D59838.html">https://docs.vmware.com/en/Unified-Access-Gateway/2.8/com.vmware.access-point-28-deploy-config/GUID-EDC244DD-07AB-4841-A893-84ADF8D59838.html</a>
$json_uag= '{"privateKeyPem":"'+ $key_oneline+ '","certChainPem":"'+ $cert_oneline+ '"}'
$API_Endpoint= "<a href="https:">https://</a>"+ $uag_host+ ":9443/rest/v1/config/certs/ssl"
Invoke-RestMethod$API_Endpoint@API_Settings
#now lets replace the certificate on Connection Server. First we install the certificate in the local machine store
Install-PACertificate
#we will check if there are an existing certificates with a friendly name of "vdm"and rename it to "replaced" with the current date for the record
Set-Locationcert:\LocalMachine\My
$old_certs= Get-Childitem|Where-Object {$_.FriendlyName -eq'vdm'}
ForEach($old_certin$old_certs){
$old_cert.FriendlyName = "Replaced $(Get-Date -format 'u')"
write-host$old_cert.Thumbprint
}
#now we will set the friendly name of this certificate to "vdm"
$cert_thumbprint_dirty= Get-PACertificate|flThumbprint
$cert_thumbprint= Out-String-InputObject$cert_thumbprint_dirty-Width100
$cert= gci $cert_thumbprint.Substring(17)
$cert.FriendlyName = “vdm”
#and the last thing is to restart Hotizon Connection Server service to pickup the new certificate
Restart-Service-Namewsbroker
Set-Location C:
Setting up parameters in the scripts
There is a number of of parameters you have to define in the script. Below is the description of all the options you have to tune.
Define domain names for Unified Communication Certificates with Subject Alternate Names:$domain = "host1.domain.com","host2.domain.com",host3.domain.com"
For Wildard certificate use this pattern:$domain = "*.domain.com","domain.com"
Define your Unified Access Server host-name with this parameter. Remember it needs to point to REST API interface on port 9443 (depending on you deployment model one-two-three NIC). Our script will change certificate only for Internet Facing interface, not for the management one.$uag_host = "uag.domain.com"
Choose a dedicated plugin for your DNS provider. Available options are here.$dnsPlugin = "GoDaddy"
Provide DNS API authentication. Most likely an API Key and API Secret, otherwise username and password depending on your provider. This could be done in a more secure way with SecureString or PSCredentails, consider this if you plan to use this approach in a production.$pArgs= @{GDKey="9ZtodG7jWNC_84teqTR1231U3tpRV4PGKJ"; GDSecret="3TPk3B123rCyCAXhKEadQa"}
Certificate Administrator email adddress – this will identify your Account at Let’s Encrypt.$email = "certadmin@domain.com"
Define Let’s Encrypt ACME server address. Start with test stating server to check if the script works ok, then switch to production (LE_PROD)Set-PAServer LE_STAGE
Provide UAG admin credentials in the format username:password and encoded with Base64 algorithm (you can use https://www.base64encode.org/). This could be done in a more secure way with SecureString or PSCredentails, consider this if you plan to use this approach in a production.$uagcred = "YWRWJDWDJDdsd13YXJlMSE="
Final tips for getting the logs, certificates and scheduling renewals in Windows Scheduler
After successful request and every renewal the Posh-ACME script will save all the variables of certificates and keys in the folder structure of the user that was executing the scripts C:\Users\Administrator\AppData\Local\Posh-ACME
As you can see, for each server (Production and Test/Staging) there is a seperate subfolder. The current-server is noted in the current-server.txt file – make sure this points to LE_PROD.
In the production server folder there is a number of subfolders with numeric names. They correspond to Let’s Encrypt Accounts created for the email address you have given in your requests. Also the current-account.txt lists the account number that is beeing used for default renewals.
In the specific Account folder you will see the actual order folder (order maps to a domain from the request). There is also a plugindata.xml file with cached GoDaddy API Key and Secret, acct.json file with cached request script and email address for the account and the
Lastly, all the certificates, keys, chains and csr’s are in the domain folder. Default password for the PFX file is poshacme – it can be also found in the order.json file along with other certificate parameters.
Using HTTP-01 challange instead of DNS-01
In some cases, you might now have the possibility to automate the DNS challenge creation using REST API from your DNS provider, or you might not have permissions to do that. In this case, you can put an acme challenge file to your web server:http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>
In our case the web server is Tomcat that is running Horizon Connection Server web services. Therefore you would have to save the challenge file to the folder:C:\Program Files\VMware\VMware View\Server\broker\webapps\.well-known\acme-challenge
To make it accessible from the Internet, we need to add additional proxy pattern on the UAG Horizon service config:
You will also have to re-write above Power Shell scripts to save the HTTP-01 challenge as a file with specific name and content. Some hints on how to do it can be found below. I might write a blog post about this method in the near term future.
https://github.com/rmbolger/Posh-ACME/wiki/%28Advanced%29-Manual-HTTP-Challenge-Validation
Sources, tools and additional reading
https://letsencrypt.org/docs/ – official documentation
https://github.com/rmbolger/Posh-ACME – Power Shell based client for ACME
https://developer.godaddy.com/getstarted – playing with GoDaddy REST API
https://www.base64encode.org/ – encoder for username:password
https://sourceforge.net/projects/xca/ – Great vault for your certificates and keys with easy options of converting to different formats. Good alternative to OpenSSL if you prefer GUI. And it works on MacOS!
