TLS Protect for Kubernetes
Log inCreate account

Issue and approve certificates with Venafi Control Plane

Learn how to configure Venafi TPP, cert-manager, approver-policy-enterprise, and venafi-enhanced-issuer so that application teams can help themselves to SSL certificates which comply with enterprise PKI policy.



Create a Venafi Connection With Vault

Choose this connection method if you would like to store the VaaS or TPP credentials in a centralized HashiCorp Vault instance. This method supports both TLS Protect Cloud (VaaS) and TLS Protect Datacenter (TPP), and requires a Vault instance to store the credentials used to connect to VaaS and TPP.

Now that you've installed all the prerequisite software in your cluster it's time to configure a Venafi Connection.

This job would typically be performed by the platform team.

The following example shows a Venafi Connection resource which uses a Kubernetes ServiceAccount Token to authenticate to HashiCorp Vault which contains the credentials for Venafi TPP.

šŸ”— venafi-connection-1-vault.yaml

# venafi-connection-1-vault.yaml
apiVersion: jetstack.io/v1alpha1
kind: VenafiConnection
metadata:
  name: application-team-1-connection
  namespace: jetstack-secure
spec:
  tpp:
    url: https://tpp1.example.com


    accessToken:
      - serviceAccountToken:
          name: application-team-1
          audiences: ["vault.vault.svc.cluster.local"]
      - hashicorpVaultOAuth:
          authInputType: OIDC
          role: application-team-1
          authPath: /v1/auth/example-corp/login
          url: http://vault.vault.svc.cluster.local:8200
      - hashicorpVaultSecret:
          secretPath: /v1/secret/data/application-team-1/tpp-username-password
          fields: ["username", "password"]
          url: http://vault.vault.svc.cluster.local:8200
      - tppOAuth:
          authInputType: UsernamePassword
          clientId: tls-protect-kubernetes # āš  This 'clientId' value has to match the value configured in TPP
          url: https://tpp1.example.com
Copy to clipboard
kubectl apply -f venafi-connection-1-vault.yaml
Copy to clipboard

āš ļø The VenafiConnection resource will not report any issues in its status as long as it is not used. Create an Issuer or CertificateRequestPolicy that references the VenafiConnection to start using the VenafiConnection, after which the VenafiConnection resource's status will be updated.

Create an example Issuer to debug our VenafiConnection with

Because the VenafiConnection is not used anywhere yet, its status will remain empty. To figure out if something is wrong with your connection and obtain the error details, we have to start using it. Therefore, we create an example issuer.

šŸ”— venafi-example-issuer.yaml

# venafi-example-issuer.yaml
apiVersion: jetstack.io/v1alpha1
kind: VenafiIssuer
metadata:
  name: example-issuer
  namespace: jetstack-secure
spec:
  venafiConnectionName: application-team-1-connection
  zone: \VED\Policy\Teams\application-team-1
Copy to clipboard
kubectl apply -f venafi-example-issuer.yaml
Copy to clipboard

Don't forget to remove this example issuer when you are done with all the steps below:

kubectl delete -f venafi-example-issuer.yaml
Copy to clipboard

Create a Role and RoleBinding

If you use kubectl describe to view the status of the VenafiClusterIssuer resource you'll see an error:

$ kubectl -n jetstack-secure describe venaficonnection application-team-1-connection
...
Status:
  Ready Conditions:
    Last Transition Time:  2023-01-10T13:43:38Z
    Message:               connection is not ready yet (building connection failed): chain element 0 (ServiceAccountToken) error: serviceaccounts "application-team-1" is forbidden: User "system:serviceaccount:jetstack-secure:venafi-connection" cannot create resource "serviceaccounts/token" in API group "" in the namespace "jetstack-secure"
    Observed Generation:   1
    Operator Id:           venafienhancedissuer.jetstack.io
    Reason:                Pending
    Status:                False
Copy to clipboard

This tells you that there's a problem with the first item in the accessToken field. The problem is that venafi-enhanced-issuer is trying to request a ServiceAccount token but it doesn't have permission. So first let's add a Role and RoleBinding to venafi-connection.yaml and reapply it:

šŸ”— venafi-connection-2.yaml

# venafi-connection-2.yaml
# create role that allows creating sa tokens for 'application-team-1'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: create-tokens-for-application-team-1
  namespace: jetstack-secure
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    verbs: ["create"]
    resourceNames: ["application-team-1"]
---
# link the controller's service account to the 'create-tokens-for-vault-sa' role
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: application-team-1-sa-rolebinding
  namespace: jetstack-secure
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: create-tokens-for-application-team-1
subjects:
  - kind: ServiceAccount
    name: venafi-connection
    namespace: jetstack-secure
Copy to clipboard
kubectl apply -f venafi-connection-2.yaml
Copy to clipboard

šŸ’” Our controllers use exponential back-off between reconciliations in case a reconciliation error occurs. You can reset the exponential back-off timer by adding an annotation to the VenafiConnection resource, resulting in an immediate reconciliation.

kubectl -n jetstack-secure annotate venaficonnection application-team-1-connection last-trigger=$(date -u +"%FT%T.000Z") --overwrite
Copy to clipboard

Create a ServiceAccount

If you use kubectl describe to view the status of the VenafiClusterIssuer resource again you'll see a different error:

$ kubectl -n jetstack-secure describe venaficonnection application-team-1-connection
...
Status:
  Ready Conditions:
    Last Transition Time:  2023-01-10T13:45:41Z
    Message:               connection is not ready yet (building connection failed): chain element 0 (ServiceAccountToken) error: serviceaccounts "application-team-1" not found
    Observed Generation:   1
    Operator Id:           venafienhancedissuer.jetstack.io
    Reason:                Pending
    Status:                False
Copy to clipboard

Add a ServiceAccount to venafi-connection.yaml and reapply:

šŸ”— venafi-connection-3.yaml

# venafi-connection-3.yaml
# create a ServiceAccount which will be used to authenticate to HashiCorp Vault
apiVersion: v1
kind: ServiceAccount
metadata:
  name: application-team-1
  namespace: jetstack-secure
Copy to clipboard
kubectl apply -f venafi-connection-3.yaml
Copy to clipboard

šŸ’” Our controllers use exponential back-off between reconciliations in case a reconciliation error occurs. You can reset the exponential back-off timer by adding an annotation to the VenafiConnection resource, resulting in an immediate reconciliation.

kubectl -n jetstack-secure annotate venaficonnection application-team-1-connection last-trigger=$(date -u +"%FT%T.000Z") --overwrite
Copy to clipboard

Install HashiCorp Vault

Once again, run kubectl describe to examine the status of the VenafiClusterIssuer. This time you will see that there is a problem with next element in the authentication chain. It can't connect to the HashiCorp Vault server.

$ kubectl -n jetstack-secure describe venaficonnection application-team-1-connection
...
Status:
  Ready Conditions:
    Last Transition Time:  2023-01-10T13:53:53Z
    Message:               connection is not ready yet (building connection failed): chain element 1 (HashicorpVaultOAuth) error: Post "http://vault.vault.svc.cluster.local:8200/v1/auth/example-corp/login": dial tcp: lookup vault.vault.svc.cluster.local on 10.96.0.10:53: no such host
    Observed Generation:   1
    Operator Id:           venafienhancedissuer.jetstack.io
    Reason:                Pending
    Status:                False
Copy to clipboard

So now we need to deploy HashiCorp vault. For simplicity we'll install Vault in the Kubernetes cluster in "dev" server mode, but you may already have an existing HashiCorp Vault server that you prefer to use.

āš ļø Never use HashiCorp Vault "dev" server mode in production.

šŸ”— hashicorp-vault.values.yaml

global:
  tlsDisable: true


injector:
  enabled: false


server:
  auditStorage:
    enabled: false


  standalone:
    enabled: true


  dev:
    enabled: true


    devRootToken: root-token-1234


  ha:
    enabled: false


ui:
  enabled: true
  serviceType: ClusterIP
  serviceNodePort: null
  externalPort: 8200
Copy to clipboard
helm upgrade vault vault \
    --install \
    --create-namespace \
    --wait \
    --namespace vault \
    --repo https://helm.releases.hashicorp.com \
    --values hashicorp-vault.values.yaml
Copy to clipboard

šŸ’” Our controllers use exponential back-off between reconciliations in case a reconciliation error occurs. You can reset the exponential back-off timer by adding an annotation to the VenafiConnection resource, resulting in an immediate reconciliation.

kubectl -n jetstack-secure annotate venaficonnection application-team-1-connection last-trigger=$(date -u +"%FT%T.000Z") --overwrite
Copy to clipboard

Configure HashiCorp Vault

Now we need to configure Vault to allow the application-team-1 ServiceAccount to authenticate.

$ kubectl -n jetstack-secure describe venaficonnection application-team-1-connection
...
Status:
  Ready Conditions:
    Last Transition Time:  2023-01-10T14:00:46Z
    Message:               connection is not ready yet (building connection failed): chain element 1 (HashicorpVaultOAuth) error: HTTP 403: permission denied
    Observed Generation:   1
    Operator Id:           venafienhancedissuer.jetstack.io
    Reason:                Pending
    Status:                False
Copy to clipboard

ā„¹ļø In this tutorial HashiCorp Vault is configured to use Kubernetes authentication, and it is running inside the Kubernetes cluster. Alternatively you can configure HashiCorp Vault to use JWT authentication, which is similar but with some important differences. The disadvantage is that with JWT authentication HashiCorp Vault will not be able to check for revoked Kubernetes JWT tokens. The advantage is that Vault won't need to connect to the Kubernetes API server which makes it easier to connect to a HashiCorp Vault server that is running outside the Kubernetes cluster where venafi-enhanced-issuer is running.

kubectl exec -n vault pods/vault-0 -- \
    vault auth enable -path=example-corp kubernetes


kubectl exec -n vault pods/vault-0 -- \
    vault write auth/example-corp/config kubernetes_host=https://kubernetes.default.svc


kubectl exec -n vault pods/vault-0 -- \
    vault write auth/example-corp/role/application-team-1 \
       role_type=jwt \
       bound_audiences=vault.vault.svc.cluster.local \
       user_claim=sub \
       bound_service_account_names=application-team-1 \
       bound_service_account_namespaces=jetstack-secure \
       policies=application-team-1-readonly \
       ttl=5m
Copy to clipboard

šŸ’” Our controllers use exponential back-off between reconciliations in case a reconciliation error occurs. You can reset the exponential back-off timer by adding an annotation to the VenafiConnection resource, resulting in an immediate reconciliation.

kubectl -n jetstack-secure annotate venaficonnection application-team-1-connection last-trigger=$(date -u +"%FT%T.000Z") --overwrite
Copy to clipboard

Store TPP Credentials in HashiCorp Vault

Now the second step in the authentication chain should be working. But the next step will fail, because we haven't yet saved the Venafi TPP credentials to a secret in vault.

$ kubectl -n jetstack-secure describe venaficonnection application-team-1-connection
...
Status:
  Ready Conditions:
    Last Transition Time:  2023-01-10T14:03:33Z
    Message:               connection is not ready yet (building connection failed): chain element 2 (HashicorpVaultSecret) error: HTTP 403: 1 error occurred:
                           * permission denied




    Observed Generation:  1
    Operator Id:          venafienhancedissuer.jetstack.io
    Reason:               Pending
    Status:               False
Copy to clipboard

šŸ”— hashicorp-vault.policy.hcl

path "secret/data/application-team-1/tpp-username-password" {
  capabilities = ["read"]
}
Copy to clipboard
kubectl exec -i -n vault pods/vault-0 -- \
    vault policy write application-team-1-readonly - < hashicorp-vault.policy.hcl


kubectl exec -n vault pods/vault-0 -- \
    vault kv put -mount=secret application-team-1/tpp-username-password username=application-team-1 password=44054994-384d-40e8-adcb-b18cc4d7b756
Copy to clipboard

šŸ’” Our controllers use exponential back-off between reconciliations in case a reconciliation error occurs. You can reset the exponential back-off timer by adding an annotation to the VenafiConnection resource, resulting in an immediate reconciliation.

kubectl -n jetstack-secure annotate venaficonnection application-team-1-connection last-trigger=$(date -u +"%FT%T.000Z") --overwrite
Copy to clipboard

Configure Venafi TPP

Assuming that you've configured HashiCorp Vault and added a valid username / password credential to a Vault secret, the issuer should be able to connect to authenticate to Venafi TPP

And assuming that the PolicyDN is correct and the user account has permission to read and write to that policy folder, the example VenafiIssuer resource should be Ready.

$ kubectl -n jetstack-secure get venafiissuer example-issuer
NAME                 READY   REASON    MESSAGE   LASTTRANSITION   OBSERVEDGENERATION   GENERATION   AGE
application-team-1   True    Checked   checked   64s              1                    1            67s
Copy to clipboard

You may encounter some of the following error messages in the Ready condition, which are described below:

  • Username/password combination not valid

Check that you have added the correct username and password to the secret in HashiCorp Vault:

Message:               connection is not ready yet (building connection failed): chain element 3 (TPPOAuth) error: HTTP 400: invalid_grant: Username/password combination not valid
Reason:                Pending
Copy to clipboard
  • Application tls-protect-kubernetes is not registered and cannot obtain grants

Check that you have configured an Application called tls-protect-kubernetes in Venafi TPP.

Message:               connection is not ready yet (building connection failed): chain element 3 (TppOAuth) error: unauthorized_client: Application 'tls-protect-kubernetes' is not registered and cannot obtain grants
Reason:                Pending
Copy to clipboard
  • zone not found

Check that the PolicyDN value is correct and that the corresponding policy folder exists in Venafi TPP. Check that the Venafi TPP user account has permission to view, read, write and revoke in that policy folder.

Message:               Issuer is not ready yet: client.ReadZoneConfiguration: vcert error: your data contains problems: zone not found
Reason:                Pending
Copy to clipboard

šŸ’” Our controllers use exponential back-off between reconciliations in case a reconciliation error occurs. You can reset the exponential back-off timer by adding an annotation to the VenafiConnection resource, resulting in an immediate reconciliation.

kubectl -n jetstack-secure annotate venaficonnection application-team-1-connection last-trigger=$(date -u +"%FT%T.000Z") --overwrite
Copy to clipboard

Next Steps

On this page