TLS Protect for Kubernetes
Log inCreate account

Approver-policy-enterprise Venafi plugin

Learn how to configure the approver-policy component of TLS Protect for Kubernetes Enterprise



Venafi Approver Policy

The Enterprise version of approver-policy has a Venafi plugin which allows it to connect to a Venafi Control Plane server (VaaS/ TPP) and apply the policy settings of a particular policy folder. The approver-policy-enterprise project uses VenafiConnection CRs to configure how to connect to the Venafi Control Plane. The configuring venafi-connection documentation explains how to create these VenafiConnection CRs.

Create a CertificateRequestPolicy containing Venafi plugin configuration.

For the snippet below, we assume that the following VenafiConnection CR exist: namespace: jetstack-secure name: application-team-1-connection

apiVersion: policy.cert-manager.io/v1alpha1
kind: CertificateRequestPolicy
metadata:
  name: venafi-policy1
spec:
  allowed:
    commonName:
      value: "*"
    dnsNames:
      values: ["*"]
    usages:
      - "signing"
      - "digital signature"
      - "server auth"
  plugins:
    venafi:
      values:
        venafiConnectionName: application-team-1-connection
        zone: \VED\Policy\Teams\application-team-1
  selector:
    issuerRef: {}
Copy to clipboard

After the connection defined in the VenafiConnection CR is tested without errors, the CertificateRequestPolicy can check the specified zone and becomes Ready.

$ kubectl wait --for=condition=Ready  certificaterequestpolicy venafi-policy1
certificaterequestpolicy.policy.cert-manager.io/tpp-policy1 condition met
Copy to clipboard

If it does not become Ready, then use kubectl describe to see what is wrong. E.g.:

$ kubectl describe certificaterequestpolicy tpp-policy1
...
Warning  NotReady  5s (x2 over 5s)
policy.cert-manager.io  CertificateRequestPolicy is not ready for approval evaluation: spec.plugins.tpp.values: Invalid value: "while
getting zone configuration for policy: while creating vcert client: vcert error: your data contains problems: auth error: failed to
authenticate: can't determine valid credentials set": Retrying in 1m0s
Copy to clipboard

In this example you can see that the credentials are invalid or are missing. If you edit the Secret and add the correct the credentials, they will be rechecked within 1 minute and the CertificateRequestPolicy will then become Ready.

Create an approved certificate

Once the CertificateRequestPolicy is Ready you can try creating a Certificate which is allowed by your TPP policy:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: approved-certificate
spec:
  commonName: hello.world
  dnsNames:
    - "hello.world"
    - "example.hello.world"
  privateKey:
    algorithm: RSA
    size: 4096
  issuerRef:
    name: my-issuer
    kind: Issuer
    group: cert-manager.io
Copy to clipboard
$ cmctl create certificaterequest approved-certificate --from-certificate-file cert-approve.yaml
Copy to clipboard

You should see an event that says that the CertificateRequest was approved:

$ kubectl get events
LAST SEEN   TYPE      REASON           OBJECT                                    MESSAGE
2m2s        Normal    Approved         certificaterequest/approved-certificate   Approved by CertificateRequestPolicy: "tpp-policy1"
Copy to clipboard

Create a denied certificate

Now let's modify the TPP policy to only allow certificates with the domain example.com:

Editing the policy settings in the TPP web interface

Delete the certificate that we created earlier and then try again.

$ kubectl delete cr approved-certificate
cmctl create certificaterequest approved-certificate --from-certificate-file cert-approve.yaml
Copy to clipboard

This time you will see the CertificateRequest has been denied and there is an error event attached to the CertificateRequest explaining what went wrong:

$ kubectl describe cr approved-certificate
...
Events:
  Type     Reason  Age   From                    Message
  ----     ------  ----  ----                    -------
  Warning  Denied  41s   policy.cert-manager.io  No policy approved this request: [tpp-policy1: common name hello.world is not allowed in this policy: [^([\p{L}\p{N}-*]+\.)*example\.com$]]
Copy to clipboard

Periodic Configuration and Connection Health Checks

As a Kubernetes platform administrator, having installed a CerticateRequestPolicy, you will want to know that approver-policy-enterprise continues to be able to connect to the Venafi API and that it continues to be able to download the Venafi policy settings. And you need to know about any problems before they affect the approval of CertificateRequest resources.

The Venafi plugin in approver-policy-enterprise periodically performs a health check which connects to the Venafi API and downloads the policy configuration. It reports errors via the CertificateRequestPolicy.Status and with structured log messages, so that you can set up your monitoring systems to either check the Ready condition in the CertificateRequestPolicy.Status, or set up alerts when errors are logged by the approver-policy-enterprise component.

You can configure the interval between health checks using the --venafi-ready-check-interval flag. The default interval is 1 hour.

If you need to reduce the frequency of Venafi API requests you can increase this interval, but this will delay your alerts and it will take longer for you to be notified if there is a problem with the connection to the Venafi API.

📖 Read more about Customizing approver-policy-enterprise using command line flags.

Troubleshooting

Configuration and Connection Validation

The Venafi plugin for approver-policy-enterprise connects to a Venafi API server and downloads policy data. If there is a failure (e.g. a connection failure, authentication error or a download failure), then CertificateRequest resources will be Denied and will not be processed by cert-manager.

You can use kubectl describe to examine the Ready condition and Event resources associated with a particular CertificateRequestPolicy. The CertificateRequestPolicy.Status is updated when ever the CertificateRequestPolicy is created or modified.

If the policy has been downloaded successfully, you will see something like:

$ kubectl describe certificaterequestpolicy application-team-1
...
Status:
  Conditions:
    Message:               CertificateRequestPolicy is ready for approval evaluation
    Reason:                Ready
    Status:                True
    Type:                  Ready
...
Events:
  Type    Reason  Age                 From                    Message
  ----    ------  ----                ----                    -------
  Normal  Ready   42m (x12 over 19h)  policy.cert-manager.io  CertificateRequestPolicy is ready for approval evaluation
Copy to clipboard

If the connection failed you will see something like:

$ kubectl describe certificaterequestpolicy application-team-1
Status:
  Conditions:
    Message:               CertificateRequestPolicy is not ready for approval evaluation: spec.plugins.venafi.values: Invalid value: "while getting zone configuration for policy: connection is not ready yet (Venafi self-test failed): error authenticating with Venafi: vcert error: server error: server unavailable: Get \"https://api.venafi.cloud/v1/useraccounts\": dial tcp: lookup api.venafi.cloud on 10.96.0.10:53: server misbehaving": Retrying in 1h0m0s
    Reason:                NotReady
    Status:                False
    Type:                  Ready
...
Events:
  Type     Reason    Age                    From                    Message
  ----     ------    ----                   ----                    -------
  Warning  NotReady  112s (x76 over 6m55s)  policy.cert-manager.io  CertificateRequestPolicy is not ready for approval evaluation: spec.plugins.venafi.values: Invalid value: "while getting zone configuration for policy: connection is not ready yet (Venafi self-test failed): error authenticating with Venafi: vcert error: server error: server unavailable: Get \"https://api.venafi.cloud/v1/useraccounts\": dial tcp: lookup api.venafi.cloud on 10.96.0.10:53: server misbehaving": Retrying in 1h0m0s
Copy to clipboard

You can also check the status of the VenafiConnection resource, as follows:

$ kubectl describe -n jetstack-secure venaficonnection
...
Status:
  Conditions:
    Last Transition Time:  2023-03-24T09:50:46Z
    Message:               connection is not ready yet (Venafi self-test failed): error authenticating with Venafi: vcert error: server error: server unavailable: Get "https://api.venafi.cloud/v1/useraccounts": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Copy to clipboard

And you can check the approver-policy-enterprise logs, as follows:

$ kubectl logs -n jetstack-secure deploy/cert-manager-approver-policy cert-manager-approver-policy
...
E0324 09:50:46.172533       1 plugin.go:286] venafi/evaluate "msg"="Internal CertificateRequestPolicy evaluation error" "error"="while getting zone configuration for policy: connection is not ready yet (Venafi self-test failed): error authenticating with Venafi: vcert error: server error: server unavailable: Get \"https://api.venafi.cloud/v1/useraccounts\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" "UID"="2405bccb-c445-4291-a9d5-4a63dc99a5b2" "certificaterequest"={"Namespace":"application-team-1","Name":"service-1-4xwxw"} "description"="The VenafiConnection is invalid." "policy"={"Namespace":"","Name":"application-team-1"}
I0324 09:50:46.176662       1 recorder.go:103] controller-manager/events "msg"="No policy approved this request: [application-team-1: Internal CertificateRequestPolicy evaluation error: 2405bccb-c445-4291-a9d5-4a63dc99a5b2]" "object"={"kind":"CertificateRequest","namespace":"application-team-1","name":"service-1-4xwxw","uid":"20d6ae73-0889-4850-8a91-331f264f3c32","apiVersion":"cert-manager.io/v1","resourceVersion":"112056"} "reason"="Denied" "type"="Warning"
Copy to clipboard

As an application developer, if you create a CertificateRequest while there is a Venafi connection failure, you may see that your CertificateRequest is denied and an associated generic Warning Event:

$ kubectl describe cr -n application-team-1 service-1-4xwxw
...
Events:
  Type     Reason              Age    From                                                Message
  ----     ------              ----   ----                                                -------
...
  Warning  Denied              7m11s  policy.cert-manager.io                              No policy approved this request: [application-team-1: Internal CertificateRequestPolicy evaluation error: 2405bccb-c445-4291-a9d5-4a63dc99a5b2]
Copy to clipboard

The warning message deliberately omits the specific details of the error, because the details may reveal sensitive infrastructure details. Instead, the message contains a UUID which can be shared with the platform administrator, so that they can diagnose the problem by looking up that UUID in the logs.

On this page