Network Policy Implementation

This directory demonstrates how to implement default deny-all network rules in a Kubernetes cluster. This is achieved using Calico’s GlobalNetworkPolicy and the Kubernetes NetworkPolicy objects. It operates based on labels attached to namespaces.

Creating rules that allow traffic is done by adding Kubernetes NetworkPolicy objects to the namespace. These objects are portable across CNI plugins that enforce Kubernetes NetworkPolicy. Calico supports mixing Calico-specific policies with Kubernetes policy. This is not true for all CNI plugins such as Cilium. Calico-specific policies honor an order attribute for specifying precedence of policy evaluation. Lower values take precedence. Kubernetes policy does not support order, as it only supports additive allow rules. Under the hood, Calico sets Kubernetes policy to an order of 100. Thus, Calico policies must have their order set above 100.

To allow the blocked traffic in the above diagram, a NetworkPolicy such as the following would be applied.

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tomcat-netpol
  namespace: team-a
spec:
  policyTypes:
    - Ingress
    - Egress
  podSelector:
    matchLabels:
      app: tomcat
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress
        - podSelector:
            matchLabels:
              app: nginx
      ports:
        - protocol: TCP
          port: 80
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: team-a
        - podSelector:
            matchLabels:
              app: mysql
      ports:
        - protocol: TCP
          port: 3306

Note: this policy, unlike GlobalNetworkPolicy, is scoped within namespace and owned by team-a.

Namespace Ownership

This design assumes administrative ownership of namespaces. In most multi-tenant environments, namespaces are provisioned for teams. This allows limits, quotas, and other sensible defaults to be configured by administrators or programmatically with automation.

Alternative Designs

a. Use only Kubernetes NetworkPolicy

This approach is portable between multiple CNI Plugins. However, it requires creating a default deny-all network policy and adding it to every namespace that gets created. Should you need to modify how default-deny all works in your cluster and your cluster hosts 500 namespaces, you now have 500 locations to update the policy. Additionally, Calico’s GlobalNetworkPolicy is far more capable than Kubernetes NetworkPolicy.

b. Other CNI Plugins

Since Calico is used by the majority of our customers and recommended in our Validated Designs, Calico is used here. Should a customer desire using a different CNI plugin, similar mechanic may be available. Do note that some plugins do not allow you to mix their CRDs with Kubernetes NetworkPolicy objects.

Deploying and Testing

This section walks you through step-by-step applying of network policies in a running cluster. Each section has a diagram demonstrating how ingress and egress traffic will behave once you complete the section. There are validation steps throughout where you will make requests to ensure policy is blocked or allowed.

This section assumes you have a Kubernetes cluster running Calico 3.2+.

Environment Setup

Start by creating the resources and network paths shown in the diagram below. For now, all traffic will be allowed.

Allow All Traffic

  1. Download calicoctl.

    curl -O -L  https://github.com/projectcalico/calicoctl/releases/download/v3.5.1/calicoctl &&\
  chmod +x calicoctl &&\
  sudo mv calicoctl /usr/local/bin

  2. Configure calicoctl.

    export DATASTORE_TYPE=kubernetes
export KUBECONFIG=~/.kube/config
calicoctl get node

    NAME
ip-10-192-192-10
ip-10-192-192-4
ip-10-192-192-8

    This assumes Calico is installed in Kubernetes API datastore mode. If it is installed in etcd datastore mode, see https://docs.projectcalico.org/getting-started/clis/calicoctl/configure/etcd

  3. Create the following GlobalNetworkPolicy

    # deny-all.yaml

# This GlobalNetworkPolicy uses Calico's CRD
# (https://docs.projectcalico.org/v3.5/reference/calicoctl/resources/globalnetworkpolicy) to
# deactivate all traffic on namespace containing the label: value of netpol_deny_all: true. The only
# exception is UDP (port 53) traffic, which is allowed.
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: global-deny-all
spec:
  # order controls the precedence. Calico applies the policy with the lowest value first.
  # Kubernetes NetworkPolicy do not support order. They are automatically converted to an order
  # value of 1000 by Calico. Setting this value to 2000, provides flexibility for 999 additional
  # GlobalNetworkPolicies to be added and ensures Kubernetes namespace-scoped policies always take
  # precedence.
  order: 2000

  # types describes the direction of traffic this policy impacts. In this case, Ingress and Egress.
  types:
  - Ingress
  - Egress

  # egress network rules
  egress:
  # Allow all egress traffic when the netpol_deny_all label is not present.
  - action: Allow
	destination: {}
	source:
	  namespaceSelector: '!has(netpol_deny_all)'

  # Allow all egress traffic when the netpol_deny_all label has value 'disabled'.
  - action: Allow
	destination: {}
	source:
	  namespaceSelector: netpol_deny_all == 'disabled'

  # Allow egress DNS traffic to any destination. This assumes DNS traffic will be over UDP and
  # never TCP.
  - action: Allow
	protocol: UDP
	source:
	  namespaceSelector: netpol_deny_all == 'enabled'
	destination:
	  nets:
		- 0.0.0.0/0
	  ports:
		- 53

  # ingress network rules
  ingress:
  # Allow all ingress traffic when the netpol_deny_all label is not present.
  - action: Allow
	destination:
	  namespaceSelector: '!has(netpol_deny_all)'
	source: {}

  # Allow all ingress traffic when the netpol_deny_all label has value 'enabled'.
  - action: Allow
	destination:
	  namespaceSelector: netpol_deny_all == 'disabled'
	source: {}

  4. Apply the GlobalNetworkPolicy

    calicoctl apply -f deny-all.yaml

    This applies the default deny-all policy described in the design section. It will have no impact initially as namespaces are not properly labeled.

  5. Create namespaces ingress and team-a.

    kubectl create ns ingress
kubectl create ns team-a

  6. Create a Contour manifest

    # contour.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ingress
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: contour
  namespace: ingress
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.contour.heptio.com
  labels:
	component: ingressroute
spec:
  group: contour.heptio.com
  version: v1beta1
  scope: Namespaced
  names:
	plural: ingressroutes
	kind: IngressRoute
  additionalPrinterColumns:
	- name: FQDN
	  type: string
	  description: Fully qualified domain name
	  JSONPath: .spec.virtualhost.fqdn
	- name: TLS Secret
	  type: string
	  description: Secret with TLS credentials
	  JSONPath: .spec.virtualhost.tls.secretName
	- name: First route
	  type: string
	  description: First routes defined
	  JSONPath: .spec.routes[0].match
	- name: Status
	  type: string
	  description: The current status of the IngressRoute
	  JSONPath: .status.currentStatus
	- name: Status Description
	  type: string
	  description: Description of the current status
	  JSONPath: .status.description
  validation:
	openAPIV3Schema:
	  properties:
		spec:
		  properties:
			virtualhost:
			  properties:
				fqdn:
				  type: string
				  pattern: ^([a-zA-Z0-9]+(-[a-zA-Z0-9]+)*\.)+[a-z0-9]{2,}$
				tls:
				  properties:
					secretName:
					  type: string
					  pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ # DNS-1123 subdomain
					minimumProtocolVersion:
					  type: string
					  enum:
						- "1.3"
						- "1.2"
						- "1.1"
			strategy:
			  type: string
			  enum:
				- RoundRobin
				- WeightedLeastRequest
				- Random
				- RingHash
				- Maglev
			healthCheck:
			  type: object
			  required:
				- path
			  properties:
				path:
				  type: string
				  pattern: ^\/.*$
				intervalSeconds:
				  type: integer
				timeoutSeconds:
				  type: integer
				unhealthyThresholdCount:
				  type: integer
				healthyThresholdCount:
				  type: integer
			routes:
			  type: array
			  items:
				required:
				  - match
				properties:
				  match:
					type: string
					pattern: ^\/.*$
				  delegate:
					type: object
					required:
					  - name
					properties:
					  name:
						type: string
						pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ # DNS-1123 subdomain
					  namespace:
						type: string
						pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ # DNS-1123 label
				  services:
					type: array
					items:
					  type: object
					  required:
						- name
						- port
					  properties:
						name:
						  type: string
						  pattern: ^[a-z]([-a-z0-9]*[a-z0-9])?$ # DNS-1035 label
						port:
						  type: integer
						weight:
						  type: integer
						strategy:
						  type: string
						  enum:
							- RoundRobin
							- WeightedLeastRequest
							- Random
							- RingHash
							- Maglev
						healthCheck:
						  type: object
						  required:
							- path
						  properties:
							path:
							  type: string
							  pattern: ^\/.*$
							intervalSeconds:
							  type: integer
							timeoutSeconds:
							  type: integer
							unhealthyThresholdCount:
							  type: integer
							healthyThresholdCount:
							  type: integer
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
	app: contour
  name: contour
  namespace: ingress
spec:
  selector:
	matchLabels:
	  app: contour
  template:
	metadata:
	  labels:
		app: contour
	  annotations:
		prometheus.io/scrape: "true"
		prometheus.io/port: "8002"
		prometheus.io/path: "/stats"
		prometheus.io/format: "prometheus"
	spec:
	  containers:
	  - image: gcr.io/heptio-images/contour:master
		imagePullPolicy: Always
		name: contour
		command: ["contour"]
		args:
		- serve
		- --incluster
		- --envoy-external-http-port=8080
		- --envoy-external-https-port=8443
		- --envoy-service-http-port=8080
		- --envoy-service-https-port=8443
	  - image: docker.io/envoyproxy/envoy-alpine:v1.9.0
		name: envoy
		ports:
		- containerPort: 8080
		  name: http
		  hostPort: 8080
		- containerPort: 8443
		  name: https
		  hostPort: 8443
		command: ["envoy"]
		args:
		- --config-path /config/contour.yaml
		- --service-cluster cluster0
		- --service-node node0
		- --log-level info
		- --v2-config-only
		readinessProbe:
		  httpGet:
			path: /healthz
			port: 8002
		  initialDelaySeconds: 3
		  periodSeconds: 3
		volumeMounts:
		- name: contour-config
		  mountPath: /config
		lifecycle:
		  preStop:
			exec:
			  command: ["wget", "-qO-", "http://localhost:9001/healthcheck/fail"]
	  initContainers:
	  - image: gcr.io/heptio-images/contour:master
		imagePullPolicy: Always
		name: envoy-initconfig
		command: ["contour"]
		args:
		- bootstrap
		# Uncomment the statsd-enable to enable statsd metrics
		#- --statsd-enable
		# Uncomment to set a custom stats emission address and port
		#- --stats-address=0.0.0.0
		#- --stats-port=8002
		- /config/contour.yaml
		volumeMounts:
		- name: contour-config
		  mountPath: /config
	  volumes:
	  - name: contour-config
		emptyDir: {}
	  dnsPolicy: ClusterFirst
	  serviceAccountName: contour
	  terminationGracePeriodSeconds: 30
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: contour
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: contour
subjects:
- kind: ServiceAccount
  name: contour
  namespace: ingress
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: contour
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - nodes
  - pods
  - secrets
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups: ["contour.heptio.com"]
  resources: ["ingressroutes"]
  verbs:
  - get
  - list
  - watch
  - put
  - post
  - patch

  7. Deploy Contour in the ingress namespace.

    kubectl apply -f contour.yaml

    Contour will be deployed as a DaemonSet attaching to a host’s port 8080. Testing assumes you will be able to route directly to a node’s IP.

  8. Create a manifest for a simple echo server

    # echoserver.yaml

apiVersion: v1
kind: Service
metadata:
  name: echoserver
  namespace: team-a
spec:
  ports:
	- port: 80
	  targetPort: 8080
	  protocol: TCP
	  name: http
  selector:
	app: echoserver

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echoserver
  namespace: team-a
spec:
  rules:
	- http:
		paths:
		  - path: /echo
			backend:
			  serviceName: echoserver
			  servicePort: 80

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  namespace: team-a
spec:
  replicas: 2
  selector:
	matchLabels:
	  app: echoserver
  template:
	metadata:
	  labels:
		app: echoserver
	spec:
	  containers:
		- name: echoserver
		  image: gcr.io/kubernetes-e2e-test-images/echoserver:2.1
		  ports:
			- containerPort: 8080

  9. Deploy the manifest in the team-a namespace.

    kubectl apply -f echoserver.yaml

    This deploys a Deployment, Service, and Ingress to satisfy the diagram above.

  10. Create a manifest for lorem-ipsum

    # lorem-ipsum.yaml

apiVersion: v1
kind: Service
metadata:
  name: lorem-ipsum
  namespace: team-a
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: lorem-ipsum

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: lorem-ipsum
  namespace: team-a
spec:
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: lorem-ipsum
              servicePort: 80

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lorem-ipsum
  namespace: team-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: lorem-ipsum
  template:
    metadata:
      labels:
        app: lorem-ipsum
    spec:
      containers:
        - name: lorem-ipsum
          image: ksdn117/lorem-ipsum:latest
          ports:
            - containerPort: 80

  11. Deploy the manifest in the team-a namespace.

    kubectl apply -f lorem-ipsum.yaml

    This deploys a Deployment, Service, and Ingress to satisfy the diagram above.

  12. Validate ingress by requesting a worker node’s public IP at port 8080 on path /echo to ensure traffic is routable.

    curl 34.219.198.115:8080/echo

    Hostname: echoserver-578989d5f6-sh5f4

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.12.2 - lua: 10010

Request Information:
        client_address=192.168.2.3
        method=GET
        real path=/echo
        query=
        request_version=1.1
        request_scheme=http
        request_uri=http://34.219.198.115:8080/echo

Request Headers:
        accept=*/*
        content-length=0
        host=34.219.198.115:8080
        user-agent=curl/7.54.0
        x-envoy-expected-rq-timeout-ms=15000
        x-envoy-external-address=174.16.148.64
        x-forwarded-for=174.16.148.64
        x-forwarded-proto=http
        x-request-id=dc54cb4c-1586-42ca-8ebd-5a831a4e3b6f
        x-request-start=t=1550689433.206

Request Body:
        -no body in request-

  13. Validate ingress by requesting a worker node’s public IP at port 8080 on path / to ensure traffic is routable.

    Validate Ingress

  14. Attach to the lorem-ipsum pod.

    LOREM_POD=$(kubectl -n team-a get pod -l app=lorem-ipsum -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n team-a -it $LOREM_POD /bin/sh

  15. Add curl to the lorem-ipsum pod

    # apt update && apt install -y curl

  16. Validate pod-to-pod egress by curling echoserver.

    # curl echoserver

    Hostname: echoserver-578989d5f6-sh5f4

Pod Information:
-no pod information available-

Enforce Network Policy

Next, apply labels to the namespace so that Calico enforces the deny-all. This would typically be handled by an administrator or automation that ensures proper labels are attached to namespaces when teams are on boarded. After labels are added, the network should behave as diagrammed below.

Network Policy Denied

  1. Add netpol_deny_all labels to namespaces.

    kubectl label ns ingress netpol_deny_all=disabled &&\
  kubectl label ns team-a netpol_deny_all=enabled

  2. Add name labels to namespaces.

    kubectl label ns ingress name=ingress &&\
  kubectl label ns team-a name=team-a

  3. Validate labels are as expected.

    kubectl get ns --show-labels

    NAME          STATUS   AGE     LABELS
default       Active   10m     <none>
ingress       Active   3m53s   name=ingress,netpol_deny_all=disabled
kube-public   Active   10m     <none>
kube-system   Active   10m     <none>
team-a        Active   3m53s   name=team-a,netpol_deny_all=enabled

  4. Validate ingress is blocked by requesting a worker node’s public IP at port 8080 on paths / and /echo.

    curl -v 34.219.198.115:8080/echo

    *   Trying 34.219.198.115...
* TCP_NODELAY set
* Connected to 34.219.198.115 (34.219.198.115) port 8080 (#0)
> GET /echo HTTP/1.1
> Host: 34.219.198.115:8080
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 57
< content-type: text/plain
< date: Wed, 20 Feb 2019 20:00:47 GMT
< server: envoy
<
* Connection #0 to host 34.219.198.115 left intact

    Note: the 503 is from envoy, representing the upstream server (echoserver) was not reachable.

  5. Attach to the echoserver pod.

    ECHO_POD=$(kubectl -n team-a get pod -l app=echoserver -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n team-a -it $ECHO_POD /bin/sh

  6. Ping google.com; verify it cannot be reached.

    # ping -w 1 google.com

    PING google.com (216.58.193.78): 56 data bytes

--- google.com ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

    You can do the same to lorem-ipsum to demonstrate that a pod in the cluster is not reachable.

  7. Resolve DNS of google.com; verify it resolves.

    # nslookup google.com

    nslookup: can't resolve '(null)': Name does not resolve

Name:      google.com
Address 1: 172.217.6.46 sfo03s08-in-f14.1e100.net
Address 2: 2607:f8b0:400a:800::200e sea15s07-in-x0e.1e100.net

    DNS (UDP:53) is allowed in all namespaces.

Allow Restricted Access

The current state of network rules for team-a is what every team will experience going forward when a namespace is created for them. In order for team-a to allow specific traffic, they must add NetworkPolicy. The diagram below demonstrates the desired network access.

Network Policy Restricted

  1. Create a network policy manifest

    # lorem-netpol.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: lorem-ipsum-netpol
  namespace: team-a
spec:
  policyTypes:
    - Ingress
    - Egress
  podSelector:
    matchLabels:
      app: lorem-ipsum

  # allow ingress traffic from the contour/envoy pod
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress
        - podSelector:
            matchLabels:
              app: contour
      ports:
        - protocol: TCP
          port: 80

  # allow egress traffic to the echoserver pod in the team-a namespace
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: team-a
        - podSelector:
            matchLabels:
              app: echoserver
      ports:
        - port: 8080
          protocol: TCP

  2. Apply the NetworkPolicy containing ingress and egress rules for lorem-ipsum pods.

    kubectl apply -f lorem-netpol.yaml

    This applies rules to allow ingress traffic originating from Contour pods to reach the lorem-ipsum pods. It also applies rules to allow egress from the lorem-ipsum pod to the echoserver pod. Note that the ingress rules have yet to be added to the echoserver pod. Thus when the lorem-ipsum pod attempts to connect to it, echoserver will not allow the traffic in.

Validation

  1. Verify ingress is allowed to lorem-ipsum through Contour.

    Verify Ingress

  2. Verify ingress is denied to echoserver through Contour.

    curl -v 34.219.198.115:8080/echo

    *   Trying 34.219.198.115...
* TCP_NODELAY set
* Connected to 34.219.198.115 (34.219.198.115) port 8080 (#0)
> GET /echo HTTP/1.1
> Host: 34.219.198.115:8080
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 57
< content-type: text/plain
< date: Wed, 20 Feb 2019 21:06:54 GMT
< server: envoy
<
* Connection #0 to host 34.219.198.115 left intact

  3. Create an echoserver network policy

    # echo-netpol.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: echoserver-netpol
  namespace: team-a
spec:
  policyTypes:
	- Ingress
  podSelector:
	matchLabels:
	  app: echoserver

  # allow ingress traffic from the contour/envoy pod
  ingress:
	- from:
	  - namespaceSelector:
		  matchLabels:
			namespace: echoserver
	  - podSelector: {}

  4. Apply the echoserver ingress NetworkPolicy to allow ingress from all pods in the team-a namespace.

    kubectl apply -f echo-netpol.yaml

    Unlike the lorem-ipsum policy, this policy allows all ingress traffic originating from pods that belong to the team-a namespace. Note that with this configuration, lorem-ipsum will be able to communicate to echoserver but echoserver will not be able to communicate with lorem-ipsum.

  5. Attach to the lorem-ipsum pod.

    kubectl exec -n team-a -it $LOREM_POD /bin/sh

  6. Verify egress is allowed to the echoserver pod.

    # curl echoserver

    Hostname: echoserver-578989d5f6-sh5f4

Pod Information:
-no pod information available-

  7. Attach to the echoserver pod.

    kubectl exec -n team-a -it $ECHO_POD /bin/sh

  8. Verify egress is denied from echoserver to lorem-ipsum.

    # wget -T 1 lorem-ipsum

    Connecting to lorem-ipsum (10.109.11.137:80)
wget: download timed out