Skip to content

Platform Setup Guide

This guide describes how to prepare K8s clusters using kind that are suitable for deploying a two-party Carbyne Stack Virtual Cloud. After completing the steps described below, you should have two kind K8s clusters called apollo and starbuck with the following pods deployed:

kubectl get pods -A
NAMESPACE            NAME                                             READY   STATUS    RESTARTS   AGE
default              istio-operator-7cf558f694-nmm9b                  1/1     Running   0          10m
default              knative-operator-f6c995cb7-bfpfq                 1/1     Running   0          7m42s
default              operator-webhook-54bfbccdf9-7fc5w                1/1     Running   0          7m42s
default              postgres-operator-5b6b676464-m22w2               1/1     Running   0          7m17s
istio-system         istio-ingressgateway-5b57d7fd4f-qhrkp            1/1     Running   0          9m30s
istio-system         istiod-5f7d759f66-h29xw                          1/1     Running   0          9m42s
knative-serving      activator-6f7db96dc5-54lqv                       1/1     Running   0          7m13s
knative-serving      autoscaler-d9c7cc55c-qbc4n                       1/1     Running   0          7m12s
knative-serving      controller-6456dbd66f-nd6nl                      1/1     Running   0          7m12s
knative-serving      domain-mapping-769cfb7fb5-d2mwc                  1/1     Running   0          7m11s
knative-serving      domainmapping-webhook-54f4564c48-86z6d           1/1     Running   0          7m11s
knative-serving      net-istio-controller-d5977cfbb-m7ttl             1/1     Running   0          7m8s
knative-serving      net-istio-webhook-5dff88bdb8-ktbgq               1/1     Running   0          7m8s
knative-serving      webhook-6447645d8d-6k4xp                         1/1     Running   0          7m10s
kube-system          coredns-787d4945fb-c8dkg                         1/1     Running   0          10m
kube-system          coredns-787d4945fb-lgjjg                         1/1     Running   0          10m
kube-system          etcd-starbuck-control-plane                      1/1     Running   0          10m
kube-system          kindnet-z7vwz                                    1/1     Running   0          10m
kube-system          kube-apiserver-starbuck-control-plane            1/1     Running   0          10m
kube-system          kube-controller-manager-starbuck-control-plane   1/1     Running   0          10m
kube-system          kube-proxy-t5259                                 1/1     Running   0          10m
kube-system          kube-scheduler-starbuck-control-plane            1/1     Running   0          10m
local-path-storage   local-path-provisioner-c8855d4bb-zhl9t           1/1     Running   0          10m
metallb-system       metallb-controller-777d84cdd5-2jjkd              1/1     Running   0          8m44s
metallb-system       metallb-speaker-vmqj4                            1/1     Running   0          8m44s

Prerequisites

Warning

Carbyne Stack has been tested using the exact versions of the tools specified below. Deviating from this battle tested configuration may create all kinds of issues.

Do not forget to perform the post installation steps for Docker.

Info

You'll need at least 3 GB of memory and 1 CPU core per kind cluster to deploy Carbyne Stack. Depending on the actual workloads you are going to deploy, these numbers can be considerably higher.

For installation instructions for these tools see the Prerequisites section.

Setting up the Clusters

Kind Clusters

You will need two Carbyne Stack Virtual Cloud Providers deployed to separate K8s clusters to complete this getting started guide. The clusters are called apollo and starbuck. You can use the --name <name> option to launch a kind cluster with K8s context name kind-<name>, as follows:

kind create cluster --name apollo --image kindest/node:v1.26.6
kind create cluster --name starbuck --image kindest/node:v1.26.6

You can switch between the clusters easily using:

kubectl config use-context kind-apollo
kubectl config use-context kind-starbuck

Important

Complete the remaining steps of this guide for the apollo cluster and then repeat for starbuck.

Configuration

  1. Before setting up the platform make some common configuration available using:

    export RELEASE_NAME=cs
    

Istio

  1. Install the Istio Operator v1.17.0 using:

    curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.17.0 TARGET_ARCH=x86_64 sh -
    helm install istio-operator istio-1.17.0/manifests/charts/istio-operator \
      --set operatorNamespace=istio-operator \
      --set watchedNamespaces="istio-system" \
      --set hub="docker.io/istio" \
      --set tag="1.17.0"
    

  2. Create an Istio Control Plane in a dedicated namespace using:

    Info

    The configuration as used below defines 5 ports for the egress gateway in the range of 80-84. These ports are used for the routing of inter-VC communication, where each port is automatically linked to the traffic of a specific partner VC. The convention for the port associated to a specific vs is 80+i, where i is the id of the partner VC.
    This given configuration limits the number of partner VCs to 5. If you need more partner VCs, you have to adjust the configuration accordingly.

    cat <<EOF > istio-control-plane.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: istio-system
    ---
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    metadata:
      namespace: istio-system
      name: cs-istiocontrolplane
    spec:
      meshConfig:
        defaultConfig:
          proxyMetadata:
            ISTIO_META_DNS_AUTO_ALLOCATE: "true"
            ISTIO_META_DNS_CAPTURE: "true"
        accessLogFile: /dev/stdout
      components:
        ingressGateways:
          - name: istio-ingressgateway
            enabled: true
            k8s:
              resources:
                requests:
                  cpu: 10m
                  memory: 256Mi
              service:
                ports:
                  ## You can add custom gateway ports in user values overrides,
                  # but it must include those ports since helm replaces.
                  # Note that AWS ELB will by default perform health checks on
                  # the first port on this list. Setting this to the health
                  # check port will ensure that health checks always work.
                  # https://github.com/istio/istio/issues/12503
                  - port: 15021
                    targetPort: 15021
                    name: status-port
                  - port: 80
                    targetPort: 8080
                    name: http2
                  - port: 443
                    targetPort: 8443
                    name: https
                  - port: 1443
                    targetPort: 9443
                    name: https-mtls
                  - port: 2379
                    targetPort: 9379
                    name: etcd-mtls
                  - port: 15443
                    targetPort: 15443
                    name: tls
                  - port: 30000
                    name: ephemeral-mpc-engine-port-0
                  - port: 30001
                    name: ephemeral-mpc-engine-port-1
                  - port: 30002
                    name: ephemeral-mpc-engine-port-2
                  - port: 30003
                    name: ephemeral-mpc-engine-port-3
                  - port: 30004
                    name: ephemeral-mpc-engine-port-4
        egressGateways:
          - name: istio-egressgateway
            enabled: true
            k8s:
              resources:
                requests:
                  cpu: 10m
                  memory: 40Mi
              service:
                ports:
                  - port: 80
                    targetPort: 8080
                    name: http-0
                  - port: 81
                    targetPort: 8081
                    name: http-1
                  - port: 82
                    targetPort: 8082
                    name: http-2
                  - port: 83
                    targetPort: 8083
                    name: http-3
                  - port: 84
                    targetPort: 8084
                    name: http-4
                  - port: 2379
                    targetPort: 8379
                    name: etcd-tcp
        pilot:
          k8s:
            env:
              - name: PILOT_TRACE_SAMPLING
                value: "100"
            resources:
              requests:
                cpu: 10m
                memory: 100Mi
      values:
        global:
          proxy:
            autoInject: disabled
            resources:
              requests:
                cpu: 10m
                memory: 40Mi
        pilot:
          autoscaleEnabled: false
        gateways:
          istio-egressgateway:
            autoscaleEnabled: false
          istio-ingressgateway:
            autoscaleEnabled: false
        sidecarInjectorWebhook:
          alwaysInjectSelector:
            - matchExpressions:
              - {key: sidecar/inject, operator: Exists}
            - matchExpressions:
              - {key: app.kubernetes.io/name, operator: In, values: [amphora, castor]}
            - matchExpressions:
              - {key: control-plane, operator: In, values: [controller-manager]}
            - matchExpressions:
              - {key: app, operator: In, values: [${RELEASE_NAME}-ephemeral-discovery]}
    EOF
    kubectl apply -f istio-control-plane.yaml
    

  3. Enable istio auto sidecar injection for the default namespace using:

    kubectl label namespace default istio-injection=enabled
    

MetalLB

  1. Install MetalLB v0.13.9 using:

    helm repo add metallb https://metallb.github.io/metallb
    kubectl create namespace metallb-system
    cat <<EOF | envsubst > metallb-values.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        pod-security.kubernetes.io/audit: privileged
        pod-security.kubernetes.io/enforce: privileged
        pod-security.kubernetes.io/warn: privileged
      name: metallb-system
    EOF
    helm install metallb metallb/metallb --version 0.13.9 -n metallb-system -f metallb-values.yaml
    

  2. Configure MetalLB using:

    export SUBNET=172.18.1.255/25
    cat <<EOF | envsubst > metallb.yaml
    apiVersion: metallb.io/v1beta1
    kind: IPAddressPool
    metadata:
      name: default
      namespace: metallb-system
    spec:
      addresses:
      - ${SUBNET}
    ---
    apiVersion: metallb.io/v1beta1
    kind: L2Advertisement
    metadata:
      name: empty
      namespace: metallb-system
    EOF
    kubectl apply -f metallb.yaml
    
    export SUBNET=172.18.2.255/25
    cat <<EOF | envsubst > metallb.yaml
    apiVersion: metallb.io/v1beta1
    kind: IPAddressPool
    metadata:
      name: default
      namespace: metallb-system
    spec:
      addresses:
      - ${SUBNET}
    ---
    apiVersion: metallb.io/v1beta1
    kind: L2Advertisement
    metadata:
      name: empty
      namespace: metallb-system
    EOF
    kubectl apply -f metallb.yaml
    
  3. Wait until an external IP has been assigned to the Istio Ingress Gateway by MetalLB:

    kubectl get services --namespace istio-system istio-ingressgateway -w
    

The public IP eventually appears in column EXTERNAL-IP.

  1. Export the external IP for later use:

    export EXTERNAL_IP=$(kubectl get services --namespace istio-system istio-ingressgateway --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
    

Knative

  1. Install the Knative Operator v1.8.2 using:

    kubectl apply -f https://github.com/knative/operator/releases/download/knative-v1.8.2/operator.yaml
    

  2. Create a namespace for Knative Serving using:

    kubectl create namespace knative-serving
    
  3. Install the patched Knative Serving component with a sslip.io custom domain using:

    cat <<EOF | envsubst > knative-serving.yaml
    apiVersion: operator.knative.dev/v1beta1
    kind: KnativeServing
    metadata:
      name: knative-serving
      namespace: knative-serving
    spec:
      version: 1.8.2
      manifests:
        - URL: https://github.com/carbynestack/serving/releases/download/v1.8.2-multiport-patch/serving-crds.yaml
        - URL: https://github.com/carbynestack/serving/releases/download/v1.8.2-multiport-patch/serving-core.yaml
        - URL: https://github.com/knative/net-istio/releases/download/v1.8.2/release.yaml
        - URL: https://github.com/knative/net-certmanager/releases/download/v1.8.2/release.yaml
      ingress:
        istio:
          enabled: true
          knative-ingress-gateway:
            selector:
              istio: ingressgateway
            servers:
              - hosts:
                - '*' 
                port:
                  number: 443
                  name: https 
                  protocol: HTTPS
                  target_port: 443
      config:
         domain:
            ${EXTERNAL_IP}.sslip.io: ""
         defaults:
            max-revision-timeout-seconds: "36000"
         istio:
            gateway.knative-serving.knative-ingress-gateway: "istio-ingressgateway.istio-system.svc.cluster.local"
    EOF
    kubectl apply -f knative-serving.yaml
    

    The configuration above will also increase Knative's default max-revision-timeout-seconds from 600 to 36000 seconds (10h). This is required as ephemeral computations are executed as so-called Knative activations and are therefore subject to its configuration.

Postgres Operator

Deploy the Zalando Postgres operator v1.9.0 using:

curl -sL https://github.com/zalando/postgres-operator/archive/refs/tags/v1.9.0.tar.gz | tar -xz
helm install postgres-operator postgres-operator-1.9.0/charts/postgres-operator

Clean Up

If you no longer need the cluster you can tear it down using:

kind delete cluster --name apollo
kind delete cluster --name starbuck

Troubleshooting

OpenVPN

In case you use OpenVPN and encounter an error message when launching a kind cluster like

ERROR: failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1
Command Output: Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the default

follow the advice given here.

kind

In case one of your kind clusters doesn't start all pods, there might an issue with limited inotify resources on Ubuntu. You can check this by printing the logs of the faulty pods by, e.g.:

kubectl logs -n kube-system <pod-name>

If there is an output similar to this

 "command failed" err="failed complete: too many open files"

follow the advice given here.