云原生

课程

阿里云原生技术公开课

https://developer.aliyun.com/learning/roadmap/cloudnative

b站的可能不太全

https://www.bilibili.com/video/BV1r7411r7h7

安装docker和k8s

docker

https://docs.docker.com/engine/install/ubuntu/

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update


sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

k8s

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

v1.32.3

sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
# If the folder `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg # allow unprivileged APT programs to read this keyring

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list   # helps tools such as command-not-found to work correctly

sudo apt-get update
sudo apt-get install -y kubectl kubeadm kubelet

VMs config

所有虚拟机(nodes)都要做的配置

由于docker-shim is deprecated for k8s

使用 containerd of docker 作为 runtime

containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

sudo vim /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"

sudo systemctl restart containerd
sudo chown root:ubuntu /run/containerd/containerd.sock

sudo vim /etc/crictl.yaml

runtime-endpoint: "unix:///run/containerd/containerd.sock"
image-endpoint: "unix:///run/containerd/containerd.sock"
timeout: 10
debug: false

测试

crictl ps

其他配置：关闭swap, 打开br_netfilter等模块

https://www.linuxtechi.com/install-kubernetes-on-ubuntu-22-04/

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf <<EOT
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOT

sudo sysctl --system

sudo systemctl restart containerd
sudo systemctl enable containerd
sudo chown root:ubuntu /run/containerd/containerd.sock

网络

iptables

https://stackoverflow.com/questions/53247682/kubernetes-calico-on-oracle-cloud-vms

controller

# Clear forwarding rejections
sudo iptables -F FORWARD
sudo iptables -P FORWARD ACCEPT

# Kubernetes ports
sudo iptables -F FORWARD
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 6443 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 2379 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 2380 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 2379:2380 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10250 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10251 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10252 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10257 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10259 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 80 -j ACCEPT

sudo netfilter-persistent save

worker

# Clear forwarding rejections
sudo iptables -F FORWARD
sudo iptables -P FORWARD ACCEPT

#Kubernetes ports
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 2379 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 2380 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10250 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10251 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10252 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 10256 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 30000:32767 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo iptables -I INPUT 1 -m state --state NEW -p tcp --dport 80 -j ACCEPT

sudo netfilter-persistent save

Oracle

ingress rule (port 6443) (port 30000-32767)

在controller初始化cluster

sudo kubeadm init --apiserver-advertise-address=10.0.0.8 --pod-network-cidr=10.244.0.0/16

这里的cidr是为了对应之后的CNI（flannel）

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

https://github.com/flannel-io/flannel

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml


kubectl apply -f kube-flannel.yml


sudo systemctl restart containerd
sudo chown root:ubuntu /run/containerd/containerd.sock

测试

kubectl get pods -o wide -A

kubectl get nodes -o wide

Wait until each pod has the STATUS of Running.

workers join cluster

根据上面controller进行初始化时候的输出，得到类似下面的命令

sudo kubeadm join 10.0.0.8:6443 --token xxx \
        --discovery-token-ca-cert-hash sha256:xxx

若失败可以进行的reset操作

controller

sudo kubeadm reset --force

sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

sudo rm -rf /etc/cni/net.d

sudo rm -rf $HOME/.kube/config

worker

sudo systemctl stop kubelet
sudo systemctl stop containerd

sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/kubelet/
sudo rm -rf /var/lib/etcd/
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

sudo systemctl start containerd
sudo systemctl start kubelet

部署

如果之后的imagePullPolicy: Never，需要提前在worker node里面通过ctr或者crictl导入镜像

创建 deployment.yaml service.yaml

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

检查

kubectl get pods -o wide
kubectl logs xxx

kubectl get services
kubectl get svc xxx
kubectl describe svc xxx

其他指令

# kubectl delete deployment xxx
kubectl scale deployment xxx --replicas=2