That being said, sometimes re-coding a service for linux just isn't practical. And while things are pretty bad now, Microsoft is a really big company who seems to want into this market; they'll put effort into making things better over time. As an example of this, their documentation is quite good, way better than most of the linux documentation in my opinion. Have a look at it as portions of this guide are lifted directly from it [https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows]
Cluster Setup
You'll need at least one linux box to serve as a master, although we're going to use a few more to host other infrastructure services. You can follow my previous blog post with one exception; you can't run Calico as Container Network Interface (CNI). Well, technically you can, but Calico for windows is provided only as a subscription service and Microsoft only documents networking with Flannel as the CNI, so that's what we'll use here.**When you initialize the cluster, be sure to use Flannel's default pod cidr of 10.244.0.0/16 or you'll have problems when setting up microsoft nodes**
[root@kube-master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16
[root@kube-master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan",
"VNI": 4096,
"Port": 4789
}
}
[root@kube-master ~]# kubectl create -f kube-flannel.yml
CNI Notes
What's wrong with running Flannel for all clusters? Flannel has a separate private network allocated to each node which is then encapsulated within UDP and passer to other nodes within the cluster. Microsoft supports two Flannel modes, vxlan mode (documented here) which creates a virtual overlay network to handle routes between nodes automatically, and host-gateway mode, which seems insane to me as you need a static route on each node to every other node's pod subnet; so I don't recommend that.Calico, on the other hand, uses simple L3 routing within the cluster so it's much easier to see where traffic is going and where it came from. I like the ideal of Calico better, but it isn't a real option without a subscription so I'll stick with Flannel on my windows cluster. There are a few decent articles on the differences between the two:
- https://medium.com/@jain.sm/flannel-vs-calico-a-battle-of-l2-vs-l3-based-networking-5a30cd0a3ebd
- https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/
Windows Nodes
You'll need to install Windows 2019. I use 2019 standard with desktop experience as I like to RDP to the box but maybe you're an extreme windows guru and can do all of this without. I've disabled the firewall and installed vmware tools. Joining a domain is entirely optional as we aren't going to use any of the domain services. If you do join, make sure you treat this as a high performance server, so take care with patch schedules and extra windows features like virus scanning. You'll also need to ensure your patch level is high enough. I recommend running microsoft update, again, and again as you'll get new patches after a reboot. The version you're running should be at least 17763.379 as provided by KB4489899. You can find this by running winver.As mentioned before, Microsoft has done a good job documenting the steps so feel free to follow along there too. Everything should be done with an elevated powershell prompt (run as administrator). These first steps will add the repository and install docker.
PS C:\Users\Administrator> Install-Module -Name DockerMsftProvider -Repository PSGallery
PS C:\Users\Administrator> Install-Package -Name Docker -ProviderName DockerMsftProvider
PS C:\Users\Administrator> Restart-Computer
PS C:\Users\Administrator> docker version
Client: Docker Engine - Enterprise
Version: 19.03.0
API version: 1.40
Go version: go1.12.5
Git commit: 87b1f470ad
Built: 07/16/2019 23:41:30
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Enterprise
Engine:
Version: 19.03.0
API version: 1.40 (minimum version 1.24)
Go version: go1.12.5
Git commit: 87b1f470ad
Built: 07/16/2019 23:39:21
OS/Arch: windows/amd64
Experimental: false
error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.39/version: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
Create Pause Image
A pause image is also run on your linux nodes, but automatically; we need to do that manually here including downloading it, tagging it, and check that it runs correctly.PS C:\Users\Administrator> docker pull mcr.microsoft.com/windows/nanoserver:1809
PS C:\Users\Administrator> docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest
PS C:\Users\Administrator> docker run microsoft/nanoserver:latest
Microsoft Windows [Version 10.0.17763.615]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\>
Download Node Binaries
You'll need several binaries available from Kubernetes' github page. The version should match the server as close as possible. The official skew policy can be found at kubernetes.io, and if you want to see your client and server version you can use this command.[root@kube-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
I've used unix to expand the node binaries, either mac or your master node would work fine using tar zxvf kubernetes-node-windows-amd64.tar.gz but you can also use windows with expand-archive. Once that's done you'll need to copy all the executables under the expanded kubernetes/node/bin/* to c:\k. I know lots of people will want to change that \k folder but don't. Microsoft has hard coded it into many scripts we'll be using. So save yourself headache and just go with it.
You'll also need to grab /etc/kubernetes/admin.conf from the master node and place that in c:\k too and download Microsoft's start script. For all of these, I used a shared folder within my RDP session but winSCP is also a good tool if you don't mind installing more software on your worker nodes. It should look like this when you're done.
PS C:\Users\Administrator> mkdir c:\k
PS C:\Users\Administrator> wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -o c:\k\start.ps1
<download and transfer kubernetes node binaries and config file>
PS C:\k> dir
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 7/23/2019 2:12 PM 5447 config
-a---- 7/18/2019 2:55 AM 40072704 kube-proxy.exe
-a---- 7/18/2019 2:55 AM 40113152 kubeadm.exe
-a---- 7/18/2019 2:55 AM 43471360 kubectl.exe
-a---- 7/18/2019 2:55 AM 116192256 kubelet.exe
-a---- 7/23/2019 2:01 PM 2447 start.ps1
Joining A Windows Node
You're finally ready to join a windows node! Again you can have a look at the documentation but if you've been following along, you'll only need two options.- ManagementIP - this is unfortunate as it'll require more scripting when you're ready to automate. It's the IP address of this worker node which you can get from ipconfig on your windows node
- NetworkMode - we're using vxlan and the default is l2bridge so this will need to be set to overlay
- ServiceCIDR - verify with kubectl cluster-info dump | grep -i service-cluster
- ClusterCIDR - check with kubectl cluster-info dump | grep -i cluster-cidr
- KubeDnsServiceIP - verify the default (10.96.0.10) with kubectl get svc -n kube-system. Cluster-IP is the field you're interested in.
PS C:\k> .\start.ps1 -ManagementIP 10.9.176.94 -NetworkMode overlay
Initial Problems
I had trouble getting the kubelet process to start. You'll notice the node doesn't go ready and if you look at the processes it will have flannel and kube-proxy but no kubelet. It seems the start-kubelet.ps1 script that's downloaded is using outdated flags, so to fix that, remove the highlighted --allow-privileged=true from start-kubelet.ps1.$kubeletArgs = @(
"--hostname-override=$(hostname)"
'--v=6'
'--pod-infra-container-image=mcr.microsoft.com/k8s/core/pause:1.0.0'
'--resolv-conf=""'
'--allow-privileged=true'
'--enable-debugging-handlers'
"--cluster-dns=$KubeDnsServiceIp"
'--cluster-domain=cluster.local'
'--kubeconfig=c:\k\config'
'--hairpin-mode=promiscuous-bridge'
'--image-pull-progress-deadline=20m'
'--cgroups-per-qos=false'
"--log-dir=$LogDir"
'--logtostderr=false'
'--enforce-node-allocatable=""'
'--network-plugin=cni'
'--cni-bin-dir="c:\k\cni"'
'--cni-conf-dir="c:\k\cni\config"'
"--node-ip=$(Get-MgmtIpAddress)"
)
I also had a problem when provisioning persistent volumes even though they weren't for the windows node. If kubernetes can't identify all nodes in the cluster, it won't do anything. The error looks like this
I0723 23:57:11.379621 1 event.go:258] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test", UID:"715df13c-8eeb-4ba4-9be1-44c8a5f03071", APIVersion:"v1", ResourceVersion:"480073", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "vsphere-ssd": No VM found
E0723 23:59:26.375664 1 datacenter.go:78] Unable to find VM by UUID. VM UUID:
E0723 23:59:26.375705 1 nodemanager.go:431] Error "No VM found" node info for node "kube-w2" not found
E0723 23:59:26.375718 1 vsphere_util.go:130] Error while obtaining Kubernetes node nodeVmDetail details. error : No VM found
E0723 23:59:26.375727 1 vsphere.go:1291] Failed to get shared datastore: No VM found
E0723 23:59:26.375787 1 goroutinemap.go:150] Operation for "provision-default/test[715df13c-8eeb-4ba4-9be1-44c8a5f03071]" failed. No retries permitted until 2019-07-24 00:01:28.375767669 +0000 UTC m=+355638.918509528 (durationBeforeRetry 2m2s). Error: "No VM found"
I0723 23:59:26.376127 1 event.go:258] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test", UID:"715df13c-8eeb-4ba4-9be1-44c8a5f03071", APIVersion:"v1", ResourceVersion:"480073", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "vsphere-ssd": No VM found
Updating A Node UUID
Like our linux nodes, you'll need to patch the node spec with the UUID of the node. Under windows you can retrieve that UUID with the following command but you'll need to reformat it.PS C:\k> wmic bios get serialnumber
SerialNumber
VMware-42 3c fe 01 af 23 a9 a5-65 45 50 a3 db db 9d 69
[root@k8s-master ~]# kubectl patch node <node_name> -p '{"spec":{"providerID":"vsphere://423CFE01-AF23-A9A5-6545-50A3DBDB9D69"}}'
Patching DaemonSets
A DaemonSet gets a pod pushed to every node in the cluster. This is generally bad because most things don't run on windows, so to prevent that you'll need to patch existing sets and use the node selector for application you produce. You can download the patch from microsoft or create your own file, it's pretty basic. If you've been following along with the code provided on github, those files already have the node selector set.[root@kube-master ~]# wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/l2bridge/manifests/node-selector-patch.yml
[root@kube-master t]# cat node-selector-patch.yml
spec:
template:
spec:
nodeSelector:
beta.kubernetes.io/os: linux
[root@kube-master ~]# kubectl patch ds/kube-flannel-ds-amd64 --patch "$(cat node-selector-patch.yml)" -n=kube-system
[root@kube-master ~]# kubectl patch ds/kube-proxy --patch "$(cat node-selector-patch.yml)" -n=kube-system
Deploying A Test Pod
I'd suggest using the Microsoft provided yaml file although I reduced the number of replicas to 1 to simplify any troubleshooting.[root@kube-master ~]# wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/l2bridge/manifests/simpleweb.yml -O win-webserver.yaml
[root@kube-master ~]# kubectl apply -f win-webserver.yaml
[root@kube-master ~]# kubectl get pods -o wide
Registering A Service
Every time you reboot you'll need to run the start command manually, which isn't all that useful. Microsoft has created some excellent instructions and a script to register the required services using the Non-Sucking Service Manager. Follow the instructions provided by Microsoft, which is basically placing both the sample script, called register-svc.ps1, and nssm.exe binary into c:\k.PS C:\k> wget https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/flannel/register-svc.ps1 -o c:\k\register-svc.ps1
.\nssm.exe set $KubeletSvc AppParameters --hostname-override=$Hostname --v=6 --pod-infra-container-image=mcr.microsoft.com/k8s/core/pause:1.0.0 --resolv-conf="" --enable-debugging-handlers --cluster-dns=$KubeDnsServiceIP --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir=$LogDir --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config
PS C:\k> .\register-svc.ps1 -ManagementIP <windows_node_ip> -NetworkMode overlay
Service "flanneld" installed successfully!
Set parameter "AppParameters" for service "flanneld".
Set parameter "AppEnvironmentExtra" for service "flanneld".
Set parameter "AppDirectory" for service "flanneld".
flanneld: START: The operation completed successfully.
Service "kubelet" installed successfully!
Set parameter "AppParameters" for service "kubelet".
Set parameter "AppDirectory" for service "kubelet".
kubelet: Unexpected status SERVICE_PAUSED in response to START control.
Service "kube-proxy" installed successfully!
Set parameter "AppDirectory" for service "kube-proxy".
Set parameter "AppParameters" for service "kube-proxy".
Set parameter "DependOnService" for service "kube-proxy".
kube-proxy: START: The operation completed successfully.
PS C:\k> .\nssm.exe edit kubelet
PS C:\k> .\nssm.exe edit kube-proxy
PS C:\k> .\nssm.exe edit flanneld
PS C:\k> .\nssm.exe remove kubelet confirm
PS C:\k> .\nssm.exe remove kube-proxy confirm
PS C:\k> .\nssm.exe remove flanneld confirm
Deleting/Re-adding A Windows Node
If you delete a windows node, such as with kubectl delete node <node_name>, adding it is pretty easy. Because the windows nodes have the kubernetes config file they re-register automatically on every service start. You might need to remove existing flannel configuration files and then reboot.PS C:\k> Remove-Item C:\k\SourceVip.json
PS C:\k> Remove-Item C:\k\SourceVipRequest.json
PS C:\k> Restart-Computer
Broken Kubernetes Things With Windows Nodes
Pretty much everything in broken. You'll be able to deploy a windows container to a windows node using the node selector spec entry like we did when patching the daemonsets above. Just place windows as the OS type instead of linux. Here's a list of things that are broken which I'll update when possible:- Persistent Volumes - you need to ensure the node is registered properly with vsphere or nothing will be able to use a persistent volume. This is because vsphere ensures all nodes can see a datastore without making a distinction between windows and linux. I can get a PV to appear on a windows node but I can't get it to initialize properly
- Node Ports - this is a documented limitation, you can't access a node port service from the node hosting the pod. Strange, yes, but you should be able to use any linux nodes as an entry for any windows pods
- Load Balancer - With version 0.8.0 it includes the os selector patch, and it should work forwarding connecting through available linux nodes but I haven't had any success yet
- DNS - untested as of yet because of load balancer problems
- Logging - should be possible as fluent bit has beta support for windows but untested yet
- Fluent bit does have some documentation to install under windows, maybe under the node itself as there isn't a docker container readily available, but none of the links work. Perhaps not yet (Aug 2019)
- Monitoring - should also be possible using WMI exporter rather than a node exporter, again, untested at this time
