It's happened, someone has asked you for a windows container. The first piece of advice to give; find a way, any way possible to run the service within a linux container. Not being knowledgeable with linux shouldn't be an excuse as most containers require very little expertise with linux anyway. Help the windows developers migrate to linux, everyone will be happier.
That being said, sometimes re-coding a service for linux just isn't practical. And while things are pretty bad now, Microsoft is a really big company who seems to want into this market; they'll put effort into making things better over time. As an example of this, their documentation is quite good, way better than most of the linux documentation in my opinion. Have a look at it as portions of this guide are lifted directly from it [
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows]
Cluster Setup
You'll need at least one linux box to serve as a master, although we're going to use a few more to host other infrastructure services. You can follow my
previous blog post with one exception; you can't run Calico as Container Network Interface (CNI). Well, technically you can, but Calico for windows is provided only as a subscription service and Microsoft only documents networking with Flannel as the CNI, so that's what we'll use here.
**When you initialize the cluster, be sure to use Flannel's default pod cidr of 10.244.0.0/16 or you'll have problems when setting up microsoft nodes**
[root@kube-master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16
Setting up Flannel is pretty easy you'll download the flannel yaml file and make some Microsoft specific changes, notably the VNI and PORT number as documented on
github and from
microsoft.
[root@kube-master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Within the ConfigMap section you'll have the following under net-conf.json:
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
We need to add the required VNI and Port information like this:
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan",
"VNI": 4096,
"Port": 4789
}
}
And install Flannel like this:
[root@kube-master ~]# kubectl create -f kube-flannel.yml
CNI Notes
What's wrong with running Flannel for all clusters? Flannel has a separate private network allocated to each node which is then encapsulated within UDP and passer to other nodes within the cluster. Microsoft supports two Flannel modes, vxlan mode (documented here) which creates a virtual overlay network to handle routes between nodes automatically, and host-gateway mode, which seems insane to me as you need a static route on each node to every other node's pod subnet; so I don't recommend that.
Calico, on the other hand, uses simple L3 routing within the cluster so it's much easier to see where traffic is going and where it came from. I like the ideal of Calico better, but it isn't a real option without a subscription so I'll stick with Flannel on my windows cluster. There are a few decent articles on the differences between the two:
Windows Nodes
You'll need to install Windows 2019. I use 2019 standard with desktop experience as I like to RDP to the box but maybe you're an extreme windows guru and can do all of this without. I've disabled the firewall and installed vmware tools. Joining a domain is entirely optional as we aren't going to use any of the domain services. If you do join, make sure you treat this as a high performance server, so take care with patch schedules and extra windows features like virus scanning. You'll also need to ensure your patch level is high enough. I recommend running microsoft update, again, and again as you'll get new patches after a reboot. The version you're running should be at least 17763.379 as provided by
KB4489899. You can find this by running
winver.
As mentioned before, Microsoft has done a good job
documenting the steps so feel free to follow along there too. Everything should be done with an elevated powershell prompt (run as administrator). These first steps will add the repository and install docker.
PS C:\Users\Administrator> Install-Module -Name DockerMsftProvider -Repository PSGallery
PS C:\Users\Administrator> Install-Package -Name Docker -ProviderName DockerMsftProvider
Reboot the machine and check docker is running properly
PS C:\Users\Administrator> Restart-Computer
PS C:\Users\Administrator> docker version
Client: Docker Engine - Enterprise
Version: 19.03.0
API version: 1.40
Go version: go1.12.5
Git commit: 87b1f470ad
Built: 07/16/2019 23:41:30
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Enterprise
Engine:
Version: 19.03.0
API version: 1.40 (minimum version 1.24)
Go version: go1.12.5
Git commit: 87b1f470ad
Built: 07/16/2019 23:39:21
OS/Arch: windows/amd64
Experimental: false
If you get an error here that looks like this:
error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.39/version: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
It just means that docker service didn't start on boot. Start it from services or from powershell using
Start-Service docker
Create Pause Image
A pause image is also run on your linux nodes, but automatically; we need to do that manually here including downloading it, tagging it, and check that it runs correctly.
PS C:\Users\Administrator> docker pull mcr.microsoft.com/windows/nanoserver:1809
PS C:\Users\Administrator> docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest
PS C:\Users\Administrator> docker run microsoft/nanoserver:latest
Microsoft Windows [Version 10.0.17763.615]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\>
Download Node Binaries
You'll need several binaries available from Kubernetes' github page. The version should match the server as close as possible. The official skew policy can be found at
kubernetes.io, and if you want to see your client and server version you can use this command.
[root@kube-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
This is saying my client and server are version 1.15.1. To download the corresponding client version, you can use this link [
https://github.com/kubernetes/kubernetes/releases/], select the
CHANGELOG-<version>.md link and download the node binaries for windows. In this case the latest is
1.15.1 , so that works out well.
I've used unix to expand the node binaries, either mac or your master node would work fine using
tar zxvf kubernetes-node-windows-amd64.tar.gz but you can also use windows with
expand-archive. Once that's done you'll need to copy all the executables under the expanded kubernetes/node/bin/* to c:\k. I know lots of people will want to change that \k folder but don't. Microsoft has hard coded it into many scripts we'll be using. So save yourself headache and just go with it.
You'll also need to grab /etc/kubernetes/admin.conf from the master node and place that in c:\k too and download Microsoft's start script. For all of these, I used a shared folder within my RDP session but winSCP is also a good tool if you don't mind installing more software on your worker nodes. It should look like this when you're done.
PS C:\Users\Administrator> mkdir c:\k
PS C:\Users\Administrator> wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -o c:\k\start.ps1
<download and transfer kubernetes node binaries and config file>
PS C:\k> dir
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 7/23/2019 2:12 PM 5447 config
-a---- 7/18/2019 2:55 AM 40072704 kube-proxy.exe
-a---- 7/18/2019 2:55 AM 40113152 kubeadm.exe
-a---- 7/18/2019 2:55 AM 43471360 kubectl.exe
-a---- 7/18/2019 2:55 AM 116192256 kubelet.exe
-a---- 7/23/2019 2:01 PM 2447 start.ps1
Joining A Windows Node
You're finally ready to join a windows node! Again you can have a look at the
documentation but if you've been following along, you'll only need two options.
- ManagementIP - this is unfortunate as it'll require more scripting when you're ready to automate. It's the IP address of this worker node which you can get from ipconfig on your windows node
- NetworkMode - we're using vxlan and the default is l2bridge so this will need to be set to overlay
Other fields that should be fine with defaults but you can check them with these commands
- ServiceCIDR - verify with kubectl cluster-info dump | grep -i service-cluster
- ClusterCIDR - check with kubectl cluster-info dump | grep -i cluster-cidr
- KubeDnsServiceIP - verify the default (10.96.0.10) with kubectl get svc -n kube-system. Cluster-IP is the field you're interested in.
When you run the start.ps1 script it'll download a lot of additional scripts and binaries eventually spawning a few new powershell windows leaving the logging one open, which can be very helpful at this stage. Run the following replacing the IP in blue with your local windows server IP address (from ipconfig)
PS C:\k> .\start.ps1 -ManagementIP 10.9.176.94 -NetworkMode overlay
Initial Problems
I had trouble getting the kubelet process to start. You'll notice the node doesn't go ready and if you look at the processes it will have flannel and kube-proxy but no kubelet. It seems the start-kubelet.ps1 script that's downloaded is using outdated flags, so to fix that,
remove the highlighted --allow-privileged=true from start-kubelet.ps1.
$kubeletArgs = @(
"--hostname-override=$(hostname)"
'--v=6'
'--pod-infra-container-image=mcr.microsoft.com/k8s/core/pause:1.0.0'
'--resolv-conf=""'
'--allow-privileged=true'
'--enable-debugging-handlers'
"--cluster-dns=$KubeDnsServiceIp"
'--cluster-domain=cluster.local'
'--kubeconfig=c:\k\config'
'--hairpin-mode=promiscuous-bridge'
'--image-pull-progress-deadline=20m'
'--cgroups-per-qos=false'
"--log-dir=$LogDir"
'--logtostderr=false'
'--enforce-node-allocatable=""'
'--network-plugin=cni'
'--cni-bin-dir="c:\k\cni"'
'--cni-conf-dir="c:\k\cni\config"'
"--node-ip=$(Get-MgmtIpAddress)"
)
I also had a problem when provisioning persistent volumes even though they weren't for the windows node. If kubernetes can't identify all nodes in the cluster, it won't do anything. The error looks like this
I0723 23:57:11.379621 1 event.go:258] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test", UID:"715df13c-8eeb-4ba4-9be1-44c8a5f03071", APIVersion:"v1", ResourceVersion:"480073", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "vsphere-ssd": No VM found
E0723 23:59:26.375664 1 datacenter.go:78] Unable to find VM by UUID. VM UUID:
E0723 23:59:26.375705 1 nodemanager.go:431] Error "No VM found" node info for node "kube-w2" not found
E0723 23:59:26.375718 1 vsphere_util.go:130] Error while obtaining Kubernetes node nodeVmDetail details. error : No VM found
E0723 23:59:26.375727 1 vsphere.go:1291] Failed to get shared datastore: No VM found
E0723 23:59:26.375787 1 goroutinemap.go:150] Operation for "provision-default/test[715df13c-8eeb-4ba4-9be1-44c8a5f03071]" failed. No retries permitted until 2019-07-24 00:01:28.375767669 +0000 UTC m=+355638.918509528 (durationBeforeRetry 2m2s). Error: "No VM found"
I0723 23:59:26.376127 1 event.go:258] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test", UID:"715df13c-8eeb-4ba4-9be1-44c8a5f03071", APIVersion:"v1", ResourceVersion:"480073", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "vsphere-ssd": No VM found
And my eventual solution was to reboot the master node. Sad, yes.
Updating A Node UUID
Like our linux nodes, you'll need to patch the node spec with the UUID of the node. Under windows you can retrieve that UUID with the following command but you'll need to reformat it.
PS C:\k> wmic bios get serialnumber
SerialNumber
VMware-42 3c fe 01 af 23 a9 a5-65 45 50 a3 db db 9d 69
And back on our kubernetes master node we'd patch the node like this
[root@k8s-master ~]# kubectl patch node <node_name> -p '{"spec":{"providerID":"vsphere://423CFE01-AF23-A9A5-6545-50A3DBDB9D69"}}'
Patching DaemonSets
A DaemonSet gets a pod pushed to every node in the cluster. This is generally bad because most things don't run on windows, so to prevent that you'll need to patch existing sets and use the node selector for application you produce. You can download the patch from microsoft or create your own file, it's pretty basic. If you've been following along with the code provided on
github, those files already have the node selector set.
[root@kube-master ~]# wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/l2bridge/manifests/node-selector-patch.yml
[root@kube-master t]# cat node-selector-patch.yml
spec:
template:
spec:
nodeSelector:
beta.kubernetes.io/os: linux
We'll need to apply it to existing DaemonSets, notably kube-proxy and kube-flannel-ds-amd64.
[root@kube-master ~]# kubectl patch ds/kube-flannel-ds-amd64 --patch "$(cat node-selector-patch.yml)" -n=kube-system
[root@kube-master ~]# kubectl patch ds/kube-proxy --patch "$(cat node-selector-patch.yml)" -n=kube-system
If you've been getting errors on your windows node from flannel saying things like
Error response from daemon: network host not found and
Error: no such container, those should now stop.
Deploying A Test Pod
I'd suggest using the
Microsoft provided yaml file although I reduced the number of replicas to 1 to simplify any troubleshooting.
[root@kube-master ~]# wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/l2bridge/manifests/simpleweb.yml -O win-webserver.yaml
[root@kube-master ~]# kubectl apply -f win-webserver.yaml
[root@kube-master ~]# kubectl get pods -o wide
Registering A Service
Every time you reboot you'll need to run the start command manually, which isn't all that useful. Microsoft has created some excellent instructions and a script to register the required services using the Non-Sucking Service Manager. Follow the
instructions provided by Microsoft, which is basically placing both the sample script, called
register-svc.ps1, and
nssm.exe binary into c:\k.
PS C:\k> wget https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/flannel/register-svc.ps1 -o c:\k\register-svc.ps1
I did have problems with the default script as it seems to reference an incorrect pause image and have a problem with the allow-privileged statement as indicated above. To fix that, edit register-svc.ps1 and under the kubelet registration replace the --pod-infra-container=kubeletwin/pause with mcr.microsoft.com/k8s/core/pause:1.0.0 and remove --allow-privileged=true. It should be line 25 and will look like this when you're done;
.\nssm.exe set $KubeletSvc AppParameters --hostname-override=$Hostname --v=6 --pod-infra-container-image=mcr.microsoft.com/k8s/core/pause:1.0.0 --resolv-conf="" --enable-debugging-handlers --cluster-dns=$KubeDnsServiceIP --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir=$LogDir --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config
Once that's fixed, you can register your services with this command where ManagementIP is the windows node IP.
PS C:\k> .\register-svc.ps1 -ManagementIP <windows_node_ip> -NetworkMode overlay
You should see the services registered and running. If you get errors like these, it's probably because register-svc.ps1 wasn't edited correctly.
Service "flanneld" installed successfully!
Set parameter "AppParameters" for service "flanneld".
Set parameter "AppEnvironmentExtra" for service "flanneld".
Set parameter "AppDirectory" for service "flanneld".
flanneld: START: The operation completed successfully.
Service "kubelet" installed successfully!
Set parameter "AppParameters" for service "kubelet".
Set parameter "AppDirectory" for service "kubelet".
kubelet: Unexpected status SERVICE_PAUSED in response to START control.
Service "kube-proxy" installed successfully!
Set parameter "AppDirectory" for service "kube-proxy".
Set parameter "AppParameters" for service "kube-proxy".
Set parameter "DependOnService" for service "kube-proxy".
kube-proxy: START: The operation completed successfully.
If you've already added the services and need to make changes, you can do that by either editing the service or removing them and re-registering with the commands listed below.
PS C:\k> .\nssm.exe edit kubelet
PS C:\k> .\nssm.exe edit kube-proxy
PS C:\k> .\nssm.exe edit flanneld
PS C:\k> .\nssm.exe remove kubelet confirm
PS C:\k> .\nssm.exe remove kube-proxy confirm
PS C:\k> .\nssm.exe remove flanneld confirm
Reboot to verify your node re-registers with kubernetes correctly and that you can deploy a pod using the test above.
Deleting/Re-adding A Windows Node
If you delete a windows node, such as with
kubectl delete node <node_name>, adding it is pretty easy. Because the windows nodes have the kubernetes config file they re-register automatically on every service start. You might need to remove existing flannel configuration files and then reboot.
PS C:\k> Remove-Item C:\k\SourceVip.json
PS C:\k> Remove-Item C:\k\SourceVipRequest.json
PS C:\k> Restart-Computer
Broken Kubernetes Things With Windows Nodes
Pretty much everything in broken. You'll be able to deploy a windows container to a windows node using the node selector spec entry like we did when patching the daemonsets above. Just place
windows as the OS type instead of linux. Here's a list of things that are broken which I'll update when possible:
- Persistent Volumes - you need to ensure the node is registered properly with vsphere or nothing will be able to use a persistent volume. This is because vsphere ensures all nodes can see a datastore without making a distinction between windows and linux. I can get a PV to appear on a windows node but I can't get it to initialize properly
- Node Ports - this is a documented limitation, you can't access a node port service from the node hosting the pod. Strange, yes, but you should be able to use any linux nodes as an entry for any windows pods
- Load Balancer - With version 0.8.0 it includes the os selector patch, and it should work forwarding connecting through available linux nodes but I haven't had any success yet
- DNS - untested as of yet because of load balancer problems
- Logging - should be possible as fluent bit has beta support for windows but untested yet
- Fluent bit does have some documentation to install under windows, maybe under the node itself as there isn't a docker container readily available, but none of the links work. Perhaps not yet (Aug 2019)
- Monitoring - should also be possible using WMI exporter rather than a node exporter, again, untested at this time
