Skip to content

Commit 7f985f3

Browse files
alexellisalexellis
committed
Initial pro readme
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
1 parent 18852ec commit 7f985f3

File tree

1 file changed

+243
-100
lines changed

1 file changed

+243
-100
lines changed

‎README.md‎

Lines changed: 243 additions & 100 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,12 @@ The tool is written in Go and is cross-compiled for Linux, Windows, MacOS and ev
88

99
How do you say it? Ketchup, as in tomato.
1010

11+
**Introducing K3sup Pro 🎉**
12+
13+
Whilst the CE edition is ideal for experimentation, we built `k3sup pro` for an IaaC/GitOps experience.
14+
15+
`k3sup pro` adds a `plan` and `apply` command to automate installations both small and large - running in parallel. The plan file can be customised and retained in Git for maintenance and updates.
16+
1117
[![Sponsor this](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&link=https://github.com/sponsors/alexellis)](https://github.com/sponsors/alexellis)
1218
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
1319
[![build](https://github.com/alexellis/k3sup/actions/workflows/build.yaml/badge.svg)](https://github.com/alexellis/k3sup/actions/workflows/build.yaml)
@@ -25,10 +31,15 @@ How do you say it? Ketchup, as in tomato.
2531
- [Demo 📼](#demo-)
2632
- [Usage ✅](#usage-)
2733
- [Pre-requisites for k3sup servers and agents](#pre-requisites-for-k3sup-servers-and-agents)
34+
- [K3sup pro](#k3sup-pro)
35+
- [K3sup plan / apply for automation and large installations](#k3sup-plan--apply-for-automation-and-large-installations)
36+
- [Rapid uninstallation / reset with `k3sup pro uninstall`](#rapid-uninstallation--reset-with-k3sup-pro-uninstall)
37+
- [K3sup pro exec](#k3sup-pro-exec)
38+
- [Get a kubeconfig from an existing installation](#get-a-kubeconfig-from-an-existing-installation)
39+
- [K3sup Community Edition (CE)](#k3sup-community-edition-ce)
2840
- [👑 Setup a Kubernetes *server* with `k3sup`](#-setup-a-kubernetes-server-with-k3sup)
2941
- [Checking if a cluster is ready](#checking-if-a-cluster-is-ready)
3042
- [Merging clusters into your KUBECONFIG](#merging-clusters-into-your-kubeconfig)
31-
- [Getting a kubeconfig from an existing installation](#getting-a-kubeconfig-from-an-existing-installation)
3243
- [😸 Join some agents to your Kubernetes server](#-join-some-agents-to-your-kubernetes-server)
3344
- [Use your hardware authentication / 2FA or SSH Agent](#use-your-hardware-authentication--2fa-or-ssh-agent)
3445
- [K3sup plan for automation](#k3sup-plan-for-automation)
@@ -74,7 +85,7 @@ K3sup runs from your local machine, without ever having to log into a remote ser
7485
* Bootstrap Kubernetes with k3s onto any VM with `k3sup install` - either manually, during CI or through `cloud-init`
7586
* Get from zero to `kubectl` with `k3s` on bare-metal, Raspberry Pi (RPi), VMs, AWS EC2, Google Cloud, DigitalOcean, Civo, Linode, Scaleway, and others
7687
* Build a Highly-Available (HA), multi-master (server) cluster
77-
* Fetch the KUBECONFIG from an existing cluster with `k3sup get-config`
88+
* Fetch the KUBECONFIG from an existing cluster with `k3sup pro get-config`
7889
* Join nodes into an existing `k3s` cluster with `k3sup join`
7990
* Build a massive cluster for automation and scale-out testing using `k3sup plan` and a JSON file with IP addresses
8091

@@ -98,7 +109,7 @@ k3sup --help
98109

99110
Windows users can use `k3sup install` and `k3sup join` with a normal "Windows command prompt".
100111

101-
## Demo 📼
112+
## Demo of K3sup CE📼
102113

103114
In the demo I install Kubernetes (`k3s`) onto two separate machines and get my `kubeconfig` downloaded to my laptop each time in around one minute.
104115

@@ -129,6 +140,235 @@ In most circumstances, cloud images for Ubuntu and other distributions will not
129140

130141
As an alternative, if you only need a single server you can log in interactively and run `k3sup install --local` instead of using SSH.
131142

143+
## K3sup Pro
144+
145+
K3sup Pro is available as a free extra to [GitHub Sponsors](https://github.com/sponsors/alexellis) on a 25 USD / mo tier and higher to individuals. A separate option to purchase on an annual basis is available via email for commercial use. Contact [contact@openfaas.com](mailto:contact@openfaas.com) for more.
146+
147+
* `plan` - take one or more JSON files and generate a YAML plan for a HA installation of K3s
148+
* `apply` - run the installation in parallel, optionally pre-downloading the K3s binary and copying it via SSH beforehand
149+
* `exec` - run a command on all nodes in the cluster
150+
* `get-config` - get a kubeconfig from an existing installation
151+
* `uninstall` - uninstall k3s from all nodes in the cluster in parallel
152+
153+
The `--predownload` flag for `k3sup pro apply` is the first step towards a fully airgapped solution, and reduces bandwidth whilst speeding up installation.
154+
155+
Walkthrough of plan, apply, get-config and exec:
156+
157+
[![asciicast](https://asciinema.org/a/HpHLjHFjXXAJ3uHiD4jKgvvaN.svg)](https://asciinema.org/a/HpHLjHFjXXAJ3uHiD4jKgvvaN)
158+
159+
### K3sup `plan` / `apply` for automation and large installations
160+
161+
The `k3sup pro plan` command reads a set of JSON files containing your hosts, and will generate a YAML plan file that you can edit to customize the installation.
162+
163+
Example input file:
164+
165+
```json
166+
[
167+
{
168+
"hostname": "k3s-server-1",
169+
"ip": "192.168.129.138"
170+
},
171+
{
172+
"hostname": "k3s-server-2",
173+
"ip": "192.168.129.128"
174+
},
175+
{
176+
"hostname": "k3s-server-3",
177+
"ip": "192.168.129.131"
178+
},
179+
{
180+
"hostname": "k3s-agent-1",
181+
"ip": "192.168.129.130"
182+
},
183+
{
184+
"hostname": "k3s-agent-2",
185+
"ip": "192.168.129.127"
186+
}
187+
]
188+
```
189+
190+
The following will create 1x primary server, with 2x additional servers within a HA etcd cluster, the last two nodes will be added as agents.
191+
192+
```bash
193+
k3sup pro plan ./n100.json ./n200.json \
194+
--user ubuntu \
195+
--servers 3 \
196+
--svclb=false \
197+
--server-extra-args "--disable traefik" \
198+
--agent-extra-args "--node-label worker=true"
199+
```
200+
201+
Example plan.yaml file:
202+
203+
```yaml
204+
version:
205+
k3s_channel: stable
206+
server_options:
207+
user: ubuntu
208+
ssh_port: 22
209+
k3s_extra_args: --disable traefik
210+
parallel: 5
211+
traefik: true
212+
agent_options:
213+
k3s_extra_args: --node-label worker=true
214+
hosts:
215+
- name: k3s-1
216+
role: server
217+
host: 192.168.138.2
218+
architecture: x86_64
219+
- name: k3s-2
220+
role: server
221+
host: 192.168.138.3
222+
architecture: x86_64
223+
- name: k3s-3
224+
role: server
225+
host: 192.168.138.4
226+
architecture: x86_64
227+
- name: k3s-agent-1
228+
role: agent
229+
host: 192.168.137.2
230+
architecture: x86_64
231+
- name: k3s-agent-2
232+
role: agent
233+
host: 192.168.137.3
234+
architecture: x86_64
235+
- name: k3s-agent-3
236+
role: agent
237+
host: 192.168.137.4
238+
architecture: x86_64
239+
240+
```
241+
242+
The YAML plan file can be edited and committed to Git for maintenance and future upgrades.
243+
244+
Then when you're ready to install, you can run `k3sup pro apply` to install in parallel.
245+
246+
The `--predownload` flag will download the k3s binary to your local machine, then copy it over SSH to each host to speed up the installation.
247+
248+
The `--parallel` flag sets how many installation steps to run at the same time.
249+
250+
```bash
251+
k3sup pro apply \
252+
--predownload \
253+
--parallel 10
254+
```
255+
256+
You can also get hold of your kubeconfig with `k3sup pro get-config` and then use `kubectl` to access your cluster.
257+
258+
Merge it into your main KUBECONFIG file:
259+
260+
```bash
261+
k3sup pro get-config \
262+
--local-path $HOME/.kube/config \
263+
--context my-k3s \
264+
--merge
265+
```
266+
267+
Or set up a local file:
268+
269+
```bash
270+
k3sup pro get-config \
271+
--local-path ./kubeconfig
272+
273+
export KUBECONFIG=`pwd`/kubeconfig
274+
```
275+
276+
Watch a demo with dozens of Firecracker VMs: [Testing Kubernetes at Scale with bare-metal](https://youtu.be/o4UxRw-Cc8c)
277+
278+
## Rapid uninstallation / reset with `k3sup pro uninstall`
279+
280+
The `k3sup pro uninstall` command will uninstall k3s from all nodes in the cluster.
281+
282+
If you have a plan YAML file, the username, SSH ports, and key files will all be read from the file, and the uninstallation will be performed in order and in parallel. Removing the agents first, then any additional servers, and finally the primary server.
283+
```bash
284+
k3sup pro uninstall
285+
```
286+
287+
If you only have devices JSON files, you may also need a `--user` and / or `--ssh-key` flag.
288+
289+
```bash
290+
k3sup pro uninstall \
291+
--user ubuntu \
292+
--ssh-key ~/.ssh/id_rsa
293+
```
294+
295+
## K3sup `pro exec` - run a command everywhere
296+
297+
The `k3sup pro exec` command allows you to run a command on all nodes in the cluster. You can specify `--servers` or `--agents` to run the command on only the servers or agents.
298+
299+
Run on all nodes:
300+
301+
```bash
302+
k3sup pro exec \
303+
"free -h"
304+
```
305+
306+
Only on servers:
307+
308+
```bash
309+
k3sup pro exec \
310+
--servers \
311+
"sudo journalctl -u k3s -n 100"
312+
```
313+
314+
Only on agents:
315+
316+
```bash
317+
k3sup pro exec \
318+
--agents \
319+
"sudo journalctl -u k3s-agent -n 100"
320+
```
321+
322+
### K3sup `pro get-config` - work with an existing cluster
323+
324+
The `k3sup pro get-config` command allows you to retrieve kubeconfig files from existing K3s installations without performing any installation steps. This is useful when you already have K3s running and just need to access the cluster configuration.
325+
326+
You can also use it if you initially created a local `./kubeconfig` file but now want to merge it under a meaningful context name to your main `$HOME/.kube/config` file.
327+
328+
Get kubeconfig from a remote server:
329+
330+
```bash
331+
k3sup pro get-config \
332+
--host 192.168.0.100 \
333+
--user ubuntu \
334+
--local-path ./kubeconfig
335+
```
336+
337+
Get kubeconfig from a local installation:
338+
339+
```bash
340+
k3sup pro get-config --local
341+
```
342+
343+
Merge kubeconfig into your main KUBECONFIG file:
344+
345+
```bash
346+
k3sup pro get-config \
347+
--host 192.168.0.100 \
348+
--user ubuntu \
349+
--merge \
350+
--local-path $HOME/.kube/config \
351+
--context my-remote-cluster
352+
```
353+
354+
Use a custom SSH key:
355+
356+
```bash
357+
k3sup pro get-config \
358+
--host 192.168.0.100 \
359+
--user ubuntu \
360+
--ssh-key $HOME/.ssh/my-key \
361+
--local-path ./kubeconfig
362+
```
363+
364+
If you do not have `k3sup pro` yet, you can also use `k3sup install` with the `--skip-install` flag.
365+
366+
## K3sup Community Edition (CE)
367+
368+
The CE edition of K3sup is available to all users for free, and the code is licensed under the MIT license so you can also adapt it for your own use, or contribute back to the project.
369+
370+
The CE edition has been available since 2019 and has been used by many different people to learn about Kubernetes, and to build their own clusters using imperative bash commands.
371+
132372
### 👑 Setup a Kubernetes *server* with `k3sup`
133373

134374
You can setup a server and stop here, or go on to use the `join` command to add some "agents" aka `nodes` or `workers` into the cluster to expand its compute capacity.
@@ -243,48 +483,6 @@ k3sup install \
243483

244484
Here we set a context of `my-k3s` and also merge into our main local `KUBECONFIG` file, so we could run `kubectl config use-context my-k3s` or `kubectx my-k3s`.
245485

246-
### Getting a kubeconfig from an existing installation
247-
248-
The `k3sup get-config` command allows you to retrieve kubeconfig files from existing K3s installations without performing any installation steps. This is useful when you already have K3s running and just need to access the cluster configuration.
249-
250-
You can also use it if you initially created a local `./kubeconfig` file but now want to merge it under a meaningful context name to your main `$HOME/.kube/config` file.
251-
252-
Get kubeconfig from a remote server:
253-
254-
```bash
255-
k3sup get-config \
256-
--host 192.168.0.100 \
257-
--user ubuntu \
258-
--local-path ./kubeconfig
259-
```
260-
261-
Get kubeconfig from a local installation:
262-
263-
```bash
264-
k3sup get-config --local
265-
```
266-
267-
Merge kubeconfig into your main KUBECONFIG file:
268-
269-
```bash
270-
k3sup get-config \
271-
--host 192.168.0.100 \
272-
--user ubuntu \
273-
--merge \
274-
--local-path $HOME/.kube/config \
275-
--context my-remote-cluster
276-
```
277-
278-
Use a custom SSH key:
279-
280-
```bash
281-
k3sup get-config \
282-
--host 192.168.0.100 \
283-
--user ubuntu \
284-
--ssh-key $HOME/.ssh/my-key \
285-
--local-path ./kubeconfig
286-
```
287-
288486
### 😸 Join some agents to your Kubernetes server
289487

290488
Let's say that you have a server, and have already run the following:
@@ -331,61 +529,6 @@ Now run any `k3sup` command, and your SSH key will be requested from the ssh-age
331529

332530
You can also specify an SSH key with `--ssh-key` if you want to use a specific key-pair.
333531

334-
### K3sup plan for automation
335-
336-
A new command was added to k3sup to help with automating large amounts of nodes.
337-
338-
`k3sup plan` reads a JSON input file containing hosts, and will generate an installation command for a number of servers and agents.
339-
340-
Example input file:
341-
342-
```json
343-
[
344-
{
345-
"hostname": "node-a-1",
346-
"ip": "192.168.129.138"
347-
},
348-
{
349-
"hostname": "node-a-2",
350-
"ip": "192.168.129.128"
351-
},
352-
{
353-
"hostname": "node-a-3",
354-
"ip": "192.168.129.131"
355-
},
356-
{
357-
"hostname": "node-a-4",
358-
"ip": "192.168.129.130"
359-
},
360-
{
361-
"hostname": "node-a-5",
362-
"ip": "192.168.129.127"
363-
}
364-
]
365-
```
366-
367-
The following will create 1x primary server, with 2x additional servers within a HA etcd cluster, the last two nodes will be added as agents:
368-
369-
```bash
370-
k3sup plan \
371-
devices.json \
372-
--user ubuntu \
373-
--servers 3 \
374-
--server-k3s-extra-args "--disable traefik" \
375-
--background > bootstrap.sh
376-
```
377-
378-
Then make the file executable and run it:
379-
380-
```bash
381-
chmod +x bootstrap.sh
382-
./bootstrap.sh
383-
```
384-
385-
Watch a demo with dozens of Firecracker VMs: [Testing Kubernetes at Scale with bare-metal](https://youtu.be/o4UxRw-Cc8c)
386-
387-
The initial version of `k3sup plan` has a reduced set of flags. Flags such as `--k3s-version` and `--datastore` are not available, but feel free to propose an issue with what you need.
388-
389532
### Create a multi-master (HA) setup with external SQL
390533

391534
The easiest way to test out k3s' multi-master (HA) mode with external storage, is to set up a Mysql server using DigitalOcean's managed service.

0 commit comments

Comments
 (0)