272 lines
5.9 KiB
Markdown
272 lines
5.9 KiB
Markdown
# 🧠 DevOps Infra Stack --- Proxmox + Ceph + Kubernetes + DNS
|
||
|
||
Fully automated self-hosted infrastructure.
|
||
|
||
|
||
|
||
The project deploys:
|
||
- Proxmox infrastructure
|
||
- Golden VM templates via
|
||
Packer - VM provisioning via Terraform
|
||
- Hardened nodes (SSH, nftables,
|
||
fail2ban)
|
||
- DNS (PowerDNS)
|
||
- NTP (chrony hierarchy)
|
||
- Ceph cluster
|
||
- Kubernetes cluster
|
||
- K8s apps (MetalLB, ingress, postgres operator,
|
||
valkey)
|
||
|
||
Everything is deployed via Makefile + Ansible + Terraform + Packer.
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 🏗 Architecture
|
||
|
||
Infrastructure components:
|
||
|
||
- Proxmox host (bare metal)
|
||
- LXC packer builder
|
||
- Golden VM templates
|
||
- VM nodes:
|
||
- DNS
|
||
- NTP
|
||
- Ceph (3 nodes)
|
||
- Kubernetes master
|
||
- Kubernetes worker
|
||
- K8s stack:
|
||
- MetalLB
|
||
- nginx ingress
|
||
- Crunchy Postgres Operator
|
||
- Valkey (Redis alternative)
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 📦 Technology Stack
|
||
|
||
- Proxmox VE
|
||
- Terraform
|
||
- Ansible
|
||
- Packer
|
||
- Docker + Docker Compose (for DNS)
|
||
- Ceph
|
||
- Kubernetes
|
||
- Helm
|
||
- PowerDNS
|
||
- Chrony
|
||
- nftables + fail2ban hardening
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 🚀 Full Infrastructure Bootstrap
|
||
|
||
Main entrypoint:
|
||
|
||
``` bash
|
||
make -f bootstrap.mk
|
||
```
|
||
|
||
It will execute:
|
||
|
||
1. VM creation
|
||
2. Hardening
|
||
3. DNS setup
|
||
4. NTP setup
|
||
5. Ceph cluster
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 🧱 Deployment Stages
|
||
|
||
## 0. Create LXC + Packer
|
||
|
||
``` bash
|
||
make -f 00_create_and_setup_lxc_container_with_packer.mk
|
||
```
|
||
|
||
- Download LXC template
|
||
- Create LXC via Terraform
|
||
- Install packer inside LXC
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
## 1. Golden VM template
|
||
|
||
``` bash
|
||
make -f 01_create_vm_golden_template.mk
|
||
```
|
||
|
||
- Download ISO
|
||
- Upload packer config
|
||
- Build golden image
|
||
- Shut down packer LXC
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
## 2. Create VMs
|
||
|
||
``` bash
|
||
make -f 02_create_vms.mk
|
||
```
|
||
|
||
- Enable cloud-init snippets
|
||
- Terraform creates VMs
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
## 3. Harden nodes
|
||
|
||
``` bash
|
||
make -f 03_harden_vms.mk
|
||
```
|
||
|
||
- Remove packer user
|
||
- SSH hardening
|
||
- nftables
|
||
- fail2ban
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
## 4. DNS
|
||
|
||
``` bash
|
||
make -f 04_setup_dns.mk
|
||
```
|
||
|
||
- PowerDNS install
|
||
- Zones + records via Terraform
|
||
- systemd-resolved config
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
## 5. NTP
|
||
|
||
``` bash
|
||
make -f 05_setup_ntp.mk
|
||
```
|
||
|
||
Hierarchy:
|
||
- edge NTP server (proxmox)
|
||
- core NTP server
|
||
- clients use core NTP server
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
## 6. Ceph
|
||
|
||
``` bash
|
||
make -f 06_setup_ceph.mk
|
||
```
|
||
|
||
- install
|
||
- bootstrap
|
||
- share keys
|
||
- cluster init
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
## 7. Kubernetes
|
||
|
||
``` bash
|
||
make -f 07_setup_k8s.mk
|
||
```
|
||
|
||
After installation:
|
||
|
||
``` bash
|
||
ssh user@k8smasternode -p 10525
|
||
```
|
||
|
||
Replace cluster endpoint with localhost tunnel.
|
||
|
||
Then:
|
||
|
||
``` bash
|
||
terraform apply -target=module.metallb_helm
|
||
terraform apply -target=module.crunchy_operator
|
||
terraform apply
|
||
```
|
||
|
||
Get credentials:
|
||
|
||
``` bash
|
||
# postgres
|
||
kubectl -n postgres-operator get secret hippo-pguser-gitlab -o jsonpath='{.data.user}' | base64 -d; echo
|
||
|
||
# valkey
|
||
kubectl -n valkey get secret valkey-users -o jsonpath='{.data.default}' | base64 -d; echo
|
||
```
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 📁 Project Structure
|
||
|
||
ansible/
|
||
terraform/
|
||
packer/
|
||
makefiles/
|
||
bootstrap.mk
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 🔐 Requirements
|
||
|
||
Before running:
|
||
|
||
- SSH access to Proxmox
|
||
- Proxmox API token
|
||
- terraform.tfvars filled
|
||
- inventory.ini filled
|
||
- kubeconfig path specified
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 🔭 Planned Services & Future Stack
|
||
|
||
The following services are planned for the next deployment stages:
|
||
|
||
- **NetBird** --- internal VPN mesh network (currently working on this
|
||
stage)
|
||
- **Keycloak** --- unified authentication and identity provider across
|
||
services
|
||
- **Monitoring stack (Grafana, Loki, Prometheus, Trickster)** ---
|
||
monitoring and observability tools\
|
||
*(previously deployed, but not yet integrated into this project)*
|
||
- **FreeIPA** --- centralized user and identity management inside
|
||
operating systems
|
||
- **Vault** --- centralized storage for passwords, tokens, and
|
||
operational credentials
|
||
- **OpenNebula** --- additional virtualization layer for providing
|
||
user VM spaces\
|
||
*(similar to AWS EC2 for internal infrastructure)*
|
||
- **Nextcloud + LibreOffice** --- Google Cloud alternative for
|
||
collaborative document editing\
|
||
*(Nextcloud deployed previously, but not yet within this project)*
|
||
- **Element + Matrix** --- Telegram-like communication platform\
|
||
*(stack deployed previously, but not yet integrated into this
|
||
project)*
|
||
- **LLM (local language model)** --- neural network for text
|
||
processing\
|
||
*(GPT‑2 already tested; LLaMA 7B planned as MVP depending on
|
||
available resources)*\
|
||
Future usage:
|
||
- LibreOffice document assistant
|
||
- Matrix/Element chatbot integration
|
||
- **Kafka** --- message queue layer between LibreOffice, Element, and
|
||
LLM services\
|
||
Ensures reliable request delivery and acts as a service integration
|
||
layer
|
||
- **OCR tools** --- document recognition and conversion pipeline\
|
||
Enables transforming documents into formats suitable for LLM
|
||
processing and search
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
# 🧠 Project Idea
|
||
|
||
Self-hosted cloud platform, own mini cloud. Fully autonomous infrastructure.
|
||
|
||
# 👤 Author
|
||
|
||
Aleksandr Hrankin
|