private-ai-platform/README.md

🧠 DevOps Infra Stack --- Proxmox + Ceph + Kubernetes + DNS

Fully automated self-hosted infrastructure.

The project deploys:

• Proxmox infrastructure
• Golden VM templates via Packer - VM provisioning via Terraform
• Hardened nodes (SSH, nftables, fail2ban)
• DNS (PowerDNS)
• NTP (chrony hierarchy)
• Ceph cluster
• Kubernetes cluster
• K8s apps (MetalLB, ingress, postgres operator, valkey)

Everything is deployed via Makefile + Ansible + Terraform + Packer.

---

🏗 Architecture

Infrastructure components:

• Proxmox host (bare metal)
• LXC packer builder
• Golden VM templates
• VM nodes:
  • DNS
  • NTP
  • Ceph (3 nodes)
  • Kubernetes master
  • Kubernetes worker
• K8s stack:
  • MetalLB
  • nginx ingress
  • Crunchy Postgres Operator
  • Valkey (Redis alternative)

---

📦 Technology Stack

• Proxmox VE
• Terraform
• Ansible
• Packer
• Docker + Docker Compose (for DNS)
• Ceph
• Kubernetes
• Helm
• PowerDNS
• Chrony
• nftables + fail2ban hardening

---

🚀 Full Infrastructure Bootstrap

Main entrypoint:

make -f bootstrap.mk

It will execute:

1. VM creation
2. Hardening
3. DNS setup
4. NTP setup
5. Ceph cluster

---

🧱 Deployment Stages

0. Create LXC + Packer

make -f 00_create_and_setup_lxc_container_with_packer.mk

• Download LXC template
• Create LXC via Terraform
• Install packer inside LXC

---

1. Golden VM template

make -f 01_create_vm_golden_template.mk

• Download ISO
• Upload packer config
• Build golden image
• Shut down packer LXC

---

2. Create VMs

make -f 02_create_vms.mk

• Enable cloud-init snippets
• Terraform creates VMs

---

3. Harden nodes

make -f 03_harden_vms.mk

• Remove packer user
• SSH hardening
• nftables
• fail2ban

---

4. DNS

make -f 04_setup_dns.mk

• PowerDNS install
• Zones + records via Terraform
• systemd-resolved config

---

5. NTP

make -f 05_setup_ntp.mk

Hierarchy:

• edge NTP server (proxmox)
• core NTP server
• clients use core NTP server

---

6. Ceph

make -f 06_setup_ceph.mk

• install
• bootstrap
• share keys
• cluster init

---

7. Kubernetes

make -f 07_setup_k8s.mk

After installation:

ssh user@k8smasternode -p 10525

Replace cluster endpoint with localhost tunnel.

Then:

terraform apply -target=module.metallb_helm
terraform apply -target=module.crunchy_operator
terraform apply

Get credentials:

# postgres
kubectl -n postgres-operator get secret hippo-pguser-gitlab -o jsonpath='{.data.user}' | base64 -d; echo

# valkey
kubectl -n valkey get secret valkey-users -o jsonpath='{.data.default}' | base64 -d; echo

---

📁 Project Structure

ansible/
terraform/
packer/
makefiles/
bootstrap.mk

---

🔐 Requirements

Before running:

• SSH access to Proxmox
• Proxmox API token
• terraform.tfvars filled
• inventory.ini filled
• kubeconfig path specified

---

🔭 Planned Services & Future Stack

The following services are planned for the next deployment stages:

• NetBird --- internal VPN mesh network (currently working on this stage)
• Keycloak --- unified authentication and identity provider across services
• Monitoring stack (Grafana, Loki, Prometheus, Trickster) --- monitoring and observability tools
  (previously deployed, but not yet integrated into this project)
• FreeIPA --- centralized user and identity management inside operating systems
• Vault --- centralized storage for passwords, tokens, and operational credentials
• OpenNebula --- additional virtualization layer for providing user VM spaces
  (similar to AWS EC2 for internal infrastructure)
• Nextcloud + LibreOffice --- Google Cloud alternative for collaborative document editing
  (Nextcloud deployed previously, but not yet within this project)
• Element + Matrix --- Telegram-like communication platform
  (stack deployed previously, but not yet integrated into this project)
• LLM (local language model) --- neural network for text processing
  (GPT‑2 already tested; LLaMA 7B planned as MVP depending on available resources)
  Future usage:
  • LibreOffice document assistant
  • Matrix/Element chatbot integration
• Kafka --- message queue layer between LibreOffice, Element, and LLM services
  Ensures reliable request delivery and acts as a service integration layer
• OCR tools --- document recognition and conversion pipeline
  Enables transforming documents into formats suitable for LLM processing and search

---

🧠 Project Idea

Self-hosted cloud platform, own mini cloud. Fully autonomous infrastructure.

👤 Author

Aleksandr Hrankin