This guide covers the essential steps for troubleshooting Proxmox boot failures and network recovery issues. Whether you’re dealing with a Proxmox VE node that won’t boot, network interfaces that fail to initialize, or connectivity problems after an update, this visual reference provides a structured approach to diagnosing and resolving common Proxmox infrastructure issue
Today I completed a BIOS update on my MSI MAG X570S Tomahawk Max WiFi motherboard, upgrading from the original BIOS Version 1.00 (dated 07/06/2021) to the latest Version 1.D1 (7D54v1D1, dated 09/19/2025). This update was performed using MSI’s M-FLASH utility as part of my Proxmox homelab infrastructure maintenance.
The visual guide above outlines the complete BIOS update process using MSI’s M-FLASH utility. Here’s a summary of the key steps performed:
For detailed step-by-step documentation of this BIOS update process, visit the NetworkThinkTank-Labs GitHub repository. This motherboard serves as the foundation for my Proxmox homelab running GPU passthrough with an NVIDIA RTX 3060 Ti for AI workloads.
Some days in the homelab are quiet — a config tweak here, a firmware update there. And then there are days like today. April 29, 2026, turned into a full-blown infrastructure marathon: eight distinct projects spanning networking, virtualization, AI deployment, storage management, and documentation. Here is a complete rundown of everything that got done.
Documentation is the backbone of any serious homelab. Today I pushed 565 lines of new documentation across multiple GitHub repositories. This included updated READMEs, configuration guides, topology diagrams, and step-by-step walkthroughs. Every project in my lab now has proper technical documentation that anyone can follow to replicate the setup. If it is not documented, it did not happen — and today, it all got documented.
My EVE-NG CCNA lab got a major overhaul. The lab now contains 29 active nodes, covering routing, switching, and network services. This includes Cisco IOS routers and switches configured for OSPF, EIGRP, BGP, VLANs, STP, ACLs, NAT, and DHCP. The lab features API troubleshooting support, Proxmox migration readiness via Terraform and Ansible, and qcow2 image management. Whether you are studying for the CCNA or just want a robust network simulation environment, this lab has you covered.
Earlier this month, I published a comprehensive 2,943-word blog post on the Network ThinkTank blog covering how to self-host AI on a Proxmox homelab with Ollama and Open WebUI. Today’s writing adds to that momentum. Consistent publishing is key to building a knowledge base that helps both myself and the broader homelab community.
Local AI is the future of privacy-conscious computing. Today I deployed four Ollama models on my Proxmox homelab, running inference entirely on local hardware. The models are served through Open WebUI, giving me a polished ChatGPT-like interface without any data leaving my network. No API keys, no cloud dependency, no privacy concerns — just pure local LLM power. The models cover different use cases from general conversation to code generation and technical assistance.
OpenClaw, an AI agent framework, was deployed and configured in the homelab today. This adds autonomous AI agent capabilities to the infrastructure, enabling task automation and intelligent workflows. The deployment involved setting up the agent runtime, configuring API endpoints, and testing basic agent interactions. This is a step toward building a more intelligent, self-managing homelab environment.
A fresh Windows Server virtual machine was built from scratch today. This VM will serve as a core infrastructure component for Active Directory, DNS, DHCP, and Group Policy management. The build process included creating the VM in the hypervisor, installing the OS, applying initial configurations, and setting up remote management. Having a Windows Server in the lab opens up enterprise-grade identity and access management capabilities.
Storage hygiene is critical in any homelab environment. Today’s cleanup operation freed approximately 20GB of space on the NAS. This involved removing outdated VM snapshots, clearing old ISO images, purging stale Docker volumes, and archiving completed project files. A clean NAS is a happy NAS — and with 20GB reclaimed, there is plenty of room for new projects.
The UniFi Network Server was installed and configured today, bringing enterprise-grade network management to the homelab. This provides centralized control over UniFi access points, switches, and security gateways. The installation included setting up the controller software, adopting network devices, configuring wireless networks, and establishing monitoring dashboards. With UniFi in place, the entire network infrastructure can be managed from a single pane of glass.
Eight projects. One day. From AI deployments to network labs, from storage cleanup to documentation — today was a masterclass in homelab productivity. Every one of these projects builds on the others, creating a more capable, better-documented, and more resilient infrastructure.
The key takeaway? Documentation makes everything better. By writing things down — both in GitHub repos and blog posts — I am building a knowledge base that pays dividends every time I need to troubleshoot, replicate, or expand my setup.
If you are running a homelab, I encourage you to document your work, share your configs, and keep building. The community is stronger when we share what we learn.
Until next time — keep labbing.
Follow the Network ThinkTank blog for more homelab guides, networking tutorials, and infrastructure deep-dives. Check out the companion GitHub repositories at github.com/jczaldivar71 for configs, scripts, and technical documentation.
By NetworkThinkTank | April 23, 2026
I got tired of sending my data to cloud AI services. Every prompt I typed into ChatGPT or Claude was being stored, analyzed, and used for training. For personal questions, code snippets with API keys, and private brainstorming sessions, that never sat well with me.
So I built my own. A fully self-hosted AI assistant running on my Proxmox homelab, powered by Ollama for local LLM inference and Open WebUI for a polished ChatGPT-like interface. The models run on my own NVIDIA GPU, the data stays on my NAS, and nothing leaves my network.
This guide walks you through exactly how I did it – from VM creation to pulling your first model and chatting with it through a clean web interface. If you have a Proxmox server and a spare GPU, you can have this running in an afternoon.
Here is what you need before starting:
The stack looks like this:
All AI processing happens locally on the GPU inside the VM. Open WebUI provides the browser-based chat interface and connects to Ollama’s API on the backend. The NAS stores all model files and conversation data so nothing is lost if the VM needs rebuilding.
First, create a new VM in Proxmox optimized for AI workloads.
Edit GRUB configuration:
nano /etc/default/grubFor AMD CPUs, set:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"For Intel CPUs, set:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"Update GRUB and reboot:
update-grubrebootAdd VFIO modules to /etc/modules:
vfiovfio_iommu_type1vfio_pcivfio_virqfdBlacklist the NVIDIA drivers on the host:
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.confecho "blacklist nvidia" >> /etc/modprobe.d/blacklist.confFind your GPU’s PCI IDs:
lspci -nn | grep NVIDIAExample output:
27:00.0 VGA compatible controller [0300]: NVIDIA [10de:2489]27:00.1 Audio device [0403]: NVIDIA [10de:228b]Bind to VFIO:
echo "options vfio-pci ids=10de:2489,10de:228b" > /etc/modprobe.d/vfio.confUpdate initramfs and reboot:
update-initramfs -u -k allrebootIn the Proxmox web UI:
VM 205 > Hardware > Add > PCI Device- Select your NVIDIA GPU- Check "All Functions"- Check "ROM-Bar"- Check "PCI-Express"- Set "Primary GPU" if this VM has no other displayAfter booting the VM, run:
lspci | grep -i nvidiaYou should see your GPU listed. If not, check IOMMU groups and VFIO binding on the Proxmox host.
With the GPU visible inside the VM, install the required software.
sudo apt update && sudo apt upgrade -ysudo rebootsudo apt install -y nvidia-driver-535 nvidia-utils-535sudo rebootnvidia-smiExpected output shows your RTX 3060 Ti with driver version and CUDA version. If nvidia-smi fails, check that the GPU passthrough is configured correctly on the Proxmox host.
sudo apt install -y ca-certificates curl gnupgsudo install -m 0755 -d /etc/apt/keyringscurl -fsSL https://download.docker.com/linux/ubuntu/gpg | \ sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpgecho "deb [arch=$(dpkg --print-architecture) \ signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/nullsudo apt updatesudo apt install -y docker-ce docker-ce-cli containerd.io \ docker-buildx-plugin docker-compose-pluginsudo usermod -aG docker $USERcurl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \ sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpgcurl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.listsudo apt updatesudo apt install -y nvidia-container-toolkitsudo nvidia-ctk runtime configure --runtime=dockersudo systemctl restart dockerdocker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smiIf this shows your GPU info inside the container, you are ready to deploy Ollama.
Create a project directory:
mkdir -p ~/ai-stackcd ~/ai-stackCreate docker-compose.yml:
version: "3.8"services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - /mnt/nas/ollama-models:/root/.ollama environment: - NVIDIA_VISIBLE_DEVICES=all deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" volumes: - /mnt/nas/openwebui-data:/app/backend/data environment: - OLLAMA_BASE_URL=http://ollama:11434 depends_on: - ollamaStart the stack:
docker compose up -dPull your first model:
docker exec -it ollama ollama pull llama3.1:8bThis downloads the Llama 3.1 8B parameter model, which is an excellent starting point for an 8GB GPU. The download is roughly 4.7GB and will be stored on your NAS mount.
ollama pull mistral:7b # Great for general tasksollama pull codellama:7b # Optimized for codingollama pull llama3.1:8b-instruct # Best for chat interactionsollama pull phi3:mini # Microsoft's compact modelollama pull gemma2:9b # Google's open modelcurl http://localhost:11434/api/generate -d '{ "model": "llama3.1:8b", "prompt": "Hello, how are you?", "stream": false}'If you get a JSON response with generated text, Ollama is working.
Open your browser and navigate to:
http://<VM-IP>:3000Open WebUI should automatically connect to Ollama using the OLLAMA_BASE_URL environment variable we set in Docker Compose. Verify by clicking Settings > Connections and confirming the Ollama URL shows http://ollama:11434 with a green status.
You can create specialized assistants using Ollama Modelfiles. Example – A coding assistant:
FROM codellama:7bSYSTEM "You are an expert programmer. You write clean, efficientcode with clear comments. When asked about code, provide workingexamples with explanations."PARAMETER temperature 0.3PARAMETER num_ctx 4096Save this as coding-assistant.modelfile and create it:
docker exec -it ollama ollama create coding-assistant \ -f /path/to/coding-assistant.modelfileThis model then appears in Open WebUI as a selectable assistant.
Storing models and data on your NAS ensures persistence and makes backups straightforward.
Install NFS client:
sudo apt install -y nfs-commonCreate mount points:
sudo mkdir -p /mnt/nas/ollama-modelssudo mkdir -p /mnt/nas/openwebui-dataAdd to /etc/fstab for persistent mounts:
192.168.1.100:/volume1/ai-models /mnt/nas/ollama-models nfs defaults,_netdev 0 0192.168.1.100:/volume1/ai-data /mnt/nas/openwebui-data nfs defaults,_netdev 0 0Mount everything:
sudo mount -aVerify mounts:
df -h | grep nasReplace 192.168.1.100 with your NAS IP and adjust the share paths to match your NAS configuration (Synology, TrueNAS, etc.).
Important: Make sure Docker containers have write permissions to these mount points. Set ownership if needed:
sudo chown -R 1000:1000 /mnt/nas/ollama-modelssudo chown -R 1000:1000 /mnt/nas/openwebui-dataBy default, the services are accessible at:
http://<VM-IP>:11434http://<VM-IP>:3000To make Ollama accessible to other machines on your network, ensure the Ollama container binds to 0.0.0.0 (default in our Docker Compose config).
Install Nginx on the VM (or use a dedicated reverse proxy VM):
sudo apt install -y nginxCreate /etc/nginx/sites-available/ai-assistant:
server { listen 80; server_name ai.homelab.local; location / { proxy_pass http://localhost:3000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_read_timeout 300s; proxy_send_timeout 300s; }}Enable the site:
sudo ln -s /etc/nginx/sites-available/ai-assistant \ /etc/nginx/sites-enabled/sudo nginx -tsudo systemctl reload nginxAdd ai.homelab.local to your DNS server or local hosts file pointing to the VM IP address.
sudo apt install -y certbot python3-certbot-nginxsudo certbot --nginx -d ai.yourdomain.comsudo ufw allow 80/tcpsudo ufw allow 443/tcpsudo ufw allow 3000/tcpsudo ufw allow 11434/tcpsudo ufw enablenvidia-smi -l 1 (updates every second)Q4_K_M - Best balance of speed and quality (recommended)Q5_K_M - Slightly better quality, slightly slowerQ8_0 - Near full quality, uses significantly more VRAMF16 - Full precision, requires 2x the VRAM (not for 8GB cards)Set in your Modelfile or at runtime:
PARAMETER num_ctx 4096watch -n 1 nvidia-smi # GPU monitoringhtop # CPU and RAM monitoringdocker stats # Container resource usageiostat -x 1 # Disk I/O monitoringAfter following this guide, you now have a fully self-hosted AI assistant running on your Proxmox homelab. Your data stays private, your models run locally on your GPU, and you have a clean web interface for interacting with multiple AI models.
The entire stack – Ollama for inference, Open WebUI for the interface, NAS for storage – runs reliably as a set of Docker containers inside a Proxmox VM with GPU passthrough. It survives reboots, updates cleanly, and scales as you add more models.
This is what homelabbing is about: taking control of your own infrastructure and running services that matter to you. A private AI assistant is one of the most practical and rewarding projects you can build today.
nvidia-smi -l 5 to a tmux session so you always see GPU memory usage. VRAM exhaustion causes silent failures that are hard to debug.alias opull='docker exec -it ollama ollama pull' Then pulling models is just: opull mistral:7bProblem: Your GPU shares an IOMMU group with other devices.
Solution: Check groups with: find /sys/kernel/iommu_groups/ -type l
If your GPU is not in a clean group, you may need an ACS override patch or a different PCIe slot. Move the GPU to a slot that isolates it in its own IOMMU group.
Problem: The Proxmox host loads NVIDIA drivers before VFIO can claim the GPU.
Solution: Blacklist nouveau and nvidia in /etc/modprobe.d/blacklist.conf and ensure VFIO modules load first. Add softdep nvidia pre: vfio-pci to modprobe configuration.
Problem: docker run --gpus all fails with “could not select device driver”.
Solution: The NVIDIA Container Toolkit is not installed or not configured. Run:
sudo nvidia-ctk runtime configure --runtime=dockersudo systemctl restart dockerProblem: Open WebUI shows “Connection failed” for the Ollama backend.
Solution: Ensure both containers are on the same Docker network (Docker Compose handles this automatically). Verify the OLLAMA_BASE_URL is set to http://ollama:11434 (using the container name, not localhost).
Problem: Downloaded models are gone after restarting the VM.
Solution: The NFS/SMB mount is not persisting across reboots. Add the mount to /etc/fstab with the _netdev option and verify with sudo mount -a after reboot.
Problem: Ollama crashes or returns errors during inference.
Solution: You are likely running a model too large for your VRAM. Stick to 7B-8B models on 8GB cards. Set OLLAMA_NUM_PARALLEL=1 to prevent concurrent requests from exceeding VRAM. Monitor with nvidia-smi.
Problem: Models take a very long time to load initially.
Solution: NFS over a 1Gbps connection is the bottleneck. Models are 4-5GB each, so initial load takes 30-40 seconds. Consider 10Gbps networking or storing frequently-used models on local SSD with NAS as backup.
Problem: GPU passthrough stops working after a Proxmox kernel update.
Solution: Kernel updates can change IOMMU behavior. After updates, verify VFIO binding:
lspci -nnk -s 27:00dmesg | grep -i vfioupdate-initramfs -u -k allAlways test GPU passthrough after host kernel updates before relying on the AI assistant for important work.
BUILDING A PRIVATE AI ASSISTANT ON MY HOMELAB
I built a self-hosted AI assistant using Ollama and Open WebUI, running on my Proxmox homelab with an NVIDIA RTX 3060 Ti.
Why? Privacy. Control. Learning.
Every prompt I type stays on my network. My models run on my GPU. My conversations are stored on my NAS. Nothing goes to the cloud.
The stack:
What surprised me:
The open-source AI ecosystem has matured to the point where running your own AI assistant is not just possible – it is practical.
If you have a homelab with a spare GPU, this is one of the most rewarding projects you can build right now.
Full setup guide on my blog: NetworkThinkTank.blog
#homelab #AI #selfhosted #Ollama #OpenWebUI #Proxmox #privacy #LLM #artificialintelligence #homelabbing
Just deployed a fully self-hosted AI assistant on my Proxmox homelab using Ollama and Open WebUI – complete with GPU passthrough and NAS storage integration. Every prompt stays private, every model runs locally, and the web interface rivals ChatGPT. Full build guide with Docker configs and real deployment tips on the blog.
After weeks of building, troubleshooting, and optimizing my CCNA lab environment, I am excited to share the entire project — now fully documented and open-sourced on GitHub. This post walks through the journey from an initial EVE-NG deployment to a fully automated Proxmox-based lab using Terraform, Ansible, and custom shell scripts.
You can find the complete repository here: github.com/jczaldivar71/eve-ng-ccna-lab
The EVE-NG CCNA Lab project started as a straightforward network simulation environment for CCNA study. It quickly evolved into a full infrastructure-as-code project covering:
generate_readme.py) to auto-generate comprehensive documentationOne of the key pieces of this project is the generate_readme.py Python script. Rather than manually maintaining a README that would inevitably fall out of sync with the actual project structure, I wrote a script that scans the repository and automatically generates a comprehensive README.md file.
The script inspects every directory — configs/, scripts/, terraform-ansible/, topology/, and images/ — and produces a fully formatted document with a table of contents, script references, setup instructions, and troubleshooting tips. Running it is as simple as:
cd scripts/
python generate_readme.pyThe generated README covers 13 sections including Overview, Project Structure, Prerequisites, Quick Start, Lab Topology, Scripts Reference, qcow2 Image Management, EVE-NG API Usage, Proxmox Deployment, Configuration Files, Troubleshooting, Known Limitations, and License information. At 340 lines, it serves as a complete guide for anyone wanting to replicate or build upon this lab.
EVE-NG is a fantastic network emulation platform, but I ran into limitations around resource management and integration with modern IaC tools. The decision to migrate to Proxmox was driven by several factors:
The migration involved exporting router and switch images from EVE-NG, converting and optimizing the qcow2 disk images, and then redeploying them on Proxmox using Terraform. The entire workflow is captured in the terraform-ansible/ directory of the repository.
The scripts/ directory contains six purpose-built shell scripts that automate every aspect of lab management:
Each script is documented with usage instructions and can be run independently or chained together for a complete deployment workflow.
One of the most impactful parts of this project was optimizing the qcow2 disk images. Network appliance images (Cisco IOSv, IOSvL2, CSR1000v, etc.) often ship with significant wasted space — preallocated but unused disk blocks that consume real storage.
The qcow2-optimize.sh script automates a multi-step optimization pipeline:
virt-sparsify to zero out unused blocks within the guest filesystemqemu-img convert -cqemu-img check to verify image health post-optimizationThe results were significant: total image storage dropped from 30GB to 18.3GB — a 39% reduction. This is especially meaningful in a home lab where storage is often limited. The optimized images boot identically to the originals but consume far less disk space on the Proxmox host.
The final piece of the puzzle is fully automated deployment using Terraform and Ansible. The terraform-ansible/ directory contains everything needed to stand up the lab from scratch:
Terraform handles the infrastructure provisioning:
Ansible manages the post-deployment configuration:
init-proxmox.yml: Initializes the Proxmox host with required packages, storage configuration, and network bridgesdeploy-vm.yml: Deploys individual VMs with their specific configurationsremove-gateway.yml: Cleans up default gateway routes that can interfere with lab routing exercisesConfiguration variables are stored in group_vars/all.yml (with a .sample template provided), and the hosts inventory file defines the Proxmox target. The ansible.cfg sets sensible defaults for host key checking and privilege escalation.
With this setup, spinning up a complete CCNA lab goes from a manual multi-hour process to a single command:
cd terraform-ansible/
terraform init && terraform apply
ansible-playbook -i hosts deploy-vm.ymlThis project is a living repository — I plan to continue adding to it as I progress through my CCNA studies and expand the lab. Future additions may include:
If you are studying for the CCNA or building your own home lab, feel free to fork the repository and adapt it to your needs. Contributions and feedback are always welcome.
GitHub Repository: https://github.com/jczaldivar71/eve-ng-ccna-lab
Hey everyone! If you have been following my blog, you know I love combining Python with network engineering. From automating backups with Netmiko to monitoring IP SLAs with DNA Center, I am always looking for ways to make our lives as network engineers easier. Today, I am excited to walk you through my latest project: an AI-Powered Network Health Checker. Don’t worry — this is totally beginner friendly. If you can write a basic Python script, you can follow along!
In a nutshell, this tool pulls real-time data from your network devices (think CPU usage, memory utilization, interface errors, etc.), feeds that data into a simple machine learning model, and tells you whether each device is healthy or if there might be an issue. The output is super straightforward — you will see messages like “Device is healthy” or “Potential issue detected.” No PhD in data science required!
Just like in my previous posts on network automation, we start by connecting to our devices and grabbing the data we need. I used the Netmiko library to SSH into each device and pull key metrics. Here is a simplified version of the script:
from netmiko import ConnectHandlerimport redevice = { 'device_type': 'cisco_ios', 'host': '192.168.1.1', 'username': 'admin', 'password': 'yourpassword',}connection = ConnectHandler(**device)cpu_output = connection.send_command('show processes cpu')cpu_match = re.search(r'CPU utilization for five seconds: (\\d+)%', cpu_output)cpu_usage = int(cpu_match.group(1)) if cpu_match else 0mem_output = connection.send_command('show processes memory')mem_match = re.search(r'Processor Pool Total:\\s+(\\d+)\\s+Used:\\s+(\\d+)', mem_output)if mem_match: mem_total = int(mem_match.group(1)) mem_used = int(mem_match.group(2)) mem_usage = (mem_used / mem_total) * 100else: mem_usage = 0intf_output = connection.send_command('show interfaces')error_matches = re.findall(r'(\\d+) input errors', intf_output)total_errors = sum(int(e) for e in error_matches)print(f"CPU Usage: {cpu_usage}%")print(f"Memory Usage: {mem_usage:.1f}%")print(f"Total Interface Errors: {total_errors}")connection.disconnect()This script connects to a Cisco IOS device, grabs CPU usage, memory utilization, and interface error counts. You can easily expand this to loop through multiple devices from an inventory file — just like we did in the backup config script project.
Here is where the AI magic comes in — but I promise it is simpler than it sounds. We are using scikit-learn’s Isolation Forest algorithm, which is perfect for anomaly detection. It learns what “normal” looks like from your data and flags anything that seems off.
import numpy as npfrom sklearn.ensemble import IsolationForesttraining_data = np.array([ [15, 40, 0], [20, 45, 1], [18, 42, 0], [22, 50, 2], [17, 38, 0], [19, 44, 1], [21, 47, 0], [16, 41, 1], [20, 43, 0], [18, 46, 2],])model = IsolationForest(contamination=0.1, random_state=42)model.fit(training_data)new_device_data = np.array([[cpu_usage, mem_usage, total_errors]])prediction = model.predict(new_device_data)if prediction[0] == 1: print("Device is healthy")else: print("Potential issue detected")The Isolation Forest works by randomly partitioning data points. Anomalies are isolated faster because they are different from the majority of the data. The contamination parameter tells the model roughly what percentage of data points are expected to be anomalies — I set it to 0.1 (10%) as a starting point, but you can tune this for your environment.
Now let us combine everything into a single script that loops through your devices, pulls the data, and runs it through the model:
from netmiko import ConnectHandlerfrom sklearn.ensemble import IsolationForestimport numpy as npimport redevices = [ {'device_type': 'cisco_ios', 'host': '192.168.1.1', 'username': 'admin', 'password': 'yourpassword'}, {'device_type': 'cisco_ios', 'host': '192.168.1.2', 'username': 'admin', 'password': 'yourpassword'},]training_data = np.array([ [15, 40, 0], [20, 45, 1], [18, 42, 0], [22, 50, 2], [17, 38, 0], [19, 44, 1], [21, 47, 0], [16, 41, 1], [20, 43, 0], [18, 46, 2],])model = IsolationForest(contamination=0.1, random_state=42)model.fit(training_data)def get_device_metrics(device): connection = ConnectHandler(**device) cpu_output = connection.send_command('show processes cpu') cpu_match = re.search(r'CPU utilization for five seconds: (\\d+)%', cpu_output) cpu_usage = int(cpu_match.group(1)) if cpu_match else 0 mem_output = connection.send_command('show processes memory') mem_match = re.search(r'Processor Pool Total:\\s+(\\d+)\\s+Used:\\s+(\\d+)', mem_output) mem_usage = (int(mem_match.group(2)) / int(mem_match.group(1))) * 100 if mem_match else 0 intf_output = connection.send_command('show interfaces') error_matches = re.findall(r'(\\d+) input errors', intf_output) total_errors = sum(int(e) for e in error_matches) connection.disconnect() return [cpu_usage, mem_usage, total_errors]for device in devices: print(f"Checking device: {device['host']}") metrics = get_device_metrics(device) print(f" CPU: {metrics[0]}% | Memory: {metrics[1]:.1f}% | Errors: {metrics[2]}") prediction = model.predict(np.array([metrics])) if prediction[0] == 1: print(" Status: Device is healthy") else: print(" Status: Potential issue detected")Here is what the output looks like:
Checking device: 192.168.1.1 CPU: 18% | Memory: 43.2% | Errors: 0 Status: Device is healthyChecking device: 192.168.1.2 CPU: 85% | Memory: 92.1% | Errors: 47 Status: Potential issue detectedThis is just the starting point. Here are some ideas to take it further:
I have uploaded the full project to my GitHub repo. Feel free to clone it, play around with it, and make it your own:
GitHub: https://github.com/NetworkThinkTank-Labs/ai-network-health-checker
If you are a network engineer who is curious about AI and machine learning, this is a great beginner project to get your feet wet. You don’t need to understand every detail of how Isolation Forest works under the hood — just know that it is a tool that can help you spot problems before they become outages.
As always, if you have questions or want to share how you have customized this for your own network, drop a comment below or reach out to me. Happy automating!
Backing up network device configurations is one of the most critical tasks in network administration. A missed backup could mean hours of manual reconfiguration after a failure. In this post, we will walk through a Python script that automates this process by connecting to a router or switch via SSH and saving the running-config to a local file.
We will use Netmiko, a popular Python library that simplifies SSH connections to network devices. Whether you manage a handful of switches or hundreds of routers, this script gives you a repeatable, automated way to capture configurations on demand.
Before getting started, make sure you have:
To install Netmiko, run:
pip install netmiko
You can also clone the full project repository and install from the requirements file:
git clone https://github.com/NetworkThinkTank-Labs/backup-config-script.git
cd backup-config-script
pip install -r requirements.txt
The core of our script uses Netmiko’s ConnectHandler to establish an SSH session. You provide the device type, hostname or IP address, and your credentials. Netmiko handles the SSH negotiation and drops you into an authenticated session.
Here is the connection function from our script:
def connect_to_device(host, username, password, device_type, port=22, enable_secret=None):
device = {
‘device_type’: device_type,
‘host’: host,
‘username’: username,
‘password’: password,
‘port’: port,
}
if enable_secret:
device[‘secret’] = enable_secret
connection = ConnectHandler(**device)
if enable_secret:
connection.enable()
return connection
Netmiko supports a wide range of device types including cisco_ios, cisco_nxos, arista_eos, and juniper_junos. You simply pass the appropriate device type string and Netmiko adapts its behavior accordingly.
Once connected, pulling the running configuration is a single method call. We use send_command to execute show running-config on the device and capture the output as a string:
def backup_running_config(connection):
running_config = connection.send_command(‘show running-config’)
return running_config
Netmiko handles paging automatically, so even if your configuration is long, you will get the complete output without needing to send space or press Enter to page through it.
With the configuration captured in memory, the next step is writing it to a file. Our script creates a backups directory automatically and saves each configuration with a timestamped filename so you never overwrite a previous backup:
def save_config_to_file(config, hostname, output_dir=’backups’):
os.makedirs(output_dir, exist_ok=True)
timestamp = datetime.now().strftime(‘%Y-%m-%d_%H-%M-%S’)
safe_hostname = hostname.replace(‘.’, ‘‘).replace(‘:’, ‘‘)
filename = f'{safe_hostname}running-config{timestamp}.txt’
filepath = os.path.join(output_dir, filename)
with open(filepath, ‘w’) as f:
f.write(config)
return filepath
This produces backup files like:
backups/192_168_1_1_running-config_2026-04-10_14-30-00.txt
The script accepts command-line arguments so you can target any device without editing the code:
python backup_config.py –host 192.168.1.1 –username admin –password mypassword
You can also specify a different device type, SSH port, output directory, or enable password:
python backup_config.py –host 10.0.0.1 –username admin –password mypassword –device-type cisco_nxos –output-dir /var/backups/network
When you run the script, you will see output like this:
[] Connecting to 192.168.1.1:22 (cisco_ios)… [+] Successfully connected to 192.168.1.1 [] Retrieving running-config…
[+] Retrieved 15234 characters of configuration
[+] Configuration saved to backups/192_168_1_1_running-config_2026-04-10_14-30-00.txt
[+] Disconnected. Backup complete!
This script provides a solid foundation for network configuration backups. Here are some ideas for extending it:
Automating network device backups does not have to be complicated. With Python and Netmiko, you can connect to any router or switch, pull the running configuration, and save it to a timestamped file in just a few lines of code.
Check out the full source code and setup instructions in the GitHub repository: https://github.com/NetworkThinkTank-Labs/backup-config-script
If you found this useful, check out my other posts on network automation including Monitoring IP SLAs with Python, DNA Center, and NetBox and Build a Home Lab Like a Pro. Stay tuned for more content from the NetworkThinkTank!
Automate IP SLA monitoring across your network using Python, Cisco DNA Center APIs, and NetBox as your source of truth.
GitHub Repo: https://github.com/NetworkThinkTank-Labs/ip-sla-monitor
If you manage a network of any size, you know that keeping tabs on performance metrics like latency, jitter, and packet loss is critical. Cisco IP SLA (Service Level Agreement) operations are the go-to feature for probing network paths and measuring these metrics directly from your routers and switches. But manually checking IP SLA statistics across dozens or hundreds of devices? That does not scale.
In this post, I will walk you through a Python-based tool I built that pulls IP SLA data from Cisco DNA Center via its REST API, enriches it with device metadata from NetBox, and generates automated performance reports. Whether you are running a handful of branch routers or a large enterprise campus, this approach gives you a scalable, repeatable way to monitor network performance.
Cisco IP SLA is a built-in feature on Cisco IOS and IOS-XE devices that allows you to generate synthetic traffic to measure network performance. You can configure operations like ICMP echo (ping), UDP jitter, HTTP GET, and more. Each operation continuously measures metrics such as round-trip time (RTT), latency, jitter, packet loss, and availability. These metrics are essential for validating SLA compliance, troubleshooting performance issues, and capacity planning.
This project brings together three key components. First, Python does the heavy lifting for API calls, data parsing, and report generation. Second, Cisco DNA Center provides a centralized REST API for pulling device inventory and running CLI commands across your entire network without SSH-ing into each device individually. Third, NetBox acts as our network source of truth, storing device metadata like site assignments, roles, platforms, and IP addresses that we use to enrich the raw SLA data.
The IP SLA Monitor tool follows a simple three-step workflow:
The project is organized into three main scripts:
ip_sla_monitor.py is the main orchestration script that ties everything together. It loads configuration from a .env file, initializes the DNA Center and NetBox clients, collects SLA data, enriches it, evaluates thresholds, and saves reports.
dnac_integration.py handles all communication with the Cisco DNA Center REST API including authentication, device inventory retrieval, and IP SLA data collection via the command-runner API.
netbox_integration.py connects to the NetBox API to look up device metadata by hostname, returning site assignments, device roles, platform types, and IP addresses.
Getting up and running is straightforward. Clone the repository, set up a virtual environment, install the dependencies from requirements.txt, and configure your .env file with your DNA Center and NetBox credentials. The only Python packages required are requests for HTTP API calls, python-dotenv for environment variable management, and urllib3. No complex frameworks or heavy dependencies. Full setup instructions are in the GitHub repo README.
One of the most useful features is configurable threshold alerting. You define your acceptable limits for latency, jitter, and packet loss in the .env file, and the tool flags any SLA operation that exceeds those limits. For example, with default thresholds of 100ms latency, 30ms jitter, and 1% packet loss, a branch router showing 115ms latency and 2.1% packet loss would be immediately flagged as an alert in the console output and reports.
The tool generates both JSON and CSV reports. The JSON report includes a summary section with total operations, passing and failing counts, and average latency, followed by detailed per-operation data enriched with NetBox metadata. The CSV report provides the same data in a tabular format that you can easily import into Excel or feed into other monitoring tools. Sample output files are included in the GitHub repository under the output directory.
This project is a solid foundation, but there is plenty of room to extend it. Some ideas for future enhancements include adding webhook or email notifications for alerts, integrating with Grafana for real-time dashboards, storing historical data in a time-series database like InfluxDB, and expanding the command-runner integration to pull live SLA statistics directly from devices.
Python automation combined with APIs from DNA Center and NetBox gives network engineers a powerful toolkit for monitoring IP SLAs at scale. Instead of manually checking IP SLA stats on individual devices, you can automate the entire workflow and get enriched reports in minutes.
Check out the full source code, sample outputs, and setup instructions in the GitHub repository: https://github.com/NetworkThinkTank-Labs/ip-sla-monitor
If you found this useful, check out my other posts on network automation including Automating Network Device Backups with Python and Netmiko and Build a Home Lab Like a Pro. Stay tuned for more content from the NetworkThinkTank!
Transform your spare room into a career-accelerating network laboratory
p>If you’re serious about leveling up your networking skills, there’s no substitute for hands-on experience. Certifications are great. Books are essential. But nothing cements your understanding of BGP peering, VXLAN fabrics, or automation workflows like building it yourself and watching packets traverse your own infrastructure.
After 20+ years in networking — from pulling cable to managing backbone infrastructure at Lumen — I can tell you that the engineers who stand out are the ones who lab. They break things on purpose, fix them under pressure, and walk into production environments with confidence.
In this guide, I’ll walk you through everything you need to build a professional-grade home lab, from hardware selection to virtualization platforms to topology design. And because I believe in open-source learning, I’ve created a companion GitHub repository with sample configs, topology files, and scripts you can use to get started immediately.
GitHub Repo: NetworkThinkTank-Labs/home-lab-guide
p>Let’s get the obvious out of the way: you can’t become a great network engineer by reading alone. Here’s why a home lab is one of the best investments you can make in your career:
p>
You don’t need to spend thousands of dollars to build an effective lab. Here’s a tiered approach based on budget and goals.
Perfect for getting started with virtualization and basic routing/switching labs.
| Component | Recommendation | Estimated Cost |
|---|---|---|
| Server | Dell OptiPlex 7050/7060 (used) or HP EliteDesk 800 G3 | $100-$200 |
| RAM | 32 GB DDR4 (upgrade if needed) | $40-$60 |
| Storage | 500 GB SSD + 1 TB HDD | $50-$80 |
| Switch | Managed switch (Netgear GS308T or TP-Link TL-SG108E) | $30-$50 |
| Misc | USB-to-serial console cable, Ethernet cables, power strip | $20-$40 |
For engineers running multiple VMs, nested virtualization, and more complex topologies.
| Component | Recommendation | Estimated Cost |
|---|---|---|
| Server | Dell PowerEdge R720/R730 or HP ProLiant DL380 Gen9 | $200-$500 |
| RAM | 64-128 GB DDR4 ECC | $100-$200 |
| Storage | 1 TB NVMe SSD + 2 TB HDD | $100-$200 |
| Network | Intel X520-DA2 10GbE NIC (SFP+) | $30-$50 |
| Switch | Cisco Catalyst 2960-X or Arista 7010T (used) | $50-$150 |
| Firewall | Netgate SG-1100 (pfSense) or old PC with OPNsense | $100-$200 |
| UPS | APC Back-UPS 600VA | $60-$80 |
This is where the magic happens. Modern network labs are predominantly virtual, and the platform you choose shapes your entire lab experience.
Why: The OG network emulator. Runs real Cisco IOS/IOU images and integrates with QEMU/KVM for full OS emulation. Best for Cisco-centric labs and certification prep (CCNA/CCNP). Pair with our BGP Fundamentals Lab and VPN IPSec/GRE Lab.
p>Why: Web-based, multi-vendor support, great for complex topologies with dozens of nodes. Best for multi-vendor labs (Cisco, Juniper, Arista, Palo Alto, Fortinet). Pair with our EVPN-VXLAN Lab for data center fabric simulation.
p>Why: Lightweight, code-defined network topologies using containers. Perfect for modern network automation workflows. Uses Docker containers running network OS images (cEOS, FRR, Nokia SR Linux). Define topologies in YAML and version control them in Git. Pair with our Network Automation with Ansible Lab.
p>A well-designed lab topology mirrors real-world architectures. Here are three topologies I recommend, progressing from simple to advanced.
Internet > Firewall (pfSense/OPNsense) > Core Switch (L3, VLAN trunking) > VLANs (Management, Lab, Servers)
What you’ll learn: VLANs, inter-VLAN routing, firewall rules, NAT, DHCP
p>Dual ISP routers with eBGP > Edge Router (BGP AS 65000) > Core switches (OSPF, LACP) > Access switches
What you’ll learn: BGP peering, OSPF areas, HSRP/VRRP, link aggregation, redundancy
p>Spine-01/Spine-02 > Leaf-01 through Leaf-04 > Server clusters
What you’ll learn: EVPN-VXLAN, BGP underlay/overlay, spine-leaf architecture, network automation at scale. Check out our EVPN-VXLAN Lab for a full walkthrough.
p>
Here’s how I approach lab exercises, and how you can use the NetworkThinkTank-Labs repositories to follow along:
As network engineers, one of our most critical yet tedious responsibilities is maintaining up-to-date backups of device configurations. Whether you manage a handful of switches or hundreds of routers across multiple sites, manually logging into each device to copy its running configuration is time-consuming, error-prone, and simply does not scale.
In this post, I will walk you through a Python automation script I built that connects to network devices via SSH, pulls their running configurations, and saves them to organized backup files, all in parallel. The full project is available on my GitHub repository.
GitHub Repository: https://github.com/jczaldivar71/network-backup-automation
Before diving into the code, let us consider why automation matters here. Manual backups are inconsistent since engineers may forget devices or skip them during busy periods. They are slow because logging into devices one at a time does not scale. They lack accountability with no automatic logging of what was backed up and when. They are error-prone since copy-paste mistakes can result in incomplete or corrupted backup files.
An automated solution runs on a schedule, covers every device in your inventory, logs every action, and produces consistent, timestamped backup files every single time.
The network backup automation script provides several key features. It supports multiple platforms including Cisco IOS, Cisco ASA, Cisco NX-OS, Juniper JunOS, and Arista EOS. It uses concurrent connections via Python ThreadPoolExecutor to back up multiple devices simultaneously. Backups are organized into date-stamped directories for easy retrieval. Each backup includes metadata headers showing the device hostname, IP address, device type, and timestamp. The script generates a JSON summary report after each run showing successes and failures. It also provides comprehensive error handling for timeouts, authentication failures, and connection issues.
The project is organized as follows. The main script is network_backup.py which contains all the automation logic. The requirements.txt file lists the Python dependencies, primarily Netmiko and Paramiko. The inventory.json file is a sample device inventory template where you define your network devices. The .gitignore file ensures backup files and sensitive data are not accidentally committed to version control. The LICENSE file contains the MIT open-source license.
The script follows a straightforward workflow. First, it loads your device inventory from a JSON file. Each device entry includes the hostname, IP address, device type, credentials, and optional enable secret. Second, it creates a date-stamped backup directory to keep backups organized by day. Third, it spawns multiple worker threads using ThreadPoolExecutor to connect to devices in parallel. Fourth, for each device, it establishes an SSH connection using Netmiko, enters enable mode if needed, and runs the appropriate show command for that platform. Fifth, the configuration output is saved to a file with a metadata header. Finally, a JSON report is generated summarizing the results of the entire backup run.
The inventory.json file is where you define all the devices you want to back up. Here is an example of what it looks like. Each device entry includes the hostname for identification, the host IP address, the device_type which tells Netmiko how to communicate with the device, the username and password for SSH authentication, the port number which defaults to 22, and an optional secret for enable mode on Cisco devices.
The BACKUP_COMMANDS dictionary maps each device type to the appropriate command for retrieving the running configuration. For Cisco IOS, ASA, and NX-OS devices, it uses “show running-config”. For Juniper JunOS devices, it uses “show configuration | display set”. For Arista EOS, it also uses “show running-config”.
The backup_device function is the core of the script. It takes a device dictionary and backup directory path, establishes the SSH connection using Netmiko ConnectHandler, enters enable mode if a secret is provided, runs the backup command, and saves the output with metadata headers to a timestamped config file.
The run_backups function orchestrates the entire process. It loads the inventory, creates the backup directory, then uses ThreadPoolExecutor to run backups in parallel across all devices. After all backups complete, it logs a summary and generates a JSON report file.
To use this script in your own environment, follow these steps. First, clone the repository from GitHub. Second, install the dependencies using pip install with the requirements.txt file. Third, edit inventory.json with your actual device information including hostnames, IP addresses, credentials, and device types. Fourth, run the script using python network_backup.py. You can also customize the behavior with command-line arguments such as specifying a different inventory file with the -i flag, changing the output directory with -o, adjusting the number of concurrent workers with -w, or enabling verbose debug logging with -v.
Since this script handles network device credentials, security is paramount. Never commit your actual inventory.json file with real credentials to version control, which is why it is included in the .gitignore file. Consider using environment variables or a secrets manager for credentials in production. Restrict file permissions on the backup directory since configuration files may contain sensitive information. Use SSH key-based authentication instead of passwords when possible. Run the script from a secured management workstation or jump box.
This script provides a solid foundation, but there are many ways to extend it. You could add email or Slack notifications for backup failures. You could integrate with a scheduling system like cron or Windows Task Scheduler to run backups automatically. You could implement configuration diff detection to alert you when configurations change unexpectedly. You could add support for additional device types. You could also store backups in a Git repository for version-controlled configuration management.
Network automation does not have to be overwhelming. Starting with a practical use case like configuration backups is an excellent way to build your Python skills while immediately adding value to your organization. The script handles the complexity of multi-vendor support, concurrent connections, and error handling so you can focus on what matters most, keeping your network safe and well-documented.
Check out the full project on GitHub: https://github.com/jczaldivar71/network-backup-automation
Feel free to fork it, customize it for your environment, and let me know how it works for you. Happy automating!