Opening raidz…

Open reading view
Skip to content

Product launches and ongoing operations

I oversee infrastructure and product operations for our cloud service and self-hosted VPN software.

My role: I sourced the hosting, analyzed costs and designed the deployment architecture used for the cloud service’s first production launch in 2020.

The established VPN product

I worked directly with customers on support, server builds, licensing, and troubleshooting, then took on broader responsibility for product releases and operations.

I brought in the company’s first enterprise customer, Google.

I helped define team roles as the software and its distribution expanded.

The 2020 cloud launch

I developed the high-level plan for how hosting, virtualization, networking and releases would fit together, then guided the engineers implementing each part. The original deployment scripts assumed an AWS environment. We kept the core product software and rebuilt the deployment tooling around dedicated servers for the VPN data plane.

Hosting selection and cost analysis

I sourced the data centers and hosting and carried out the cost analysis. Bandwidth costs and locations near our customers drove the choice of dedicated hardware. For our service, dedicated hosting offered better bandwidth economics and more location choice than the public-cloud options available to us.

Virtualization and deployment tooling

I chose KVM because it was open source, well maintained and lightweight for our use. Full virtual machines gave us the guest-kernel and operating-system control needed for customizations that containers, including LXC, could not support for our requirements.

The team used Ansible to bootstrap the servers and Puppet to manage their configuration. This replaced the AWS-oriented deployment assumptions with tooling for the dedicated-host environment.

Blue/green releases with stable IPs

My design used two KVM virtual machines on each dedicated host as part of the blue and green environments. One environment served customers while the other received the new software and was tested before taking traffic.

The public-facing IP stayed consistent through the switch. Using iptables and NAT, we directed new connections to the updated environment while established connections continued on the original one. We then moved existing clients through controlled reconnects on a chosen schedule. That separated the switch for new traffic from the migration of customers already connected.

This architecture was already running for the first production launch in 2020. It gave the team a repeatable process for releasing software, recovering from bad releases and bringing new locations online. As demand grew, I directed capacity expansion with the teams supporting the service.

Decisions across teams

A release involves more than infrastructure. I’ve worked across product, engineering, marketing, and finance on releases, billing changes, and marketplace delivery.

I work out how those dependencies affect the release and establish who is responsible for the next step. That includes deciding what needs to be checked before the work is ready.

Operations after release

Resilience across data centers

The cloud service now spans more than 30 regions. I distribute physical nodes across multiple data centers within each region to reduce the impact of a facility failure. When choosing hosting, I balance proximity to customers and bandwidth costs against local infrastructure risks.

During incidents, I manage escalations and explain the situation to leadership. The work continues through corrective actions and a clear record of what needs to change.

Read about applied AI