Infrastructure engineering for East Africa's operators, platforms and regulators See the work →
Engineering Record

Infrastructure Work You Can Verify

Six engagements from our Kenyan team: payments code we maintain in public, an internet provider's own cloud, two platform rebuilds, a payments platform whose infrastructure we run end to end, and a 565-truck workshop taken off paper. One of them you can check in a git log without asking us for a reference. Below them, the cloud, security and ERP work we do for businesses across East Africa.

01
Flagship engagement · Open source

Upstream in Mojaloop, the Payment Switch Central Banks Build On

A central bank cannot buy a national payment switch the way it buys laptops. Mojaloop is the open-source software those switches are built from, a digital public good with the Gates Foundation behind it, adopted by the central banks of Rwanda and Tanzania and by the COMESA bloc. The payments code is free to read and free to run. Standing it up is the part that stalls, because the infrastructure has to satisfy a regulator before it satisfies an architect.

Payment data usually may not leave the country, which deletes the ordinary answer of managed cloud databases in whichever region is cheapest. The switch has to run on hardware inside the border while the same platform still runs on AWS for test environments, participant sandboxes and the markets where cloud is permitted. Two substrates, one set of manifests, or you maintain the platform twice and watch the copies drift. In the project that problem is called the storage cluster, and it is most of what we work on.

One storage claim resolving to two substrates A single MysqlCluster claim passes through a Crossplane composition and resolves either to a managed database in a public cloud region or to a Percona XtraDB cluster on hardware inside the country, presenting the same secret and service name to the application in both cases. MysqlCluster sc.mojaloop.io/v1alpha1 Crossplane composition readiness gates first, then resources Public cloud region test, sandbox, permitted markets Managed MySQL the provider runs it Hardware inside the border where the data may not leave Percona XtraDB on K8s disks you own, in country Same secret, same service name. The application never learns which one it got.
One claim, two substrates A service asks for a MysqlCluster. On AWS that becomes a managed database; on Kubernetes standing on hardware you own it becomes a Percona XtraDB cluster, same name, same credentials in the same secret, same service name the application already resolves. Object storage resolves the same way, to Ceph on your own disks or to S3. That is what lets one platform be lawful where the data may not leave and economical where it may.

The resources are the easy half

A Crossplane composition has to converge, and that is the part that catches people: create a MySQL cluster before its bucket and the bucket's secret report ready and you get a database with nowhere to write its backups, quietly, until the day you need a restore. Most of our commits in that package are ordering and readiness conditions rather than resources. The same discipline runs through the hardware we place in country for clients who cannot use a region at all.

Check it yourself

Everything else on this page is a claim. This part is not, because the history is public and the reviewers were the project's maintainers rather than our colleagues. Contributor positions consolidate author aliases and keep moving as the project does.

What we are not claiming

We do not operate any country's payment switch, and nothing here should be read as saying we do. We maintain part of the code those deployments start from, at the level where deployments actually fail. If your tender needs a systems integrator to staff a program office, we would rather be the specialist inside somebody else's bid than the prime contractor who over-promised.

If Mojaloop is on your roadmap, the useful conversation is not about the application. It is about where the law says the data may live, what your regulator will accept as a control, and who is awake when the switch is not.

02
Flagship engagement · Service provider

A Kenyan ISP's Own Cloud on Proxmox HA, Ceph and OpenStack

An internet provider already has the customers, the fibre and the rack. What it does not have is anything to sell them once the link is live. The usual answer is to resell a cloud from another continent, which hands the margin and the customer relationship to a company the customer has never heard of. This provider went the other way and built its own.

Owning it means a hypervisor cluster you can bet a business on, billing that provisions rather than only invoicing, a control panel customers use without being trained, domains, and a straight answer to where the backups are. Each of those is a solved problem alone. The joins are where these builds die, and a customer buying at two in the morning is the test.

A paid order becoming a running service Four stages: WHMCS opens the account and raises the invoice, the platform provisions a Proxmox virtual machine or an OpenStack instance, DirectAdmin and the registrar complete the hosting and domain, and backups, monitoring and the recovery plan attach at creation. 01 Order WHMCS account Invoice raised The customer picks a plan and pays. 02 Build Proxmox VE machine or OpenStack instance Payment clears and the platform provisions. 03 Host DirectAdmin account Registrar, DNS, mail Records already correct when the customer looks. 04 Protect Backups, monitoring A place in the DR plan Attached when the service is created, not when it fails. The joins are where these builds die: a paid order that raises an invoice and then waits for somebody to read a ticket, a cluster with no quorum, a backup nobody has restored.
WHMCS, DirectAdmin and the registrar on one order path The customer picks a plan and pays; WHMCS opens the account and raises the invoice against it. Payment clearing is what provisions, not a ticket somebody reads in the morning. Which platform the service lands on depends on what was bought: a Proxmox virtual machine for most, an OpenStack instance for a customer who wants an API.

Proxmox for the servers, OpenStack for the API, ZFS and Ceph underneath

ZFS gives a node local NVMe speed with checksums and snapshots, replicating to a partner node on a schedule. Ceph spreads the same data across every node, so a machine can die mid-write and the guest keeps its disk. A customer on a cheap single VPS gets the first. A customer running their own billing system gets the second and pays for it. Putting everything on Ceph is the mistake in most first clusters, because it is right for the workloads that need it and an expensive way to serve the ones that do not. Which workload sits where is a Proxmox cluster design decision, taken before anybody buys disks.

OpenStack sits beside Proxmox rather than under it. Proxmox is where virtual machines live and where high availability is cheapest to run. OpenStack is what you add when customers stop asking for a server and start asking for tenants, quotas and an API of their own. Two platforms cost more to run than one, so it is a decision about the customer base rather than about the technology.

When reselling is the better business

There is a size below which none of this is worth owning. Under a rack and a few dozen customers, reselling wins, because somebody else carries the capital, the spares and the disk that fails at 3am. The economics turn when you already have the space, the power and enough customers to keep it busy, and when your buyers start asking where the data physically sits, which in Kenya they increasingly do.

Two things we say out loud on every one of these. High availability is not backup: Proxmox restarts a guest on another node, it does not keep it running, and a dropped database is dropped on every replica inside a second. And we do not sell shared hosting or register domains ourselves. We build the platform an operator sells them from, and the operator owns it: the cluster, the credentials, the runbooks and the recovery plan.

If you have a rack and a customer list, the conversation worth having is which workloads justify Ceph, what the billing has to automate before it pays for itself, and what your recovery plan says when a node does not come back. Send the shape of the business and we will scope the platform.

03
Flagship engagement · Rebuild

From Manual Cloud Management to Infrastructure as Code and GitOps

Everything ran on Google Cloud and everything had been built by hand. The program serves subsidized school meals to children in Kenyan public schools, and the software behind that is a live product with five engineering teams shipping into it. Projects, networks, databases, clusters and IAM bindings all existed because somebody had once clicked them into being, and the reason had left with the browser tab.

That is survivable right up until it is not. Staging had drifted far enough from production that testing there proved very little. Pipelines authenticated to the cloud with service account keys sitting in repository secrets. The honest answer to who can reach the production database was that somebody would have to go and look. The team was not careless. They were shipping, and the platform had outgrown anyone's ability to write it down.

How a change reaches the cloud after the rebuild Three lanes: pipelines authenticate to Google Cloud through Workload Identity Federation with no static key, changes travel from a pull request through an Atlantis plan and an approval before applying to eight projects, and cluster state is reconciled from git by ArgoCD across five clusters. IDENTITY CI pipeline Workload Identity Federation Google Cloud no key to leak or rotate CHANGES Pull request Atlantis plan attached to the PR Approval Apply 8 GCP projects STATE Manifests in git ArgoCD, 54 apps ordered by sync wave 5 clusters Drift corrected nobody is paged Nobody applies from a laptop, us included. The hard part is never writing the Terraform: it is the order of operations, which module exists before the cluster, which secret lands before the provider.
What the platform looks like written down Thirty-three OpenTofu modules and fifty-three Terragrunt stacks across eight Google Cloud projects, fifty-four ArgoCD applications over five clusters ordered by sync waves, and every pipeline reaching the cloud through Workload Identity Federation so there is no key to leak, rotate or find later in a git history. A single OIDC provider fronts ArgoCD, Vault and internal tooling; repositories, teams and branch protection are code, so joining and leaving are pull requests.

What changed on a Tuesday morning

Day to day Before After
Creating a resource Console, by whoever had access A pull request with the plan attached
Standing up an environment Never successfully done twice A checklist and a fixed apply order
CI authentication Long-lived JSON keys in repo secrets Federated identity, nothing to rotate
Who can reach production Ask around and hope Read the code and the SSO grants
Undoing a bad change Remember what you changed Revert the commit

Ten months so far and roughly 2,700 commits, with the work continuing. Anyone selling a cloud rebuild as a six-week engagement is selling the first layer and leaving before the difficult part starts, which is never writing the Terraform. It is the order of operations: which module must exist before the cluster, which secret has to land before the identity provider will start, what comes up first on a cold start when nothing is running. Getting that right is most of what our DevOps and CI/CD engineering is for.

We would also tell you to skip it

If you run a handful of servers in one project and one person touches them, the console is fine and this apparatus is overhead you will resent. The return starts at the second environment, the second team, or the first auditor. Below that line we will say so and quote you something smaller. Above it, the client owns all of it: the repositories, the documentation, and each decision recorded as the pull request that made it. Nothing in the path is CloudSpinx tooling, which is the same co-managed shape we recommend to any team that already employs good engineers.

If your cloud estate was built by hand and you have started to feel it, that is the conversation. Tell us what worries you most and we will say what a rebuild would genuinely involve, before anyone quotes a number.

04
Flagship engagement · Capacity

Embedded in a Live AWS Platform Team, Shipping to Production in Week One

This one runs the other way round, because the platform was already good. Twenty-one thousand lines of OpenTofu describing an AWS estate, fifty Atlantis projects gating the applies behind a pull request, ArgoCD reconciling four EKS clusters, 67 services owned by nine engineering teams across two AWS accounts, and both the GitHub organization and the SSO grants declared as code. What the team did not have was enough senior hands for the backlog sitting behind it.

Shipping one safe change into an estate that size means holding all of it. Which module owns the parameter group. Which sync wave has to bring a secret in before the pod that mounts it starts. Which of the two accounts production actually lives in, and which of the nine teams reviews the pull request. Nobody hands you that on arrival. You read the code until you can predict what a plan will say before Atlantis prints it, and our first change to the shared Helm chart all 67 services render from was merged inside week one.

The layers we worked in

Not one subsystem. Over seven weeks the work landed in seven of them, because in an estate wired this tightly the problems do not respect the boundaries.

Data
RDS parameter groups across three environments: statement statistics loaded, anything past two seconds logged, Aurora ceilings adjusted.
Compute
Karpenter node pools written as OpenTofu structures. Arm64 and amd64 spot across three zones, disruption budgets, consolidation, and the root volume size that stopped pods being evicted.
Identity
IAM roles per environment across two AWS accounts, production being the separate one, each scoped to exactly the read calls its job makes.
Delivery
Atlantis projects, ArgoCD applications and kustomize overlays, sync-wave ordered so a secret exists before the pod that mounts it starts looking.
Secrets
External Secrets pulling from Vault through the chart all 67 services render from, extended for the services the naming convention never fitted without changing one of the others.
Observability
Alloy collectors, Loki labels and Mimir rules declared in the Nix repository that builds the monitoring hosts. Editing a dashboard by hand was not an option.
Documentation
137 databases inventoried by a job that reads the live estate, regenerates the markdown, signs the commit and pushes the same page into Confluence.

Where it actually broke

One of those items was a slow-query alert across 137 Postgres databases, which written down sounds like an afternoon. Most of the seven weeks went on failures that report success.

Where a slow-query pipeline breaks between a database and a chat channel Logs travel from 137 Postgres databases through CloudWatch log groups, an Alloy collector, Loki and a Mimir recording rule to an alert. Three silent failures sat in the collector stage and a fourth followed a label rename in the rule. Every hop reported success. A pipeline dropping half its events looks identical to a healthy one. RDS 137 databases CloudWatch log groups Alloy collector Loki labels and streams Mimir rule and the alert Three silent failures, all here A JSON envelope, not the message execute name: parsed, statement: not The receiver needs the experimental flag And one after a rename env became environment, so the alert routed nowhere, and the threshold moved into LogQL
Four silent failures between a database and a chat channel None of this appears in the final diff. The collector ran, the log groups were right, and nothing arrived. Then half the traffic stayed invisible, because Postgres logs a prepared statement in a different shape from a direct one. The work was never writing the configuration; it was proving what the configuration did at every hop, in an environment where being wrong is quiet. That is the difference between an engineer who has run this and one who has read the documentation.
This is the wrong purchase in two cases

With no platform team, an extra pair of hands has nobody to hand to. What you want is the rebuild above, or the engagement below where we run the estate ourselves. With a handful of servers and one administrator, it is expensive help for a problem better solved by buying the outcome. We will tell you which you are on the first call, including when the answer is none of them.

Otherwise it is the same work we do on AWS and multi-cloud estates for operators and platforms in the region, at whatever depth the team does not already have covered. Every change went through the client's own controls: their tickets, their branch protection, their approvals, their audit trail. Procurement usually asks that before it asks about the engineering, and the answer has to already be yes.

Your platform team is probably not the problem. There are just fewer of them than there is work. Send us the item that keeps slipping from one sprint to the next, or send the tender, and we will tell you whether it needs a person, a workstream or a rebuild.

05
Flagship engagement · Managed infrastructure

Managed Infrastructure for a Payments Platform, From the Metal to the Pager

The company builds payment products. We run everything underneath them: the dedicated servers, the virtualization on top of them, the database, the caching and queues in front of it, and the monitoring that says one of those is unhappy before a customer does.

The scope is deliberately whole, because the split version is what fails. Hardware from one supplier, a systems person who set the servers up and has since left, a database somebody tuned once: that is the outage where three parties each explain it is somebody else's part. We plan the capacity, place the machines, build every box from automation, harden it, and keep it running.

Two things break a platform that is growing: it falls over when the traffic arrives all at once, or it drifts until nobody can rebuild what is actually there.

What absorbs load and what holds the data on a managed platform Customer traffic reaches application servers running as virtual machines on dedicated hardware. A cache answers reads and a queue holds writes that can wait, so bursts do not reach the database directly. The database primary replicates to a live replica and backs up off the box, and monitoring covers every component. Dedicated servers, virtualized, every box configured from automation Customer traffic Cache reads answered here Application servers on virtual machines Queue and workers writes that can wait Database primary where writes land Live replica the same data, again Backups off the box restores get drilled replication Monitoring on every box above A rebuild runs the automation. Nobody puts a box back together from memory.
What absorbs the load, and what holds the truth Scale and stability are the same problem approached from two ends. The cache and the queue exist so a burst becomes a longer queue rather than a slower database, which is the difference between a payment that takes an extra second and a payment that fails. The replica exists so the primary is not the only copy of the truth. The backup exists because a replica faithfully copies a bad delete. None of it is exotic. What makes it hold is that every box in the picture is built from automation, so the one you rebuild under pressure comes back the same as the one beside it.

What a managed infrastructure contract is actually for

It is a set of answers to questions you only ask on a bad night. These are the ones this platform has.

Traffic arrives all at once

The cache answers the reads and the queue holds the writes, so a spike becomes a longer queue rather than a slower database.

The primary database fails

A replica is already carrying the same data, which is a different thing from having last night's dump.

A server has to be rebuilt

The configuration is automation, so the box comes back identical to the ones beside it rather than the way somebody remembers it.

A vulnerability is published

Hardening and patching sit on our side of the line. The platform team usually hears about it after it is done.

Something breaks at midnight

Monitoring sees it and it reaches us. Finding out from your own customers is the failure worth paying to avoid.

Data has to come back

Restores are drilled rather than assumed. A backup nobody has ever restored is a hope with a filename on it.

Hardening is the part that gets done once everywhere else. Every box we run gets the same base: no password logins, a firewall that denies by default, security updates applied on a schedule somebody owns, and logs that outlive the server they were written on. That is the same discipline as the rest of our Linux server work, applied to a platform that moves other people's money.

Where infrastructure stops helping

Most "the server is slow" tickets are not the server. They are a query with no index, an N+1 in the application, or a job that runs at the same time as the backup. We will find it and tell you, but the fix is in your code, and no amount of RAM buys it back. Infrastructure has a ceiling and the application sets it.

Two other limits. If you already employ people who do this, you want hands beside them rather than us running it, which is the engagement above. And if your traffic is genuinely spiky, quiet most of the week and ten times that for a few hours, dedicated servers charge you for the peak every day of the month. A cloud region is cheaper then, and we would rather say so than sell you the servers.

If your platform is growing and nobody really owns the servers under it, that is the conversation. Send us what you run today and who gets called when it stops, and we will tell you what we would change first, and what we would leave alone.

06
Flagship engagement · Custom software

Custom Fleet Management Software Running a 565-Truck Workshop

Every truck that came into the bay left with a service card written out by hand. Parts came off a spreadsheet. Oil came out of a drum and onto a page. None of it was dishonest and all of it was invisible: nobody could say what a truck had cost over a year, which tire brand was worth buying again, or who had signed for the twenty liters that left the store on Tuesday.

A workshop runs like that for years and it works, right up to the fleet size where the gaps cost more than the fix. A single plant outside Dar es Salaam distributes soft drinks across the region, and its fleet had grown past the point where anyone could hold it in their head. We built the fleet management system it runs on now, from an empty database, around how that workshop actually works.

The work order as the gate every material passes through A truck arriving or a service interval falling due opens a work order on that truck. Parts, tires and dispensed fluids can only be recorded against an open work order, each deducting stock as it happens, and closing the order prints the service card and writes the audit row. Truck into the bay Interval due 11 per truck Work order open on this truck nothing moves without one Parts issued Tire fitted Fluid dispensed the pump will not release oil without an open job Stock down, alerted Position, retreads, km Liters off at the hose Closing prints the service card. Every change keeps old value, new value, user and address.
The oil pump had to agree with the work order The requirement nobody writes into a brief is usually the one that pays for the project. Fluids were the leak here: a drum, a hose and a signature. So the dispensing equipment talks to the system through a pair of APIs, and it will not release oil except against a job open on that truck. The liters come off stock as they leave the hose, not at month end when somebody tries to reconcile a page. No packaged product does that, and it is the single reason the build was worth commissioning.

What the software replaced on the workshop floor

None of it is clever software. All of it was leaking money.

A service card written out by hand, then lost.

A branded PDF the system prints when the work order closes.

Parts counted in a spreadsheet and reconciled later, if at all.

Stock deducted as parts are issued, with an alert before a line runs out.

Tires fitted and scrapped with no record of where they sat or how far they ran.

A code per tire, its wheel position, every retread, and the kilometers behind it.

Oil and fluids dispensed on trust.

Each dispense authorized against an open job and deducted at the hose.

Service intervals remembered, or noticed when something broke.

Eleven intervals per truck, each raising its own card when it comes due.

Who changed what, answered by asking whoever was on shift.

Old value, new value, user and address, on every table.

It runs on the plant's own server, not on a subscription, and the client owns the source, the schema and the data. Adding a hundred trucks costs a hundred rows. We keep that server patched and upgraded under a support agreement, with daily backups on a thirty-day retention, and the last restore was actually tested rather than assumed. That is the only backup statistic worth quoting to anyone.

What it still does not do

Servicing, parts and tires are digitized. Vehicle intake is not, so a fault is still found halfway through a repair instead of at the gate. There is no requisition chain, so a parts request is still a conversation. Permits, insurance and speed governor certificates are still tracked by whoever remembers, on a fleet where a lapsed certificate is a truck that cannot legally move. We have scoped that work and it is not built. A buyer should read that here rather than find it in month three.

The wider version of the same honesty, said before we quote anybody: most workshops this size should buy a fleet package rather than commission one. A custom build earns its cost when the process is genuinely yours, which here meant eleven service intervals per truck model, a tire lifecycle no vendor sells as a product, and a fuel pump that had to be inside the system. If your workshop looks like everyone else's, we will tell you to buy the package and help you pick it.

We build software around how a business actually works, and the honest first step is checking whether something you can buy already covers you. When it does, we say so. When it does not, send us the rules, the workflow and the reporting that are genuinely yours, and we will scope a system around them.

The Rest of the Practice

Cloud, security, ERP and multi-site IT

The work above is what we publish in depth, because that evidence is ours to show. The rest of what we do sits below, written as what the engagement actually involves rather than as a case study with numbers attached. When another client clears us to publish theirs, it moves up there with the other six.

Cloud migration and cost control

Servers past their warranty, or a cloud bill nobody can account for.

We inventory what actually runs before anything moves, because the cheapest migration is usually the one where a third of the estate turns out to be retired rather than lifted. Then it goes in waves, each with a way back. AWS, Azure and Google Cloud.

See the service →

Security audits and incident response

A breach you have just found, or a customer questionnaire asking for controls you do not have.

Containment first, then how they got in, then closing it. Outside an incident it is an audit scoped to what a regulator or your board will actually ask about, the Kenya Data Protection Act included, rather than a scanner report with four hundred findings and no order of priority.

See the service →

ERP, CRM and custom business systems

Finance in QuickBooks, stock in a spreadsheet, and a month-end that takes a week.

Odoo, ERPNext or Frappe, with KRA VAT and statutory deductions configured for how Kenyan payroll actually runs, and a custom module only where the standard system genuinely does not fit. These projects fail on scope rather than on software, so the first phase covers the processes costing you money now and nothing else.

See the service →

Managed IT and helpdesk

No monitoring, no patch schedule, and a helpdesk that is whoever picks up.

Monitoring, endpoint protection, patching and a ticket queue with a response time attached to it. Single site or several, including teams split across Nairobi, Mombasa, Kampala and Dar es Salaam.

See the service →

Web and mobile development

A business that sells in person and has no way to take a booking or a payment online.

Sites, e-commerce and apps with the local rails wired in: M-Pesa for payment, SMS for reminders and confirmations. Built so the next person to maintain it does not have to call us.

See the service →

IT strategy and program management

A project that already failed once, or five offices running five different systems.

An audit, a roadmap you can budget against, vendor selection where we are not the vendor for every line of it, and someone accountable for delivery. The useful output is usually what to stop doing.

See the service →
WhatsApp