a software architect cares about the unseen things

// architecture · observability · resilience

---

A software architect cares about the things other people don’t worry about… especially when everything is working fine.

But don’t just take my word for it, Wikipedia says:

A software architect is a software engineer responsible for high-level design choices related to overall system structure and behaviour.

It’s a software architect’s responsibility to match architectural characteristics (aka non-functional requirements) with business requirements.

Once, when our Cloud provider turned off a managed database during normal business hours for maintenance, management asked, “Why didn’t you build this system so it can work without a database?”

Most software engineers I know laugh when I tell them this story (most systems will blow up horribly when the database is down), but actually, Business was right… a consumer app should be able to function without a working database. And surely, 5 weeks later, during our next database outage, the system still allowed mission-critical functionality while degrading performance in less critical areas.

Does your app have a status page? Are you able to put your software into maintenance mode when downstream vendors are having issues (to stop the bleeding)? Do you know how many API tokens you are burning through, or what your servers will cost you this month?

When your system goes down, who lets you know first? Your ops team or your customers?

These days, as a single mistake in a prompt could kick off a loops in AI workflows that shoot your token usage through the roof, will you even know?

It’s not all doom and gloom. Here are a few recommendations of where you can start:

1. Observability is more important than testing. (Though testing is still critical) You need to wire in mechanisms into your ecosystem to determine whether the system is up (liveness), responding to requests (readiness), and healthy (healthchecks). Depending on where and how you host your system, you should have some options available. For example, Kubernetes will probe your containers for you if you tell it how to do it, and there are lots of free benefits to doing this (like zero downtime deploys)
2. Pull metrics out of your apps. Log request times and statuses, database response times, and errors. Silent errors are the worst! It’s common practice in programming to have mechanisms to catch errors (AI agents love adding them). However, in an attempt to make code robust, exceptions often go unnoticed, and bugs get “Weird”. Start measuring on a macro level, and remember: where there is smoke, there is fire. Customers don’t complain, they just leave. Make sure to actively look for things going wrong in your system all the time - or let an AI agent parse your logs for you.
3. Your system should still work if one or two components are down. If you can’t do this, you will never be able to deploy changes without downtime (trust me, you want to be able to do that). The key to doing this is for subsystems to communicate ASYNCRONOUSLY. Queues are great for this (e.g. RabbitMQ), and I highly recommend you look at implementing queues for any process/request that takes longer than 100ms
4. Timeouts are scary but necessary. Especially in transactional systems, a timeout can be a very difficult error to recover from, since a transaction may have succeeded or failed, and you may not have the means to determine the actual outcome. However, not having the timeout could pull down your entire system. It is essential to work through these error scenarios and make sure the system does not retry successful transactions or report a successful transaction as a failed one, or you may double-bill customers or double-provision virtual goods.
5. Maintenance mode is your safety net. The first thing a user does when something doesn’t work is they try again. As a result, most outages cause a spike in traffic that actually worsens the problem to a point that you cannot realistically even see what’s wrong. By cutting off the inbound flow into your system, you can calm things down enough to actually fix the problem. A practical way to achieve this is to add an environment variable that switches your gateway into maintenance mode so that all page requests redirect users to a maintenance page, and API requests return a 503 response (helping your upstream consumers to also degrade gracefully)