The Technical Architect

I had to write the article about the Technical architect position, especially because the position has been recently questioned in many circles. And it is not because I am a serving technical architect, as that means very little. I find immense value in the position, as do many other developers. If you asked many of my colleagues–whom I had the pleasure to work with–they would not envision the world without a Technical architect.

So why are our experiences of working with an architect so vastly different? I will try to explain this in the following article.

Let me preamble this by saying that when we say things like "Technical architects are useless", or "Typescript is bad" in the industry, those aren't truth functional statements. They aren't logically equivalent to "Mary has two legs". Rather, they are emotive statements. They are more like "Booooo! Technical Architect" or "Yaaaay! Typescript", which reflect the speaker's internal emotional attitude towards the idea, rather than any real quantitive analysis. Providing anecdotal examples of bad or good interaction with a Technical architect is folk statistics, and folk science is no real method of getting to the Truth.

(Besides, good luck getting a real "objective" account of the superiority of any similar matter, such as „Rust sucks" or "C# is better than Java" with a scientific approach; it is close to impossible.)

But let's go back to the specimen at a glance, an architect.

The Role in Question

We understand software architect as a role, but roles get a bit hazier the higher the level we go. For instance, a Technical Architect's role is vaguer than that of a Software Architect. According to Simon Brown, the author of the C4 model, Technical architects can include aspects of software and cloud architecture. If we go higher than that, roles like a Solutions Architect can even vary significantly more. I've seen Solutions architects being fully hands-on tech people deep in designing AWS solutions, but also firmly on the other side of the fence; advising businesses on their needs. Again, a Software Architect is the most obvious and has commonality with the Technical architect, at least in a few key points. It is also the position that interacts with engineers the most, daily.

In this post, I'll discuss Technical architecture as described above, which broadly includes cloud and software architecture.

Architecture and Design

Even though not entirely the same, architecture and design are similar concepts. A common definition of design in the tech context is shaping the system. Architecture, on the other hand, is the current shape of the system. It can be the shape of a UI system, underlying infrastructure, or the entire fleet of systems. Things are shaped all the way to code components. Code components, and their arrangement, are the concern of software architecture. Things are also shaped at a distance, which concerns cloud or infrastructure architects. (There can be even more zooming out, even to the enterprise level, but this is not the architect we have in mind in this post)

The technical architect's job is to oversee how the system is shaped on varying levels of abstraction. Because of this, an architect is naturally in a hands-on position. Designing is an activity, after all. As Uncle Bob explained in his famous Clean Architecture, an architect should typically be among the most competent developers in a company. There's a simple reason for that. Any technical leader has to earn a degree of engineers' trust. Trust is demonstrated by repeated activity that leads to outcome. This, in turn, creates eagerness to follow the leader (engagement), so that the leader can effectively guide the company's digital vision. It is a cat-and-mouse game.

Without all of the above, there's little success for a leader. Without trust, it's nearly impossible to get a vote of confidence. Moreover, it follows naturally that interpersonal relationship building is important in the role of the technical architect. Relationships are key to understanding people's problems and to boost engagement.

(By the way, all above is not only true of a Technical architect but any Staff+ role. Staff+ means as much as individual contributor roles. There is a great book Staff Engineer by Will Larson that describes some of those topics and roles in depth).

Generally speaking, whenever I hear about negative "experiences" involving working with an architect, I immediately know where the root problem is (or was). It is typically the lack of trust, which is the lack of track record or results. You can be as smart as you'd like, but one is always measured against something material.

Now that we have defined the specimen at large let's talk about the architect's daily practice.

Daily Bread

Skills-wise, an architect is the most full-stack or generalist type of engineer, stretching across multiple domains–whatever design requires. An architect is a prime example of a T-shaped position. T-shaped people consider themselves cross-functional; they have broader knowledge in many areas.

Chances are that you read about the dreaded days of Big Architecture. Back in the day, an architect would write holy commandments and hand them over to common folk (the developers). Architects were the white wizards in the ivory tower. Designs were top-down decrees that the development team had to adhere to. No matter what.

To nobody's surprise, this approach failed spectacularly. Perhaps the common complaint was that architects created problems for the engineering teams but did not solve any of them with the teams.

I mentioned before that an architect is a hands-on role. I should elaborate on why. The biggest reason that software architecture must be a hands-on responsibility is that, contrary to buildings, software is not aimed to be forever set in stone. With buildings, unless a hurricane or some other catastrophe happens, one can expect no drastic changes to the building structure and the overall construction process. Plan it once, and it is done. Houses stand many years before they need any changes.

Software is nothing like it. Software exists in a perpetual hurricane where the primordial forces of business, networks, customer demand, and other volatile factors meet. Architects must make small or large adjustments along the way, and navigate the corrective course. They should be the guiding light for the project's implementation. They should possess good soft skills and communication and offer expertise whenever necessary. They must be good team players if they are to be good architects.

The architect is then a guide. The word guide here is of vivid importance. A modern architect does not tell anybody what to do and shouldn't own every technological choice as a bottleneck. Neither does an architect push for the one-directional style of engagement characteristic of Big Architecture days. Instead, an architect should specify the constraints, principles (e.g. systemic qualities), and a pool of design choices that will guide teams to the right technological decision, under a common consensus. It is always possible that the architect is wrong on all accounts and in a healthy environment, an architect should be successfully challenged by the team. After all, leaving choices to a single person can never be good as nobody is omnipotent.

Lastly, I'd like to clarify that I am not saying that an architect won't ever have to make technological choices. Yes, an architect will do that occasionally, which is unavoidable, but that shouldn't be the default expectation. The default expectation is guidance toward the decision. The decision becomes a document artifact of the system.

Just as an architect needs to guide towards something, he has to stray away from a few things. We'll now explore what the architect should avoid at all costs.

Architect’s grave enemy

The biggest enemy of an architect is the early decision. Big early decisions limit the pool of late options. When the time comes, an architect must capitalise on the number of decisions that haven't yet been made. The best time to make the right decision is with the right amount of information. Knowing that exact time takes experience and it’s not a trivial skill.

Naturally, in the early stages of development, most projects have many paths of evolution. This translates to nearly countless opportunities. At the same time, only limited data is available. We rarely have the complete picture when starting with anything at all. When decisions are made, the windows of opportunity naturally close. The job of an architect is to keep them open for as long as possible, so business goals can be achieved even if requirements change late in the game. 

The approach to late design decisions was pioneered by a Japanese car manufacturer, Toyota. It has become known as the Toyota Way. It stands in contrast to Taylorism, a management philosophy characterised by strict top-down decision-making and standardized, immutable processes. Lean principles, integral to Toyota's Way, stress flexibility and continuous improvement. This radical departure from Taylorism influenced the Agile software movement, leading to the collapse of Waterfall, a model associated with big early decisions and rigid structure. The birth of Agile gave rise to more adaptive and iterative development methodologies. We call these methodologies empirical.

Architecture as an empirical science

Empiricism is the philosophical stance that emphasises the importance of observation as the primary source of knowledge. It states that knowledge is derived from sensory perceptions, experimentation, and evidence gathered from the external world. In the context of Agile and Lean methodologies, empiricism is associated with an iterative and feedback-driven approach.

Agile frameworks work through empiricism by promoting continuous adaptation. They borrow heavily from the Toyota Way. Teams gather real-world feedback and through frequent reviews, they adjust based on what was observed.

Two big concepts rooted in Lean; Just-In-Time (JIT) and a pull system, are important here. JIT aligns orders with production schedules, being driven by customer demand. It applies a waste-reducing strategy to do that. It means that resources are only worked on when we know they will be immediately consumed. Pull systems complement this by responding to actual customer trends and requests. These systems work on real-time data and continuous customer feedback. Without the customer pull, no work should be ordered. It is all empirical and observational by virtue.

Empiricism has become a dominant paradigm in our lives. We strive to get data at every step and make small adjustments once we have additional information. Just like with the advent of empiricism in modern science, we now apply the same methodology to designing systems. Designing computer systems is on the perpetual revision course.

With this in mind, it is clear that designing systems is rather hard. It requires the constant involvement and feedback of leaders who can guide the system’s shape at every step. From the use cases and conceptual designs to the detailed code reviews, technical architects ensure that capabilities are implemented correctly.

As you see, architectural guidance in this sense described in prior sections perfectly matches the Agile practice picture I have just drawn. An architect guides and helps to correct the course of the ship by specifying possible decisions, exhausting the number of outcomes, and carving out implications for the future roadmap. The architect is successful if he/she helps the team make the right decisions. The right decisions are ultimately measured in KPIs and other metrics defined by the team.

Alan Kay’s Extreme Late-binding principle

If you need a more geeky principle to live by than something manufacturing-oriented (e.g. Lean), I will give you Alan Kay’s late-binding principle.

Alan Kay was the original inventor of one of the most influential languages ever–Smalltalk, and the core tenet of Smalltalk was extremely late-binding.

As Kay explains, extreme late-binding enables you to avoid prematurely committing yourself 100% to a single right solution for any problem. This flexibility not only simplifies revising those decisions later since you anticipate them but also allows you to potentially create systems that can be modified while still operational. It is an intentional design of those systems that makes them operable as such.

That is a great principle to have. The entire Agile enterprise is, in one way or another, an attempt to apply the late-binding principle, to get comfortable that the only constant is change.

(I know I talked about Agile in-depth quite a few times here, and many developers loathe the very term, likely because of Scrum, and other mini-waterfall frameworks like SaFe. Others even say the Agile movement should be disbanded and started anew because of this. Time will tell. For now, just imagine that Agile is all about late-binding.)

The Power of Reason

We talked a great deal about the empirical (or late bound) approach in sections before. Empiricism is all the rage these days, but empiricism is not the only scientific stance that humans tend to follow. The opposite tendency is rationalism.

Rationalism holds that primary knowledge begins in our heads, independently from experience (as Descartes famously declared). Rationalism establishes the primacy of reason and logic over sensory data, which it holds secondary.

Peter Thiel is a good example of a rationalist. His bestseller book "Zero to One" champions the primacy of reason over empiricism while criticising Lean and Agile methodologies. They are attacked for being overly reactive and not forward-thinking enough. There's a grain of truth in what Thiel says.

Sometimes experiential data is not enough and we need a broader design approach. This is where we need to spend more time in our heads to pave the way for innovation, rather than follow what the competition is doing.

Forward-looking is where Architectural enablers come into play. Namely, there are times when a person needs to headway the design of future business capability which cannot be achieved now. If a company has an architect, that person is responsible for making sure we achieve the desired functionality in time. When an architect is absent, this can be the responsibility of a technical leader, which is a much more generic position than an architect.

Simply put, a company might have a big plan and that plan needs plenty of forethought. The job, in this case, is to de-risk the planned delivery of the feature or capability. The artifacts that are borne at this point are architectural enablers. They are likely to start as Epics and then boil down to tickets on a task board. Eventually, they will result in early fat marker directions on the possible futures of digital capability. An architect will work through the architectural principles and constraints that give birth to the new capability.

As a rule of thumb, we should never design too much too early, and we should design just enough, but not more. It's difficult to strike that balance and this comes with experience.

Dave Thomas captures this sentiment in just one sharp quote:

"Big design up front is dumb, but doing no design up front is even dumber."

What determines whether we utilise architectural enablers (or design upfront) is the size of the business ask, its complexity, and how much of a gap exists between the current and the future. If it's possible to achieve the vision without a larger design, there is no need to design upfront. Designing upfront should not be a default approach in modern practice. It is always best to design Just-In-Time for multiple reasons.

This is where keeping in mind Lean principles can always be beneficial to us, even when having to design early. Generally speaking, there should be no work pulled unless there is a clear demand for it. If we design too much too early for no particular reason, the system will change multiple times over a few quarters. By that time, much of what we did could be either thrown away or require redoing. Even worse is the fact that the demand for the capability might entirely vanish, and we end up having invested ourselves in an item of no value. And this is neither good for us nor the company.

The Cost of No-Architecture

We’ve been talking about the architecture, but maybe let’s talk about zero architecture whatsoever. There is a famous formula that is relevant here and was described by Uncle Bob in his book Clean Architecture. It states that, over time, it costs more and more to update badly designed systems. At some point, teams grow, but the output continuously diminishes. In other words, systems with little to no design will experience an ever-increasing cost of change. This can become the Achilles' heel of not just a single application, but the whole company.

Paradoxically still, certain people say "No to Architects!".

According to these folks, architects have no place in modern tech companies. In my view, they either use an obsolete definition of an architect straight from the days of Big Architecture or never actually worked with a successful architect on the team.

Let's look at a particularly strong statement from ThePrimeagen's Youtube video:

"There is like no architects [at Netflix], we just don't have architects. Architects are the ones that go around in lab coats telling you how you should write your code and its very fantastic and it's very beautiful and it works perfect on a whiteboard, but when it actually comes to implement, its horrifying an awful. I don't like architects, okay, I don't want to have anything to do with architects. I've never had a good experience involving an architect. I swear every single time it's always just awful."

For someone who spent the last decade in Netflix, which has no architects, it begs the question of when was the last time the man in question worked with an architect? How accurate is the picture he's painting, especially regarding modern practice?

There are many mistakes and misunderstandings in just this single YouTube quotation above. It distorts the modern practice of an architect. As I described in depth in the earlier sections of this article, architects are not "telling" anybody what to do. They do not create big "whiteboard" designs upfront that are to be followed or implemented religiously. This is not the modern practice I have been describing in my article.

To reiterate, architects are guides who aid the team in making better decisions and crafting principles that help in making better decisions over time. There is no point at which an architect sits in a dark room and finally comes out with "The Immutable Laws of Implementing System X". Architecture is an ongoing collaborative effort between an individual contributor and the rest of the team. Early, constant iterative feedback, is what we architects do.

Is there any hard data on the topic? Other than anecdotal examples? There is, albeit indirectly. There's been research on adequate system design. Let's have a look.

A study by McKinsey found that as much as 20-40% of the value of the entire technology budget for legacy systems could be consumed by carrying costs of inadequate technical solutions. Companies incur a "rework" cost of about 30-50% in future projects due to bad decisions taken in the past. Bad and irreversible design decisions.

Another study by Capers Jones researched software development practices and productivity. His findings indicate that the cost of fixing an error after deployment is often 100 times higher than fixing it during the design phase. This exponential increase shows the cost implications of no architecture.

Thirdly, the Consortium for Information & Software Quality (CISQ) released a report that tells us that improperly designed systems cause huge costs to businesses, ranging up to $2.84 annually in America. If we don't need architects we need to try harder to find an alternative.

A Few Field Examples

Architects are very efficient and successful by a few examples. We already mentioned Netlifx before in the context of no-architects (Primeagen's video), but how about we twist it around?

To recall an inspiring story, let’s look at the example of said Netflix, a member of the FAANG gang. The success of Netflix as a cloud platform was largely paved by Adrian Cockcroft, who was, at the time, a serving Cloud Architect at Netflix. His design paper is widely available and well-known. It made Netflix what it is today. Cockcroft’s work in designing Netflix’s cloud architecture influenced the entire industry, and Netflix cloud patterns are still widely referred to as a golden standard for microservices. After his success at Netflix, Cockcroft became a vice president of cloud architecture strategy at Amazon Web Services, where he continued his work.

Still, you can sometimes get away with not having dedicated system designers in your company. This is only possible if you have exceptional, design-minded engineers. If everyone is design-minded, you don't need an architect. In any case, this is very rare. Netflix does pay top dollar for engineers and every single one of those guys is likely a superb software architect.

My personal experience after conducting dozens of interviews in my career is that I know that engineers up to a senior level are excellent at programming, but do not necessarily excel at systems design. I understand that in some companies, the role of an architect might be redundant, but if an architect is successful at bringing design thinking to the forefront where it was absent, then architecture becomes the first consideration before any work is done.

To conclude, even if a company, like many FAANG companies, does not have an official person who is a systems designer, it never means it doesn’t have architects. It is common for top designers to be Staff Engineers or Principal Engineers, so don’t get bogged down by definitions, as every company has a slightly different structure. Since an architect is frequently a technical leader, these positions are often used instead but can mean the same thing. At some point, these definitions become blurry and unimportant. Any systems designer should be able to prove his or her design through hands-on activity (i.e. coding).

There is one fact of the matter, however. The issue of too little design-mindedness is most problematic for startups that are tight on budgets. It is less of an issue for big FAANG corporations who are already established because if they face a slowdown, they have enough budget to throw more money into the problem. As a result, the increasing number of resources can keep the pace of development, even though the system could be increasingly difficult to change.

Some companies such as Basecamp understand this problem very well and prefer to stay small, so their systems don't become too complicated. They both have splendid product design methodologies (see Shape-up), but also keep their development teams small, in the range of couples of dozens of people at most. David Heinemeier Hansson, the CEO of Hey and Basecamp, prides himself on creating small enough environments of change, where change is quick to enact without throwing many resources at it. Design is simple, teams are small, and companies are modest in size. Every developer can understand the entirety of such a system end-to-end.

Good design is then simple yet scalable and flexible. It has just enough abstraction balanced out with pragmatism (YAGNI–You aren't going to need it). But make no mistake, it's hard to strike that balance. It's a common misconception that simple designs are "easy" to achieve. Nothing further from the truth; simple, elegant designs that work for a company's vision are hard to get right. Due to complexity, nearly every company I had the pleasure to work at within the last ten years faced a need for at least one general system rewrite. There was eventually a tipping point when things could not be tolerated anymore, and couldn’t be updated easily. Rather than being maintained and extended then, entire libraries were decommissioned and replaced with new ones.

If backend systems are challenging to design, then designing UI systems is even harder. UI systems are volatile and susceptible to rapid modification. UIs change much more frequently than the system’s business logic. Hence, on the client side, it is easy to fall into the trap of endless volatility (or hype). Regarding the UI systems, we haven’t lately paid much attention to designing those properly. Frontend applications are frequently written as firmware and not software. They can manifest incredible levels of coupling that lead to entire project rewrites by a straw of few business requirements. Developers rewrite frontend applications every few years or so, frequently because of a whim (hype) or because the project was badly written (which means as much as it was badly designed!).

Indeed, if we didn't need architects, then why all of the above is the case?

When we look back at some of the most successful software in the world, we can see that it changed very little over time. (By little, I mean that it sustained its lifetime without total platform rewrites). Take two examples; Microsoft Windows, and Apple’s native platform. Their success is based on relative stability or simplicity. The former promises backward compatibility, the latter has a philosophy of keeping the system small and simple (UNIX). In either case, the design had to be careful and it provided a foundation for the future success of the business.

The Cost of Hype

The cost of change can also be directly associated with the cost of hype which can indirectly hurt the system’s design. I will illustrate this with a simple example.

A popular acronym is now the developer experience (DX). Focus on DX has been obsessing over the UI industry for quite a while. Bundlers, preprocessors, build tools, test runners, frameworks, and transpilers. They all bring developer experience. The danger of DX fixation is that engineers can rewrite the whole app into a different technological stack just for the sake of the experience.

In the end, this all hurts the end user. Alex Russell has an excellent article on the topic. In his “Developer Experience” Bait-and-Switch, Russell warns everyone that heavy focus on DX can come at the cost of end User Experience (UX).

Recently, however, things have returned to extreme simplicity in terms of DX. Old, rusty technologies such as Django or Rails are coming back with a bang.

Have a look at the recent explosion in HTMX. What is the entire HTMX trend if not slightly more interactive server-side templating, back from the year 2010? It goes back to the basics, easy design; it shares no gratification over DX whatsoever. Atop of that, #nobuild becomes used more and more often, as everyone is tired of spending the majority of time on perfecting the build system instead of serving the needs of the company so it doesn't slide into oblivion.

What we need instead is to build solid foundations for our system. Our system should require little change over a long period, it needs to be simple, but not a volatile mess. It needs to be simple but open to sharp turns imposed by the business. If changes are required, we make sure we leave our options open. Don’t get me wrong, the developer experience must be more than adequate; otherwise, we won’t have happy and productive engineers. Poor developer experience can be costly to the project since it directly impacts the lead time. Lead time is the time from when work is picked up and when it hits the market. We need to have an adequate developer experience, so we can be agile and rapid, but the entire developer's practice shouldn't revolve around DX.

Anecdotally, I experienced the downfall of a struggling startup where developers, instead of working on new features–which were requested by investors–spent the next couple of months migrating the entire codebase to TypeScript. Not only did the effort take six times the planned time, as it usually does, but the company did not survive, so to speak. Perhaps it would’ve been wiser to first deliver the promise, buy some time, earn some dollars, and then migrate the codebase.

Rewrites and revamps can be dangerous. Do you remember the story of the Netscape web browser in the 1990s? Because of neglect of the browser’s architecture and the ultimate decision to rewrite the browser, Internet "Exploder" gained absolute market dominance.

This is not against proper developer experience, modern frameworks, or tooling. This is a reminder that we should prefer system design and tech leadership over shiny jewelry. Design qualities are, for example, extensibility, testability, scalability, and so on. Or maybe I am stupid that I prefer services with good design patterns and practices over how cool they are built.

Let me know.

Conclusion

Hopefully, this rather long rant convinced some why the role and craft of Technical architects are very valuable in this day and age.

I won't be able to convince folks like Primeagen, but trust me, one negative experience from a long time ago, does not invalidate the entire profession off the bat.