Technology Evaluation Criteria

Congratulations! You’ve heard of a great new technology promising a better development experience. Before you dive in and start to convince your team to use the technology in your next project, let’s take a few seconds to consider some questions.

Is the technology’s API stable?

A technology with a rapidly changing API presents significant development challenges. Every time you upgrade, you run the risk of breaking core portions of your system, forcing you to rewrite the things that change over and over. The alternative to that is to not upgrade at all, which leaves you stuck using a technology that may be missing features or security patches that you need. Make sure that the API is stable and has a proven track record of long-term support and proper deprecation roadmaps, or you may find yourself struggling to build castles on constantly shifting sands.

How much would it cost to keep up with any changes?

Development time costs money. Developers need to read and learn changes, code needs to change. If the technology changes very rapidly, you have to keep up. Most of the time keeping up is a good thing, but it’s important to consider the business cost.

Are major pieces still being added or removed?

Like the changing API, if you don’t know what the platform will look like even in the short term, chances are you’ll suffer from it. If they suddenly remove a key component your software is relying on, or they add a way of doing something that becomes core to the platform after you’ve already written code, then it can become costly to switch.

If you had to hire a team of developers to use this technology within 2 weeks, could you do so easily?

Using an amazing language that only 3 people in the world know isn’t a sound business decision. Consider the bus factor — if the only dev that knew the technology got hit by a bus, could you find a replacement for them quickly enough to not suffer significant business losses? Make sure that you have access to a steady supply of available talent that knows the technology.

If you had to train a team of developers to use this technology within 2 weeks, could you do so easily?

If you can’t hire, you’ll have to train — how complex a technology is to learn should be considered before adopting it. If it takes a year of intense training to learn and understand a technology, then access to a talent supply suddenly becomes a lot more important.

How much do developers who know this technology cost to hire compared to current developers?

If a technology improves development effectiveness by 20 percent, but the developers that know the technology cost 80 percent more, it’s simply not worth it. Factor in the cost of hires.

Is there a healthy amount of 3rd-party libraries and tooling available for this technology?

A base technology might be great, but if it doesn’t have a vibrant 3rd-party ecosystem, it might be a problem. Your only options would be to roll your own tooling, wait for the vendor to provide the tooling, or make do without them. Sometimes it’s worth it — many times it isn’t. Make sure there’s enough libraries and tooling that’ll allow you to accomplish what you want to accomplish.

Is there a healthy amount of tutorials and written content available?

Developers learn from examples. Make sure any new technology you adopt has ample documentation, tutorials, and code samples. Look for blog posts, questions, and articles about the technology to ensure that if your developers have an issue, they have the materials needed to figure it out. It’s also important to ensure that the materials are up-to-date and accurate — rapidly changing APIs can make tutorials even a few weeks old completely irrelevant.

Is the technology’s ecosystem growing or shrinking?

A growing ecosystem is a good indicator of healthiness — people are moving to the technology for a reason. However, if people are abandoning the technology in droves, it’s time to figure out why. If you see everyone running in the same direction, make sure you shouldn’t be running as well.

Is the language the technology is based on widely used and growing in popularity?

Sometimes a technology might itself seem stable, but be built on a completely unstable base. Check to see that the technologies your technology uses are solid and reliable. If those fall through, chances the technology you are evaluating will also suffer.

Is the technology backed by significant sponsors?

Technology backed by corporate sponsors and large communities are often a lot more reliable than technologies built by a single developer over the weekend.

Is the language the technology is based on backed by significant sponsors?

Same as above — is the base going to be around for a while?

Does your company have the skills to utilize the technology?

A technology might be perfect, but if your company doesn’t have the skills needed to take advantage of the benefits of the technology, you might as well be using a virtual paperweight. Find or train talent.

Is an expert in the technology available?

Generalists can only get you so far. You’ll eventually run into a situation where you’ll need an expert’s help. Make sure you can find one before you need them. Experts can help avoid pitfalls and poor design decisions that may take months or even years to reveal themselves.

Does the technology play nicely with existing technologies in use within your organization?

If you buy a Windows-based IDE for your team of developers that all use Macs, it’s a poor purchase decision. If your organization primarily uses Sharepoint, it might not be a good idea to purchase Peoplesoft modules.

Does the technology meet the organization’s current and future needs?

Don’t invest heavily into card swiping systems if your organization is moving to a card-not-present business model. Make sure that technology decisions you make consider the direction of the organization.

Does the technology offer a significant advantage over currently used technologies?

Changing technologies for the sake of changing technologies is a waste of time, effort, and money. Unless the technology gives a massive benefit, chances are it isn’t worth it. Switching from relational database vendor A to relational database vendor B when you don’t and won’t use any differentiating features of either is a waste and akin to programming for your resume.

Are currently used technologies being fully leveraged or utilized?

Why change technologies if your current technologies can handle the situation just fine with a bit more utilization? Make sure that the reasons you are evaluating the technology for still exist even after accounting for the capabilities of your current technologies.

Does the technology have any fees or costs associated with it?

Some technologies are expensive — tens of thousands of dollars per month in licensing fees is no joke. Even smaller monthly fees from service providers add up especially considering you’ll likely need dozens of other services all charging smaller fees.

Can the organization afford to pay these fees or costs?

The technology might be great, but if it bankrupts your organization, find an alternative.

Does using the technology lock the organization in to that technology’s ecosystem?

Flexibility is important. Sometimes moving to a technology means a very long business relationship with the vendor and ecosystem. You begin to rely on everything provided by the vendor, including consultants, training, support, modules, patches, technologies, etc. Make sure you’re OK with vendor lock-in and make sure you’ll still be OK with it 15–20 years from now.

How easy is it to change away from the technology?

Being able to quickly reverse a poor technology decision is an advantage. If the technology doesn’t intertwine itself into your systems and can be easily rolled-back or replaced, it reduces the risk of adopting the technology.

How easy is it to change existing systems to use the new technology?

If integrating with your existing systems is incredibly fast and easy, it’s an advantage as it provides an opportunity to realize value from the technology much faster.

How much customization is required to make the technology work for your current and future needs?

Off-the-shelf solutions sometimes don’t quite do what you need, and you need to work hard on transform it to do what you want. Consider the costs associated with modifications.

Can your technology be used for production use?

Some technologies just aren’t appropriate for production use. If you’re tinkering, that’s fine, but chances are you want the technology to be used in production. Make sure the technology you are choosing is mature enough for that, and not just a hobbyist’s tinkering tool.

Does the technology rely on other technologies being present?

When you bring in one dependency, you bring in all of its other dependencies, and their dependencies, and so on and so on. Make sure to evaluate those as well.

Can you easily change behavior of the technology based on the target or environment?

Chances are you have multiple environments for development, including staging and test servers. You may have multiple environments for production as well — different clients may require different behaviors from the technology. If the technology is difficult to change based on the target or environment, you may need to find an alternative solution.

Can the technology be deployed in the way you need to be deployed?

Make sure the technology can easily support your deployment scenarios and can bend where you need it to bend. If it needs to be manually deployed and your process relies on automatic deployments, you might have trouble.

Does the technology require changes to other technologies in use?

Technology choices don’t happen in a vacuum. Deciding on a technology cascades into other decisions as some possibilities open up and others close.

Does the technology have any competitors?

Don’t buy the first cow you see at the market. If a technology has competitors, see what they’re offering. Compare prices, features, and offerings.

Does the decision to use this technology need to be made now?

If you can defer a technology decision, you can make the decision when you have more information. If you don’t have to decide on a technology now, it might make sense to wait. Of course, don’t let yourself wait just because you can’t decide.

Is the technology secure?

If you’re using an insecure technology, you’ll need to ensure that breaches don’t cause damage or lead to breaches of other systems. If the technology isn’t secure, it better not be used for anything that does require secure technologies.

Is the technology usable?

Make sure you can use the technology. If the technology requires an always-on internet connection and your environment deals with offline or spotty connections, you’re going to have a problem.

Is the technology performant?

Your organization likely has certain performance requirements. Make sure the technology meets those requirements.

Is the technology maintainable?

Maintainability is paramount — if you can’t figure out how to keep the technology running for the next 10–20 years, it might not be the right choice.

Is the technology being updated?

If patches, features, bug-fixes, and security updates are regularly being released, it’s a big advantage over having to live with vulnerabilities and bugs.

Committing to a technology can be a long-term decision with far-reaching consequences. Be sure to spend enough time considering the situation before you make the decision.