Hans Otharsson on the Questions Every CIO Should Ask a Modernization Vendor

I have spent nearly four decades in enterprise IT, most of it in legacy modernization, professional services, and the commercial and delivery machinery required to make these programs succeed. Software AG, ModernSystems, OpenLegacy, CoEnterprise, and now CloudFrame have given me different views of the same durable problem: large organizations depend on systems-built decades ago, and moving off them is much harder than the pitch decks suggest.

I joined CloudFrame because its approach matched what experience had taught me: generating new code is not the hardest part. Proving that the new system behaves correctly is. The other thing experience taught me is that legacy modernization is a business problem, a corporate culture problem, and a technology problem, roughly in that order of difficulty. The code is rarely what defeats these programs.

What keeps me engaged is that these systems run payroll, claims, settlements, and other work where there is no partial credit. Getting that right, at scale and in production, remains a genuinely difficult problem. I have never been very good at walking away from one of those.

The most common failure pattern is treating a legacy system as a collection of source code rather than as a living production system.

The project is presented as straightforward: inventory the programs, convert the code, compile it on the target platform, test it, move it into production. That sounds reasonable. It is also where a lot of modernization programs quietly go off the rails.

The source code does not tell you everything about how the system behaves in production. Over decades, business rules become distributed across programs, data structures, job-control logic, schedulers, interfaces, operational procedures, and sometimes manual workarounds known only to a few people. Two programs with similar line counts can have dramatically different complexity because of transaction volumes, data conditions, restart requirements, timing dependencies, or downstream consumers.

I have seen this happen more than once. In one representative project at a financial institution, a nightly settlement run was pitched as a straightforward conversion, and the conversion itself appeared to succeed. The code compiled. Unit tests passed. The new system produced the expected results for the test cases provided.

The serious problems showed up at production scale.

On the mainframe, the run took roughly 20 to 25 minutes in standard daily processing and 30 to 40 minutes at peak season. On the target platform, the same run took close to 300 minutes — eight to ten times longer than a peak-season run. It overran the overnight window and pushed into the business day.

The overrun was bad enough. What everyone fixated on was restart. When a job failed partway through, the new system did not resume from the last checkpoint the way the original did, so recovery meant reprocessing large portions from the beginning.

But restart was the symptom, not the cause. What cost the real time and money was the long, almost archaeological hunt to find out why — with every stakeholder circling the same uneasy question: “I thought we tested this.”

Underneath all of it was the data. The volume, and the way the mainframe absorbed that volume seamlessly, never translated to the target environment, and no one had modeled it before the conversion. None of this was visible from a static review of the source code. Functional equivalence had been proven. Operational equivalence had never been baselined.

What had been sold as a conversion project became a prolonged stabilization effort. The technology changed. The understanding of production behavior did not exist before the change was made.

That is the recurring pattern: teams discover the difficult truths late, when they are most expensive to fix. I hate surprises, and in this work the expensive ones are almost always avoidable. A credible program establishes the behavioral, data, performance, interface, and operational baseline at the beginning. Time spent on reconnaissance is never wasted. Otherwise, “straightforward” is usually just another word for “we have not found the surprises yet.”

The concept is not hard to grasp. The mistake persists because the system rewards certainty before the evidence exists.

Vendors and integrators face pressure to price competitively. Sponsors need a clean, approvable business case. Delivery teams often inherit a date before operations has been heard. Complexity is not necessarily concealed; it is progressively rounded down until “straightforward” becomes the working assumption.

The other problem is who is not in the room. The people who actually know how the system behaves — the operations staff who handle the 2 a.m. failures, the production-control or scheduling team that knows which jobs cannot slip, and the analyst who remembers why that one workaround exists — are rarely consulted before scope is set. Their knowledge is not in the source code. It is in their heads, and often only in their heads.

So the early reconnaissance is mostly about surfacing what the code cannot tell you:

• Production run times and batch windows
• Transaction volumes and actual data characteristics at peak, not just average
• Restart, recovery, and production incident history
• Upstream and downstream interfaces
• Service-level commitments
• Operational procedures and the manual workarounds that keep the thing running

Do that work before you commit to scope, not after. It is comparatively cheap at the start and ruinous at the end.

AI delivers genuine value when it is used to accelerate analysis and reduce undifferentiated manual work.

It can be effective in:

• Explaining unfamiliar code and creating initial documentation
• Identifying patterns, dependencies, duplicate logic, and likely dead code
• Supporting application inventory and classification
• Generating test cases and identifying gaps in existing test coverage
• Assisting developers with target-language code, documentation, and remediation
• Helping subject-matter experts search large codebases and trace business concepts across programs

Those are real gains. In a large estate, reducing weeks of manual investigation to days can materially change the economics of a program.
The narrative overshoots when it suggests AI can ingest an application, understand it completely, convert it automatically, and produce a production-ready replacement with limited human intervention. That is the “Easy Button” pitch, and after three decades of this work, I have not seen an Easy Button that holds up against a real production system. These capabilities are advancing quickly, so the honest question is what the technology can reliably do today, not whether it might get there eventually. Even now, generating plausible code is not the same as preserving the behavior of a production system.

AI models are probabilistic. They may generate code that looks correct, compiles, and passes a limited test suite while still changing an edge-case calculation, a transaction sequence, a database behavior, or a restart condition. That risk matters most in systems involving money, claims, taxes, benefits, inventory, or regulatory reporting. A small difference repeated across millions of transactions is not a small problem.

AI also cannot infer production behavior from source code alone. It needs evidence: runtime execution, actual data characteristics, interfaces, volumes, service levels, failure recovery, operational dependencies. Without that grounding, it is making an informed guess. Sometimes a very good guess. Still a guess.

AI is a force multiplier, not an autonomous modernization authority. It helps people understand and execute the work faster. It does not eliminate the need for deterministic transformation — rule-driven conversion that produces the same, repeatable output for a given input — where appropriate, experienced practitioners, representative testing, and measurable proof of functional and operational equivalence.

Over the next five years, generative AI will move from being primarily a code explainer to becoming an analyst and workflow orchestrator. It will inventory applications, draft documentation, map dependencies, propose tests, triage exceptions, and suggest remediation. Those outputs can tolerate iteration because they are advisory and can be reviewed before they affect production.

High-consequence conversion has a different requirement. The transformation must be repeatable, traceable, and controlled at portfolio scale. Deterministic techniques are well suited to applying approved rules consistently, but determinism alone does not prove equivalence.

The handoff is where the two approaches become powerful together. AI identifies and proposes. Engineers approve. Deterministic tooling applies the approved change consistently. Verification measures the result. Exceptions become reusable transformation rules rather than one-off code repairs.
Better models will move the boundary, but they will not remove the need for evidence, engineering controls, and accountable production decisions.

1. “How will you prove that the modernized system behaves like the existing production system?”

I want an answer that defines functional equivalence in measurable terms and explains how it will be demonstrated across business outputs, numeric precision, interfaces, performance, error handling, and restart behavior. The vendor should describe how it builds the baseline, generates or expands test coverage, compares results, handles exceptions, and maintains traceability from the original system to the target.

I would walk away if the answer is essentially “the generated code compiles,” “our AI is highly accurate,” or “your existing test suite will validate it.” Compilation is not validation, accuracy percentages are meaningless without a defined denominator, and legacy test suites are almost always incomplete.

2. “What evidence do you use beyond the source code to understand how the application operates in production?”

A credible answer includes runtime behavior, data characteristics, transaction and batch volumes, interfaces, scheduling, performance requirements, failure recovery, and operational procedures — and how those findings shape the modernization scope and target design.

I would walk away if the approach is primarily code ingestion followed by AI-generated documentation and conversion. Static analysis can tell you what the application could do. It does not reliably tell you what it does in production, how often it does it, or under which data and operating conditions.

3. “When the generated result is wrong, how will we detect it, correct it, and reproduce the correction across the rest of the portfolio?”

This is where the demo ends and factory-scale delivery begins. I want clear answers on traceability, repeatability, model and prompt controls, human review, regression testing, exception management, retransformation, and ownership of the generated code. The vendor should also explain what happens when the source system changes during a multi-year program.

I would walk away if corrections depend on manually repairing generated code one application at a time, if results cannot be reproduced consistently, or if the vendor treats every exception as additional services work. At portfolio scale, small exceptions become a boa constrictor. They accumulate slowly, then consume the schedule, the budget, and the business case.

The biggest warning sign across all three questions is certainty without evidence. A modernization vendor should be able to show not only how it generates a result, but how it proves it, reproduces it, operates it, and stands behind it. The tools will keep improving. That discipline is what separates a modernization that holds up in production from one that quietly becomes a stabilization project.

The programs that succeed maintain a single accountable owner who is senior enough to make decisions stick. Just as important, they preserve the reasoning behind those decisions when people or reporting structures change. These programs often outlast reorganizations, budget cycles, and executive tenures. When ownership changes and the decision history disappears with it, the program starts relitigating settled questions.

The first non-obvious trait: the successful programs give the operations people real authority, not just a seat. On the programs that stabilize badly, the people who know how the system behaves are treated as an information source to be consulted. On the programs that succeed, those same people have the standing to stop the line, to say “that is not how the batch window works” and be heard. The knowledge is the same. The difference is whether the organization lets it govern decisions.

On modernization programs, experienced application and operations specialists are often labeled the “grumpy old men.” They interrupt the clean design with inconvenient statements: “It works that way, but not always.” “You are forgetting this.” “We do it this way because it works.” To a team eager to move, that sounds like resistance.

On one program, one of those specialists challenged a design that automatically released a month-end file. The source showed no reason to stop it. What it did not show was a manual reconciliation, introduced after an earlier incident, that existed nowhere in the code or runbook. Without it, incomplete transactions could move downstream.

Once the team made that specialist part of the design authority, the control became explicit and testable. Ignore these people, and they will be standing at every failure saying, “I told you so.” Bring them into the project, and what looked like resistance becomes knowledge. That is the corporate-culture part of modernization: deciding whether experience is treated as obstruction or evidence.

The second: successful sponsors treat bad news as early information rather than failure. When surfacing a problem gets you punished, people stop surfacing problems, and the surprises simply arrive later and more expensive. The organizations that get this right have leaders who visibly reward the person who found the issue in month two instead of shooting the messenger. That cultural habit is worth more than another layer of governance or another methodology workshop.

At the board level, legacy modernization usually loses the argument when it is framed as “IT wants to upgrade its infrastructure.” It is abstract, it is expensive, and it competes with initiatives that have visible revenue attached. So the first job is to stop pitching it that way.

Modernization earns its place when it is framed as risk and optionality, not maintenance. The legacy estate is often the thing quietly blocking the priorities the board actually cares about: the AI initiative that cannot readily and safely use governed mainframe data, the compliance obligation that is one retirement away from nobody understanding the code, or the cost line that grows every year while the talent to maintain it shrinks. Modernization is rarely the goal. It is the precondition for the goals.

A simple lens for sequencing is to rank each legacy system on two axes: the business risk it carries if it fails or cannot be changed, and how much it blocks something the business is actively trying to do. The systems that are both high-risk and actively blocking are where you start, because their value can be expressed in terms the CFO already speaks: loss exposure, regulatory deadlines, revenue delay, and cost avoided. Within that group, sequence the work by the amount of exposure or constraint removed relative to the cost, delivery risk, and time to measurable benefit. Systems that are low-risk and blocking nothing can wait, no matter how old they are. Age is not the trigger. Consequence is.

That reframing also solves the justification problem. You are not asking the board to fund a technology project. You are asking them to remove a constraint on things they have already decided to do.

CloudFrame’s principle has not changed: the business behavior is the asset, while the platform is the variable.

Mechanically, CloudFrame starts by establishing how the application actually behaves, including runtime execution and relevant data characteristics. Rule-driven transformation then produces repeatable target code, and verification compares the modernized result with that established baseline. AI can assist with analysis, test design, and remediation, but approved corrections must be traceable and reusable across the portfolio.

To be fair, model-driven approaches are often better and faster at handling irregular, one-off code that does not fit a repeatable pattern, especially when developing a deterministic rule would cost more than resolving the exception; that is precisely why we use AI for analysis, triage and remediation.
Many AI-first approaches begin with the source code and use a model to infer intent and generate the replacement directly. That can produce an impressive demonstration, but production modernization requires more than plausible code. It requires repeatability, measurable verification, and a way to apply corrections consistently when the same pattern appears thousands of times.

The distinction is not AI versus no AI. It is whether AI-generated output becomes the production authority or serves as an accelerator inside a controlled, evidence-based process.

Five to ten years from now, most modernization analysis and drafting will be AI-assisted by default. Large estates will be understood faster and at lower cost, and much of the manual archaeology that consumes programs today will shrink substantially.

More importantly, modernization will become a continuous enterprise capability rather than an episodic, multiyear rescue project. The organizations that do this well will maintain current behavioral baselines, test assets, dependency maps, and clear ownership of business rules instead of rebuilding that knowledge every time a platform reaches end of life.

CIOs making decisions today should preserve optionality. Own the evidence, transformation rules, tests, and data contracts that allow you to change models, vendors, or platforms later. Avoid tying the enterprise to a tool whose economics and capabilities cannot be predicted several years out. And do not allow today's AI platform to become tomorrow's legacy constraint.

Buy tools for today. Build evidence, portability, and discipline for the decade.