I finished Phase 2 of the NerdDinner modernization, which was the final phase in the process (you might want to read about Phase 1 first). The app went from targeting .NET Framework 4.5 and using outdated dependencies to targeting the latest version of .NET and running up to date dependencies. During the process, functionality was preserved by creating characterization tests and then ensuring they passed before closing each milestone. At the end of the project, all characterization tests had passed and manual testing confirmed the app was working as expected.
For the most part, the process went smoothly. However, there was a snag that took longer than anticipated. When testing Milestone 8, a lot of tests were failing in Visual Studio. During Claude's code-writing cycle, it runs existing tests using the CLI. The CLI executed tests were passing, so it never flagged any failing tests. The stack trace in the Visual Studio test error messages indicated that a Microsoft Visual C++ Redistributable was missing. I had Claude Code analyze the error messages and it suggested a few things to try including making sure an MSVC Redistributable was installed, restarting Visual Studio, and rebuilding the solution. But, after doing what Claude suggested the tests were still failing. Claude suggested a couple more things to try, but at this point I decided that we needed more precise information.
I manually debugged the tests and found that the path that AppDomain.CurrentDomain.BaseDirectory resolved to was not what was expected (see DL-023). It was resolving to the test runner's directory, not the app's directory where certain native DLLs were located. Assembly.Location was tried next, but that failed when running the tests via vstest.console.exe due to the xUnit adapter shadow-copying the assemblies to a temporary cache. The actual fix was to use Assembly.CodeBase, which resolved correctly under Visual Studio's test runner and vstest.console.exe. This fixed loading the native DLLs. So, the fact that the CLI-run tests were passing at first ended up being a coincidence because AppDomain.CurrentDomain.BaseDirectory happened to resolve as expected.
Further test runs in Visual Studio uncovered other issues of the same type: built-in .NET APIs resolving different paths than expected under various environments (see DL-024, DL-025, and DL-026). Now that we were clued in on what was happening, the subsequent issues were addressed more quickly since the root cause was known. Still, getting all the tests passing consumed almost half of the time it took to complete Phase 2.
So, what can we learn from this? The first is that the app and tests must run successfully in all required environments. Since NerdDinner shipped with a .sln file, one of the required environments was Visual Studio. The second is that, just like humans, AI tools have blind spots. In this case, it was that Claude can't run Visual Studio. If a required environment happens to fall into a blind spot and you don't account for that, you are risking problems manifesting later. And in software, the earlier problems are caught, the better.
While Phase 1's lesson was strategic, breaking the project into two phases to lower risk, the lesson from Phase 2 is tactical. The AI tooling was limited by what environments it could run. In this case, the human needed to help bridge a gap by manually debugging and giving Claude the stack trace. This is why Decimal Zero takes all your environments into account, and doesn't just rely on where the tooling happens to run successfully.