Deployment Smoke Test: When a Clean Build Still Breaks

A deployment smoke test starts the built application the way production will run it, requests its pages, and blocks the release if any answer is wrong. Ours runs inside every build of mkdirhq.com, after the build and before the deploy. It exists because an upgrade once built cleanly and then served an error on every page.

I'm Dheer, co-founder at mkdir (mkdirhq.com). On 2026-10-12 we tried version 16.4.0 of the web framework this site is built on. Lint passed. The type check passed. The production build finished and listed every route. Then the built site answered every page with a server error.

None of those checks had ever started the site. Each one looked at the code, and the code was fine. What broke was the step after the code: running the output on the platform that hosts it. So we wrote a script that does exactly that, and made the release depend on it.

What is a deployment smoke test?

The name comes from hardware. You switch the new board on and watch for smoke before you test anything finer. For a website the question is the same. Does the thing you are about to release start, and do its pages answer?

A smoke test is deliberately shallow. Headlines, layout and button placement belong to other tests. This one asks whether the release can serve traffic at all, the one failure no visitor forgives.

Many teams run theirs after the deploy: load the homepage, expect a 200, roll back on anything else. That works, but it finds the problem after visitors could already see it. We run ours before the deploy, against the exact files that would be uploaded. If it fails, nothing ships and the live site stays on the previous version.

Why can a build pass and the site still fail?

A build proves the code compiles and that the pages built ahead of time can be rendered. It does not prove the server code runs in the runtime it will be deployed to. On a plain server those are close to the same thing. On a serverless platform they are not.

Our site reaches its host through an adapter: a separate package that turns the framework's build output into something the platform can run. The framework and the adapter come from different teams and ship on different schedules. Version 16.4 of the framework changed something our adapter release did not handle yet. The build was happy. The adapter's output was not.

The fix arrived later. The first adapter release with support for 16.4 is 1.20.12, and its package metadata declares that it needs framework version 16.4.1 or newer. We moved to that pair on 2026-10-23, as part of a security upgrade, and the smoke test passed all 75 of its checks on it.

The lesson we wrote into the repo is about pairs. When two packages have to agree, bumping one of them changes the pair. Our instructions now name five packages that only move together, and only when a founder asks for it.

What does our smoke test check?

It is one script with no test framework and no new dependencies. It uses the runtime's built-in fetch and child processes. Here is what it does, in order.

• > Refuses to run if the built server file is missing, so it can never pass by testing nothing.
• > Fills the local page cache the way a deploy fills it. Prerendered pages are served from that cache, and without it every one of them would look missing.
• > Starts the built output in the hosting platform's own local runtime, on 127.0.0.1 port 8788, and waits up to 90 seconds for it to answer.
• > Reads sitemap.xml and collects every page address in it. If the sitemap lists fewer than 20 pages, that alone is a failure.
• > Requests each page and expects a 200 with an h1 heading in the HTML. An empty shell that returns 200 still fails.
• > Checks robots.txt, the default share card image, and the trial mode of the start page.
• > Requests a page that does not exist and a blog post that does not exist, and expects a 404 for both.
• > Posts an empty form to the contact route and expects the route's own JSON refusal.

Reading the sitemap was the most useful decision in the script. A scheduled agent adds blog posts twice a week, and new pages appear often. Nobody has to edit the test when they do. This post will be in the list the next time the test runs.

The 404 checks matter more than they look. A site that answers 200 for a page that does not exist tells search engines the page is real. Google calls that a soft 404, and a broken route can produce one on every bad link.

The form check is the subtle one. An empty submission can be refused three ways: 422 from validation, 429 from the rate limiter, or 500 with "Failed to send message" when the mail credentials are absent, which they always are locally. All three are fine. They are JSON written by our own code, which proves the route loaded and ran. A crashed server answers with an error page, and that fails.

When something fails, the script prints up to 25 problems, each with the page and the status code. It adds the last error lines from the server log, then exits with a non-zero code. That is what stops the deploy.

Why does a local preview redirect to the live site?

This one is easy to miss. By default, the platform's local dev server presents the production hostname to the site, over plain http. Our site redirects http to https for that hostname, as it should. So every request to the local copy comes back as a redirect to www.mkdirhq.com.

In a browser that is worse than an error. The redirect is followed, the live site loads, and you believe you are looking at your local build. The test sets its requests to not follow redirects, so it would see a 3xx and fail. Either way you are not testing what you think.

The fix is one flag that tells the local server which hostname to present: 127.0.0.1 and the port the test uses. The manual preview command does the same on port 8787. Our repo instructions say to open that exact address, for this reason.

Should a smoke test run before or after the deploy?

Before, if your platform lets you run the exact build output locally. A push to main on this site runs lint, the blog post check, the build and the smoke test, then deploys. If any step fails, nothing deploys. Other branches are built and uploaded as non-live versions, with a preview link on the pull request.

An after-deploy check still has a place. Some things only exist in production: the real domain, the real secrets, the real network path to other services. If you can only test those after release, keep the check small and make rollback one step. Ours is the deployments list in the hosting dashboard, or a revert of one commit.

What can a smoke test not catch?

Anything that needs a real secret or a real outside service. We learned this eleven days after the first incident. On 2026-10-23 we upgraded the email library behind our two forms to clear its security advisories. Our checks passed, and both emails rendered the same as on the old version.

Then we sent a real contact form from the pull request's preview, and it returned 500. The older version had sent fine from previews the same day. The new one opens its connection to the mail server differently, and that difference only shows against a real mail server from the real runtime.

The smoke test could not have caught it. Locally there are no mail credentials, so it accepts a 500 from the form route by design. We reverted that one upgrade, kept the framework and adapter security fixes, and left the email advisory open and tracked. The rule is now written next to the dependency: before moving it, send one real form email from a preview build.

Other things sit outside the script too. It does not render pages in a browser, so it cannot see a 3D scene that fails to mount or a layout that collapses. For the 2026-10-23 upgrade we also loaded 14 pages in a real browser and compared status, headings and page heights against the version before. It does not measure speed either. The free plan allows 10 ms of CPU time per request, and the weekly report watches that number separately.

How do you add a smoke test to your own deploy?

The script is under 150 lines. Most of the value comes from a few choices you can copy to any stack.

• > Run the artifact you will ship, in the runtime you will ship it to. A dev server running different code proves little.
• > Take the page list from the sitemap, not from a list someone has to maintain. Google's limit is 50,000 addresses per sitemap file, so a very large site may want to sample.
• > Check one thing in the body that every real page has. A status code alone passes an empty page.
• > Test the failure paths too: an unknown address must return 404.
• > Call one server route with an input that cannot cause a side effect, like an empty form, and check the answer came from your code.
• > Put it between the build and the deploy, so a failure means nothing ships.
• > Fail loudly, with the page, the status and the server's own error lines in the output.

If you do not write code yourself, there is one question to ask whoever runs your site or internal tools: what runs between the build and the release? If the answer is "the build", ask what happens when the build passes and the site does not start. On 2026-10-12, for us, the answer would have been every page.

Sources

• The incidents, the script, its 75 checks and the build steps: this site's own repository and build output, 2026-10-12 to 2026-10-23
• The adapter's supported framework versions: the peerDependencies field in its published package metadata, version 1.20.12
• Sitemaps list absolute URLs, with a limit of 50,000 URLs or 50MB per file: Google Search Central, Build and submit a sitemap (developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap)