Silently

Part 6 of 7 in the Building Novi Kvadrat series.

The word “silently” is in 35 of Novi Kvadrat’s commit messages, as of 19 October. The first, on 12 March 2026, explains why municipality rows never reached the database: “This caused municipality INSERTs to silently fail.” After that it keeps coming back, about filters, sitemaps, row counts and keystrokes.

The bugs that cost us the most didn’t crash. They answered 200 OK, kept the build green or printed a number that looked fine. Here is each one: what it looked like, how it was found, why it happened, the fix, and what changed afterwards.

The sitemap that kept the build green

A new sitemap went in on 23 April. It listed the static pages, then, inside a try, read every project, unit and developer from the database; blog posts joined it on 4 May. On 5 May one line in the production build log gave it away:

sitemap: dynamic entries failed, returning static only Error: Dynamic server usage: Route /sitemap.xml couldn’t be rendered statically because it used cookies.

The build was green. The deployed sitemap held “the 5 static paths × 5 locales = 25 URLs. Every project, unit, blog post, developer, category, tag, and author URL was silently dropped.”

The cause was one import. The sitemap built its database client with the helper that reads the visitor’s cookies, and reading cookies during next build trips Next.js’s guard for static pages. The guard threw, and the catch turned that into a log line and a valid-looking file:

// the sitemap, April 2026 (simplified)
try {
  const supabase = await createClient()       // reads cookies() → the static guard throws
  const { data: projects } = await supabase.from('projects').select('slug')
  // ...push every dynamic URL
} catch (e) {
  console.error('sitemap: dynamic entries failed, returning static only', e)
}
return entries                                  // build green, 25 URLs

The fix that day, “Fix sitemap silently dropping all dynamic entries on Vercel build”, switched to a client that reads no cookies; row-level security still limits it to public rows.

It came back in a quieter form. An SEO audit on 6 October found the sitemap “truncated, stale and fragile”: its units were capped at 1,000, a cap known since 27 July, and query errors were swallowed with data ?? []. The rebuild the same day split it into an index and 12 files and turned the old rule around: “A family that cannot be read THROWS — on a background revalidation Next keeps serving the last good file.” The units file no longer stops at 1,000 units.

A thousand rows and a 200 OK

The app reads Supabase through PostgREST, and on this project PostgREST returns at most 1,000 rows per request. Our note on it reads: “A .limit(N) above that does NOT raise the cap: the response is silently cut to 1,000 rows, no error, no flag.” Any total summed over that array is a partial sum, shown as a total.

// the bug (simplified)
const { data } = await db.from('daily_site_stats').select('*').gte('day', from).limit(20_000)
const visitors = data.reduce((sum, r) => sum + r.visitors, 0)  // 1,000 rows, no error

// the fix (simplified): page through it, and fail above a ceiling
for (let from = 0; ; from += 1000) {
  const { data, error } = await page(from, from + 999)
  if (error) return { error }                   // never a silently partial list
  rows.push(...data)
  if (rows.length > CEILING) return { error: 'over the ceiling' }
  if (data.length < 1000) break                 // a short page is the last one
}

The cap turned up again and again:

• In the portal’s sales hub, which “could report fewer sold units than the org actually sold”, fixed on 30 July.
• On the homepage in September, where the header’s Buy count turned out to be exactly 1,000 units plus the sale listings. The test log puts it best: “The number could not have been made right by fixing the cap; the predicate was the bug.” Fixing only the cap would have counted more of the wrong rows.
• In a review of the Pages admin on 12 October, where our notes record that “Neither builder nor the three reviewers named the 1,000 cap — the code already documents it”.

It ended up with three paging helpers, on 30 July, 29 September and 6 October, each solving the same cap again. The rule now: “In plans and reviews, treat any .limit(>1000) as a bug”, and give a read “a ceiling above which the read FAILS rather than renders partial data”.

The last sighting was mine, and it had more underneath.

One number, three layers

On 14 October I opened the admin analytics and saw seven days with more visitors than thirty. A shorter range can’t hold more visitors than a longer one that contains it.

The first layer was the cap. The 30-day range had 1,943 rows and the page got 1,000 of them: it showed 4,650 visitors where the truth was 12,045. Every fetcher there now pages through, with a ceiling of 100,000 rows, and a test runs them against “a fake PostgREST with the real cap”.

Then we measured what the corrected number was made of. Over seven days, server-side listing views came from 5,975 visitor ids, and only 191 of them, 3%, ran the site’s JavaScript that day. 96% had no referrer and 84% were desktop: “scrapers with browser user agents.” Listing views were counted on the server on every render, filtered only by a user-agent pattern. An audit the same day, over thirty days, found that “Visitors” added up daily visitor-days, and “about 90% of them are crawlers”. The same daily table fed the analytics that developers and agencies see in their portal, whose listing-view counts were “inflated about 20×”.

who the visitors were · seven days

server-side visitors · 5,975 ids

• 191 ran the site’s JavaScript that day

their referrer

• 96% had no referrer

their device

• 84% were desktop

each square is about 1% of the server-side visitors

04/04Over seven days, server-side listing views came from 5,975 visitor ids.Only 191 of them, 3%, ran the site’s JavaScript that day.96% had no referrer.84% were desktop: “scrapers with browser user agents.”

The diagram, step by step:

1. Over seven days, server-side listing views came from 5,975 visitor ids.
2. Only 191 of them, 3%, ran the site’s JavaScript that day.
3. 96% had no referrer.
4. 84% were desktop: “scrapers with browser user agents.”

The third layer was us. Local development and staging were writing into the production analytics: tab sessions with a localhost referrer were 35% of all tab sessions over thirty days.

The fix shipped the same day as a verified layer. A visitor counts only after sending at least one event from the page itself that day, with a country and no referrer from localhost or the test site. The headline metric became “Visits”, and an ingest guard drops any event that doesn’t come from production on novikvadrat.rs. I took the audit’s advice that portal customers see only verified numbers, because “showing both keeps selling crawler hits as demand.”

one number, three layers · 14 oct 2026

1. seven days beat thirtymore visitors in the shorter range
2. layer 1: the row cap4,650 shown, 12,045 true
3. still wronglayer 2: crawlersonly 3% ran the site’s JavaScript
4. layer 3: uslocalhost, 35% of tab sessions
5. same daya verified layera visitor counts only after a page event

05/0514 Oct: the admin analytics show seven days with more visitors than thirty.Layer one, the row cap: the page got 1,000 of 1,943 rows, and showed 4,650 visitors where the truth was 12,045.The fix made the number bigger and still wrong. Layer two, crawlers: only 3% of server-side visitors ran the site’s JavaScript.Layer three, us: local development and staging wrote into the production analytics. Tab sessions from localhost were 35% of all tab sessions.The same day, a verified layer: a visitor counts only after sending an event from the page itself.

The diagram, step by step:

1. 14 Oct: the admin analytics show seven days with more visitors than thirty.
2. Layer one, the row cap: the page got 1,000 of 1,943 rows, and showed 4,650 visitors where the truth was 12,045.
3. The fix made the number bigger and still wrong. Layer two, crawlers: only 3% of server-side visitors ran the site’s JavaScript.
4. Layer three, us: local development and staging wrote into the production analytics. Tab sessions from localhost were 35% of all tab sessions.
5. The same day, a verified layer: a visitor counts only after sending an event from the page itself.

Fixing the first bug had made the number bigger and still wrong.

Green build, red deploy

On 12 October two deploys went to ERROR right after “Deploying outputs…”, and the build log showed no error at all. The reason lived only on the deployment object: errorCode: too_many_routes, “Max is 2048, received 2108”. Production kept serving the previous build while 26 commits waited. The first theory was that “~60+ routes” had to go.

None had to go. Every page under the [locale] segment is a dynamic route, and so is every API route with a parameter: 659 in all. Vercel’s Next.js builder writes two entries for each, the page and its .rsc payload. If the app has a Pages Router and a middleware, it adds a third, a _next/data copy. It learns whether there is a Pages Router from one field in the routes manifest, appType, and when that field is missing it assumes yes. Next.js only writes the field from 16.1, and the app runs an earlier 16.0 release. So 3 × 659, plus about 130 fixed entries, made 2,108.

// the deploy builder's decision (simplified)
const hasPagesRouter = ['pages', 'hybrid'].includes(manifest.appType ?? 'pages')  // missing → yes
// with a middleware too, every dynamic route gets page + .rsc + a _next/data copy

// the fix: a stamp after next build (simplified)
if (!manifest.appType) manifest.appType = 'app'   // the value Next ≥ 16.1 writes itself

Measured with Vercel’s build tool, the stamp took the count from 2,106 to 1,435; all 671 removed entries were _next/data rules for a router the app doesn’t have. The fix was live early on 13 October, my time.

green build, red deploy · 12–13 oct 2026

1. no appTypeso the builder assumes a Pages Router
2. with a middleware2,108 routes3 × 659 dynamic, plus about 130 fixed
3. deploy refusedmax 2,048, and the build log green
4. the fixa stamp writes appTypeafter next build
5. 1,435 routesmeasured, from 2,106; a budget in every build log

05/05Next.js writes appType into the routes manifest only from 16.1. The app runs 16.0, so Vercel’s builder assumes a Pages Router.With a middleware too, each of 659 dynamic routes gets three entries. With about 130 fixed ones, that makes 2,108.12 Oct: the build log is green, and the deploy is refused. “Max is 2048, received 2108.”The fix: a stamp after next build writes appType, the value Next.js writes itself from 16.1.Measured with Vercel’s build tool, 2,106 entries become 1,435. A route budget now prints in every build log.

The diagram, step by step:

1. Next.js writes appType into the routes manifest only from 16.1. The app runs 16.0, so Vercel’s builder assumes a Pages Router.
2. With a middleware too, each of 659 dynamic routes gets three entries. With about 130 fixed ones, that makes 2,108.
3. 12 Oct: the build log is green, and the deploy is refused. “Max is 2048, received 2108.”
4. The fix: a stamp after next build writes appType, the value Next.js writes itself from 16.1.
5. Measured with Vercel’s build tool, 2,106 entries become 1,435. A route budget now prints in every build log.

A route budget now runs after every build and prints “Vercel routes (estimate): N of 2,048 (M left)” in the log. It warns from 1,850 and fails the build above 2,048, “with the reason in plain words”. Checked against Vercel’s tool on the same output, it was one off: 1,559 against 1,560. And a rule: “A deploy ERROR with a green build log: read errorCode on the deployment first.”

Silent for weeks

The hourly analytics rollup failed on every run from 09:07 UTC on 1 October. Its daily table stopped at that day, so the admin and portal analytics froze, and “nothing alerts on a failing pg_cron job”. It was found on 3 October only because the agent building the next analytics step reasoned about the rollup’s key.

There were two causes, one behind the other. First, a foreign key: events from a hard-deleted project sat inside the rollup’s two-day window. Behind that, Postgres’s “ON CONFLICT DO UPDATE command cannot affect row a second time”: the rollup grouped by owner columns that weren’t in its upsert key, an earlier backfill had given five entity-days two owners, and one statement tried to update the same row twice. The new version groups by the upsert key, takes the latest owner it can find and skips projects and units that are gone. On a throwaway local Postgres 17 the old version reproduced the error, and the new one passed 11 of 11 checks and was safe to rerun. The standing rule: before touching any analytics table, check cron.job_run_details for the rollup first.

The sitemap’s unit cap had been quiet for longer: known since 27 July, it lasted until the rebuild on 6 October.

silent for weeks · jul – oct 2026

1. the sitemap’s 1,000-unit cap known 27 Jul · rebuilt 6 Oct
2. the analytics rollup failing from 09:07 UTC, 1 Oct · found 3 Oct

Jul 2026AugSepOct

03/0327 Jul: the sitemap’s 1,000-unit cap is known. It lasts until the rebuild on 6 Oct.09:07 UTC, 1 Oct: the hourly analytics rollup starts failing on every run, and nothing alerts.3 Oct: it’s found only because an agent building the next analytics step reasons about the rollup’s key.

The diagram, step by step:

1. 27 Jul: the sitemap’s 1,000-unit cap is known. It lasts until the rebuild on 6 Oct.
2. 09:07 UTC, 1 Oct: the hourly analytics rollup starts failing on every run, and nothing alerts.
3. 3 Oct: it’s found only because an agent building the next analytics step reasons about the rollup’s key.

The same boundary, three times

Next.js splits an app into server and client modules. Across that line, only plain data travels: a Server Component can render a client component but cannot hand it a function as a prop, and a client module’s exports arrive on the server as references, so anything but rendering a component throws. Type-checks and unit tests can’t see it, because they run every module in one Node world. We learned it at least three times.

On 5 May the admin home returned a 500: a Server Component passed icon components into Client Components as props, and production stripped the message down to a digest. On 28 August the portal’s inspections page “served a correct page to a probe while every real visitor with zero inspections hit the portal error boundary.” That commit added a test because “none of the three existing gates can see this class of bug.” In the pass that shipped on 23 September, the first real request to a rebuilt project page failed, after tsc, the unit tests and next build had all passed:

// overview-part.tsx (simplified)
'use client'
export function isOverviewEmpty(vm) { return !vm.text }   // becomes a client reference

// project-page-view.tsx, a Server Component (simplified)
if (isOverviewEmpty(vm)) { … }                             // throws on the first real request

A ratchet test now requires every value the project pages’ server files import from a client module to be a component; it was proven to fail on the original import. And the rule: “A twin is not verified until its route has served one real request on a next start build.” A twin is the new version of a page, built beside the old one.

Every gate green

The admin dashboard was rebuilt in September as “the Command deck”. On 21 September a measuring pass with a phone harness, on a production build, found what its commit title calls “the bug an eyeball cannot find”. On a cold load at 390 px, five of fifteen cards stayed at their desktop size. Resizing the window fixed all five, “which is exactly why an eyeball and every click-through pass”.

The hook that says “this is a phone” answers no on the server and during hydration, then flips. But the deck streams through eleven Suspense boundaries, so server HTML keeps arriving after the client has flipped. React hydrates that late HTML, logs that the mismatch “won’t be patched up”, and keeps the server’s attributes. The unit test passed “no matter what the prop said”, because a jsdom render never asks for the server’s answer. The fix tells the server the answer, with a per-device cookie; a device’s very first load still gets the default and corrects on the next navigation.

Then, with every gate green, I opened the live deck and found four defects in minutes. The commit at 00:56 on 22 September, “AD3 shipped, and the four the owner found after every gate was green”, is written to me: “Then you opened the live deck and found four more, all real”. A card drew “This section could not load.”, because the deck’s own filter for nameless rows threw on a null title and a .catch() swallowed it: “A filter that crashes on its own subject.” Four prefetch 404s. Two more dead links. 99 px of empty page under one card. A ratchet test now stops a deck link from pointing at a 404 again, and the commit doesn’t soften the rest:

tsc, 23,688 tests, axe at any impact, two harnesses and the gallery all agreed the deck was fine, and four real defects were sitting on the live page. The ratchet closes the door class; the rest is the argument for the eyeball.

the command deck · 21–22 sep 2026

when	what happened
21 Sep	“the bug an eyeball cannot find”: 5 of 15 cards desktop-sized on a phone
then	every gate green: tsc, 23,688 tests, axe, two harnesses, the gallery
in minutes	the live deck: a card that couldn’t load, four prefetch 404s, two dead links, 99 px of empty page
00:56, 22 Sep	the commit for the four, and a ratchet: no deck link to a 404 again

04/0421 Sep: a phone harness on a production build finds “the bug an eyeball cannot find”. Five of fifteen cards stay desktop-sized.Then every gate is green: tsc, 23,688 tests, axe, two harnesses and the gallery.I open the live deck and find four defects in minutes, all real.00:56, 22 Sep: the commit for the four, with a ratchet test so a deck link can’t point at a 404 again.

The diagram, step by step:

1. 21 Sep: a phone harness on a production build finds “the bug an eyeball cannot find”. Five of fifteen cards stay desktop-sized.
2. Then every gate is green: tsc, 23,688 tests, axe, two harnesses and the gallery.
3. I open the live deck and find four defects in minutes, all real.
4. 00:56, 22 Sep: the commit for the four, with a ratchet test so a deck link can’t point at a 404 again.

Two lines went into our notes from it. “Guard the VALUE (typeof x === 'string'), not the TYPE.” And the one that stuck: “A known defect with no owner is a shipped defect.”

Testing the tests

A passing test has to earn trust too. The method is mutation: break each guard on purpose, run the tests, restore the file. If the suite stays green with the guard broken, the test was decoration. The ledgers record the scores: 11 of 11 caught in the blog editor, 27 of 27 on the new admin sign-in, and 16 of 16 on a database migration for the 3D buildings, proved on a throwaway local Postgres 17 with 337 of 337 assertions.

The gates got the same suspicion. A build piped into tail printed “Next.js build worker exited with code: null and signal: SIGABRT”, and the shell exit was 0, because a pipe reports the status of its last command. An audit script whose route filter matched nothing printed “6 checks — 0 blocking failures”: “a green that tested nothing”. Builds now run unpiped, and a filter that matches nothing exits with an error.

What I’d pass on

A fallback is a decision to fail quietly. The sitemap’s catch, its data ?? [] and the deck’s swallowed .catch() all turned an error into something that looked normal. Let the read fail, and keep serving the last good output instead.

A 200 OK can carry a wrong number. The row cap never raised an error. Page every read that can pass a server limit, and make it fail above a ceiling rather than render part of the data.

When a number looks wrong, keep digging after the first bug. Fixing the cap made the visitor count bigger and still wrong. Measure what the corrected number is made of before anyone sees it.

Put every limit and every job where you already look. The route cap was reported only on a deployment object, and the rollup’s failures only in cron.job_run_details. A budget line in the build log costs almost nothing.

Green gates are a floor, not a verdict. Before calling something done, check one real request and one real number, and give every known defect an owner.

The pace all of this happened at, and what it cost, is Part 7.

The series

1. Three flats and a platform
2. Research before anything
3. See it before you build it
4. Five languages, five of every mistake
5. Drawn, not recognized
6. Silently
7. More hours than I should