What we measure, and what it returned
Three published measurements, the method behind each one, and the conclusion we got wrong and had to fix.
We build calculators. Somewhere around the twentieth one, guessing at things stopped being something we could do with a straight face, so we started measuring them instead. This page collects what came out of that, and the rules we hold ourselves to when we publish it.
Published so far
-
We tested 39 online calculators on four arithmetic traps
Thirty-eight of thirty-nine returned arithmetic we agree with. The one failure was a correct formula sitting under a label from a different mode, which is worse than a wrong formula: the number looks calculated and the user has no way to see the mismatch.
-
Library research guides catalogue sources, not tools
71 of 35,640 outbound links pointed to a calculator, converter or spreadsheet, or 0.199 percent of the total. Librarians curate readings. An outreach plan that assumes they curate utilities fails before the first email.
-
Link decay on reference pages: eight failure modes
A request against the linked domain reports a healthy site while every path under it returns 404. Six of twenty-six apparent failures were the checker getting a different answer from the one a browser gets, and anyone who skipped the recheck would have published them as broken.
What gets published here
A question qualifies when two things are true at once. Observation has to be able to settle it, and the answer has to be missing. Plenty of interesting questions fail the first test. Which calculator has the nicest interface is a real question, and we have opinions about it, but no measurement closes it, so it will never be a piece here.
The second test throws out more than the first. Before the calculator audit we looked for an existing survey of whether online calculators return correct arithmetic, and couldn't find one that named the tools and printed the inputs. That gap is what made it worth doing. When a solid answer already exists, linking to it is the honest move, and it takes an afternoon instead of a week.
How a measurement becomes a piece
The rule that costs the most time: every failure is rechecked before it gets counted. In the link decay work, 26 URLs came back as failures on the first pass and 6 of them were fine. Those six were sites answering a crawler differently from a browser. One host, app.dimensions.ai, returns 404 to an identified crawler and 202 to a browser, and any survey that skips the recheck publishes it as broken.
That mistake has a real cost. A maintainer acting on our table would have deleted six working citations. So the recheck is mandatory, and each piece reports how many candidates it removed. That number tells you how far to trust surveys that never mention doing one.
Counting rules get stated too. Broken links are counted by unique URL, not by occurrence. One OWASP address appeared in six campus guides through a shared template, and counting it six times would record six defects where one edit fixes all of them. A choice like that can quietly double a headline number, so it goes in the open.
The one we got wrong
The link decay study said seven failure modes when it went up. It says eight now, and the eighth exists because our own conclusion was wrong.
We had observed that nrel.gov and three hosts under it resolve with no address record on 8.8.8.8, 1.1.1.1 and 9.9.9.9. That observation was correct and still is. What we implied from it was that citations to those hosts were dead ends. Then a search for something else turned up pvwatts.nlr.gov, which only made sense if the laboratory had been renamed. It had. NREL became the National Laboratory of the Rockies and moved to nlr.gov, and PVWatts runs there, serving the same calculator.
The old zone left DNS. The resource never left the web. No DNS query can tell those two apart, and the successor host shares no string with the one that failed, so no automated checker closes that gap by itself. What closes it is a person searching for the resource by name before declaring it gone.
We were two days from emailing municipal offices to tell them their solar tool was dead and offer ours instead. That is the version of this failure that actually costs something, and it's why the piece describes the correction openly instead of patching it quietly.
What does not go up
No sponsored measurement, and no measurement of anything we have a commercial relationship with. No comparison where our own tool is both an entry and the referee. The calculator audit tested margin, discount, post spacing and fee gross-up, four things we build tools for, and our tools are left out of the results table on purpose. The bench at the top of that piece runs the same four calculations, so you can check the correct answers without taking our word for any of it.
We also don't publish a piece we can't date. Every table is a snapshot of one day. Sites change, and a result presented as permanent will be wrong later without anyone noticing.
Checking any of it without trusting us
Each piece prints the URLs and the exact values entered. Open the tool, type the same numbers, read the output. If a site has changed behavior since our snapshot, we want to hear about it: write to us with the input and the output, and we'll rerun the test and correct the page.
Nobody has taken up that offer so far. We mention it because a claim about responsiveness with no record behind it is worth as much as any other unbacked claim.
Being measured now
The audit covered whether calculators return the right number. The harder question is whether they tell you what the number can't do. A mortgage tool that prints a payment to the cent while its assumptions carry a forty percent spread is arithmetically correct and still misleading, and that failure is far more common than a wrong formula. We're working out how to score disclosure without the score collapsing into taste.
Frequently asked questions
Who writes this and who pays for it?
Douglas Santos writes it. The operator is MetaSecurity LTDA, registered in Brazil under CNPJ 38.660.173/0001-09, and advertising pays for the network. Nobody sponsors a piece, no tool covered here is a client, and no result has ever been shown to an operator before publication. The one calculator we found returning a wrong number is named in the audit, with its URL.
Why publish research on a site full of calculators?
We hit the same traps while building the calculators, and the answers weren't written down anywhere we could find. How often are public calculators actually wrong? Do librarians link to tools? Which link checks catch which failures? We had to measure those to do our own work, and the measurements are worth more published than kept in a folder.
Can I reuse the data?
Yes. Every piece lists the inputs, the URLs and the dates, so you can rebuild the tables from scratch. If you republish a table, link back to the piece so readers can see the method and the date of the snapshot. A result with no date on it stops being true at some point, and nobody notices when.
What happens when you get something wrong?
We change the piece and say what changed. The link decay study went from seven failure modes to eight because our conclusion about one of them was wrong: the observation held, the inference didn't. The correction is described in the piece itself and in the commit that made it, dated August 18, 2026.
How do you decide what to measure next?
The question has to be checkable and still unanswered. If a reliable source already has the answer, we link to it instead of redoing the work. If only opinion can settle the question, it doesn't become a piece. That rules out most of what would be easy to write, and that's the idea.