What we measure, and what it returned
Three published measurements, the method behind each, and the one conclusion we got wrong and had to fix.
We build calculators. Somewhere around the twentieth one it stopped being possible to guess at things and keep a straight face, so we started measuring them instead. This page collects what came out of that, along with the rules we hold ourselves to when we publish it.
Published so far
-
We tested 39 online calculators on four arithmetic traps
Thirty-eight of thirty-nine returned arithmetic we agree with. The single failure was not a wrong formula but a right formula under a label from a different mode, which is worse: the number looks calculated and the user has no way to see the mismatch.
-
Library research guides catalogue sources, not tools
71 of 35,640 outbound links pointed to a calculator, converter or spreadsheet — two hundredths of the total. Librarians curate readings, not utilities, and an outreach plan built on the opposite assumption fails before the first email.
-
Link decay on reference pages: eight failure modes
A request against the linked domain reports a healthy site while every path under it returns 404. Six of twenty-six apparent failures were the checker being served differently from a browser, and would have been published as broken by anyone who skipped the recheck.
What gets published here
A question qualifies when two things are true at once. It has to be settleable by observation rather than argument, and the answer has to be missing. Plenty of interesting questions fail the first test. Which calculator has the nicest interface is a real question and we have opinions about it, but there is no measurement that closes it, so it will never be a piece here.
The second test throws out more than the first. Before the calculator audit we looked for an existing survey of whether online calculators return correct arithmetic, and could not find one that named the tools and printed the inputs. That gap is what made it worth doing. When the answer already exists somewhere solid, linking to it is the honest move and takes an afternoon instead of a week.
How a measurement becomes a piece
The rule that costs the most time: every failure gets rechecked before it gets counted. In the link decay work, 26 URLs came back as failures on the first pass and 6 of them were fine. Those six were sites answering a crawler differently from a browser. One host, app.dimensions.ai, returns 404 to an identified crawler and 202 to a browser, and any survey that skips the recheck publishes it as broken.
Getting that wrong is not a rounding error. A maintainer acting on our table would have deleted six working citations. So the recheck is not optional and the pieces report how many candidates it removed, because that number tells you how much to trust surveys that do not mention doing it.
Counting rules get stated too. Broken links are counted by unique URL rather than by occurrence: one OWASP address appeared in six campus guides through a shared template, and counting it six times would record six defects where one edit fixes all of them. It is the sort of choice that quietly doubles a headline number, so it belongs in the open.
The one we got wrong
The link decay study said seven failure modes when it went up. It says eight now, and the eighth exists because our own conclusion was wrong.
We had observed that nrel.gov and three hosts under it resolve with no address record on 8.8.8.8, 1.1.1.1 and 9.9.9.9. That observation was correct and still is. What we implied from it was that citations to those hosts were dead ends. Then a search for something else turned up pvwatts.nlr.gov, which only makes sense if the laboratory had been renamed. It had. NREL became the National Laboratory of the Rockies, moved to nlr.gov, and PVWatts is running there, serving the same calculator.
The old zone left DNS. The resource never left the web. No DNS query can tell those two apart, and the successor host shares no string with the one that failed, so no automated checker closes that gap on its own. The check that closes it is a person searching for the resource by name before declaring it gone.
We were two days from emailing municipal offices to tell them their solar tool was dead and offer ours instead. That is the version of this failure that actually costs something, and it is why the correction is described in the piece rather than quietly patched.
What does not go up
No sponsored measurement, and no measurement of anything we have a commercial relationship with. No comparison where our own tool is one of the entries and also the referee. The calculator audit tested margin, discount, post-spacing and fee gross-up, which are four things we build tools for, and our tools are deliberately absent from the results table. The bench at the top of that piece runs the same four calculations so a reader can check the correct answers without taking our word for any of it.
We also do not publish a piece that we cannot date. Every table is a snapshot of a day. Sites change, and a result presented as permanent is a result that will be wrong later without anyone noticing.
Checking any of it without trusting us
Each piece prints the URLs and the exact values entered. Open the tool, type the same numbers, read the output. If a site has changed behaviour since our snapshot, we would rather hear about it: write to us with the input and the output and we will rerun the test and correct the page.
That offer has been taken up zero times so far, which we mention because a claim about responsiveness with no record behind it is worth as much as any other unbacked claim.
Being measured now
The audit covered whether calculators return the right number. The harder question is whether they tell you what the number cannot do. A mortgage tool that prints a payment to the cent while the underlying assumptions carry a forty per cent spread is arithmetically correct and still misleading, and that failure is far more common than a wrong formula. We are working out how to score disclosure without the score collapsing into taste.
Frequently asked questions
Who writes this and who pays for it?
Douglas Santos writes it. The operator is MetaSecurity LTDA, registered in Brazil under CNPJ 38.660.173/0001-09, and the network pays for itself with advertising. Nobody sponsors a piece, no tool covered here is a client, and no result has ever been shown to an operator before publication. The one calculator we found returning a wrong number is named in the audit along with its URL.
Why publish research at all on a site full of calculators?
Because we hit the same traps building the calculators, and the questions we had to answer were not written down anywhere we could find. How often are public calculators actually wrong. Whether librarians link to tools. Which link checks catch which failures. We had to measure those to do our own work, and the measurements are more useful published than kept.
Can I reuse the data?
Yes. Every piece lists the inputs, the URLs and the dates, so the tables can be rebuilt from scratch. If you republish a table, link back to the piece so a reader can see the method and the date the snapshot was taken. A result with no date attached to it stops being true at some point without anyone noticing.
What happens when you get something wrong?
We change the piece and say what changed. The link decay study went from seven failure modes to eight because our conclusion about one of them was wrong: the observation held, the inference did not. That correction is described in the piece itself and in the commit that made it, dated 18 August 2026.
How do you decide what to measure next?
Something has to be both checkable and unanswered. If the answer is already published somewhere reliable, we link to it instead of redoing it. If the question can only be settled by opinion, it does not become a piece. That rules out most of what would be easy to write, which is the point.