Gary Marcus

Rank 20 of 47
|
Score 61
5/10/2025, 1:40:28 AM
X
In reply to:
Daniel Litt
@littmath
·
29d
@GaryMarcus @Wasgo @polynoamial I’m sorry, I’m finding this discussion very frustrating. “Graph axes misleadingly labeled” is not actually a response to “if you understand correctly what they’re measuring, it’s consistent with logarithmic growth.”
Gary Marcus
@GaryMarcus
·
29d
@littmath @Wasgo @polynoamial The metr task time thing was a train wreck, as discussed in my substack.
Daniel Litt
@littmath
·
29d
@Wasgo @GaryMarcus @polynoamial Same, but nonetheless I think it’s consistent with a logarithmic scaling law, broadly construed.
Cameron Williams
@Wasgo
·
29d
@littmath @GaryMarcus @polynoamial You’d know more about the math benchmarks than I would, but on the generalized novel tasks for METR I found the results highly misleading as a graph.
Daniel Litt
@littmath
·
30d
@Wasgo @GaryMarcus @polynoamial IMO this is an overstatement, at least where math benchmarks are concerned, not to mention e.g. METR.
Cameron Williams
@Wasgo
·
30d
@littmath @GaryMarcus @polynoamial Right. But the problem isn’t the exact benchmark, it’s a question of whether or not we’re evaluating the model on novel problems, or ones within the dataset.

Never models are still improving on problems within the dataset (no wall) but not consistently on novel ones (wall).

The statement is part of a technical discussion about the interpretation of graph axes in relation to mathematical benchmarks and model evaluation. It critiques the meaningfulness of the y-axis in a specific graph, contributing to a broader conversation about data representation and model performance.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm with words or actions. It is a technical critique rather than a personal attack.
  2. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by addressing the technical aspect of the graph's y-axis measurement. It does not engage in personal attacks or ad hominem arguments. [+1]