Gary Marcus

Rank 17 of 47
|
Score 91
1/19/2025, 9:42:46 PM
X
In reply to:
FeltSteam0
@FeltSteam
·
3d
@Nicky_Bonez @GaryMarcus But I think it's fine for labs to fund benchmarks, if anything they need to put more money into developing good benchmarks and evals. I guess it should be disclosed publicly though.
FeltSteam0
@FeltSteam
·
3d
@Nicky_Bonez @GaryMarcus Well if they trained on the benchmark (which they didn't necessarily do), then yeah it could be contaminated. But since the benchmark is (all else) private contamination can't leak through large scale pretraining picking up some of the benchmark on the web.
Nick
@Nicky_Bonez
·
4d
@FeltSteam @GaryMarcus The problem is their models are contaminated. Almost every new benchmark that partners with a model designing group puts that group on the top. Benchmarks should be independently developed.
FeltSteam0
@FeltSteam
·
4d
@GaryMarcus What "robustness" issues are you thinking may be a problem here? And honestly OpenAI should fund more benchmarks and initiates to create benchmarks, but they should probably be more openly accessible (though I guess keeping them more private is a way to help avoid contamination)
Gary Marcus
@GaryMarcus
·
4d
If this is true it stinks, and adds to the concerns re the transparency and robustness of o3 demo.

The statement critiques the funding of private access to problems by a company, suggesting it creates an unfair advantage over competitors. This implies concerns about transparency and fairness in the development and evaluation of benchmarks.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement highlights a potential harm in the form of unfair competitive advantage, aligning with the principle of striving to do no harm. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    It respects privacy and dignity by focusing on the actions of organizations rather than individuals. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by pointing out issues of transparency and fairness, encouraging a more equitable approach. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    It engages in constructive criticism by addressing the issue without personal attacks. [+1]