Gary Marcus

Rank 15 of 47
|
Score 76
5/3/2024, 4:12:12 PM
X
In reply to:
Crypto Facts
@Crypt0_Facts
·
554d
@RomeoLupascu @GaryMarcus No, I mean why 99.9999%?

Why wouldn't 99.9% or 99.99% work?
Romeo Lupascu
@RomeoLupascu
·
554d
@Crypt0_Facts @GaryMarcus Because if not then I'll need to verify every single "answer" and how am I going to do that? Google it? If so what's the point of asking the LLM in the first place? Not to say about how much time I spend checking. This is borderline insanity unless we can reduce the error rate.
Crypto Facts
@Crypt0_Facts
·
555d
@RomeoLupascu @GaryMarcus Why must it go down to 1/1000000?
Romeo Lupascu
@RomeoLupascu
·
555d
@GaryMarcus I do not understand how we can show a model that is correct only 80% of the time and then push it in production to handle sensitive decisions like health, financial, legal. The necessary error rate must go under 1/1000000 before we can take any of these models seriously.
Gary Marcus
@GaryMarcus
·
556d
Scaling “laws” were never laws.

They were trends. Trends whose time may be up.

13.5 months and $50B post GPT-4, GPT-4 remains on top.
Noam Brown
@polynoamial
·
556d
Many have pointed out that LLM benchmarks are broken and gamed. Happy to see my former resident @hughbzhang, @summeryue0, and the great @scale_AI folks do something about it! They made a private version of GSM8k and evaled GPT-4, Claude, Mixtral, Phi, etc: https://arxiv.org/pdf/2405.00332

The statement in question is part of a technical discussion on the appropriate error thresholds for language models in production, specifically addressing the need for high accuracy in domains that handle sensitive decisions. The conversation involves multiple participants debating the necessity of achieving extremely low error rates to ensure the reliability of language models in critical applications.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement contributes to a constructive discussion on the reliability of language models, aiming to minimize harm by advocating for higher standards in model accuracy. [+2]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the dignity of others by maintaining a professional tone and focusing on the technical aspects of the discussion without any personal attacks or derogatory comments. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by clarifying the positions in the ongoing debate, thereby fostering a better comprehension of the issues at stake. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by supporting a point made by another participant, thereby enriching the dialogue. [+1]
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    The statement acknowledges the validity of another participant's point, demonstrating a willingness to correct or align one's views in light of new information or valid arguments. [+1]
  6. Principle 6:
    I will use my influence for the betterment of society.
    The statement uses its influence to advocate for higher standards in technology deployment, which could lead to better societal outcomes by reducing the risk of errors in critical applications. [+2]
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds the principles of free speech by participating responsibly in a public technical debate, contributing substantively to the discussion. [+1]