Gary Marcus

Rank 16 of 47
|
Score 72
4/7/2024, 4:21:52 PM
X
In reply to:
Limelihood ⏸️ Function
@StatsLime
·
449d
@GaryMarcus @Simeon_Cps @DanraeP @WSJ Data augmentation is an ugly hack to introduce approximate invariances into a model that doesn't have the correct inductive biases

Synthetic data is about creating training data completely from scratch, e.g. with self-play or by creating and then reading proofs for theorems
Gary Marcus
@GaryMarcus
·
449d
@Simeon_Cps @DanraeP @StatsLime @WSJ @Simeon_Cps I don’t that I fully follow your distinction. People often use the terms more or less interchangeably, don’t they? How are you distinguishing?
Siméon
@Simeon_Cps
·
449d
@DanraeP @StatsLime @GaryMarcus @WSJ That's not what people are doing with synthetic data and LLMs. People are using models to generate maths proofs than they then check with theorem provers that you can then throw in the dataset if it's correct. It's not "rearrangement".

Don't conflate synthetic data and data…
Danrae Pray
@DanraeP
·
449d
@StatsLime @Simeon_Cps @GaryMarcus @WSJ Yes, precisely! So data augmentation can be helpful for reinforcing information that's already in the data set, but not for teaching a gazillion edge cases which aren't already in the dataset.
Limelihood ⏸️ Function
@StatsLime
·
449d
@DanraeP @Simeon_Cps @GaryMarcus @WSJ It's more about hoping to teach it that those slightly rearranged sentences can still mean basically the same thing
Danrae Pray
@DanraeP
·
450d
@Simeon_Cps @GaryMarcus @WSJ That is different than data augmentation though.

Data augmentation is like taking a US history book, changing up the order of the sentences inside it, and hoping you can now use it to teach the kids physics 😅

Might learn some physics, but not much

The statement is part of a technical discussion on data augmentation and synthetic data within the field of data science and machine learning. The tone is inquisitive, and the intent is to seek clarification on the distinction between two concepts in data processing. The content is focused on the definition and application of data augmentation and synthetic data, which is a specialized topic of interest to professionals in the field.