As larger models require pretraining on trillions of tokens, it is unclear how scalable is curation of “high-quality” corpora, such as social media conversations, books, or technical papers, and whether we will run out of unique high-quality data soon. Falcon shows that properly filtered and deduplicated web data alone can lead to powerful models; even … Continue reading Paper – Falcon