Latest Posts

Code Llama is a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. There are multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama…

Instruction back translation is a scalable method to build a high-quality instruction following language model by automatically labeling human written text with corresponding instructions. Finetuning LLaMa on two iterations of this approach yields a model that outperforms all other LLaMa-based models on the Alpaca leaderboard, demonstrating highly effective self-alignment. Instruction Back Translation Self-Augmentation (generating instructions):…

As larger models require pretraining on trillions of tokens, it is unclear how scalable is curation of “high-quality” corpora, such as social media conversations, books, or technical papers, and whether we will run out of unique high-quality data soon. Falcon shows that properly filtered and deduplicated web data alone can lead to powerful models; even…

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large-scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. The relative importance of these two stages is measured by training LIMA, a 65B parameter LLaMa language model fine-tuned with…

Alpaca is fine-tuned from Meta’s LLaMA 7B model. The Alpaca model is trained on 52K instruction-following demonstrations generated in the style of self-instruct using text-davinci-003. On the self-instruct evaluation set, Alpaca shows many behaviors similar to OpenAI’s text-davinci-003 but is also surprisingly small and easy/cheap to reproduce. Alpaca is intended only for academic research and…

LLaMA is a collection of foundation language models ranging from 7B to 65B parameters, trained on trillions of tokens using publicly available datasets exclusively. Training Data English CommonCrawl [67%] Five CommonCrawl dumps, ranging from 2017 to 2020, are preprocessed using the CCNet pipeline. This process deduplicates the data at the line level, performs language identification…

The State Space Model taking on Transformers Right now, AI is eating the world. And by AI, I mean Transformers. Practically all the big breakthroughs in AI over the last few years are due to Transformers. Mamba , however, is one of an alternative class of models called State Space Models ( SSMs ). Importantly,…

Model Architecture Gemini 1.5 Pro is built upon a sparse mixture-of-experts Transformer based architecture, inheriting and enhancing the multimodal capabilities of its predecessor, Gemini 1.0. The MoE approach employs a learned routing function to direct inputs to a subset of the model’s parameters, enabling conditional computation. This architecture allows for the expansion of the model’s…

In an era where digital content transcends borders, the ability to share videos in multiple languages has become an invaluable asset for creators worldwide. The recent breakthroughs by Elevenlabs in AI video dubbing and translation technology have opened new horizons, making it simpler and more efficient for creators to reach a global audience. This technology…