We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual...

Promising increase in context, obviously we’ve seen other methods like yarn and rope scaling, but nice to see Meta validating some methods and hopefully releasing the models themselves!