🚇Long Context Challenge in LLMs
Oct 10, 2024
| Oct 10, 2024
Words 21Read Time≈ 1 min
type
Post
status
Published
date
Oct 10, 2024 05:52 AM
slug
long-context
summary
Existing solutions and possible directions to long-context generation
tags
LLM
Long-Context
category
Research
icon
password
Information Original

Benchmarks

LongBench

6 different task types:
notion image
notion image

RULER

  1. Retrieval: needle-in-a-haystack (from
    LLMTest_NeedleInAHaystack
    Github
    LLMTest_NeedleInAHaystack
    Owner
    gkamradt
    Updated
    Oct 10, 2024
    )
  1. Multi-hop tracing: tracing variables with a given value
  1. Aggregation (summarization)
  1. QA (with distracting information)
notion image

Infrastructure

Position Interpolation

Using RoPE:
notion image
 
notion image
notion image
 
Relate Posts :
Research
  • LLM
  • Long-Context
    • Paper Reading: Code Pretraining in LLMsAlibaba Qwen: Generalizing an LLM from 8k to 1M Context using Qwen-Agent
    Loading...
    Catalog
    0%