Previous Card
Why Language Models Hallucinate
Extract-0: A SPECIALIZED LANGUAGE MODEL FOR DOCUMENT INFORMATION EXTRACTION
Extract-0 is a 7-billion parameter language model specifically optimized for document information extraction. It achieves performance exceeding larger general-purpose models like GPT-4.1 on diverse extraction tasks. This is accomplished through a novel combination of synthetic data generation, parameter-efficient fine-tuning (LoRA), and reinforcement learning (GRPO) with a semantic similarity-based reward function. ✨
Article Points:
1
Extract-0: 7B model excels in document information extraction.
2
Outperforms GPT-4.1 and o3 on 1,000 tasks with 0.573 mean reward.
3
Achieves performance via synthetic data, LoRA fine-tuning, and GRPO.
4
Memory-preserving synthetic data pipeline generates 280K examples.
5
Parameter-efficient LoRA fine-tuning modifies only 0.53% of weights.
6
Semantic similarity reward function addresses extraction ambiguity.
Extract-0: A SPECIALIZED LANGUAGE MODEL FOR DOCUMENT INFORMATION EXTRACTION
Core Innovations
Synthetic Data Generation
LoRA Fine-tuning
GRPO with Custom Reward
Methodology
Memory-preserving Data Pipeline
Parameter-efficient Adaptation
Semantic Similarity Reward
Performance
Outperforms GPT-4.1 & o3
Achieves 0.573 Mean Reward
147% Improvement over Baseline
Resources
7B Parameters
$196 Training Cost
Single H100 GPU
Limitations
Training Data Coverage
Reward Function Nuances
Single-document Focus
Implications