import torch
from model import TransformerLM
# Head pruning configuration
config = {
'prune_ratios': [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],
'batch_size':32,
'max_epochs':50
}
defrun_experiment(prune_ratio):
model =TransformerLM().prune_heads(prune_ratio)
results = model.evaluate(config)
return results.perplexity
Results
perplexity · 6 pruning levels
Validation Perplexity by Pruning Ratio
Eval Suite Builder
4 evals configured
Eval
Category
Metric
Threshold
Active
Quality Retention
Output accuracy after pruning
Quality
Accuracy (%)
≥ 95%
Inference Speed
Tokens per second improvement
Performance
Speedup (x)
≥ 1.5x
Model Compression
Parameter count reduction
Efficiency
Size (%)
≤ 70%
Benchmark Results
Overall
76%
↑ 4.2% vs v2.3
Passed
3/4
Regressions
1
Eval Scores vs Threshold
Reasoning86%
80%
Factuality79%
75%
Code Gen64%
70%
Evals
Easily benchmark every step.
Work closely with Soren's team to design custom evals and benchmarks. Or, let Soren's agents set them up for you.
Infrastructure
Managed infrastructure. Built for scale.
We take care of orchestration, scaling, and reliability behind the scenes. Stay focused on results, not maintenance.
Soren — Research Overview
All systems running
12 projects
218 experiments
Project
Progress
Latest Finding
Status
Attention Optimization
2 researchers
6 / 9 runs
40% of heads removed, quality maintained
active
Sparse Autoencoder
4 researchers
14 / 17 runs
L1 penalty 0.01 best recon loss
active
RL Curriculum Design
3 researchers
3 / 11 runs
Needs review — reward plateau
review
Tokenizer Benchmarks
1 researcher
8 / 8 runs
BPE-64k best on all 3 benchmarks
done
Results
Attention Opt. maintained quality at 40% pruning
1m
Tokenizer all benchmarks complete
9m
RL Curriculum reward plateau detected
14m
Sparse Autoencoder 3 new promising features
22m
Soren is an applied research and product lab building AI to democratize R&D.
Soren is built for teams tackling complex technical problems. It’s ideal for research-driven organizations, but also for general engineers, researchers, and R&D teams that want to move faster.
Yes. Your data is fully isolated and encrypted at rest and in transit. We never use your research data to train models. Soren is designed for institutional-grade security and compliance.
Soren supports many computational research workflows, ranging from ML experimentation and molecular design. Contact us with any questions or use cases.
Pricing depends on your use case, domain, and scale. Please reach out or book a demo, and we’ll get the best plan for you.
