REVIEW 2 cited by
H2O-Danube3 Technical Report
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present H2O-Danube3, a series of small language models consisting of H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T tokens. Our models are pre-trained on high quality Web data consisting of primarily English tokens in three stages with different data mixes before final supervised tuning for chat version. The models exhibit highly competitive metrics across a multitude of academic, chat, and fine-tuning benchmarks. Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a modern smartphone, enabling local inference and rapid processing capabilities even on mobile devices. We make all models openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically.
Forward citations
Cited by 2 Pith papers
-
Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed
Quantizing a larger pretrained model preserves trustworthiness better than pruning, and it produces small models that score higher on trustworthiness than small models trained from scratch.
-
Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization
Uncertainty-based routing thresholds for small-to-large LLM offloading can be bootstrapped from a calibration set built on other datasets, because confidence distributions depend more on the small model and uncertaint...
Discussion (0). Continue with ORCID to comment.