REVIEW 12 cited by
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option. This has recently given rise to a range of techniques for injecting new facts through updating model weights. Current evaluation paradigms are extremely limited, mainly validating the recall of edited facts, but changing one fact should cause rippling changes to the model's related beliefs. If we edit the UK Prime Minister to now be Rishi Sunak, then we should get a different answer to Who is married to the British Prime Minister? In this work, we present a benchmark, MQuAKE (Multi-hop Question Answering for Knowledge Editing), comprising multi-hop questions that assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. While we find that current knowledge-editing approaches can recall edited facts accurately, they fail catastrophically on the constructed multi-hop questions. We thus propose a simple memory-based approach, MeLLo, which stores all edited facts externally while prompting the language model iteratively to generate answers that are consistent with the edited facts. While MQuAKE remains challenging, we show that MeLLo scales well with LLMs (e.g., OpenAI GPT-3.5-turbo) and outperforms previous model editors by a large margin.
Forward citations
Cited by 12 Pith papers
-
Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings
A ground-truth-first synthetic memory benchmark shows that agent-memory architecture rankings invert with history length: short-horizon leaders lose at nine weeks.
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
The Power of Power Law: Asymmetry Enables Compositional Reasoning
Power-law data sampling creates beneficial asymmetry in the loss landscape that lets models acquire high-frequency skill compositions first, enabling more efficient learning of rare long-tail skills than uniform distr...
-
Don't Use a Cannon to Kill a Fly: Lightweight Model Editing for LLMs to Correct Deprecated API Recommendations
AdaLoRA-L restricts edits to API-specific layers and raises specificity by 33–836% (relative) on a new 3,000+ instance benchmark while staying close to AdaLoRA's effectiveness.
-
Towards Meta-Cognitive Knowledge Editing for Multimodal LLMs
CogEdit and MIND shift multimodal knowledge editing toward evaluating and enabling meta-cognitive skills: self-awareness, boundary monitoring, and noise robustness.
-
Towards a Principled Evaluation of Knowledge Editors
The choice of evaluation metric, generation length, and edit batch size changes the ranking of knowledge editors, and exact string matching produces false positives.
-
PropMEND: Hypernetworks for Knowledge Propagation in LLMs
PropMEND meta-trains a gradient-editing hypernetwork with propagation questions as the outer-loop objective, improving multi-hop knowledge propagation in LLMs in-domain but generalizing poorly to unseen relations.
-
Efficient Knowledge Editing via Minimal Precomputation
Precomputing only a few thousand hidden vectors instead of 44 million is enough for MEMIT, ROME, and EMMET editing to match full-precomputation scores on CounterFact.
-
CODEMENV: Benchmarking Large Language Models on Code Migration
CODEMENV provides 922 examples and three tasks for evaluating LLMs on cross-version code migration, finding models are much better at migrating old code to new versions (up to 43.84% pass@1) than the reverse.
-
COMPKE: Complex Question Answering under Knowledge Editing
COMPKE is a new benchmark with 11,924 complex questions that tests knowledge editing through one-to-many relations and logical operations, where existing editing methods often fail.
-
Break Through the Compression Bottleneck: From Theory to Practice
The paper asserts a first proof that low-rank decomposition and quantization are non-orthogonal tools for LLM compression, recommends low-rank-first ordering, and adds a diagonal scaling fix (DAM) that reduces the com...
-
Robust Knowledge Editing via Explicit Reasoning Chains for Distractor-Resilient Multi-Hop QA
Teaching an LLM to emit a fixed four-stage reasoning chain during fine-tuning makes single-pass multi-hop knowledge editing robust to distractor facts.
Discussion (0). Sign in to comment.