Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Transfer in Deep Reinforcement Learning using Knowledge Graphs

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Knowledge graphs let reinforcement learning agents transfer skills across text-adventure games.

desk verdict A credible ablation study and recipe for KG-based transfer in text-adventure RL, but the headline 80% step gain rests on a dense oracle reward; the paper needs a sparse-reward full-transfer row and a completed horror table before the claim holds up. read the letter →

arxiv 1908.06556 v1 pith:7REHYFLK submitted 2019-08-19 cs.CL cs.AIcs.LG

Prithviraj Ammanabrolu , Mark O. Riedl This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords text-adventuregamesdeepreinforcementlearningknowledgegraphstransferquestion-answeringpretrainingKG-DQNinteractivefictionactionpruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that knowledge graphs can carry reusable knowledge between text-adventure games, making a deep reinforcement learning agent learn better policies faster. It combines three transfer mechanisms: seeding the agent's knowledge graph from static walkthrough-style text, pre-training part of the deep Q-network with a question-answering task on oracle play traces, and initializing the target game's network with parameters trained on a source game in the same genre. Across computer-generated and human-authored games in two genres, the full combination reduces completion steps by up to 80 percent relative to training from scratch. The contribution is an explicit, interpretable mapping between state and action spaces as the medium for transfer.

What carries the argument

The knowledge graph is a directed graph of RDF triples extracted automatically from game observations and static text resources via open information extraction. It serves two roles: it is part of the state representation for the deep Q-network, and it prunes the combinatorial action space by ranking actions according to whether their objects appear in the graph. Transfer works because the graph creates an explicit mapping between the state and action vocabularies of different games; the paper additionally transfers network weights by pre-training the encoder with a question-answering system and by initializing the target network with source-game parameters.

What would settle it

Leave the games unpartitioned and use only the original sparse reward, then compare full transfer against no-transfer training on the same game; if the full system still completes the game in far fewer steps, the transfer claim holds, and if neither converges, the reported speedup depends on the oracle checkpoints and manual truncation.

Watch

Extended reading notes

Core claim

The paper claims that knowledge graphs are a suitable medium for transferring domain knowledge between text-adventure games, and that a combination of graph seeding, question-answering pretraining, and source-to-target parameter transfer lets a deep Q-network learn a higher-quality control policy faster than training without transfer, with up to an 80 percent reduction in completion steps on some games. The authors further claim that the knowledge graph itself provides the explicit, interpretable mapping between state and action spaces that makes such transfer possible, overcoming the partial observability and large action spaces typical of text-adventure games.

Load-bearing premise

The results assume that adding oracle-derived dense reward and manually shrinking the games does not already account for the speedup attributed to transfer, since every transfer comparison is run with those modifications.

Editorial extensions

If this is right

  • A new game in a known genre can be trained faster and to higher final reward by seeding its knowledge graph and initializing from a source game, rather than training from scratch.
  • Pre-training on oracle traces and seeding from guides produce similar early-training gains, so either can serve as a cheaper substitute when a source game is unavailable.
  • The full pipeline is more efficient in total episodes than training the target alone: source training followed by 50 target episodes beats more than 150 target-only episodes.
  • Because the knowledge graph gives an interpretable state-action mapping, transfer failures can be diagnosed by inspecting which triples are missing or mis-linked.
  • The action-pruning benefit of a seeded graph should grow with game complexity, since the graph narrows a larger action space more dramatically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial inference: because dense reward comes from perfect walkthroughs, the transfer gain might shrink if the reward were replaced with a domain-independent shaping signal; this is testable by ablating the oracle-derived reward.
  • The paper does not demonstrate transfer across genres; a natural extension is to test whether a graph seeded from horror-theme guides helps an agent on a science-fiction game.
  • The knowledge-graph state representation suggests the method could transfer to other partially observable language environments beyond games, but that is an extrapolation rather than a claim of the paper.
  • Seed graph quality may matter more than its size; generic guides might yield diminishing returns compared to genre-specific guides, which the current experiments do not fully separate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a transfer-learning pipeline for deep Q-networks that play text-adventure games, building on the authors' prior KG-DQN system. Three transfer mechanisms are combined: seeding the knowledge graph from static walkthrough guides, pre-training parts of the network with a question-answering system on oracle traces, and transferring parameters from a source game to a target game in the same genre. The method is evaluated on two domains using TextWorld and Jericho: a slice-of-life domain (TextWorld generated games and 9:05) and a horror domain (Lurking Horror, Afflicted, Anchorhead). The central claim is that the full pipeline lets the agent learn a higher-quality control policy faster, with up to an 80% reduction in completion steps relative to a no-transfer KG-DQN baseline.

Significance. If the results hold, the paper makes a useful contribution to transfer in text-based reinforcement learning: it treats the knowledge graph as an explicit, interpretable medium for cross-game transfer, evaluates on human-authored interactive fiction, and releases code. A particular strength is that the internal comparisons are fair: the no-transfer baseline receives the same dense reward as the full system, so the ablation design does not fit the transfer result into existence. The main significance caveat is that the headline speedup is measured under a modified task (oracle-derived dense reward, manual checkpoints, pre-pruned action sets), so the external claim about learning game-playing competence faster is only partially supported.

major comments (4)
  1. [§7, Table 2] The headline claim of an 80% gain in completion steps is supported only by the dense-reward comparison in the slice-of-life domain: KG-DQN Full (D) takes 274.76 ± 21.45 steps versus 1267.2 ± 7.5 for KG-DQN no transfer (D), while no 'KG-DQN Full (S)' row is reported. Because the dense reward is constructed from oracle checkpoints (§6.3), this comparison shows faster exploitation of the shaped reward, but it does not establish that the transfer mechanism improves game-playing competence under the original sparse reward. Please add a sparse-reward full-pipeline run, or explicitly scope the claim to the dense-reward setting.
  2. [Table 3 and §6.2] In the horror domain, the no-transfer baseline's steps column is missing, so the speedup claimed in the text cannot be measured there; the final-reward gain (39.9 vs 6.8) is also obtained after the games are manually partitioned at intermediate checkpoints and their action sets pre-pruned in §6.2. Without the no-transfer steps under identical partitioning and pruning, the results cannot separate transfer gains from task simplification. Please report the missing baseline and, if possible, a comparison on the unpartitioned games.
  3. [§6.2 and §6.3] The experimental protocol substantially changes the learning task relative to the paper's opening claim about text-adventure games: games are ended at manually selected checkpoints, action sets are pre-pruned, and a proportional reward is given for matching oracle-generated state-action pairs. These modifications apply equally to all compared conditions, so the internal ablation is fair, but the external claim that the full method 'learns a higher-quality control policy faster' on text-adventure games requires evidence that the gains survive with the game's own reward signal and full action set; otherwise the measured improvement may reflect the oracle prior rather than knowledge-graph transfer. Please clarify that the evaluation is on the modified task, or add an experiment on a complete game under an unmodified reward.
  4. [§7] The statement that 'a series of t-tests ... show that all results are significant with p<0.05' is based on only three random seeds per condition, which gives a t-test very low power, and no correction for multiple comparisons is described. This weakens the statistical support for the incremental claims about seeding and QA pretraining; please report effect sizes, per-metric tests with the full number of runs, or use a procedure appropriate for small samples.
minor comments (5)
  1. [Abstract] The first sentence repeats 'through text descriptions' for both making sense of the world and declaring actions; the second occurrence should presumably be 'through text commands'.
  2. [§6.3] The scaling factor for the dense reward augmentation is described only as 'less than the smallest reward given by the original reward function'; the actual per-game scaling values should be reported for reproducibility.
  3. [Table 1] The column grouping in Table 1 is hard to read; the two slice-of-life games and the three horror games should be separated with clearer headings or subheadings.
  4. [§7] The discussion compares the untuned dense evaluation with the no-transfer sparse baseline, which confounds reward type with transfer; the comparison should be made within the same reward condition.
  5. [§5 and §2] There are minor typos: 'within in same domain' in §5 and 'using a both deep Q-networks' in §2; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the transfer gains are measured against within-paper no-transfer baselines under identical dense-reward conditions, so the headline result does not reduce to its inputs.

full rationale

The paper's central claim—that knowledge-graph seeding, QA pretraining, and source-to-target parameter transfer yield faster learning of a higher-quality policy—is an empirical claim tested by ablations in Tables 2 and 3, not a derivation that equates a conclusion with an input. The strongest candidate for circularity is the oracle-derived dense reward in Section 6.3, which gives the agent proportional reward for matching perfect-walkthrough state-action checkpoints. However, the no-transfer baseline receives exactly the same augmented reward (e.g., "KG-DQN no transfer (D)" in Table 2 and the dense-reward no-transfer row in Table 3), so the reported step reductions, such as 1267.2 ± 7.5 down to 274.76 ± 21.45 for 9:05, are controlled comparisons rather than quantities defined to equal themselves. The paper explicitly acknowledges the sparse-reward limitation via the statement "a sparse reward signal usually results in ineffective exploration by the agent" and "To make experimentation feasible, we augment the reward," which is a generality and external-validity concern, not circularity. The self-citations to Ammanabrolu and Riedl (2019) supply the KG-DQN architecture, graph-extraction rules, and QA-pretraining recipe, but the present paper's experiments ablate these components against no-transfer baselines and do not invoke any prior uniqueness theorem or fitted parameter to force the result. No equation, metric, or reported gain is identical by construction to a training input, and no fitted parameter is renamed as a prediction. The absence of a sparse-reward "KG-DQN Full (S)" row weakens the scope of the 80% claim, but that is an experimental-coverage issue, not circular reasoning.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central empirical claim rests on the KG-DQN framework from the authors' prior work and on several domain assumptions about graph extraction, oracle guidance, and game simplification. No invented physical entities. The free parameters are experimental choices, not fitted constants in a derivation.

free parameters (2)
  • Dense reward augmentation scale = Not reported; set per game below the smallest game reward
    Used in Section 6.3 to scale oracle-based checkpoint rewards. Chosen by hand per game and affects all dense-reward experiments, including the no-transfer baseline.
  • Word embedding dimensionality = 100 (horror), 50 (slice of life)
    Reported in Section 6 as a hyperparameter inherited from prior work. Not central to the transfer effect but an experimental choice.
assumptions (6)
  • domain assumption Open-world assumption for knowledge graphs (missing triples can be true or false).
    Stated in Section 3. Underlies how the KG state and action pruning are interpreted.
  • domain assumption Knowledge graphs extracted by OpenIE and hand rules adequately represent game state and object affordances.
    Inherited from Ammanabrolu and Riedl (2019). Not revalidated independently in this paper.
  • domain assumption Oracle traces and online walkthroughs provide correct state-action supervision for QA pretraining and dense reward.
    Assumed in Sections 5 and 6.3. The method depends on access to near-perfect solutions.
  • ad hoc to paper Manually selected intermediate checkpoints and pre-pruned action sets preserve the game's learning challenge.
    Described in Section 6.2. If these modifications trivialize the tasks, the transfer results may not extend to full games.
  • domain assumption Graph embeddings and network weights transfer across games within a genre despite differences in graph structure.
    Assumed when the source-task network weights initialize the target-task network in Section 5. This is the mechanism the experiments are designed to test.
  • standard math Text-adventure games are POMDPs with the 7-tuple formalization of TextWorld.
    Section 2, standard model in the literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transfer in Deep Reinforcement Learning using Knowledge Graphs." pith.science (2026). https://pith.science/paper/7REHYFLK

@misc{pith2026190806556,
  author       = {Pith},
  title        = {Pith review of: Transfer in Deep Reinforcement Learning using Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7REHYFLK}},
  note         = {Machine review of arXiv:1908.06556}
}
read the original abstract

Text adventure games, in which players must make sense of the world through text descriptions and declare actions through text descriptions, provide a stepping stone toward grounding action in language. Prior work has demonstrated that using a knowledge graph as a state representation and question-answering to pre-train a deep Q-network facilitates faster control policy transfer. In this paper, we explore the use of knowledge graphs as a representation for domain knowledge transfer for training text-adventure playing reinforcement learning agents. Our methods are tested across multiple computer generated and human authored games, varying in domain and complexity, and demonstrate that our transfer learning methods let us learn a higher-quality control policy faster.

Figures

Figures reproduced from arXiv: 1908.06556 by the authors.

Figure 1
Figure 1. The KG-DQN architecture. possession or in their immediate surrounding en￾vironment. We make minor modifications to the rules used in Ammanabrolu and Riedl (2019) to better achieve such a graph in general interactive fiction environments. The agent also has access to all actions accepted by the game’s parser, following Narasimhan et al. (2015). For general interactive fiction environ￾ments, we develop our own method … view at source ↗
Figure 2
Figure 2. Select partial example of what a seed knowl [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Partial unseeded knowledge graph example [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Reward curve for select experiments in the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interactive Language Learning by Question Answering

    cs.CL 2019-08 conditional novelty 6.0 of 10

    QAit turns question answering into an interactive text-game task, and the paper's baselines show current agents cannot generalize beyond memorized games, while humans can.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Prithviraj Ammanabrolu and Mark O. Riedl. 2019. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings of 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019

    work page 2019

  4. [4]

    Gabor Angeli, Johnson Premkumar, Melvin Jose, and Christopher D. Manning. 2015. Leveraging Linguistic Structure For Open Domain Information Extraction . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

    work page 2015

  5. [5]

    Antoine Bordes, Nicolas Usunier, Ronan Collobert, and Jason Weston. 2010. Towards understanding situated natural language. In Proceedings of the 2010 International Conference on Artificial Intelligence and Statistics

    work page 2010

  6. [6]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL)

    2017

  7. [7]

    Marc-Alexandre C \^ o t \' e , \' A kos K \' a d \' a r, Xingdi Yuan, Ben Kybartas, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. http://arxiv.org/abs/arXiv:1806.11532 TextWorld : A Learning Environment for Text-based Games . In Proceedings of the ICML/IJCAI 2018 Workshop on Computer Games, page 29

    arXiv 2018

  8. [8]

    Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. 2017. https://doi.org/10.24963/ijcai.2017/144 What can you do with a rock? affordance extraction via word embeddings . In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17 , pages 1039--1045

    work page doi:10.24963/ijcai.2017/144 2017

Show all 31 references
  1. [9]

    Milica Gasic, Catherine Breslin, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Steve J. Young. 2013. Pomdp-based dialogue manager adaptation to extended domains. In SIGDIAL Conference

    2013

  2. [10]

    Matan Haroush, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. 2018. Learning How Not to Act in Text-Based Games . In Workshop Track at ICLR 2018, pages 1--4

    2018

  3. [11]

    Matthew Hausknecht. 2018. Jericho: A learning environment for man-made interactive fiction games. https://github.com/microsoft/jericho

    2018

  4. [12]

    Williams

    Matthew Hausknecht, Ricky Loynd, Greg Yang, Adith Swaminathan, and Jason D. Williams. 2019. Nail: A general interactive fiction agent. CoRR, abs/1902.04259

    2019 arXiv

  5. [13]

    Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2016. Deep Reinforcement Learning with a Natural Language Action Space . In Association for Computational Linguistics (ACL)

    2016

  6. [14]

    Girish Joshi and Girish Chowdhary. 2018. https://doi.org/10.1109/ICRA.2018.8462977 Cross-domain transfer in reinforcement learning using target apprentice . In Proceedings of the International Conference on Robotics and Automation, pages 7525--7532

    2018

  7. [15]

    George Konidaris and Andrew G. Barto. 2007. Building portable options: Skill transfer in reinforcement learning. In IJCAI

    2007

  8. [16]

    George Konidaris, Ilya Scheidwasser, and Andrew G. Barto. 2012. Transfer in reinforcement learning via shared features. The Journal of Machine Learning Research, 13:1333--1371

    2012

  9. [17]

    Long-Ji Lin. 1993. https://doi.org/10.1021/ja01613a031 Reinforcement learning for robots using neural networks . Ph.D. thesis, Carnegie Mellon University

    1993 doi

  10. [18]

    Yaxin Liu and Peter Stone. 2006. Value-function-based transfer for reinforcement learning using structure mapping. In AAAI

    2006

  11. [19]

    Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. 2017. https://doi.org/10.1613/jair.1.11263 Deep transfer in reinforcement learning by language grounding . Journal of Artificial Intelligence Research, 63

    2017 doi

  12. [20]

    Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. 2015. Language Understanding for Text-based Games Using Deep Reinforcement Learning . In Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2015

  13. [21]

    Trung Thanh Nguyen, Tomi Silander, and Tze-Yun Leong. 2012. Transferring expectations in model-based reinforcement learning. In NIPS

    2012

  14. [22]

    Salakhutdinov

    Emilio Parisotto, Jimmy Ba, and Ruslan R. Salakhutdinov. 2016. Actor-mimic: Deep multitask and transfer reinforcement learning. CoRR, abs/1511.06342

    2016 arXiv

  15. [23]

    Lakshminarayanan, Mitesh M

    Janarthanan Rajendran, Aravind S. Lakshminarayanan, Mitesh M. Khapra, P. Prasanna, and Balaraman Ravindran. 2017. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. In ICLR

    2017

  16. [24]

    Rusu, Neil C

    Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. CoRR, abs/1606.04671

    2016 arXiv

  17. [25]

    Richard S Sutton and Andrew G Barto. 2018. Reinforcement Learning: An Introduction . MIT Press

    2018

  18. [26]

    Ruo Yu Tao, Marc-Alexandre C \^ o t \' e , Xingdi Yuan, and Layla El Asri. 2018. Towards solving text-based games by producing adaptive action spaces. In Proceedings of the 2018 NeurIPS Workshop on Wordplay: Reinforcement and Language Learning in Text-based Games

    2018

  19. [27]

    Taylor, Nicholas K

    Matthew E. Taylor, Nicholas K. Jong, and Peter Stone. 2008. Transferring instances for model-based reinforcement learning. In ECML/PKDD

    2008

  20. [28]

    Taylor and Peter Stone

    Matthew E. Taylor and Peter Stone. 2009. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10:1633--1685

    2009

  21. [29]

    Zhuoran Wang, Tsung-Hsien Wen, Pei hao Su, and Yannis Stylianou. 2015. Learning domain-independent dialogue policies via ontology parameterisation. In SIGDIAL Conference

    2015

  22. [30]

    Yin and S

    H. Yin and S. J. Pan. 2017. Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, pages 1640--1646. AAAI Press

    2017

  23. [31]

    Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. 2019. https://open...

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.