VUSFA:Variational Universal Successor Features Approximator to Improve Transfer DRL for Target Driven Visual Navigation

Denys J.C. Matthies; Rivindu Weerasakera; Shamane Siriwardhana; Suranga Nanayakkara

REVIEW 4 major objections 6 minor 18 references

VUSFA:Variational Universal Successor Features Approximator to Improve Transfer DRL for Target Driven Visual Navigation

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VUSFA improves transfer in target-driven visual navigation by conditioning the A3C policy on universal successor features, lifting zero-shot goal success from about 15% to 20% in AI2THOR.

desk verdict A plausible architectural idea with consistent-but-weak empirical gains; the successor-feature interpretation is not actually supported because the USF head is effectively trained by the value loss, not the TD loss. read the letter →

arxiv 1908.06376 v1 pith:ETUOA47A submitted 2019-08-18 cs.LG cs.AIcs.NEstat.ML

Shamane Siriwardhana , Rivindu Weerasakera , Denys J.C. Matthies , Suranga Nanayakkara This is my paper · ORCID

classification cs.LGcs.AIcs.NEstat.ML

keywords UniversalSuccessorFeaturestransferreinforcementlearningtarget-drivenvisualnavigationA3Cactor-criticAI2THORvariationalinformationbottleneckpolicyconditioning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a reinforcement-learning agent can transfer to new navigation goals much better when its policy is allowed to read the expected future-state occupancy vector that drives its value estimate. The proposed architecture, VUSFA, combines Universal Successor Features with an A3C actor-critic, adds a Successor Feature Dependent Policy (SFDP) that feeds the successor feature vector directly into the policy head, and applies a variational information bottleneck to the shared state-goal encoder. Evaluated in AI2THOR after training on only 20 goal states, about 1.15% of the tested states, VUSFA reaches unseen goals from random starts in 20.01% of episodes, versus 15.04% for the General Value Function baseline, and adapts to new goals in fewer training steps. If correct, the result suggests successor features can serve not only as a value-decomposition tool but also as useful perceptual input for action selection, and that compressing the state-goal embedding helps generalization.

What carries the argument

The load-bearing object is the Universal Successor Feature vector $\mathbf{y}(s,g)$, defined as the expectation of discounted future state features $\mathbf{f}(s_{t+1})$ the agent will encounter from state $s$ when heading toward goal $g$. It carries the argument because the value function is written as $V(s,g)=\mathbf{y}(s,g)^\top\mathbf{w}(g)$, so $\mathbf{y}$ encodes transferable environment dynamics and $\mathbf{w}$ encodes the goal's reward preference. Around this object, the paper adds two mechanisms: a Successor Feature Dependent Policy (SFDP) that concatenates $\mathbf{y}$ into the policy network's input while stopping gradients, and a Variational Siamese Bottleneck (VSB) that constrains the mutual information between the image inputs and the shared encoder's latent codes. The variational bottleneck acts as a regularizer on the embeddings that produce $\mathbf{f}$, $\mathbf{w}$, and $\mathbf{y}$.

What would settle it

Run the trained VUSFA agent on held-out goals while recording, at each state, the predicted $\mathbf{y}(s,g)$ and the actual discounted sum of successor state embeddings $\sum_{k\ge0} \gamma^k \mathbf{f}(s_{t+k})$ along the rollout. If the two disagree appreciably, or if setting $\lambda_y=0$ and removing $\mathbf{y}$ from the policy leaves the zero-shot success rate unchanged, then the reported transfer gains do not depend on learning true successor features.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a Universal Successor Feature based A3C agent can be made stable and effective in photorealistic visual navigation by learning state features and goal-encoding vectors jointly rather than pretraining an autoencoder, by training the goal-encoding vector through the actor-critic value loss instead of a separate sparse reward-regression loss, by conditioning the policy on the successor feature vector while blocking policy gradients from flowing back into it, and by applying a variational information bottleneck to the siamese encoder so that the embeddings used for the features, reward vector, and successor features stay compressed. The authors report zero-shot success rates of 16.16%, 17.23%, and 20.01% for their three model variants, against 15.04% for the GVF baseline, and faster adaptation to new goals as shown in their learning curves.

Load-bearing premise

The approach assumes that the internal vector called the successor feature really learns discounted future state occupancies, even though its own temporal-difference loss is weighted almost to zero and the paper never verifies this.

Editorial extensions

If this is right

Held-out goals are reachable after training on a tiny fraction of states: with 20 training goals, zero-shot success reaches 20.01% overall, up from 15.04% for the GVF baseline.
Feeding the successor feature vector to the policy adds transfer value on top of USFA alone: the SFDP variant improves zero-shot success from 16.16% to 17.23%, and the full model to 20.01%.
The variational bottleneck adds further value while keeping training stable: VUSFA outperforms USFA with SFDP in most environments.
If the claim is correct, the same recipe should transfer to other visual control tasks with sparse goal rewards, since the added components are task-agnostic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test this, but the architecture invites a check: with $\lambda_y=0.0005$, the successor-feature branch is trained almost entirely through the product $\mathbf{y}^\top\mathbf{w}$, so the gain attributed to $\mathbf{y}$ may instead come from the goal-conditioned embeddings or the bottleneck.
A natural next experiment is Monte Carlo validation: record rollouts and compare $\mathbf{y}(s,g)$ to $\sum_{k\ge0}\gamma^k \mathbf{f}(s_{t+k})$; agreement would confirm the successor-feature interpretation, disagreement would not.
The SFDP idea could be combined with generalized policy improvement over a library of previously learned policies, which the paper leaves unexplored, potentially improving both zero-shot and fine-tuning transfer.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper proposes VUSFA, a combination of Universal Successor Features with A3C for target-driven visual navigation in AI2THOR, plus two new bits: feeding the successor-feature vector into the policy with a stop-gradient (SFDP) and a variational bottleneck on the siamese encoder. The new architecture is real; I don't see those pieces in the cited USFA work. The authors also ship code. That's worth some credit.

What's the actual evidence? In Table 1, zero-shot success rates across four scenes: baseline GVF 15.04% overall, their USFA version 16.16%, +SFDP 17.23%, +VSB 20.01%. Consistent positive trend, but no error bars, no significance tests, and only a single training run per configuration, it seems. And the only external baseline is Zhu et al. 2017. The paper doesn't compare against the USFA methods it builds on (Ma et al., Borsa et al.). So we don't know if the gain is from USFA or from the other changes.

The bigger concern is whether y is actually a successor feature. The total loss is lp*Lp + ly*Ly + lV*LV with ly=0.0005 and lV=0.5. The value loss LV only fits the product y^T w to returns. Gradients to y from Ly are about a thousand times weaker than from LV. So the USF head is trained almost entirely by the value loss, not by the TD error in Eq. (6). The paper offers no diagnostic that y behaves like an expected discounted sum of future features. If y is just an arbitrary factorization feature, then the SFDP mechanism and the USFA interpretation lose their grounding. That's not a minor nit; the transfer claim is specifically attributed to successor features.

There is also the usual simulator-tuning worry: hyperparameters (Ic, ly) were tuned on the same evaluation scenes, so the zero-shot numbers may partially reflect that. The 500-step success criterion and 10 random starts are quite permissive, but at least the setup is transparent.

Who should read this? Someone working on transfer RL for embodied navigation might find the SFDP idea worth experimenting with, especially with code available. But as a paper, the evaluation needs a proper USFA baseline, error bars, and a verification that y satisfies the TD property or a revision that drops the successor-feature interpretation. If the authors can show that, it could become a useful contribution. As it stands, I'd give it a major-revision review, not a desk reject.

My recommendation: send it to peer review, but tell the authors the successor-feature verification is load-bearing.

Referee Report

4 major / 6 minor

Summary. The paper proposes VUSFA, an architecture that combines Universal Successor Features (USFA) with A3C for target-driven visual navigation in the AI2THOR simulator. It introduces a Successor Feature Dependent Policy (SFDP) that conditions the policy on the learned successor feature vector, and a Variational Siamese Bottleneck (VSB) that imposes an information constraint on the siamese encoder. The authors evaluate zero-shot navigation success rates and fine-tuning adaptation curves, comparing the full VUSFA model against the GVF baseline of Zhu et al. (2017), a plain USFA implementation, and USFA with SFDP. They report that VUSFA outperforms all these variants, with an overall zero-shot success rate of 20.01% versus 15.04% for the baseline.

Significance. If the central mechanism were verified, the paper would be a useful empirical extension of USFA to high-dimensional visual navigation, and the SFDP idea plus the variational bottleneck could offer practical improvements for transfer reinforcement learning. The authors provide an open-source implementation and ablate each claimed component, which are clear strengths. However, the current evidence does not establish that the learned vector y is a true successor feature, the baseline comparison omits the USFA methods the paper builds on, and the statistical support is thin. The significance of the stated contribution is therefore not yet demonstrated.

major comments (4)

[§5.2, Eq. (6), Algorithm 1, Table 2] Equation (6) defines the USF TD loss, but Table 2 sets λ_y = 0.0005 while λ_V = 0.5, so gradients from L_y are about three orders of magnitude smaller than those from L_V. Because L_V only constrains the product y^T w to match returns, the paper gives no evidence that y converges to the discounted future state occupancy described in §4 and used by SFDP. This is load-bearing, as the transfer gains are attributed to successor features. Please provide a direct diagnostic such as the Bellman residual ||y(s) − f(s) − γ y(s′)|| under the trained policy, or an ablation with a comparable λ_y, to show that y is a true successor feature.
[§7, Table 1, Figure 2] Table 1 compares Models 01–04 only against the GVF baseline of Zhu et al. (2017) and the authors' own ablations; there is no comparison with the USFA methods of Ma et al. (2018b) or Borsa et al. (2018), despite the Conclusion claiming improvement over 'previous state-of-the-art GVF and USF-RL based research.' Table 1 also reports success rates without variance, standard errors, or significance tests, and Figure 2 shows adaptation curves with no quantitative or statistical comparison. The empirical core of the paper is therefore under-supported; please add at least one external USFA baseline and report confidence intervals or standard errors for the zero-shot results and the transfer curves.
[Supplementary Algorithm 1] Algorithm 1 in the Supplementary Material is inconsistent with Eqs. (5) and (6). In lines 14–18, R_y and R_V are bootstrapped from y(s_t) and y(s_t)^T w(g), and the loop then accumulates R_y ← y(i,g) + γ R_y and R_V ← y(s_t,g)^T w(g) + γ R_V. No scalar reward r_t and no state feature f(s_i) enter these targets, and the loop index i is used for R_y but not for R_V. As written, the y-loss is circular (y predicts a discounted sum of y) and can have trivial fixed points; it does not implement the successor feature TD loss of Eq. (6). The authors must correct the pseudocode or clarify how the actual implementation differs from the text.
[§6.1, Algorithm 1 line 22] Section 6.1 states that the reparameterized embedding is used only to predict w, while the policy and the USFA head use the mean vectors of the encoder; Algorithm 1 line 22 computes the KL term only from the goal encoding f_µ(g) and f_σ(g). As a result, the Variational Siamese Bottleneck as implemented regularizes only the goal embedding that feeds w, not the state/goal representation that produces y. This is in tension with the claim in Section 5 that the VSB improves 'the quality of USFA and the reward prediction vector.' Please clarify whether the KL is applied to both state and goal encodings, and how the mean-based y and p are affected by the bottleneck.

minor comments (6)

[Title and Abstract] The title and abstract use 'VUSF A' with an extra space; please correct to 'VUSFA' throughout.
[§2.1.2, Eq. (4)] The TD loss in Eq. (4) is written as a bare expectation without a norm or square, inconsistent with Eq. (6); please unify the notation.
[§5.1] The Information Bottleneck is attributed to Tishby and Zaslavsky (2015), but the original IB framework is Tishby, Pereira, and Bialek (1999); please cite the original source.
[§6.1, Figure 1] The caption of Figure 1 and the labels inside the figure are garbled and repeated, making it hard to verify the gradient-stopping mechanism; please redraw the figure.
[§7.1] The text uses 'weather' instead of 'whether' in Section 7.1, and 'cumulutive' in Section 4 should be 'cumulative'.
[Algorithm 1] The symbol b is used both for the Lagrangian multiplier and for the beta update function in line 21; please use distinct symbols, and specify the prior r(z) used in the KL term.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical architecture comparison whose transfer claims are evaluated on held-out goals, not constructed from fitted quantities.

full rationale

VUSFA's central claim is that the proposed architecture improves zero-shot transfer and adaptation speed in AI2THOR relative to GVF and USFA baselines. The evidence is empirical: Table 1 reports success rates on goal states that were not used in training, and Figure 2 measures episode length during fine-tuning to new goals. No equation in the paper defines the reported success rates in terms of the calibrated hyperparameters or trained heads. The hyperparameter search in Table 2 (lc = 0.2, ly = 0.0005, lV = 0.5) is ordinary tuning; it does not make the zero-shot success rates true by construction. The SFDP contribution is an architectural hypothesis tested by ablation (Model 02 vs Model 03 and Model 04), and the paper explicitly stops policy gradients from flowing into the successor-feature head, so the policy's use of y is not a disguised identity. There are no load-bearing self-citations, and the cited prior USFA work is used as background formalism rather than as the sole justification for the empirical result. The most serious vulnerability is a soundness gap, not circularity: with ly = 0.0005 and lV = 0.5, the USF head y is trained almost entirely through the value loss, so the paper does not demonstrate that y actually satisfies the successor-feature TD relation of Equation 6. That missing diagnostic undermines the theoretical interpretation of SFDP, but it does not make the reported transfer improvements reduce to the paper's inputs. Therefore no circular step is present.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper imports the successor feature decomposition and the variational information bottleneck from prior work without re-deriving them. The central burden is empirical: the architecture and loss weights are fitted to the navigation task. No new conceptual entities such as particles or forces are introduced.

free parameters (5)

Information constraint I_c = 0.2
Tuned over {0.1, 0.2, 0.5} and selected for best performance (Table 2). Used in the VIB KL penalty.
USF TD loss weight lambda_y = 0.0005
Tuned over {0.0005, 0.0006, 0.0009}; the chosen value makes the successor-feature loss almost negligible relative to value and policy losses.
Value loss weight lambda_V = 0.5
Only value explored; reported as giving good results.
Policy loss weight lambda_p = 1.0
Fixed at 1.0, not tuned.
Lagrangian update rate alpha_beta = not reported
Used in Equation 11 but no value is given, so the adaptive update cannot be reproduced exactly.

assumptions (6)

domain assumption Reward can be expressed as a linear function of state features and a goal-dependent weight vector (Equation 2).
Inherited from successor feature literature (Barreto et al. 2017, Ma et al. 2018b) without re-derivation.
domain assumption The goal-conditioned value function decomposes as the inner product of successor features y and reward weights w (Equation 3).
Core factorization of USFA, used throughout the losses in Algorithm 1.
standard math The variational information bottleneck objective with a KL constraint (Equation 10) is a valid regularizer for the encoder.
Taken from Alemi et al. (2016); the paper adapts it but does not prove it.
domain assumption AI2THOR is a representative photorealistic environment for evaluating visual navigation transfer.
The evaluation is confined to four AI2THOR scenes, with no real-world validation.
domain assumption A3C with 100 threads converges to a stable policy for this task.
Standard RL practice; no convergence analysis is provided.
domain assumption The prior r(z) in the KL term is tractable and fixed.
The paper does not specify the prior distribution (e.g., standard normal), which is required to compute the KL term.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VUSFA:Variational Universal Successor Features Approximator to Improve Transfer DRL for Target Driven Visual Navigation." pith.science (2026). https://pith.science/paper/ETUOA47A

@misc{pith2026190806376,
  author       = {Pith},
  title        = {Pith review of: VUSFA:Variational Universal Successor Features Approximator to Improve Transfer DRL for Target Driven Visual Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ETUOA47A}},
  note         = {Machine review of arXiv:1908.06376}
}

read the original abstract

In this paper, we show how novel transfer reinforcement learning techniques can be applied to the complex task of target driven navigation using the photorealistic AI2THOR simulator. Specifically, we build on the concept of Universal Successor Features with an A3C agent. We introduce the novel architectural contribution of a Successor Feature Dependant Policy (SFDP) and adopt the concept of Variational Information Bottlenecks to achieve state of the art performance. VUSFA, our final architecture, is a straightforward approach that can be implemented using our open source repository. Our approach is generalizable, showed greater stability in training, and outperformed recent approaches in terms of transfer learning ability.

Figures

Figures reproduced from arXiv: 1908.06376 by the authors.

**Figure 2.** Agent’s transfer learning ability: No. of train [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** For the transfer learning tasks, 20 new goals were randomly selected excluding the trained [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 3.** Figure 3: The twenty goal locations used for training [PITH_FULL_IMAGE:figures/full_fig_p012_3.png]

**Figure 4.** Figure 4: Example of a trajectory of an agent inside the AI2THOR simulator [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Architecture used for the SFDP-A3C model [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

**Figure 6.** Figure 6: Proposed architecture by Ma et al. [2018b] to train an actor-critic agent with USF (image from Ma [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 8 canonical work pages

[1]

Deep variational information bottleneck

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint:1612.00410,

arXiv
[2]

lV = 0.5, ly = 0.0005 and Ic = 0.2 gave good results and were used in the plots. Parameters V alues Explored V alues Used Ic 0.2,0.5,0.1 0.2 ly 0.0005,0.0006,0.0009 0.0005 lV 0.5 0.5 lp 1.0 1.0 Table 2: Explored Hyperparameters 12 g st Shared f (g) f (st ) FC-1024 FC-512 wg FC-512 FC-512 ... · ˆVR FC-512 a1 a2 a3 a4 p FC-512 y(st+1) SR Loss Return Loss Po...

work page 2011
[6]

The ﬁrst one is the state representation network with parameters F

The architecture consists of three sub-networks that need to be trained. The ﬁrst one is the state representation network with parameters F. This network is an autoencoder which gets trained before the training of the DRL agent. Then there are two heads to approximate the USF ( qp ), and policy (qy ). These two heads share parameters in the early layers w...

work page 2016
[9]

Playing atari with deep reinforcement learning

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint:1312.5602,

arXiv
[14]

Transfer in RL has been evaluated on multiple similar tasks by Barreto et al

were able to learn task-speciﬁc features that were able to quickly adapt to distal changes in the reward by ﬁne-tuning only the reward prediction feature vector. Transfer in RL has been evaluated on multiple similar tasks by Barreto et al. [2017], who introduced Successor Features (SF). They adapted SR to be applicable in the continuous domain and were ab...

work page 2017
[15]

was used by Zhang et al. [2017]. Unlike our approach, they showcase their solution in simple maze-like environments using DQN as the baseline method, while we use actor-critic methods in a photorealistic simulation environment. DQN-based techniques frequently suffer from stability issues when applied to complex problems, such as large-scale navigation [Ba...

work page 2017
[16]

[Borsa et al., 2018] work describes a USFA training method by adapting e-greedy Q learning for set of simple tasks in Deepmind-Lab [Beattie et al., 2016] simulator

also extended the idea of SF-RL in to USFA by adapting the concepts of GVF and generalized policy improvement theorem. [Borsa et al., 2018] work describes a USFA training method by adapting e-greedy Q learning for set of simple tasks in Deepmind-Lab [Beattie et al., 2016] simulator. We found their method is hard to adapt to our problem domain mainly due t...

work page 2018
[18]

Moreover, Alemi et al

J(q, E)min = E(z⇠E(z|x)) [Jp(q(y|z))] s.t E(x⇠p(x)) [KL[E(z|x)kr(z)]]  Ic (18) The constraint term of E(x⇠p(x)) [KL[E(z|x)kr(z)]]  Ic mentioned in Equation 18 can be subsumed into the loss function J(q, E)min with a Lagrangian Multiplier [Bertsekas, 2014] b that needed to be updated in an adaptive manner with the training procedure of the neural network...

work page 2014

Show all 18 references

[1991]

Universal successor features approximators

Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint:1812.07626,

arXiv
[1993]

Crowdmove: Au- tonomous mapless navigation in crowded scenarios

Tingxiang Fan, Xinjing Cheng, Jia Pan, Dinesh Monacha, and Ruigang Y ang. Crowdmove: Au- tonomous mapless navigation in crowded scenarios. arXiv preprint:1807.07870,

arXiv
[1995]

V ariational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information ﬂow

Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. V ariational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information ﬂow. arXiv preprint:1810.00821,

arXiv
[2003]

Auto-Encoding V ariational Bayes

Diederik P Kingma and Max Welling. Auto-Encoding V ariational Bayes. arXiv e-prints, art. arXiv:1312.6114, Dec

arXiv
[2009]

Deep learning and the information bottleneck principle

Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5. IEEE,

2015
[2013]

Ai2-thor: An interactive 3d environment for visual ai

Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Y uke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint:1712.05474,

arXiv
[2015]

Asynchronous methods for deep reinforcement learning

9 V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proc of ICML, pages 1928–1937,

1928
[2016]

Universal successor features for transfer reinforcement learning

Chen Ma, Dylan R Ashley, Junfeng Wen, and Y oshua Bengio. Universal successor features for transfer reinforcement learning. 2018a. Chen Ma, Junfeng Wen, and Y oshua Bengio. Universal successor representations for transfer reinforcement learning. arXiv preprint:1804.03758, 2018...

arXiv
[2017]

Deep successor reinforcement learning

Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint:1606.02396,

arXiv
[2018]

Deepmind lab

Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küt- tler, Andrew Lefrancq, Simon Green, Víctor V aldés, Amir Sadik, et al. Deepmind lab. arXiv preprint:1612.03801,

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Deep variational information bottleneck

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint:1612.00410,

arXiv

[2] [2]

lV = 0.5, ly = 0.0005 and Ic = 0.2 gave good results and were used in the plots. Parameters V alues Explored V alues Used Ic 0.2,0.5,0.1 0.2 ly 0.0005,0.0006,0.0009 0.0005 lV 0.5 0.5 lp 1.0 1.0 Table 2: Explored Hyperparameters 12 g st Shared f (g) f (st ) FC-1024 FC-512 wg FC-512 FC-512 ... · ˆVR FC-512 a1 a2 a3 a4 p FC-512 y(st+1) SR Loss Return Loss Po...

work page 2011

[3] [6]

The ﬁrst one is the state representation network with parameters F

The architecture consists of three sub-networks that need to be trained. The ﬁrst one is the state representation network with parameters F. This network is an autoencoder which gets trained before the training of the DRL agent. Then there are two heads to approximate the USF ( qp ), and policy (qy ). These two heads share parameters in the early layers w...

work page 2016

[4] [9]

Playing atari with deep reinforcement learning

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint:1312.5602,

arXiv

[5] [14]

Transfer in RL has been evaluated on multiple similar tasks by Barreto et al

were able to learn task-speciﬁc features that were able to quickly adapt to distal changes in the reward by ﬁne-tuning only the reward prediction feature vector. Transfer in RL has been evaluated on multiple similar tasks by Barreto et al. [2017], who introduced Successor Features (SF). They adapted SR to be applicable in the continuous domain and were ab...

work page 2017

[6] [15]

was used by Zhang et al. [2017]. Unlike our approach, they showcase their solution in simple maze-like environments using DQN as the baseline method, while we use actor-critic methods in a photorealistic simulation environment. DQN-based techniques frequently suffer from stability issues when applied to complex problems, such as large-scale navigation [Ba...

work page 2017

[7] [16]

[Borsa et al., 2018] work describes a USFA training method by adapting e-greedy Q learning for set of simple tasks in Deepmind-Lab [Beattie et al., 2016] simulator

also extended the idea of SF-RL in to USFA by adapting the concepts of GVF and generalized policy improvement theorem. [Borsa et al., 2018] work describes a USFA training method by adapting e-greedy Q learning for set of simple tasks in Deepmind-Lab [Beattie et al., 2016] simulator. We found their method is hard to adapt to our problem domain mainly due t...

work page 2018

[8] [18]

Moreover, Alemi et al

J(q, E)min = E(z⇠E(z|x)) [Jp(q(y|z))] s.t E(x⇠p(x)) [KL[E(z|x)kr(z)]]  Ic (18) The constraint term of E(x⇠p(x)) [KL[E(z|x)kr(z)]]  Ic mentioned in Equation 18 can be subsumed into the loss function J(q, E)min with a Lagrangian Multiplier [Bertsekas, 2014] b that needed to be updated in an adaptive manner with the training procedure of the neural network...

work page 2014