{"id":"f10a4f55-17fe-48bc-bee1-5628d66dfb9a","arxiv_id":"2506.07046","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"QForce-RL is a quantized, reconfigurable FPGA compute engine for reinforcement learning inference that claims up to 2.6x FPS and substantial resource savings versus prior accelerators.","lead":"This paper presents QForce-RL, an FPGA accelerator design that combines quantization with a reconfigurable SIMD multiply-accumulate unit and a CORDIC-based activation unit to run reinforcement learning policies faster and with lower power. The authors report up to 2.6x higher FPS and better energy efficiency than prior RL accelerators, but the comparison uses different FPGAs and does not validate RL accuracy on the hardware.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hardware-specific quantization and CORDIC approximations are never run in an RL episode, so the claim of 'no significant performance degradation' rests on software-only QuaRL results.","rationale":"The paper's strongest claim is that QForce-RL delivers up to 2.6x FPS and 2.3x performance over SoTA without significant RL performance degradation. The reader's weakest assumption correctly identifies that the reward-preservation part is not validated on the actual hardware: Figure 3 shows QuaRL software results, and Section III-C reports only synthesis, implementation, and CPU emulation metrics. Unit-level QoR values in Section III-A cannot certify end-to-end RL correctness because small MAC and activation errors can be amplified by LSTM recurrence and the RL environment feedback loop. This is the most load-bearing concern because a high-FPS accelerator that no longer solves the RL task would not constitute a valid RL compute engine. The FPS comparison in Table V is also unfair due to mismatched FPGAs and precisions, but that issue would affect only the magnitude of the speedup claim, whereas missing RL validation affects the validity of the whole design. The concrete test proposed above directly supplies the missing evidence by running the actual QForce-RL datapath on representative RL benchmarks and comparing to a matched FP32 baseline. Since this concern supports the reader's REJECT verdict without requiring any change, the verdict remains UNCHANGED.","tokens_in":11503,"tokens_out":2827,"duration_ms":31818,"concrete_test":"Re-implement the QForce-RL RTL (Q-MAC plus V-ACT at FxP8/16/32) with the trained PPO policies from Section III, and run it in RTL simulation or on the VC707 board across the Fig. 3 environments (e.g., CartPole, Pong, Walker2D), comparing episode rewards and FPS against the FP32 E2HRL baseline on the same FPGA and input size. If mean reward drops more than a pre-registered tolerance (e.g., 5%) or the speedup disappears under matched conditions, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim combines two parts: a large speedup over SoTA and no significant RL performance loss. Section III-C reports only FPGA synthesis, post-implementation resource/power numbers, and CPU emulation; no QForce-RL RTL simulation or on-FPGA RL episode is presented. Figure 3's reward curves are inherited from QuaRL [1], not from the Q-MAC/V-ACT datapath. The approximate multiplier variants in Section III-A are characterized by unit-level QoR of 98.4-99.2%, which is not evidence that end-to-end RL reward survives when errors propagate through LSTM recurrent state, softmax, and environment feedback loops. The claim 'without significant performance degradation' is therefore unsupported for the actual proposed hardware. Table V's headline 2.6x FPS comparison is also confounded by different FPGAs (Virtex-7 vs Artix-7), different bit-widths, and different workloads, but the reward-preservation gap is the more load-bearing issue: if quantization or CORDIC approximation degrades policy quality, the compute-engine claim collapses regardless of FPS.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QForce-RL, an FPGA-oriented quantized reinforcement-learning inference engine built from two hardware blocks: a configurable SIMD multiply-accumulate unit (Q-MAC) supporting FxP8/16/32 precisions and a CORDIC-based activation unit (V-ACT) supporting ReLU, Sigmoid, Tanh, and Softmax. The authors claim up to 2.3x performance enhancement and 2.6x better FPS compared with state-of-the-art accelerators, with no significant degradation in RL reward. The evidence consists of FPGA synthesis and post-implementation resource/power numbers for the Q-MAC and V-ACT, a Table V comparison of FPS/throughput/energy across several prior designs, CPU emulation latency results, and software quantization results taken from prior work (QuaRL).","tokens_in":11685,"tokens_out":7625,"duration_ms":74005,"significance":"If the empirical claims were properly supported, this work would address a real need: low-cost, reconfigurable compute elements tailored to RL workloads, where multi-precision SIMD and diverse activation functions matter. The proposed Q-MAC and V-ACT are plausible building blocks, and the reported resource numbers are modest. However, the central claims are not established by the evidence in the manuscript. The reward-preservation claim is borrowed from software quantization results, not from the proposed hardware datapath, and the headline speedups come from uncontrolled comparisons across different FPGAs and different application workloads. The paper's contribution is therefore best viewed as a preliminary hardware proposal rather than a validated RL compute engine.","major_comments":[{"comment":"The claim that QForce-RL operates \"without significant performance degradation\" is not supported by any RL experiment on the proposed hardware. The reward plots in Fig. 3 are inherited from QuaRL [1] and concern software quantization, not the Q-MAC/V-ACT datapath with CORDIC approximations. Section III-C reports only FPGA synthesis, post-implementation resource/power numbers, and CPU emulation; no RTL simulation or on-FPGA RL episode is presented. Since quantization and CORDIC errors propagate through LSTM recurrent state, softmax, and environment feedback, the unit-level QoR values in Section III-A are insufficient evidence for reward preservation. Please add an end-to-end RL evaluation, such as RTL simulation or FPGA-in-the-loop with standard RL environments, to substantiate this central claim.","section":"Section II, Fig. 3, and Section III-C"},{"comment":"The headline 2.6x FPS and related performance comparisons are not controlled. The proposed Q-FC/Q-LSTM results are on Virtex-7, whereas E2HRL [2] is on Artix-7 and other baselines use Zynq, Arria 10, KCU15, ZCU102, or Nexys A7-100T; the workloads also differ (object detection versus RL). This contradicts the statement in Section III-C that \"We have also re-implemented SoTA designs with the similar parameters for a fair comparison.\" For a valid comparison, the baselines should be re-implemented on the same FPGA family, with the same bit-width, and using the same RL policy and environment.","section":"Table V and Section III-C"},{"comment":"The approximate multiplier alternatives (ILM, CORDIC-mult, Posit-mult, Quant-MAC) are evaluated only by unit-level QoR between 98.4% and 99.2% on MAC operations. This is not evidence that end-to-end RL reward is preserved, because errors can accumulate in recurrent inference, interact with the environment, and affect policy quality. The claim of a \"modest 1.8% accuracy loss\" must be re-measured at the policy or episode level, not inferred from per-multiplier QoR.","section":"Section III-A"}],"minor_comments":[{"comment":"The claim of \"performance enhancement up to 2.3x\" is ambiguous; Table V shows 2.8x throughput for Q-FC versus E2HRL and 2.6x FPS, while 2.3x appears closest to the energy-efficiency ratio (26.1/11.4). Please state explicitly which metric the 2.3x refers to.","section":"Abstract and Table V"},{"comment":"The manuscript contains numerous typos and grammatical errors, including \"became\" in the first sentence, \"efforts efforts\" in Section I, and \"Rge energy consumption\" in Section III-C. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The caption \"Off-chip Components\" does not match the plot, which shows convergence-time speedup across RL tasks. Please correct the caption or the figure content.","section":"Fig. 3(b)"},{"comment":"The uniform affine quantization formula is not clearly defined: specify whether the min/max are taken over the weight tensor, define the scale and zero-point notation, and clarify the meaning of Qn and Wq.","section":"Section II, Eq. (1)"},{"comment":"The convergence-cycle comparison \"3n/8+1 cycles compared to (n/2+1) stages\" is stated without derivation or a supporting reference. Please provide the basis for this claim.","section":"Section III-B"},{"comment":"The ASIC synthesis results should state the synthesis corner (SS/TT/FF), supply voltage, and cell library version for the TSMC 28nm technology, so that power and area numbers are reproducible.","section":"Table III"}],"recommendation":"major_revision","confidential_remarks":"The missing evidence is substantial: the reward-preservation claim requires an end-to-end RL evaluation of the actual Q-MAC/V-ACT datapath, and the speedup claims require controlled comparisons on a common FPGA and workload. If the authors cannot provide such experiments in a revision, the paper should be rejected. I also note that the manuscript relies heavily on the authors' own prior work (Flex-PE, Quant-MAC); the editor may wish to check that the novel contribution relative to those works is clearly delineated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jha et al. have put together a plausible but unproven piece of engineering. The architecture is a sensible merging of E2HRL's hierarchical RL template with QuaRL-style quantization, wrapped in a SIMD Q-MAC and a CORDIC-based V-ACT. The unit descriptions are clear, the synthesis numbers for the MAC and activation blocks are new, and the paper is honest about its lineage—Flex-PE, Quant-MAC, and LPRE are cited as the source of the building blocks. That is real incremental value: the integration is not a new idea, but it is a reasonable one.\n\nThe load-bearing problem is that the headline claims are not supported by what is actually measured. Table V compares the proposed design on Virtex-7 at 250 MHz with 8-bit Q-MACs against E2HRL on Artix-7 at 100 MHz with 32-bit SISD, and against object-detection accelerators on entirely different workloads. The 2.6x FPS figure is therefore not a controlled speedup; it is a confounded one. The same applies to 'no significant performance degradation.' Figure 3's reward curves are QuaRL's, not QForce-RL's. No RL episode is ever run on the FPGA or on an RTL simulation of the Q-MAC/V-ACT datapath. Unit-level QoR of 98.4–99.2% tells us nothing about whether policy quality survives after errors propagate through an LSTM recurrent state, softmax, and environment feedback. The authors implicitly confirm this by reporting only synthesis, emulation, and CPU measurements.\n\nThere are smaller issues: the CPU comparison uses a modern Ryzen against older mobile cores, and Table IV's arithmetic-intensity column contains numbers that look implausible next to the delay and power values. These are minor next to the missing end-to-end RL validation.\n\nMy take: the paper is coherent and the hardware idea is worth a serious look, but as submitted it does not establish its two central claims. A fair revision would add controlled comparisons (same FPGA, same workload, same precision baseline) and at least an RTL or emulation-level RL rollout with reward curves for the actual datapath. I'd send it to referees, expect major revision, and not cite the headline numbers in the meantime.","headline":"A credible incremental FPGA RL accelerator whose headline speedups and reward-preservation claims are not supported by the evidence as presented.","tokens_in":12251,"tokens_out":3021,"would_cite":false,"duration_ms":32262,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A quantized FPGA engine for reinforcement-learning policies is claimed to reach up to 2.6x the frame rate of prior accelerators at a fraction of the energy.","keywords":["reinforcement learning","FPGA accelerator","quantization","SIMD processing element","multiply-accumulate unit","CORDIC activation function","hierarchical deep reinforcement learning","edge AI"],"falsifier":"Run the quantized policy network on the FPGA in a closed-loop RL environment, such as an Atari or continuous-control task from the software study, and compare cumulative reward against the FP32 baseline; a materially larger reward drop, or FPS/energy numbers that do not reproduce the reported 2.6x and 0.15 mJ figures on the same board, would refute the central claim.","tokens_in":11282,"feed_emoji":"🤖","tokens_out":9399,"duration_ms":92291,"temperature":0.7,"pith_summary":"The paper claims that reinforcement-learning policy inference can be made cheap enough for edge FPGAs by combining quantized fixed-point arithmetic with a configurable SIMD multiply-accumulate unit and a CORDIC-based activation unit. QForce-RL is built from these units into a hierarchical CNN/LSTM agent, and the reported results are 2.3x throughput and up to 2.6x frame rate over prior accelerators, with frame energies of 0.15 mJ for the fully connected configuration and 0.26 mJ for the LSTM configuration. The payoff, if the claims hold, is an FPGA design that lets resource-constrained devices run RL agents in real time without the memory and power cost of 32-bit floating-point computing. The accuracy evidence is the paper's load-bearing link: reward preservation is taken from a software quantization study, while the hardware itself is demonstrated through synthesis, emulation, and resource tables.","feed_headline":"Quantized FPGA engine claims 2.6x the frame rate for RL","feed_subtitle":"A configurable SIMD MAC and CORDIC activation unit cut energy per frame to 0.15 mJ on edge FPGAs.","key_machinery":"The load-bearing mechanisms are Q-MAC and V-ACT. Q-MAC is a runtime-configurable SIMD multiply-accumulate unit that reuses sixteen 8-bit multipliers to deliver 16/4/1 MACs per cycle at FxP8/16/32 precision, shrinking LUT and flip-flop counts while keeping throughput up. V-ACT is a unified activation unit that derives ReLU, sigmoid, tanh, and softmax from low-latency CORDIC hyperbolic and linear stages, with an HOAA path that folds exponent handling into the hyperbolic output for a reported 21% gain. Together the two units let one quantized datapath serve conv, FC, and LSTM layers at three precisions, which is what turns quantization from a software choice into a hardware resource and energy saving.","core_discovery":"QForce-RL is presented as a complete FPGA-oriented compute engine for hierarchical deep RL in which convolution, fully connected, LSTM gating, and softmax all run on quantized fixed-point hardware with selectable 8/16/32-bit precision. The core datapath is a Q-MAC array of sixteen 8-bit multipliers that can be fused into four 16-bit or one 32-bit MAC per cycle, so a single unit delivers SIMD parallelism at low precision without a multiplier per lane. Activations go through V-ACT, a low-latency CORDIC-based unit that computes ReLU, sigmoid, tanh, and softmax and converges in (3n/8+1) cycles; the paper reports a 21% performance gain from its HOAA mechanism and up to 15% resource savings over an existing multi-precision SIMD processing element. In an 8-PE configuration at 232 MHz the authors report 11 GOPS and 2.8 GOPS for Q-FC and Q-LSTM, energy efficiencies of 26.1 and 7.8 GOPS/W, and 0.15/0.26 mJ per frame. The conclusion that RL performance is preserved rests on a software quantization study; the hardware evidence is synthesis and emulation, not closed-loop RL episodes.","pith_inferences":["If the hardware really preserves the software-measured rewards, the Q-MAC/V-ACT pair could be reused in existing CNN/LSTM accelerators to give edge robots and drones real-time RL inference without retraining.","The paper validates inference only; an online training check would test whether quantized actors also preserve learning convergence when the accelerator interacts with the environment, not just when it replays a stored policy.","Because V-ACT covers softmax and tanh/sigmoid and Q-MAC is precision-scalable, the unit is not RL-specific, and the reported DNN numbers suggest the larger payoff may be a mixed-precision DNN accelerator.","The FPS gains are reported against different baselines in different tables; a direct same-board comparison against the hierarchical 32-bit baseline would separate the contribution of quantization from that of SIMD parallelism."],"forward_implications":["At 8-bit precision the synthesized 28nm Q-MAC reaches 6.37 TOPS/W and 6.28 TOPS/mm$^2$, about 3.8x the energy efficiency of the nearest multi-precision MAC baseline in the comparison table.","Replacing the standard multiplier inside Q-MAC with approximate alternatives such as CORDIC-mult, Quant-MAC, ILM, or posit-mult keeps quality of results at 98.4–99.2% and saves up to 42% area, so the accelerator can trade accuracy for resource use at deployment time.","The same Q-MAC and V-ACT blocks, assembled into a DNN accelerator, are claimed to yield 112 ms inference latency (69.3 GOPS) on a representative workload, beating three published edge accelerators with about 1% accuracy difference.","Parameterized PE count and precision selection let one architecture span FPGA and mobile CPU deployments, with 8-bit SIMD giving 2.6x latency improvement over 32-bit on a CPU core."],"supporting_citations":[{"why":"the software quantization study that supplies the reward-loss curves showing quantized RL policies stay close to FP32.","marker":"[1]"},{"why":"the hierarchical RL baseline whose CNN/LSTM topology and 32-bit CPU latency the proposed engine extends and compares against.","marker":"[2]"},{"why":"the multi-precision SIMD processing element whose precision-scalable design Q-MAC adapts and that serves as the resource-saving comparison for V-ACT.","marker":"[11]"},{"why":"the low-latency CORDIC algorithm whose cycle count V-ACT inherits for hyperbolic and linear activation paths.","marker":"[34]"},{"why":"a quantized multiply-accumulate unit design used both as a comparison point and as one of the approximate-multiplier options Q-MAC can host.","marker":"[17]"},{"why":"a CORDIC-based DNN accelerator that appears as a baseline for the claimed 112 ms DNN inference extension.","marker":"[24]"}],"fun_headline_variants":["QForce-RL: quantized RL hits 2.6x FPS on edge FPGAs","FPGA RL engine: 2.6x FPS and 0.15 mJ per frame","Quantized MAC and CORDIC: 2.6x FPS for RL on FPGA","RL on FPGA: 2.6x FPS, 0.15 mJ/frame with quantization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central accuracy claim depends on the assumption that the rounding and activation approximations built into the hardware change the policy's reward by only the negligible amount seen in the software quantization study; the paper's reward curves come from that software study, not from running the FPGA or its RTL simulation.","fun_headline_variants_meta":{"raw":{"variants":["QForce-RL: quantized RL hits 2.6x FPS on edge FPGAs","FPGA RL engine: 2.6x FPS and 0.15 mJ per frame","Quantized MAC and CORDIC: 2.6x FPS for RL on FPGA","RL on FPGA: 2.6x FPS, 0.15 mJ/frame with quantization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001332,"raw_usage":{"total_tokens":5444,"prompt_tokens":999,"completion_tokens":4445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":4339}},"tokens_in":615,"tokens_out":4445,"duration_ms":30492,"temperature":1.0,"reasoning_tokens":4339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:42:45.838332+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the quantized policy network on the FPGA in a closed-loop RL environment, such as an Atari or continuous-control task from the software study, and compare cumulative reward against the FP32 baseline; a materially larger reward drop, or FPS/energy numbers that do not reproduce the reported 2.6x and 0.15 mJ figures on the same board, would refute the central claim.","supporting_citations":[{"cited_title":"QuaRL: Quantization for fast and environmentally sustainable reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"the software quantization study that supplies the reward-loss curves showing quantized RL policies stay close to FP32."},{"cited_title":"E2HRL: An energy-efficient hardware ac- celerator for hierarchical deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"the hierarchical RL baseline whose CNN/LSTM topology and 32-bit CPU latency the proposed engine extends and compares against."},{"cited_title":"Flex-PE: Flexible and SIMD Multi-Precision Processing Element for AI Workloads,","cited_arxiv_id":null,"evidence_quote":"the multi-precision SIMD processing element whose precision-scalable design Q-MAC adapts and that serves as the resource-saving comparison for V-ACT."},{"cited_title":"Low Latency Hybrid CORDIC Algorithm,","cited_arxiv_id":null,"evidence_quote":"the low-latency CORDIC algorithm whose cycle count V-ACT inherits for hyperbolic and linear activation paths."},{"cited_title":"QuantMAC: Enhancing Hardware Perfor- mance in DNNs With Quantize Enabled Multiply-Accumulate Unit,","cited_arxiv_id":null,"evidence_quote":"a quantized multiply-accumulate unit design used both as a comparison point and as one of the approximate-multiplier options Q-MAC can host."},{"cited_title":"An Empirical Approach to Enhance Performance for Scalable CORDIC-Based Deep Neural Networks,","cited_arxiv_id":null,"evidence_quote":"a CORDIC-based DNN accelerator that appears as a baseline for the claimed 112 ms DNN inference extension."}],"review_version":1}