{"id":"17104959-bb22-4124-8c11-dc88344f7a9c","arxiv_id":"2411.18002","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors swap optical flow for a learned representation flow layer in an egocentric action recognition network, reducing inference time by orders of magnitude with accuracy roughly unchanged.","lead":"This paper replaces the expensive optical flow computation in a two-stream action recognition model with a trainable representation flow layer, cutting per-video inference time from about 100 to 200 seconds down to about 0.15 seconds while roughly matching accuracy. A smart generalist would read it to see a practical path toward faster video action recognition, especially for egocentric or wearable camera systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5's 'without representation flow layer' baseline (20.08%) contradicts Table 4's RGB-only accuracy (50.00%), undermining the ablation that selects 2 flow layers and the HMDB accuracy gain.","rationale":"The reader's weakest assumption correctly identifies that hyperparameters were chosen on HMDB and may not generalize. My concern sharpens this: the ablation table that justifies the chosen hyperparameters contains a direct internal contradiction with Table 4, so the selection is not merely untested but its evidential basis is broken. This does not overturn the speed claim, which is well motivated and orders-of-magnitude robust, nor does it prove the accuracy numbers are false; it means the accuracy comparison on HMDB, and by extension the borrowed hyperparameters on GTEA61 and EGTEA GAZE+, cannot be verified from the paper alone. A single corrected ablation row would settle whether representation flow layers actually help. The reader's CONDITIONAL verdict remains appropriate, with the condition now extended to require correcting the ablation baseline and releasing the protocol/code. I do not see a reason to reject the paper outright: the central architectural idea is plausible, the runtime numbers are consistent with eliminating external optical flow, and the internal inconsistency may be a reporting error. But the paper should not be accepted as final until this baseline is resolved.","tokens_in":7537,"tokens_out":10311,"duration_ms":95481,"concrete_test":"Re-run the Table 5 row 'Without representation flow layer' under exactly the same architecture, fusion, and training protocol as the winning '2 representation flow layers' row, and also report the RGB-only branch under that same protocol. If the corrected 'without' value is approximately 50% rather than 20.08%, the reported benefit of representation flow layers collapses and the hyperparameter selection is invalid. If the corrected value is 20.08%, report the full configuration and training details so the discrepancy with the 50.00% RGB-only result can be explained.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim rests on the choice of two representation flow layers, 20 iterations, and ResNet-34, all selected by ablation on HMDB (Section III-C, Tables 5-7). Table 5 is internally inconsistent with Table 4 in a way that makes this selection unreliable. In Table 4, the RGB-only branch on HMDB is 50.00%, and the two-stream representation-flow model is 50.71%. In Table 5, the row 'Without representation flow layer' reports 20.08% for 'the proposed model.' If this row means the flow branch is removed, it should match the RGB-only 50.00%; if it means the representation branch is retained with the flow layers deleted, it is an unexplained configuration that is 30 points below the RGB-only baseline and far below every other row. Either way, the ablation does not compare like with like, so the 4.26-point gain attributed to adding two flow layers (20.08 to 24.34) may be an artifact of an unreported protocol change or training bug rather than evidence for the representation flow branch. Because the same HMDB set is used both to tune and to report the headline 50.71% result, the claimed 0.84% improvement over the optical-flow two-stream model is not an out-of-sample result. The speed advantage (0.15-0.19 s vs 25-204 s) is real and independent of this issue, but the 'exceeds accuracy' part of the central claim is not currently supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end two-stream action recognition network that replaces the optical-flow branch of the EgoRCNN model with a differentiable representation-flow branch. The RGB branch uses ResNet-34 with CAM-based spatial attention and ConvLSTM temporal encoding; the flow branch uses representation flow layers followed by a convolutional layer. On GTEA61, EGTEA GAZE+, and HMDB, the authors report that the proposed model matches or slightly exceeds the optical-flow two-stream baseline (69.84%, 61.09% vs 60.44%, 50.71% vs 49.87%) while reducing per-video prediction time from tens or hundreds of seconds to roughly 0.15-0.19 seconds. Ablation studies on HMDB select two flow layers, 20 iterations, and ResNet-34 as the flow backbone.","tokens_in":7861,"tokens_out":1760,"duration_ms":17625,"significance":"The central speed claim is credible and potentially valuable: removing offline optical-flow extraction is a well-motivated way to make two-stream egocentric action recognition practical, and the reported runtime reduction (roughly three orders of magnitude) follows directly from eliminating a dominant pre-processing cost. The architecture itself is a straightforward integration of published components (representation flow from Piergiovanni and Ryoo; EgoRCNN from Sudhakaran and Lanz), and the paper does not claim a new theory or a new dataset. If the accuracy parity/improvement were established with proper statistical care, the contribution would be a useful engineering result for real-time egocentric recognition. However, the accuracy evidence is not currently reliable enough to support the headline claim of exceeding the optical-flow baseline, because the ablation that motivates the key design choices is internally inconsistent and the main comparison is performed on the same HMDB split used for hyperparameter selection.","major_comments":[{"comment":"The ablation that selects the number of representation flow layers is internally inconsistent with the main results. Table 4 reports RGB flow only at 50.00% on HMDB and representation flow only at 24.34%; Table 5 reports 'Without representation flow layer' at 20.08%. If this row means the flow branch is the RGB-only model, it should match 50.00%; if it means something else, such as a model with the flow layers removed but other components retained, the configuration is not defined and is 30 points below the RGB-only baseline. Either way, the 4.26-point gain from adding two flow layers (20.08 to 24.34) is not a controlled comparison, so the choice of two flow layers is not supported by the evidence presented.","section":"Section III-C, Tables 4 and 5"},{"comment":"The HMDB hyperparameters (number of flow layers, iterations, and backbone size) are selected by ablations on HMDB and then the same HMDB results are used to claim a 0.84% improvement over the optical-flow baseline. This makes the reported HMDB accuracy an in-sample, hyperparameter-tuned number rather than an out-of-sample evaluation. A held-out validation split or a separate test set is needed before the accuracy improvement on HMDB can be credited.","section":"Section III-B, Table 4 and Section III-C"},{"comment":"All accuracy comparisons are reported as point estimates without error bars, multiple seeds, or significance testing. On GTEA61 the proposed and baseline models both report exactly 69.84%, and the claimed gains on EGTEA GAZE+ (0.65%) and HMDB (0.84%) are well within the range that could arise from random seed variation, especially for small datasets. The paper should report the variance across runs or demonstrate that the differences are statistically meaningful before claiming that the proposed model 'exceeds' the original model.","section":"Section III-B, Tables 1-4"},{"comment":"The state-of-the-art comparison is not usable as presented. The text states that Table 8 shows GTEA61 results and Table 9 shows EGTEA GAZE+ results, but the tables are not reproduced with their numeric entries, and the text claims 'a significant improvement' without giving the compared accuracies. The comparison should either be removed or presented with concrete numbers, splits, and citations.","section":"Section III-C, Tables 8-9"}],"minor_comments":[{"comment":"The keyword list contains 'CAM' twice; the duplicate should be removed.","section":"Abstract and Keywords"},{"comment":"The survey of prior work attributes 'Heng et al.' to dense trajectories, but the correct author is Heng Wang; the reference list itself uses 'Wang, Heng' correctly. Consider fixing the in-text name.","section":"Section I, references"},{"comment":"The cross-entropy loss sums over k categories with indicator (c == i), but the notation for the summation index and the use of k are not defined consistently; please clarify the index range and the meaning of k.","section":"Section II-B, Equation (18)"},{"comment":"There are typographical issues such as 'pre video' and 'state-of-art'; these should be corrected during revision.","section":"Section III-B, GTEA61 paragraph"},{"comment":"Some table captions are incomplete or generic ('Table 1: Comparison ...') and the reader must interpolate the configuration from the text; adding explicit column headers and dataset split information inside each table would improve clarity.","section":"Tables 1-4"}],"recommendation":"major_revision","confidential_remarks":"The speed advantage is real and well explained by the architecture, but the accuracy claim is the load-bearing part of the advertised contribution and it is currently undermined by the Table 4/Table 5 inconsistency and by in-sample hyperparameter selection on HMDB. I would not reject the paper: the central idea is sound, and the accuracy evidence could be fixed by re-running with held-out validation and reporting variance. However, the authors should be asked to either reconcile the ablation protocol or remove the unsupported accuracy-improvement claim and frame the contribution as a speed/parity result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the runtime reduction is real and welcome, but the accuracy claim is not as solid as the abstract suggests. The internal inconsistency between Table 4 and Table 5 is load-bearing.\n\nWhat's actually new: the paper replaces the optical flow branch of EgoRCNN with representation flow, making the whole thing trainable end-to-end, and measures the speed/accuracy trade-off on GTEA61, EGTEA GAZE+, and HMDB. It's a straightforward engineering combination, not a new method, but a legitimate practical variant.\n\nWhat the paper does well: the speedup numbers are credible and well-motivated. Removing precomputed optical flow obviously eliminates the dominant cost, and the reported classification times are on the right order. The method section is clear enough to replicate the architecture, and comparing against the original EgoRCNN on the same datasets is the correct evaluation setup.\n\nThe soft spots are real. Table 5 reports 20.08% for the 'without representation flow layer' configuration, while Table 4 reports 50.00% for the RGB-only branch. Those numbers describe the same thing and are 30 points apart. That makes the ablation selecting '2 flow layers' unreliable, and the 4.26-point gain attributed to adding flow layers is likely an artifact of an unreported protocol change or a typo. On top of that, HMDB is used to select hyperparameters (flow layers, iterations, backbone), then the same split is used to report the headline 50.71% accuracy, so the 0.84% gain over the optical-flow baseline is not out-of-sample. There are no error bars or significance tests, and the exact match of 69.84% on GTEA61 adds to the suspicion that the accuracy differences are within noise.\n\nNone of this undercuts the efficiency story. The paper is worth a referee's time because the speedup is real and the comparison is honest in intent, but the accuracy claims need major revision: fix the ablation, add variance, and separate tuning from evaluation.\n\nRecommendation: send to peer review with a request for major revision. The reader's conditional verdict is right; I lean skeptical on the accuracy numbers but fully accept the speedup.","headline":"Real speedup, shaky accuracy: the Table 4/5 ablation inconsistency undermines the flow-layer selection, and HMDB tuning contaminates the headline accuracy.","tokens_in":8382,"tokens_out":2518,"would_cite":false,"duration_ms":21657,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the optical-flow branch of a two-stream egocentric action recognition model with a differentiable representation flow layer preserves accuracy on three benchmarks while shrinking per-video prediction time from minutes to under…","keywords":["two-stream network","egocentric action recognition","representation flow","optical flow replacement","class activation map","ConvLSTM","end-to-end training","video action recognition"],"falsifier":"A held-out egocentric dataset with fast camera motion would separate the two motion representations: if the representation-flow branch trails the optical-flow branch by more than a few absolute accuracy points there, the claim that learned flow generalizes as well as classical flow for egocentric action is falsified.","tokens_in":7330,"feed_emoji":"🎬","tokens_out":6159,"duration_ms":47316,"temperature":0.7,"pith_summary":"The paper proposes replacing the optical flow branch of a two-stream egocentric action recognition network with a differentiable representation flow layer, enabling end-to-end training. The resulting model matches the original accuracy on GTEA61 (69.84%) and improves it by 0.65 and 0.84 percentage points on EGTEA GAZE+ and HMDB, respectively. Per-video prediction time drops from 101.68–203.99 seconds to 0.15–0.19 seconds. The authors argue this makes accurate two-stream action recognition practical for egocentric video.","feed_headline":"Swap optical flow for representation flow, cut runtime 1,000x","feed_subtitle":"Two-stream action recognition matches optical-flow accuracy on GTEA, EGTEA, and HMDB at a fraction of the cost.","key_machinery":"The representation flow layer is a fully differentiable motion estimator that computes a flow field $u$ by iteratively solving a total-variation energy minimization on deep feature maps, using learnable parameters rather than fixed hyperparameters. The iteration alternates between updating the flow estimate and the dual vector field $p$, with Sobel-filtered gradients of the feature maps as the data term. Because the layer is differentiable, the entire two-stream network can be trained end-to-end, and at inference time it eliminates the expensive step of computing and storing optical flow from raw video.","core_discovery":"On its own terms, the paper's claim is that a two-stream network for egocentric action recognition can replace precomputed optical flow with a learnable representation flow layer operating on deep feature maps, without sacrificing accuracy and while gaining a three-orders-of-magnitude speedup. The hybrid model uses class activation maps to guide spatial attention and a ConvLSTM for temporal encoding in the RGB stream, and it stacks two representation flow layers separated by a convolutional layer in the motion stream. After joint training, the model attains 69.84% on GTEA61 (identical to the optical-flow baseline), 61.09% average accuracy on EGTEA GAZE+ (versus 60.44%), and 50.71% on HMDB (versus 49.87%), with inference times of 0.1881 s, 0.1503 s, and 0.1459 s per video respectively.","pith_inferences":["The transferability assumption is the riskiest part: the representation flow hyperparameters were tuned on HMDB, and the paper does not report tuning on egocentric data, so a re-tuning on GTEA61 or EGTEA GAZE+ might change the reported comparisons.","The large runtime reduction suggests that for practical systems, the main bottleneck was precomputed optical flow, not the recognition network itself; removing it could enable on-device egocentric action recognition.","A direct comparison against an optical-flow branch with the same ResNet-34 backbone and no other architectural changes would isolate the effect of the flow representation more cleanly than the current comparison against the original model."],"forward_implications":["Two-stream action recognition for egocentric video no longer needs a separate optical-flow preprocessing stage, so models can be trained and deployed directly on raw RGB input.","Per-video prediction time falls from the order of minutes to well under a second, making real-time egocentric activity analysis feasible.","The accuracy gains on EGTEA GAZE+ and HMDB suggest representation flow can capture motion cues that are at least as informative as classical optical flow for a range of action types.","The two-branch design with joint training by cross-entropy loss is general; the same replacement could be applied to other two-stream architectures."],"supporting_citations":[{"why":"Supplies the representation flow algorithm, the central differentiable motion layer that replaces optical flow.","marker":"[11]"},{"why":"Defines the EgoRCNN model, the two-stream egocentric architecture into which the representation flow layer is integrated.","marker":"[12]"},{"why":"Provides class activation mapping, used in the RGB stream to generate the spatial attention map that sharpens feature responses.","marker":"[13]"},{"why":"Introduces ego-flow input for egocentric two-stream recognition, one of the baselines the proposed model is compared against.","marker":"[16]"},{"why":"Basis for the I3D baseline, a modern 3D convolutional action recognition architecture used as a comparison in the experiments.","marker":"[17]"},{"why":"Temporal Segment Network baseline, whose sparse sampling strategy is compared on the same datasets.","marker":"[18]"}],"fun_headline_variants":["Replace optical flow with representation flow for 1000x faster action recognition","Learnable representation flow replaces optical flow, 1000x faster inference","Cut runtime 1000x by swapping optical flow for representation flow","Representation flow stream matches optical flow accuracy at 1000x speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the representation flow algorithm, originally designed for third-person action recognition, transfers effectively to egocentric action recognition inside the base two-stream model, and that the hyperparameters chosen on HMDB (two flow layers, 20 iterations, ResNet-34) remain optimal on the egocentric datasets without re-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Replace optical flow with representation flow for 1000x faster action recognition","Learnable representation flow replaces optical flow, 1000x faster inference","Cut runtime 1000x by swapping optical flow for representation flow","Representation flow stream matches optical flow accuracy at 1000x speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3333,"prompt_tokens":990,"completion_tokens":2343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2274}},"tokens_in":606,"tokens_out":2343,"duration_ms":13476,"temperature":1.0,"reasoning_tokens":2274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:36:01.122579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A held-out egocentric dataset with fast camera motion would separate the two motion representations: if the representation-flow branch trails the optical-flow branch by more than a few absolute accuracy points there, the claim that learned flow generalizes as well as classical flow for egocentric action is falsified.","supporting_citations":[{"cited_title":"Representation flow for action recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the representation flow algorithm, the central differentiable motion layer that replaces optical flow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides class activation mapping, used in the RGB stream to generate the spatial attention map that sharpens feature responses."},{"cited_title":"First person action recognition using deep learned descriptors","cited_arxiv_id":null,"evidence_quote":"Introduces ego-flow input for egocentric two-stream recognition, one of the baselines the proposed model is compared against."},{"cited_title":"Quo vadis, action recognition? a new model and the kinetics dataset","cited_arxiv_id":null,"evidence_quote":"Basis for the I3D baseline, a modern 3D convolutional action recognition architecture used as a comparison in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Temporal Segment Network baseline, whose sparse sampling strategy is compared on the same datasets."}],"review_version":1}