REVIEW 3 major objections 5 minor 35 references
FiFTy: Large-scale File Fragment Type Identification using Neural Networks
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A compact 1-D CNN with a trainable byte-embedding layer identifies file types from raw blocks, reaching 77.5% accuracy on 75 classes and running an order of magnitude faster than the previous tool.
desk verdict Useful empirical contribution, but the headline accuracy may be inflated by a non-file-disjoint split; worth citing for the dataset and architecture, and worth sending to peer review with a request for a file-level re-run. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the embedding layer: a learned lookup table that maps each of the 256 byte values to a dense real-valued vector before a stack of one-dimensional convolution and max-pooling blocks processes the sequence. This replaces sparse one-hot encoding and lets the network compress the input while convolutional filters learn characteristic local byte transitions. The final representation comes from average pooling with dropout and two fully connected layers, and per-scenario hyper-parameters are selected by the Tree-structured Parzen Estimator.
What would settle it
Recompute FFT-75 accuracy with a file-level split in which every block of a given source file is assigned wholly to training, validation, or testing, and compare the resulting FiFTy accuracy against the reported 77.5%; a substantial drop would show that the block-level shuffle inflated the headline generalization numbers.
Extended reading notes
Core claim
The central discovery is that a trainable embedding layer—one small real-valued vector per byte value, used in place of a 256-entry one-hot code—allows a compact one-dimensional convolutional network to learn the byte-transition patterns that distinguish 75 file types directly from raw blocks. On 4,096-byte blocks the model averages 77.5% accuracy across all classes, on 512-byte blocks it averages 65.6%, and in two-class photo-carving scenarios it reaches 99% to 99.6% JPEG detection accuracy. FiFTy also beats a neural network trained on byte co-occurrence matrices and one trained on global statistics, in both accuracy and runtime.
Load-bearing premise
The load-bearing premise is that randomly shuffling individual byte blocks into training and test subsets creates independent samples; blocks from the same original file can appear on both sides, so the network could memorize file-specific patterns and the 77.5% and 65.6% accuracies would overstate performance on files never seen during training.
Editorial extensions
If this is right
- File carvers and memory-forensics tools can drop explicit feature extraction and still classify fragments, removing the largest runtime bottleneck of older systems.
- At roughly 38 seconds per gigabyte on a server GPU, whole-disk triage of memory dumps becomes practical for high-volume investigations.
- FFT-75 gives the community a balanced, publicly available 75-type benchmark with both 512-byte and 4,096-byte blocks, making future tools directly comparable.
- The focused scenarios show that a two-class JPEG-versus-other detector can exceed 99% accuracy, which is directly relevant to photo carving on memory cards from cameras and IoT devices.
- The remaining confusions are concentrated in predictable groups—HEIC/MOV, high-entropy archives, and compound office formats—so downstream carving logic can be designed around them.
Reading between the lines
- The reported accuracies may be optimistic because the FFT-75 split shuffles individual blocks rather than whole files, so blocks from one source file can appear in both training and test sets; a stricter file-level split is the natural follow-up test.
- If the result transfers to genuinely unseen files, the embedding-plus-convolution recipe should also work for other raw-byte classification tasks, such as network protocol identification or malware-family detection.
- A natural extension is to combine FiFTy's per-block probabilities with context across neighbouring blocks, for example a conditional random field or sliding-window voting, which could further reduce errors in compound formats.
- Because the hyper-parameter analysis found that convolution kernel size and stride matter more than embedding size, smaller and faster variants could probably shrink the embedding dimension with only a small accuracy cost.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents FiFTy, a 1-D convolutional neural network for file fragment type identification that operates directly on raw byte blocks with a trainable embedding layer, avoiding hand-crafted features. The authors introduce FFT-75, a balanced public dataset of 75 file types, and evaluate FiFTy on six application scenarios at block sizes of 512 and 4,096 bytes. They report 77.5% average accuracy on the 75-class scenario with 4,096-byte blocks at roughly 38 sec/GB, outperforming Sceadan (69%, 9 min/GB) and two neural baselines (NN-CO, NN-GF) under the same split and hardware. Additional experiments on the external GovDocs corpus and a DFRWS carving challenge are used to probe generalization. The paper claims state-of-the-art accuracy and speed, and releases the code and dataset publicly.
Significance. If the reported results hold, FiFTy would be the best open-source file fragment type classifier to date, and FFT-75 would be the largest balanced public benchmark for the task. The paper has notable strengths: the methodology is described in detail, the comparison to baselines is performed on the same splits and hardware, the implementation and dataset are publicly available, and the external GovDocs and DFRWS evaluations provide independent evidence for the multimedia carving scenarios. The architecture is compact (around 450k parameters for the main 75-class model) and the speed advantage over feature-based baselines is substantial and plausibly robust. The main risk is the block-level train/test split, which can leak file identity and inflate the headline accuracy on unseen files.
major comments (3)
- [Section IV-A] The train/validation/test split is formed by shuffling individual byte blocks after sampling them from source files, rather than by partitioning at the file level. Since blocks from the same source file can therefore appear in both training and testing, adjacent blocks sharing container headers, compression state, embedded objects, or content allow the network to memorize file-specific cues. This is load-bearing because the headline 77.5% accuracy and the comparison against Sceadan and NN-CO in Table I are all computed on this split. The authors should either provide a file-disjoint split (all blocks of a file in one split) or quantify the cross-split file overlap and show that the reported accuracies are stable under a file-disjoint split.
- [Section IV-E and Table VI] The external generalization results weaken the claim that FiFTy 'consistently outperforms all baselines' on unseen files. On GovDocs, scenario #1 accuracy drops from 77.5% on FFT-75 to 55.9%, and on DFRWS (512-byte blocks) to 41.4%; the paper itself notes that external results are comparable only for scenarios #3-#6. The 21.6-point drop on the 75-class scenario is the pattern expected if part of the FFT-75 score comes from file-memorization rather than generalizable type discrimination. The authors should add a file-disjoint evaluation on FFT-75 or another 75-class corpus and explicitly report the gap; if the drop persists, the headline accuracy should be qualified as an upper bound for block-level memorization rather than a claim of unseen-file generalization.
- [Section IV-C] No error bars, confidence intervals, or repeated splits are reported for any of the accuracy numbers, despite the fact that the test set is a random sample and the effective sample size is reduced by file-level correlation. Given that the main claim rests on a single split and a small accuracy margin over NN-CO (77.5% vs 75.3%), the authors should report results over multiple random splits or provide bootstrapped confidence intervals so that the uncertainty from the sampling procedure is visible.
minor comments (5)
- [Section I] In the first paragraph, 'Koglomorov complexity' should be 'Kolmogorov complexity'.
- [Supplementary Materials] The phrase 'inlcuding source code' contains a typo; it should be 'including source code'.
- [Table I] The table footnotes state that training time was computed on a Tesla P40, while Section III-B states that all experiments were run on a Tesla V100; please clarify which GPU was used for training versus inference.
- [Table VII] The 'Total Wins' row breaks ties by runtime speed; this rule should be stated in the table caption for clarity.
- [Figure 5] The figure caption lists subfigures (a)-(d) for 512-byte blocks and (e)-(h) for 4096-byte blocks, but the text references the subfigures without the block size in some places; adding the block size to each subfigure label would improve readability.
Circularity Check
No circularity found: FiFTy's accuracy is a measured test-set statistic, and the approach is independently re-evaluated on GovDocs and DFRWS corpora.
full rationale
The paper does not derive any predicted quantity from a fitted parameter or from a self-citation chain. The central claim (77.5% average accuracy on 75 file types with 4,096-byte blocks) is an empirical measurement on a hold-out test set: 'The blocks are shuffled to evenly distribute the file-types within the training (80%), validation (10%) and hold-out testing (10%) subsets' (Section IV-A). The baselines (Sceadan, NN-GF, NN-CO) are evaluated on the same dataset and the same split, so the relative comparison is not constructed from FiFTy's own outputs. The paper additionally tests generalization on external corpora: 'We tested generalization capabilities of FiFTy on the GovDocs corpus and a file-system memory dump from a DFRWS1 carving challenge' (Section IV-E), with results reported in Table VI. These external evaluations are independent of the FFT-75 training data and provide a check that does not reduce to the training inputs. The only notable methodological weakness is that the random block shuffle can place blocks from the same source file into both training and testing, which may inflate absolute accuracy; however, this is a generalizability/validity concern, not circularity, because the reported accuracy is measured rather than derived from the training objective or from a fitted parameter renamed as a prediction. No load-bearing self-citation is used: the FFT-75 dataset citation [21] is a public, externally usable resource, and the cited prior works are standard baselines rather than forced premises. Accordingly, no circular step meets the evidentiary bar of exhibiting an equation that reduces to its own input, so the circularity score is 0.
Assumptions & free parameters
free parameters (7)
- Embedding size =
16, 32, 48, or 64 depending on scenario (Table II)
- Convolution kernel size =
11, 19, 27, or 35 depending on scenario (Table II)
- Convolution stride =
3, 11, 19, 27, or 35 depending on scenario (Table II)
- Number of convolutional blocks =
1, 2, or 3 depending on scenario (Table II)
- Max pooling size =
2, 4, 6, or 8 depending on scenario (Table II)
- Number of dense units =
16, 32, 64, 128, or 256 depending on scenario (Table II)
- Dropout probability =
0.1
assumptions (4)
- domain assumption Byte-level statistics contain enough signal to distinguish 75 file types.
- domain assumption Shuffled block-level splitting yields independent training and test examples.
- standard math Backpropagation and TPE optimize the model as intended.
- domain assumption FFT-75 is representative of real carving workloads.
Cite this review
Pith. "Pith review of FiFTy: Large-scale File Fragment Type Identification using Neural Networks." pith.science (2026). https://pith.science/paper/3FQJIP25
@misc{pith2026190806148,
author = {Pith},
title = {Pith review of: FiFTy: Large-scale File Fragment Type Identification using Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/3FQJIP25}},
note = {Machine review of arXiv:1908.06148}
}
read the original abstract
We present FiFTy, a modern file type identification tool for memory forensics and data carving. In contrast to previous approaches based on hand-crafted features, we design a compact neural network architecture, which uses a trainable embedding space, akin to successful natural language processing models. Our approach dispenses with explicit feature extraction which is a bottleneck in legacy systems. We evaluate the proposed method on a novel dataset with 75 file types - the most diverse and balanced dataset reported to date. FiFTy consistently outperforms all baselines in terms of speed, accuracy and individual misclassification rates. We achieved an average accuracy of 77.5% with processing speed of approx 38 sec/GB, which is better and more than an order of magnitude faster than the previous state-of-the-art tool - Sceadan (69% at 9 min/GB). Our tool and the corresponding dataset are available publicly online.
Figures
Figures from the paper (13 more)
Reference graph
Works this paper leans on
-
[1]
Sceadan: Using Concate- nated N-Gram Vectors for Improved File and Data Type Classification,
N. L Beebe, L. A Maddox, L Liu, and M Sun, “Sceadan: Using Concate- nated N-Gram Vectors for Improved File and Data Type Classification,” IEEE Tran. Inf. Forensics and Security , vol. 8, no. 9, pp. 1519–1530, 2013
work page 2013
-
[2]
Using NLP techniques for file fragment classification,
S Fitzgerald, G Mathews, C Morris, and O Zhulyn, “Using NLP techniques for file fragment classification,” Digital Investigation , vol. 9, pp. S44–S49, aug 2012
work page 2012
-
[3]
Statistical Disk Cluster Classification for File Carving,
C. J Veenman, “Statistical Disk Cluster Classification for File Carving,” in IEEE Int. Symposium on Information Assurance and Security , 2007
work page 2007
-
[4]
Sparse Coding for N-Gram Feature Extraction and Training for File Fragment Classification,
F Wang, T.-T Quach, J Wheeler, J. B Aimone, and C. D James, “Sparse Coding for N-Gram Feature Extraction and Training for File Fragment Classification,” IEEE Tran. Inf. Forensics and Security , vol. 13, no. 10, pp. 2553–2562, oct 2018
work page 2018
-
[5]
Cocost: a computational cost efficient classifier,
L Li, U Topkara, B Coskun, and N Memon, “Cocost: a computational cost efficient classifier,” in 2009 Ninth IEEE International Conference on Data Mining . IEEE, 2009, pp. 268–277
work page 2009
-
[6]
Jpgcarve: An advanced tool for automated recovery of fragmented jpeg files,
J De Bock and P De Smet, “Jpgcarve: An advanced tool for automated recovery of fragmented jpeg files,” IEEE Tran. Inf. Forensics and Security, vol. 11, no. 1, pp. 19–34, Jan 2016
work page 2016
-
[7]
File Type Identification of Data Fragments by Their Binary Structure,
M Karresand and N Shahmehri, “File Type Identification of Data Fragments by Their Binary Structure,” in 2006 IEEE Information Assurance Workshop
work page 2006
-
[8]
Predicting the types of file fragments,
W. C Calhoun and D Coles, “Predicting the types of file fragments,” Digital Investigation, vol. 5, pp. S14–S20, sep 2008
work page 2008
Show all 35 references
-
[9]
A File Fragment Classification Method Based on Grayscale Image,
T Xu, M Xu, Y Ren, J Xu, H Zhang, and N Zheng, “A File Fragment Classification Method Based on Grayscale Image,” Journal of Computers, vol. 9, no. 8, aug 2014
2014
-
[10]
JPEG Recovery Lab,
“JPEG Recovery Lab,” https://www.disktuna.com/photo-recovery- easeus-data-recovery-vs-jpeg-recovery-lab/, Accessed Jun 20, 2019
2019
-
[11]
Carving orphaned jpeg file fragments,
E Uzun and H. T Sencar, “Carving orphaned jpeg file fragments,” IEEE Tran. Inf. Forensics and Security , vol. 10, no. 8, pp. 1549–1563, 2015
2015
-
[12]
Every shred helps: Assembling evidence from orphaned jpeg fragments,
E Durmus, P Korus, and N Memon, “Every shred helps: Assembling evidence from orphaned jpeg fragments,” IEEE Tran. Inf. Forensics and Security, vol. 14, no. 9, pp. 2372–2386, 2019
2019
-
[13]
Efficient estimation of word representations in vector space,
T Mikolov, K Chen, G Corrado, and J Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781 , 2013
2013 arXiv
-
[14]
File fragment classification using neural networks with lossless representations,
L Hiester, “File fragment classification using neural networks with lossless representations,” 2018, Bachelor Thesis, East Tennessee State University
2018
-
[15]
File Fragment Classification Using Grayscale Image Conversion and Deep Learning in Digital Forensics,
Q Chen, Q Liao, Z. L Jiang, J Fang, S Yiu, G Xi, R Li, Z Yi, X Wang, L. C Hui, D Liu, and E Zhang, “File Fragment Classification Using Grayscale Image Conversion and Deep Learning in Digital Forensics,” in IEEE Security and Privacy Workshops , 2018
2018
-
[16]
Overview of the high efficiency image file format,
M Hannuksela, E Aksu, J Lainema, and V . M Vadakital, “Overview of the high efficiency image file format,” in JCTVC-V0072. 2015
2015
-
[17]
Algorithms for hyper- parameter optimization,
J. S Bergstra, R Bardenet, Y Bengio, and B K ´egl, “Algorithms for hyper- parameter optimization,” in Advances in neural information processing systems, 2011, pp. 2546–2554
2011
-
[18]
The jpeg still picture compression standard,
G. K Wallace, “The jpeg still picture compression standard,” IEEE Tran. consumer electronics, vol. 38, no. 1, pp. xviii–xxxiv, 1992
1992
-
[19]
A Fragment Classification Method Depending on Data Type,
N Zheng, J Wang, T Wu, and M Xu, “A Fragment Classification Method Depending on Data Type,” in IEEE Int. Conf. on Computer and Information Technology Ubiquitous Computing and Communications Dependable Autonomic and Secure Computing Pervasive Intelligence and Computing, oct 2015
2015
-
[20]
Data Type Classification: Hierarchical Class-to-Type Modeling,
N Beebe, L Liu, and M Sun, “Data Type Classification: Hierarchical Class-to-Type Modeling,” in IFIP Advances in Information and Com- munication Technology, pp. 325–343. Springer Int. Publishing, 2016
2016
-
[21]
File fragment type (fft) - 75 dataset,
G Mittal, P Korus, and N Memon, “File fragment type (fft) - 75 dataset,” http://dx.doi.org/10.21227/kfxw-8084, 2019
2019 doi
-
[22]
GovDocs Dataset,
“GovDocs Dataset,” https://digitalcorpora.org/corpora/files, Accessed Jun 20, 2019
2019
-
[23]
Intel Movidius Myriad X VPU,
“Intel Movidius Myriad X VPU,” https://www.movidius.com/myriadx, Accessed Jun 20, 2019
2019
-
[24]
F Chollet et al., “Keras,” https://keras.io, 2015
2015
-
[25]
Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,
J Bergstra, D Yamins, and D. D Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” 2013
2013
-
[26]
Sequential model-based optimization for general algorithm configuration,
F Hutter, H. H Hoos, and K Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” in Proc. Int. Conf. on Learning and Intelligent Optimization , Berlin, Heidelberg, 2011, LION’05, pp. 507–523, Springer-Verlag
2011
-
[27]
Bz2 package,
“Bz2 package,” https://docs.python.org/3/library/bz2.html, Accessed Jun 20, 2019
2019
-
[28]
Zlib package,
“Zlib package,” https://docs.python.org/3/library/zlib.html, Accessed Jun 20, 2019
2019
-
[29]
Sceadan - open source implementation,
“Sceadan - open source implementation,” https://github.com/ UTSA-cyber/sceadan
-
[30]
Everything you need to know about the JPEG-killing HEIF for- mat Apple is adopting,
“Everything you need to know about the JPEG-killing HEIF for- mat Apple is adopting,” https://www.cultofmac.com/487808/heif-vs- jpeg-image-files/, Accessed Jun 20, 2019
2019
-
[31]
Overview of the high efficiency video coding (hevc) standard,
G. J Sullivan, J.-R Ohm, W.-J Han, and T Wiegand, “Overview of the high efficiency video coding (hevc) standard,” IEEE Tran. Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649–1668, 2012
2012
-
[32]
Progressive neural architecture search,
C Liu, B Zoph, M Neumann, J Shlens, W Hua, L.-J Li, L Fei-Fei, A Yuille, J Huang, and K Murphy, “Progressive neural architecture search,” in Proc. European Conf. Computer Vision , 2018, pp. 19–34
2018
-
[33]
Exploring randomly wired neural networks for image recognition,
S Xie, A Kirillov, R Girshick, and K He, “Exploring randomly wired neural networks for image recognition,” arXiv preprint arXiv:1904.01569, 2019
1904 arXiv
-
[34]
Benchmarking state-of-the-art deep learning software tools,
S Shi, Q Wang, P Xu, and X Chu, “Benchmarking state-of-the-art deep learning software tools,” in Int. Conf. Cloud Computing and Big Data (CCBD). IEEE, 2016, pp. 99–104
2016
-
[35]
Conditional random fields: Probabilistic models for segmenting and labeling sequence data,
J Lafferty, A McCallum, and F. C Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” 2001. IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY 13 TABLE VII FILE -TYPE BREAKDOWN OF CLASSIFICATION ACCURACY FOR ALL MODELS ; MIS...
2007 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.