A Retrospective Recount of Computer Architecture Research with a Data-Driven Study of Over Four Decades of ISCA Publications

Jinjun Xiong; Omer Anjum; Wen-Mei Hwu

arxiv: 1906.09380 · v1 · pith:HZL6QSXQnew · submitted 2019-06-22 · 💻 cs.AR · cs.DL· cs.IR

A Retrospective Recount of Computer Architecture Research with a Data-Driven Study of Over Four Decades of ISCA Publications

Omer Anjum , Wen-Mei Hwu , Jinjun Xiong This is my paper

Pith reviewed 2026-05-25 18:23 UTC · model grok-4.3

classification 💻 cs.AR cs.DLcs.IR

keywords ISCA publicationsdata-driven analysisNLP pipelinecomputer architecture trendsdocument understandingretrospective studyresearch patterns

0 comments

The pith

An NLP pipeline applied to all ISCA papers from 1973 to 2018 extracts computation patterns and research trends that the authors judge worth sharing despite acknowledged limits in document understanding.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper describes building an NLP-based pipeline originally for the DISCvR project to analyze unstructured technical documents and identify computation needs in modern systems. An early application was a request to review 50 years of MICRO papers, which led to a fuller study of every ISCA publication spanning 1973 through 2018. The authors note that true understanding of unstructured data remains difficult yet their analysis still produced enough concrete findings on historical trends to merit publication. They present the work as a starting point meant to encourage wider community use of similar data-driven approaches to track how computer architecture research has changed.

Core claim

The central claim is that a practical NLP pipeline for document understanding, when run on the complete set of ISCA papers from 1973 to 2018, yields identifiable computation patterns and research trends that remain interesting enough to report even while the underlying natural language processing technology is still limited.

What carries the argument

The NLP-based AI pipeline developed in the DISCvR project, used here to process and extract trends from the full historical corpus of ISCA publications.

If this is right

Computer architecture researchers gain a quantitative baseline for how topics such as parallelism, memory systems, and power have risen or fallen across four decades.
Conference organizers obtain a repeatable method for reviewing past programs and identifying underrepresented areas.
The same pipeline can be reapplied to later ISCA volumes or to related venues to track ongoing evolution.
Limitations identified in the current analysis point to specific improvements needed in handling technical terminology and citation context.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach could be extended to papers from other architecture or systems conferences to test whether the same trends appear outside ISCA.
If the pipeline improves, it might eventually support automated generation of research roadmaps by comparing extracted patterns against emerging hardware capabilities.
The work implicitly suggests that community-maintained labeled datasets of architecture papers would help future NLP efforts succeed where the current prototype falls short.

Load-bearing premise

The pipeline can pull reliable patterns and trends out of the unstructured text of technical papers even though the authors themselves note that current natural language understanding remains imperfect.

What would settle it

Expert reviewers or independent re-analysis of the same ISCA corpus finds that the reported trends do not align with documented historical shifts in the field or that the extracted patterns are too noisy to be actionable.

Figures

Figures reproduced from arXiv: 1906.09380 by Jinjun Xiong, Omer Anjum, Wen-Mei Hwu.

**Figure 1.** Figure 1: An Overview of the Pipeline Recently we conducted a study of some notable publication trends for ISCA from 1973, when it was inaugurated, to 2018. The main questions we were trying to answer was how the topics and thus the community interests evolved over these 45 years. Our data set includes all the abstracts of papers published in the conference. The source of our data set is Microsoft Academic Graph [1]… view at source ↗

**Figure 2.** Figure 2: Number of Papers Each Year [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Average Number of Authors in a Paper 3 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Figure 4: Percentage of industry vs. academia affiliation of first authors [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: A long-term history of selected top topics [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

**Figure 6.** Figure 6: Phrases and trends, 1973-1975. In the early 1970’s, numerous designs were proposed to improve efficiency of running programs written in those high-level languages, but only a handful were actually implemented [2]. Burroughs E-mode machines for Algol 60, Burroughs B2000 for COBOL, LISP machines and Intel 432 for Ada are some of those examples. A number of ideas based on microprogramming can be found in the… view at source ↗

**Figure 7.** Figure 7: Top Phrases and Trends, 1976-80 7 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗

**Figure 8.** Figure 8: Top topics and trends, 1981-85 of attention by the ISCA community was paid to the interconnection network between processing elements. The driving applications behind those systems were the military and scientific applications including image processing, astrophysics and weather prediction. Individual processors were not capable of providing the required computation speed at that time. In industry, mini-s… view at source ↗

**Figure 9.** Figure 9: The RISC designs matched well with the transistor budget of the microprocessor chips during this time. Industry companies began to produce chips based on new ISAs like SPARC by SUN Microsystems, MIPS by MIPS, Inc., Spectrum by Hewlett-Packard, and 960 by Intel. However, the simpler instruction sets also increased the pressure for increased memory bandwidth for instruction fetch. Pipelining allowed the C… view at source ↗

**Figure 9.** Figure 9: Top topics and trends, 1986-1990 11 [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗

**Figure 10.** Figure 10: Top topics and trends, 1991-95 13 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗

**Figure 11.** Figure 11: Top topics and trends, 1996-2000 14 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗

**Figure 12.** Figure 12: Top topics and trends, 2001-2005 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗

**Figure 13.** Figure 13: Top topics and trends, 2006-10 17 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗

**Figure 14.** Figure 14: Top Topics and Trends 2011-2015 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗

**Figure 15.** Figure 15: Top topics and trends, 2016-18 quickly developed cuDNN. Several other fields such as personalized medicine, genomics, physics, economics also realized that GPUs may help to consume lots of existing data for scientific breakthroughs. Some of the Top10 supercomputers also got equipped with GPUs. However, power consumption for large computing clusters was still a bottleneck. In 2011-15, we saw that power con… view at source ↗

read the original abstract

This study began with a research project, called DISCvR, conducted at the IBM-ILLINOIS Center for Cognitive Computing Systems Reseach. The goal of DISCvR was to build a practical NLP based AI pipeline for document understanding which will help us better understand the computation patterns and requirements of modern computing systems. While building such a prototype, an early use case came to us thanks to the 2017 IEEE/ACM International Symposium on Microarchitecture (MICRO-50) Program Co-chairs, Drs. Hillery Hunter and Jaime Moreno. They asked us if we can perform some data-driven analysis of the past 50 years of MICRO papers and show some interesting historical perspectives on MICRO's 50 years of publication. We learned two important lessons from that experience: (1) building an AI solution to truly understand unstructured data is hard in spite of the many claimed successes in natural language understanding; and (2) providing a data-driven perspective on computer architecture research is a very interesting and fun project. Recently we decided to conduct a more thorough study based on all past papers of International Symposium on Computer Architecture (ISCA) from 1973 to 2018, which resulted this article. We recognize that we have just scratched the surface of natural language understanding of unstructured data, and there are many more aspects that we can improve. But even with our current study, we felt there were enough interesting findings that may be worthwhile to share with the community. Hence we decided to write this article to summarize our findings so far based only on ISCA publications. Our hope is to generate further interests from the community in this topic, and we welcome collaboration from the community to deepen our understanding both of the computer architecture research and of the challenges of NLP-based AI solutions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a modest exploratory NLP scan of 45 years of ISCA papers that extends their prior MICRO work and stays honest about technical-document limits, but shows little validation or concrete results.

read the letter

The paper runs their DISCvR NLP pipeline over the full ISCA corpus from 1973 to 2018 and reports some historical trends in research topics and computation patterns. It builds directly on the MICRO-50 analysis they did earlier, so the core move is simply applying the same tool to a second major conference archive. They are clear that the goal is to spark community interest rather than deliver a finished, validated study. That framing is fair and keeps expectations in line with what an observational scan can actually deliver. The work is new in the narrow sense that no one has published this exact ISCA-wide extraction before. It also does one thing well: it states upfront that current NLP still struggles with unstructured technical papers and treats the output as preliminary observations only. That honesty is useful. The main soft spot is the absence of any visible checks on the pipeline itself. The abstract gives no error rates, no manual validation samples, and no comparison against human-coded trends, so it is hard to judge whether the extracted patterns are stable or mostly noise. Because the claim is explicitly modest—an initial effort to share findings and invite collaboration—this gap does not sink the paper, but it does limit how much weight the trends can carry. The paper is mainly for people already interested in the history of computer architecture or in applying document-understanding tools to scientific literature. A reader looking for rigorous trend quantification or reproducible methods will find it thin. It is worth sending to peer review so the community can see the actual outputs and comment on both the architecture history and the NLP reliability questions.

Referee Report

2 major / 0 minor

Summary. The manuscript describes an exploratory observational study applying an NLP pipeline (developed in the DISCvR project) to ISCA papers from 1973–2018. Motivated by an earlier MICRO-50 analysis, it aims to surface computation patterns and research trends, acknowledges inherent difficulties in technical-document understanding, and positions the work as a preliminary effort intended to stimulate community interest rather than a fully validated analysis.

Significance. If the reported trends prove reproducible and the pipeline’s error characteristics are characterized, the study could serve as a useful seed for data-driven meta-research in computer architecture. Its modest framing and explicit recognition of NLP limitations are appropriate for an initial effort.

major comments (2)

[Abstract] Abstract: the central claim that the analysis produced 'enough interesting findings' to share with the community is not supported by any concrete examples, quantitative trends, or error analysis of the NLP pipeline; without these the reader cannot evaluate whether the findings are reliable or novel.
[Abstract] The manuscript supplies no validation metrics, inter-annotator agreement, or manual spot-checks for the extraction of computation patterns; this absence is load-bearing because the weakest assumption identified is precisely the reliability of the NLP pipeline on unstructured technical text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback on our exploratory study. We agree that the abstract requires concrete support for its claims and that the reliability of the NLP pipeline needs better characterization. We will revise the manuscript accordingly while preserving its positioning as a preliminary effort.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that the analysis produced 'enough interesting findings' to share with the community is not supported by any concrete examples, quantitative trends, or error analysis of the NLP pipeline; without these the reader cannot evaluate whether the findings are reliable or novel.

Authors: We agree that the abstract should include concrete examples to substantiate the claim. In the revision we will add specific quantitative trends identified in the ISCA corpus (such as shifts in dominant computation patterns across decades) and reference the pipeline error characteristics already analyzed in the body of the paper. This will enable readers to assess reliability and novelty directly from the abstract. revision: yes
Referee: [Abstract] The manuscript supplies no validation metrics, inter-annotator agreement, or manual spot-checks for the extraction of computation patterns; this absence is load-bearing because the weakest assumption identified is precisely the reliability of the NLP pipeline on unstructured technical text.

Authors: We acknowledge the absence of explicit validation metrics in the current version. The work is framed as preliminary and we already note the inherent difficulties of NLP on technical documents. For the revision we will add manual spot-check results on a sample of extracted patterns together with basic agreement statistics. A full inter-annotator study remains outside the scope of this initial effort but will be listed as future work. revision: partial

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is an exploratory observational study applying NLP to extract trends from external ISCA publication data spanning 1973-2018. No equations, derivations, fitted parameters, or predictions appear in the provided text or abstract. The central claim is explicitly modest (interesting findings worth sharing despite NLP limitations), with no load-bearing steps that reduce to self-defined inputs, self-citations, or ansatzes. The work is self-contained against external benchmarks as a retrospective recount without internal reductions.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No details on model parameters, background assumptions, or new entities are provided in the abstract; the work relies on standard NLP techniques whose internals are not described.

pith-pipeline@v0.9.0 · 5871 in / 1043 out tokens · 26576 ms · 2026-05-25T18:23:42.299839+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MA) and Applications. In Proceedings of the 24th International Con- ference on World Wide Web (WWW ’15 Companion). ACM, New York, NY, USA, 243-246. DOI=http://dx.doi.org/10.1145/2740908.2742839

work page doi:10.1145/2740908.2742839 2015
[2]

Ditzel and David A

David R. Ditzel and David A. Patterson. 1980. Retrospective on high-level language computer architecture. In Proceedings of the 7th annual symposium on Computer Architecture (ISCA ’80). ACM, New York, NY, USA, 97-104. DOI: http://dx.doi.org/10.1145/800053.801914 23

work page doi:10.1145/800053.801914 1980

[1] [1]

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MA) and Applications. In Proceedings of the 24th International Con- ference on World Wide Web (WWW ’15 Companion). ACM, New York, NY, USA, 243-246. DOI=http://dx.doi.org/10.1145/2740908.2742839

work page doi:10.1145/2740908.2742839 2015

[2] [2]

Ditzel and David A

David R. Ditzel and David A. Patterson. 1980. Retrospective on high-level language computer architecture. In Proceedings of the 7th annual symposium on Computer Architecture (ISCA ’80). ACM, New York, NY, USA, 97-104. DOI: http://dx.doi.org/10.1145/800053.801914 23

work page doi:10.1145/800053.801914 1980