[2305.04241] Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens

2024.12.12 21:05

Skip to main content We gratefully acknowledge support from the Simons Foundation, member institutions , and all contributors. Donate cs arXiv:2305.04241

quick links

Login Help Pages About

Computer Science > Computation and Language

arXiv:2305.04241 (cs) [Submitted on 7 May 2023 ( v1 ), last revised 27 May 2023 (this version, v2)]

Title: Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens

Authors: Zhanpeng Zeng , Cole Hawkins , Mingyi Hong , Aston Zhang , Nikolaos Pappas , Vikas Singh , Shuai Zheng View a PDF of the paper titled Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens, by Zhanpeng Zeng and 6 other authors View PDF Abstract: Transformers are central in modern natural language processing and computer vision applications. Despite recent works devoted to reducing the quadratic cost of such models (as a function of the sequence length), dealing with ultra long sequences (e.g., with more than 16K tokens) remains challenging. Applications such as answering questions based on a book or summarizing a scientific article are inefficient or infeasible. Here, we propose to significantly improve the efficiency of Transformers for ultra long sequences, by compressing the sequence into a much smaller representation at each layer. Specifically, by exploiting the fact that in many tasks, only a small subset of special tokens (we call VIP-tokens) are most relevant to the final prediction, we propose a VIP-token centric compression (VCC) scheme which selectively compresses the sequence based on their impact on approximating the representation of the VIP-tokens. Compared with competitive baselines, our algorithm is not only efficient (achieving more than $3\times$ efficiency gain compared to baselines on 4K and 16K lengths), but also offers competitive/better performance on a large number of tasks. Further, we show that our algorithm scales to 128K tokens (or more) while consistently offering accuracy improvement. Comments: 10 pages main text, 12 pages appendix, preprint Subjects: Computation and Language (cs.CL) ; Machine Learning (cs.LG) Cite as: arXiv:2305.04241 [cs.CL] (or arXiv:2305.04241v2 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2305.04241 Focus to learn more arXiv-issued DOI via DataCite

Submission history

From: Zhanpeng Zeng [ view email ]
[v1] Sun, 7 May 2023 10:32:18 UTC (320 KB)
[v2] Sat, 27 May 2023 04:17:13 UTC (532 KB)
Full-text links:

Access Paper:

View a PDF of the paper titled Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens, by Zhanpeng Zeng and 6 other authors View PDF TeX Source Other Formats view license Current browse context: cs.CL prev | next
new | recent | 2023-05 Change to browse by: cs
cs.LG

References & Citations

NASA ADS Google Scholar Semantic Scholar a export BibTeX citation Loading

BibTeX formatted citation

× Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) Links to Code Toggle Papers with Code ( What is Papers with Code? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos

Demos

Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers

Recommenders and Search Tools

Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) About Help Contact Subscribe Copyright Privacy Policy Web Accessibility Assistance

arXiv Operational Status
Get status notifications via email or slack