Home
Loading

aVenture is in Alpha: During this preview period, you should expect the research data to be limited and may not yet meet our exacting standards. We've made the decision to provide early access to our data to showcase the product as we build, but you should not yet rely upon it alone for your investment decisions.

aVenture is in Alpha: During this preview period, you should expect the research data to be limited and may not yet meet our exacting standards. We've made the decision to provide early access to our data to showcase the product as we build, but you should not yet rely upon it alone for your investment decisions.

Get in touch

  • Contact

  • Request a demo

  • Request data updates

  • Add a company

Research

  • Companies

  • Investors

  • People

aVenture

  • Sitemap

  • Feature requests

Member

Backed by

© aVenture Investment Company, 2026. All rights reserved.

San Francisco, CA, USA

Privacy Policy

aVenture Investment Company ("aVenture") is an independent research platform providing detailed analysis and data on startups, venture capital investments, and key industry individuals. It is not a registered investment adviser, broker-dealer, or investment advisor and does not provide investment advice or recommendations. The data provided by aVenture does not constitute recommendations or advice, whether by methodology, analysis, AI-generated content, or a statement written by a staff member of aVenture.

aVenture is not affiliated with any of the people, companies, organizations, government agencies, regulatory bodies, or investment funds we provide coverage for on this site unless explicitly stated otherwise. Users assume full responsibility for decisions made based on information obtained from this platform. Links to external websites do not imply endorsement or affiliation with aVenture. Any links that provide the ability to invest in a primary or secondary transaction in a company are for convenience only and do not constitute solicitations or offers to buy or sell an investment. Investors should exercise heightened precaution and due diligence when investing in private companies, especially those not independently audited.

While we strive to provide valuable insights with objectivity and professional diligence, we cannot guarantee the accuracy of the information provided on our platform. Before making any investment decisions, you should verify the accuracy of all pertinent details for your decision. To the fullest extent permitted by law, aVenture shall not be liable for any direct, indirect, incidental, consequential, or financial damages arising from use of this site, whether by consumers of its contents directly or by persons or organizations covered by our research, even if we are advised of the possibility. Our best-efforts processes and correction request forms do not create a warranty or duty of care.

Profiles on this platform may include content generated in part by large language models (LLMs, artificial intelligence) that aggregate publicly available sources (e.g., SEC EDGAR, public filings, press releases). Source attribution is provided where known; always verify statements and claims here against original sources before relying on any data. Content on our site may contain inaccuracies, omissions, or what are commonly called 'hallucinations' if generated in part or in full by AI / LLMs. The risk can also exist even when content is written by a human, as internal and third-party sources may also have inaccuracies for the same or different reasons. While we randomly audit a proportion of content, this is not exhaustive.

We recommend that an independent auditor be hired to verify the accuracy of the information before relying on it for any sensitive decisions. By accessing this platform, you agree not to rely solely on any information generated by AI, aggregated, or sourced or written otherwise on this site, for investment, financial, or other decisions. aVenture assumes no responsibility for inaccuracies, omissions, or hallucinations. You must independently verify all data from primary sources. Use of this platform constitutes your waiver of claims for reliance-based damages, including negligent misrepresentation. To report an error, request a correction, or dispute information about a company or individual, contact us via our request data updates form.

Loading
Loading
Home
News
Efficient Pretraining with Token Superposition

From Nous Research Blog

By Nous Research Team

May 10, 2026

Efficient Pretraining with Token Superposition

Efficient Pretraining with Token Superposition
Nous Research introduces Token Superposition Training (TST), a method that achieves 2-3x wall-clock speedup on LLM pretraining at matched FLOPs without changing model architecture, optimizer, tokenizer, or training data. During an initial superposition phase, the model processes bags of contiguous tokens rather than individual tokens, then resumes standard next-token training. Validated at 270M, 600M, 3B dense, and 10B-A1B MoE scales.

View original article on nousresearch.com

Most Recent

Long Context Pre-Training with Lighthouse Attention

Long Context Pre-Training with Lighthouse Attention

Nous Research introduces Lighthouse Attention, a selection-based hierarchical attention mechanism that runs ~17x faster than standard attention at 512K context and delivers 1.4-1.7x end-to-end pretraining speedup.

May 9, 2026

BofA Global Research Ranked No. 1 in the 2026 Extel Asia Research Survey

BofA Global Research Ranked No. 1 in the 2026 Extel Asia Research Survey

BofA Global Research ranked No. 1 overall in the 2026 Extel Asia Research Survey, with leading results across all 35 categories.

Apr 26, 2026

What VC Firms Actually Told Us About AI

What VC Firms Actually Told Us About AI

Everyone in venture has a take on AI. We wanted the data behind the takes.

Apr 22, 2026

Field Notes on Scaling MoE Expert Parallelism with DeepEP

Field Notes on Scaling MoE Expert Parallelism with DeepEP

Nous Research shares detailed field notes on scaling Mixture-of-Experts training using DeepEP, achieving 15,057 tokens/sec per GPU and 57% of theoretical NVLink bandwidth on B200 clusters.

Dec 8, 2024

Similar Posts

Why Cohere’s ex-AI research lead is betting against the scaling race

Why Cohere’s ex-AI research lead is betting against the scaling race

Cohere's former VP of AI research, Sara Hooker, is launching a new startup to build AI models that can adapt to their environment.

Oct 22, 2025

Fastino trains AI models on cheap gaming GPUs and just raised 17.5M led by Khosla

Fastino trains AI models on cheap gaming GPUs and just raised 17.5M led by Khosla

TechCrunch profiles Fastino 17.5M seed round led by Khosla Ventures and its task-specific AI model architecture trained on sub-100K gaming GPU clusters.

May 7, 2025

Stability AI releases ChatGPT-like language models

Stability AI releases ChatGPT-like language models

Stability AI, the startup behind the generative AI art tool Stable Diffusion, today open sourced a suite of text-generating AI models intended to go head to head with systems like OpenAI’s GPT-4. Called StableLM and available in “alpha” on GitHub and Hugging Face, a platform for hosting AI models and code, Stability AI says that […]

Apr 19, 2023

Long Context Pre-Training with Lighthouse Attention

Long Context Pre-Training with Lighthouse Attention

Nous Research introduces Lighthouse Attention, a selection-based hierarchical attention mechanism that runs ~17x faster than standard attention at 512K context and delivers 1.4-1.7x end-to-end pretraining speedup.

May 9, 2026

Most Recent

Long Context Pre-Training with Lighthouse Attention

Long Context Pre-Training with Lighthouse Attention

Nous Research introduces Lighthouse Attention, a selection-based hierarchical attention mechanism that runs ~17x faster than standard attention at 512K context and delivers 1.4-1.7x end-to-end pretraining speedup.

May 9, 2026

BofA Global Research Ranked No. 1 in the 2026 Extel Asia Research Survey

BofA Global Research Ranked No. 1 in the 2026 Extel Asia Research Survey

BofA Global Research ranked No. 1 overall in the 2026 Extel Asia Research Survey, with leading results across all 35 categories.

Apr 26, 2026

What VC Firms Actually Told Us About AI

What VC Firms Actually Told Us About AI

Everyone in venture has a take on AI. We wanted the data behind the takes.

Apr 22, 2026

Field Notes on Scaling MoE Expert Parallelism with DeepEP

Field Notes on Scaling MoE Expert Parallelism with DeepEP

Nous Research shares detailed field notes on scaling Mixture-of-Experts training using DeepEP, achieving 15,057 tokens/sec per GPU and 57% of theoretical NVLink bandwidth on B200 clusters.

Dec 8, 2024

Similar Posts

Why Cohere’s ex-AI research lead is betting against the scaling race

Why Cohere’s ex-AI research lead is betting against the scaling race

Cohere's former VP of AI research, Sara Hooker, is launching a new startup to build AI models that can adapt to their environment.

Oct 22, 2025

Fastino trains AI models on cheap gaming GPUs and just raised 17.5M led by Khosla

Fastino trains AI models on cheap gaming GPUs and just raised 17.5M led by Khosla

TechCrunch profiles Fastino 17.5M seed round led by Khosla Ventures and its task-specific AI model architecture trained on sub-100K gaming GPU clusters.

May 7, 2025

Stability AI releases ChatGPT-like language models

Stability AI releases ChatGPT-like language models

Stability AI, the startup behind the generative AI art tool Stable Diffusion, today open sourced a suite of text-generating AI models intended to go head to head with systems like OpenAI’s GPT-4. Called StableLM and available in “alpha” on GitHub and Hugging Face, a platform for hosting AI models and code, Stability AI says that […]

Apr 19, 2023

Long Context Pre-Training with Lighthouse Attention

Long Context Pre-Training with Lighthouse Attention

Nous Research introduces Lighthouse Attention, a selection-based hierarchical attention mechanism that runs ~17x faster than standard attention at 512K context and delivers 1.4-1.7x end-to-end pretraining speedup.

May 9, 2026