OpenAI just released GPT-OSS, its first open weights model since GPT-2 back in 2019. For developers, researchers, and hobbyists, this is a big deal: one of the leading AI labs finally put something tangible back into the open-source ecosystem.

But GPT-OSS didn’t arrive into an empty landscape. In just the last year, two other open source heavyweights—Alibaba’s Qwen 3 and DeepSeek V3—have made serious waves. Each brings different architectural tricks, context strategies, and post-training approaches to the table.

So how do these models compare? Let’s dig in.

GPT-OSS: OpenAI’s Long-Context MoE

After years of closed releases, GPT-OSS feels like a statement: OpenAI hasn’t forgotten its open source roots. The model comes in two sizes:

  • 120B parameters (with about 5B active per token)
  • 20B parameters (with about 3.6B active per token)

Both use a mixture of experts (MoE) design, meaning only a fraction of parameters fire at once. This makes inference more efficient while keeping the benefits of a large network in reserve.

Architecturally, GPT-OSS packs all the modern upgrades we’ve come to expect:

  • Decoder-only transformer backbone
  • Grouped Query Attention (GQA) to cut memory usage
  • SwigLU activations in feed-forward networks for smoother learning
  • Rotary positional embeddings (RoPE) for flexible sequence handling
  • RMSNorm with pre-normalization for stability

The standout is its 131,000 token context window, trained directly with YaRN scaling during pretraining instead of patched on later. That makes GPT-OSS “born long context,” rather than fine-tuned into it.

It ships with the O200K Harmony tokenizer, a 200K vocabulary byte-pair encoding system optimized for text, code, and math. Training details remain vague—OpenAI admits only that it used trillions of tokens with heavy focus on STEM and coding—but the model is released quantized by default, making it lightweight enough for consumer-grade GPUs.

Qwen 3: Alibaba’s Multistage Reasoning Lab

Alibaba Cloud’s Qwen 3 family landed earlier this year with benchmark scores that rival the best open models on the market. Unlike GPT-OSS, Qwen offers both dense and MoE variants:

  • Dense models from 6B to 32B parameters
  • Sparse MoE models at 30B and 235B parameters, with 8 experts active per token

Architecturally, Qwen 3 builds on its predecessor, Qwen 2.5, but with refinements:

  • Same core stack—GQA, SwigLU, RoPE, RMSNorm
  • Replaces static QKV bias with QK-Norm, dynamically rescaling attention vectors for stability
  • Trained on 36 trillion tokens—double the previous version—including multilingual text, STEM data, reasoning tasks, and synthetic data generated by earlier Qwen models

Training was staged in three phases:

  1. General stage: 30T tokens, 119 languages, 4K sequence length
  2. Reasoning stage: 5T higher-quality math, logic, and coding tokens
  3. Long context stage: extended to 32K tokens, then scaled to 128K at inference with YaRN and dual-chunk attention

Qwen’s real innovation is in post-training. It uses a four-step process:

  • Chain-of-thought cold start: challenging reasoning problems with verified answers
  • Reasoning RL with GRPO, using only ~4K query-verifier pairs
  • Thinking mode fusion: integrates reasoning-heavy and lightweight inference into one model, with a toggle
  • General RL: instruction following, alignment, and tool use

The result is a model family that balances reasoning ability, multilingual coverage, and efficiency.

DeepSeek V3: Scale and Efficiency on Steroids

Then there’s DeepSeek V3, released at the end of last year and still one of the largest open source LLMs ever. At 671B parameters, it’s enormous. But like GPT-OSS and Qwen, it’s MoE—only about 37B parameters are active per token.

What makes V3 stand out is its focus on efficiency:

  • Trained natively in 8-bit precision, cutting costs without killing performance
  • Introduces MLA (Multi-head Latent Attention), which compresses keys and values into a latent space before caching, then decompresses during inference—saving memory compared to GQA
  • Extended context gradually: 32K tokens during training, then staged expansions up to 128K tokens

The more recent V3.1 update layered in:

  • Two-phase long context training for more reliable 128K sequences
  • Hybrid thinking mode, allowing dynamic switching between heavy reasoning and lightweight inference
  • Enhanced tool use and agent capabilities

In short, DeepSeek’s bet is clear: go massive, then optimize aggressively to make huge models usable.

Comparing the Big Three

Here’s how GPT-OSS, Qwen, and DeepSeek stack up:

Model Sizes & Parameters

  • GPT-OSS: 20B and 120B MoE (3.6–5B active)
  • Qwen 3: Dense (6B–32B) and MoE (30B, 235B; 8 active per token)
  • DeepSeek V3: 671B MoE (37B active)

Context Handling

  • GPT-OSS: 131K tokens, trained natively with YaRN
  • Qwen 3: 32K trained, 128K scaled at inference
  • DeepSeek V3: staged training to 128K

Attention Mechanism

  • GPT-OSS: GQA
  • Qwen 3: GQA with QK-Norm
  • DeepSeek V3: MLA (compressed KV cache)

Post-Training & Alignment

  • GPT-OSS: alignment layers for safety and control
  • Qwen 3: chain-of-thought bootstrapping, reasoning RL, thinking toggle, general RL
  • DeepSeek V3: hybrid thinking mode, stronger tool use, staged reasoning RL

Takeaways

What’s fascinating isn’t just the size of these models, but the different roads they take to achieve competitive results.

  • GPT-OSS is long-context by design, efficient enough to run on consumer GPUs, and tuned for safety.
  • Qwen 3 pushes reasoning with innovative post-training and mixes dense and MoE variants.
  • DeepSeek V3 is a giant, betting on MLA and 8-bit training to make scale affordable and long context feasible.

Despite their differences, they converge on similar benchmarks. That’s the nature of today’s LLM race: less about first principles theory, more about pragmatic engineering choices and empirical discoveries.

The big story here? Open weights are back in a big way. And for developers, that means choice: different architectures, different tradeoffs, different tools to build with.

Stay on track with more model deep dives at LaunchPerk Magazine—where we break down the architecture so you can focus on building.