Hacker News

Breaking the 1.58-bit Barrier for Ternary LLMs

• 15 Comments
• So they get down from 1.58 to 1.48 bits per weight by exploiting the fact that actual weights in practice are 0 51% of the time. Neat.
  If ternary llms work out and are baked into hardware as custom silicon I bet they'll be shockingly efficient.

  By infogulch 2 hours ago

• By “work out” you mean no accuracy degradation? That’s a big ask - currently we can barely quantize to dynamic fp4 with small block size - still not completely lossless on all benchmarks.

  By kadushka 2 hours ago

• QAT, which bitnet training is a form of, helps a ton in preserving accuracy at such low bits per parameter. There are also better quantization approaches that try to preserve the most sensitive weights† but are computationally expensive and so not typically done. Another complementary option is, if the model is fast enough, we should be able to push up correctness by self-consistency voting at close to T=1. Smart/fast Zero-shot classifiers like the recent Jev could help with aggregation across answers too, extending applicability.
  †Every paper I've read estimates the average information content of transformer LLMs at about 3-4 bits per parameter. Curiously, biological synapses are also estimated to be about 4-5 bits per synapse, possibly a bit lower.

  By Vetch 39 minutes ago

• You are conflating post training quantization and low bit training.

  By danielmarkbruce 17 minutes ago

• Well, you could train directly at this bitrate.

  By montroser an hour ago

• Pushing past log2(3) for real. This could drastically shrink LLMs for embedded systems, making them truly portable.

  By Marchant_hq 16 minutes ago

• Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.

  By om8 2 hours ago

• PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no-op so it is fast on CPU.
  If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.

  By janalsncm 2 hours ago

• which is important though since sending it across the wire over and over and over is actually the main bottleneck.

  By mitxela 2 hours ago

• If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS

  By om8 2 hours ago

• sounds like a perfect fit for ASIC-optimized models (where matrix ops could be supported directly in BITCOS format, potentially) & achieving record power efficiency for on-device inference.
  And it looks like per [0], a model needs only ~30% more weights to be at comparable quality, if quantization-aware training is done...
  0. https://arxiv.org/pdf/2402.17764 - The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

  By yalok an hour ago

• Only a presence bitmap? If we're contemplating packing schemes I'm tempted to write a paper that uses arithmetic coding to squeeze out a few more centi-bits.

  By wgd an hour ago

• Very interesting, I was just exploring this to hopefully fit one of the latest quantized models in 16GB of VRAM.

  By plqbfbv 2 hours ago

• This is the only time "1.58 bit" phrase makes more sense than "1 trit"
  Who knew that if you actually look at information entropy you can pack stuff better!

  By NooneAtAll3 2 hours ago

• Woah. Good science.

  By Kevcmk 3 hours ago