DIS. TBDJUN 2026

Distributions

DIS. TBDJUN 2026

The Bernoulli Distribution

The Distribution That Broke Deep Learning

Introduction

Bernoulli is ruthlessly simple. You are a 1 or you are a 0. There is no negotiation.

This makes it the most honest way to model a binary decision — a neuron either fires or it doesn't, a weight either exists or it doesn't. The L0 norm, the most truthful measure of a network's complexity, is just a sum of Bernoulli gates. Turn off the weights that don't matter. Count what's left.

The problem: backpropagation is the chain rule. The chain rule needs derivatives. And Bernoulli has none. The moment you put a Bernoulli gate inside a neural network, the gradient hits it and dies. The colab session implodes. The most honest regularizer is mathematically incompatible with the only training method that works at scale.

This is the distribution that broke deep learning. Understanding why requires going back to the coin flip.

The Historical Context

The Bernoulli distribution is named after the Swiss mathematician Jacob Bernoulli (1655–1705). His magnum opus, Ars Conjectandi (The Art of Conjecturing), published posthumously in 1713, laid the foundations for probability theory and combinatorics. Before Bernoulli, probability was largely constrained to specific games of chance; Bernoulli generalized these concepts to apply to civil, moral, and economic domains.

Crucially, within Ars Conjectandi, Bernoulli introduced the concept of sequences of independent trials with two possible outcomes—what we now call Bernoulli trials. He used this foundation to formally prove the first version of the Law of Large Numbers, demonstrating that as the number of trials increases, the empirical proportion of successes mathematically converges to the theoretical probability $p$ .

§1. The Basics

Consider an experiment that can result in only one of two states. Yes or no. 1 or 0.

For our example, we are going to work with the infamous coin—a standard coin with absolutely no special attributes. Let us take Heads (H) to be the success state, occurring with probability $p$ . Consequently, Tails (T) is the failure state, occurring with probability $1-p$ (often denoted as $q$ ).

P(X=1) = p, \quad P(X=0) = 1-p

It is that simple.

Heads (1): 0%Tails (0): 0%

Total Flips: 0

p0.50

Fig. 01 — A single Bernoulli trial with adjustable parameter p.

Observe the Law of Large Numbers. As the number of flips increases, watch the wild empirical average flatten toward the theoretical bias p.

The Law of Large Numbers

1.0

0.0

n=1000

Theoretical: 0.500Empirical: ---

P(1) = 0.50P(0) = 0.50

Bias Parameter (p)

Fig. 02 — Simulating the Law of Large Numbers through successive Bernoulli trials.

§2. The Expected Value

If you wanted to empirically calculate the probability that the next flip of our coin is Heads, you could throw it 10-15 times, write down each outcome, and divide the number of heads by the total number of trials. Simple enough!

But what if you wanted to know what to expect for the next flip? Intuitively, the expected value should simply be the probability of getting Heads. Why? Because we specifically defined Heads as $X=1$ and Tails as $X=0$ .

When you calculate an expected value, you are taking a weighted average. You multiply the value of Heads ( $1$ ) by its probability ( $p$ ), and add the value of Tails ( $0$ ) multiplied by its probability ( $1-p$ ). The zero obliterates the Tails term, leaving only the probability of Heads:

E[X] = 1 \cdot p + 0 \cdot (1-p) = p

$E[X] = p$ is almost disappointingly simple. The expected value of a coin flip is just the probability of getting Heads. Boring. But this sets up the contrast for what comes next.

§3. Variance

The interesting one is variance. To understand why it behaves the way it does, don't look at the algebra first. Think about it this way: variance measures how surprised you get on average.

When are you most surprised by a coin flip? When $p = 0.5$ . A fair coin. You genuinely have no idea what's coming. Maximum surprise, maximum variance. When are you least surprised? When $p = 0$ or $p = 1$ . A two-headed coin. You already know the answer before you flip. Zero surprise, zero variance.

So variance should be a function that is zero at the edges and maximum at 0.5. What simple function does that? $p(1-p)$ . When $p=0$ : $0 \times 1 = 0$ . When $p=1$ : $1 \times 0 = 0$ . When $p=0.5$ : $0.5 \times 0.5 = 0.25$ . Maximum.

\text{Var}(X) = p(1-p)

The multiplication here isn't just an algebraic trick—it is the direct product of "how likely is success" and "how likely is failure." When both are substantial, uncertainty is maximum. When either collapses to zero, certainty takes over.

The Compression of Uncertainty

0.25

p=0

p=1

p=0.5

Max Uncertainty

Variance Var(X)

P(1) = 0.50Var(X) = 0.2500P(0) = 0.50

Bias Parameter (p)

Fig. 03 — The state of uncertainty, peaking at p=0.5.

Probability Mass Function

1.00.50.0

0.50

X=0

0.50

X=1

P(1) = 0.50P(0) = 0.50

Bias Parameter (p)

Fig. 04 — The Probability Mass Function mapping discrete states.

§4. The PMF

Now, the compression. We need to unify these two distinct states—success and failure—into a single mathematical function: the Probability Mass Function (PMF). We want an equation that spits out $p$ when $x=1$ , and $1-p$ when $x=0$ .

In programming, this is just an if/else statement. But math requires a single, continuous-looking expression. How do we build a switch without logic gates? We use exponents.

Anything to the power of 0 is 1, and anything to the power of 1 is itself. If we use our binary outcome $x \in \{0,1\}$ as an exponent, it acts exactly like an on/off switch. $p^x$ turns on when $x=1$ . To get the inverse switch for failure, we use $1-x$ , which turns $(1-p)^{1-x}$ on when $x=0$ .

Multiply them together, and you get a single elegant equation that perfectly routes the inputs:

P(X=x) = p^x (1-p)^{1-x}

It is not an arbitrary definition. It is a mathematical trick to compress a binary state into a single line.

§5. Entropy

Before the flip, how much do you notknow? That's the question entropy answers.

A fair coin — $p=0.5$ — is the most uncertain object imaginable. You know nothing about the next outcome. Your best guess is still just a guess. Now bias it toward heads — say $p=0.9$ — and suddenly the flip is barely surprising. You'd bet heads every time and you'd be right nine times out of ten. The uncertainty collapsed.

Shannon [1948] showed that this intuition can be made precise. For any binary trial, the amount of uncertainty — the information entropy — is:

H(X) = -p \log_2(p) - (1-p) \log_2(1-p)

At $p=0.5$ , this returns exactly 1 bit — the purest atomic unit of uncertainty. Push $p$ toward either extreme and the entropy falls to 0. A coin that always lands heads carries no information at all; you already know what's coming.

Entropy runs much deeper than coin flips — it underpins cross-entropy loss, KL divergence, and the information-theoretic foundations of machine learning. We'll return to it properly in a later module.

1.00

bits of surprise

Fig. 06 — Entropy Field. Maximum unpredictability is reached at

p=0.5

(1 bit).

P(1) = 0.50H(X) = 1.00P(0) = 0.50

Bias Parameter (p)

Fig. 05 — The measurement of uncertainty, peaking at 1 bit when p=0.5.

H, H0.25

H, T0.25

T, H0.25

T, T0.25

Flip 2 (H/T) →

Flip 1 (H/T) →

Fig. 05 — Independence Area Map. The joint probability

P(A \cap B)

is exactly the area

P(A) \times P(B)

P(1) = 0.50P(0) = 0.50

Bias Parameter (p)

Fig. 06 — Two independent trials. The joint probability is simply their product.

§6. Independence

A single Bernoulli trial is the fundamental unit of probability, but its real power emerges when we start stacking them.

Imagine flipping our coin twice. If the flips are independent, it means the outcome of the first flip has absolutely zero influence on the second. The coin has no memory.

Mathematically, independence allows us to simply multiply the probabilities together. The probability of getting Heads and then Tails is precisely the probability of Heads multiplied by the probability of Tails: $p \times (1-p)$ .

This innocent property of multiplication is what allows us to scale a single coin flip into complex, massive systems. And that leads directly to the Binomial distribution.

The Missing GradientSo why did Bernoulli break deep learning? Because you cannot differentiate a coin flip. To train a network that uses Bernoulli gates—for things like sparse routing or hardware-efficient quantization—we are forced to cheat. We use techniques like the Gumbel-Softmax trick or the Straight-Through Estimator to pretend the function is continuous during the backward pass. The pure, discrete Bernoulli trial remains fundamentally hostile to the calculus of backpropagation.

Interactive Laboratory

Put the theory to work. We've written some Python simulating the concepts above. The code is running entirely in your browser via WebAssembly. Break it, change the probabilities, and see what happens.

Output

Output will appear here after running the code

Output

Output will appear here after running the code

Output

Output will appear here after running the code

Output

Output will appear here after running the code

What comes next?

From Atomic to Aggregate

ΣXᵢ ~ Binomial

We have understood the single biased coin. What happens when we flip n independent coins and count the total number of successes? This leads us naturally to the Binomial Distribution.

Enter the Binomial Lab