Application of Mathematics in Real-World Usage · Guide 43 · BTT Mathematics Hub
Information theory asks a precise question: how much uncertainty is removed when a message is observed, and how efficiently can messages be represented or transmitted when symbols have unequal probabilities and channels can make errors?
This guide uses ideal discrete sources and channels for Mathematics teaching. It is not a communications-system, cryptographic or storage-system implementation guide. Real systems include finite block lengths, latency, synchronization, hardware constraints, burst errors, protocol overhead and security requirements.
Self-information grows when an event is less probable
For an event with probability p, self-information in bits is I=−log₂p.
If p=1/8, I=−log₂(1/8)=3 bits. A rarer event carries more surprise under this measure.
Entropy is expected self-information
For a discrete source with probabilities pᵢ, entropy is H(X)=−Σpᵢlog₂pᵢ.
A fair binary source has probabilities0.5 and0.5, giving H=1 bit per symbol.
A biased binary source has entropy below one bit
For p=0.75 and1−p=0.25, binary entropy is H₂(p)=−0.75log₂0.75−0.25log₂0.25≈0.8113 bits.
The source is more predictable than a fair coin, so its uncertainty per symbol is lower.
Maximum binary entropy occurs at equal probabilities
For a Bernoulli source, H₂(p) reaches its maximum1 bit at p=0.5 and falls toward0 as p approaches0 or1.
This is a statement about uncertainty, not about whether an outcome is valuable or important.
Average code length is probability-weighted
If symbols have probabilities0.5,0.25,0.125,0.125 and codeword lengths1,2,3,3, average length is0.5×1+0.25×2+0.125×3+0.125×3=1.75 bits/symbol.
A simple average of lengths would be2.25 and would ignore how often each codeword occurs.
Kraft’s inequality checks whether prefix-code lengths are possible
For binary prefix codes, lengths lᵢ must satisfy Σ2^(−lᵢ)≤1.
For lengths1,2,3,3, the sum is1/2+1/4+1/8+1/8=1, so the lengths satisfy the equality case.
Huffman coding assigns shorter codewords to more probable symbols
MIT’s information-theory materials treat Huffman coding as a central optimal-prefix-code construction for known symbol probabilities.
For probabilities0.5,0.25,0.125,0.125, one valid optimal length pattern is1,2,3,3, giving the1.75-bit average above.
Compression ratio needs a declared baseline
If a fixed representation uses8 bits per symbol and a compressed code averages1.75 bits, size ratio is8/1.75≈4.571.
The fractional size reduction is1−1.75/8=0.78125, or78.125%, before metadata and framing overhead.
Joint entropy measures uncertainty in a pair
Two independent fair bits have four equally likely pairs, each probability1/4, so joint entropy is2 bits.
If the second bit were an exact copy of the first, the pair would have only two possible equiprobable outcomes and joint entropy would be1 bit.
Conditional entropy measures uncertainty left after observing another variable
If Y is an exact copy of fair bit X, then H(Y|X)=0 because X completely determines Y.
If X and Y are independent fair bits, H(Y|X)=H(Y)=1 bit.
Mutual information is the uncertainty removed by observation
I(X;Y)=H(X)−H(X|Y)=H(Y)−H(Y|X).
For identical fair bits, mutual information is1 bit. For independent fair bits, it is0.
A binary symmetric channel flips bits with probability p
In the ideal binary symmetric channel, each transmitted bit flips independently with probability p.
For p=0.1 and equiprobable inputs, channel capacity is C=1−H₂(0.1)≈1−0.4690=0.5310 bits per channel use.
Capacity is not the same as raw symbol rate
If a channel is used1,000 times per second with theoretical capacity0.531 bits/use under its model, the asymptotic information-rate ceiling is about531 bits/s.
Actual finite-length codes operate with overhead and nonzero error constraints, so capacity is a theorem-level limit rather than a promise for every code.
Repetition coding trades rate for error reduction
Repeat one bit three times and decode by majority vote. If independent flip probability is p=0.1, decoding fails when two or three copies flip.
Error probability=3(0.1²)(0.9)+0.1³=0.028. The raw code rate is1/3 information bit per transmitted bit.
Block-code rate is information symbols divided by transmitted symbols
A(7,4) binary block code carries4 information bits in7 transmitted bits, so rate R=4/7≈0.5714.
The remaining three transmitted bits provide redundancy according to the particular code design.
Parity detects some errors but does not identify all of them
One even-parity bit appended to four data bits can detect any odd number of bit flips in the five-bit word, but an even number of flips may leave parity unchanged.
Error detection and error correction are therefore different mathematical capabilities.
A complete information-theory calculation states the alphabet and probability model
State symbol probabilities, logarithm base, code constraints, channel transition probabilities, independence assumptions and whether the quantity is entropy, average length, mutual information, code rate or capacity.
The return path is source uncertainty → representation/coding → channel noise → decoded message → residual error and rate.
Practice: twenty information-theory Mathematics questions
- p=1/8: find self-information in bits.
- Find entropy of a fair binary source.
- Find H₂(.75).
- What p maximises binary entropy?
- For probabilities.5,.25,.125,.125 and lengths1,2,3,3, find average length.
- Evaluate Kraft sum for lengths1,2,3,3.
- Fixed8-bit baseline compressed to1.75 bits average: find compression ratio8/1.75.
- Find fractional size reduction.
- Two independent fair bits: find joint entropy.
- Two identical fair bits: find joint entropy.
- If Y=X exactly, find H(Y|X).
- If X,Y independent fair bits, find H(Y|X).
- Find mutual information for identical fair bits.
- Find mutual information for independent fair bits.
- For BSC p=.1, estimate H₂(p).
- Find capacity1−H₂(.1).
- At1000 channel uses/s, convert capacity.531 bits/use to bits/s.
- For triple repetition with p=.1, find majority-decoding error probability.
- Find rate of a(7,4) code.
- Why can one parity bit miss an even number of bit flips?
Worked answers
- 3 bits.
- 1 bit.
- About0.8113 bits.
- p=0.5.
- 1.75 bits/symbol.
- 1.
- About4.571:1.
- 78.125%.
- 2 bits.
- 1 bit.
- 0.
- 1 bit.
- 1 bit.
- 0.
- About0.4690 bits.
- About0.5310 bits/use.
- About531 bits/s.
- 0.028.
- 4/7≈0.5714.
- Because an even number of flips can restore the original even/odd parity state.
Sources and connected applications
For entropy, source coding, Huffman coding, channel coding and capacity, see MIT OpenCourseWare: Information Theory lecture notes and MIT Digital Communication Systems lecture slides. All symbol distributions and channel examples here are original teaching constructions.
Continue with Aerodynamics, Lift, Drag, Reynolds Number and Scaling; Machine Learning, Regression, Classification, Loss Functions and Gradient Descent; and Game Theory, Payoffs, Nash Equilibrium, Mixed Strategies and Auctions. Return to the BTT Mathematics Hub.