Neural Networks: The Intuition
"Neural network" sounds like a different universe from the regression you already know. It isn't. The building block is a model you met in Stats 3 (logistic regression), and the whole trick is to stack a handful of them so that straight decision boundaries can bend into any shape you need. This lesson builds the intuition and stops there: no backpropagation calculus, just the ideas that make the rest make sense.
A neuron is just logistic regression
An artificial neuron takes some inputs, multiplies each by a weight, adds them up with a bias, and pushes the total through an activation function. With a sigmoid activation, which squashes any total into the range 0 to 1, that is exactly logistic regression: output = σ(w₁x₁ + w₂x₂ + b). The weights are the slopes, the bias is the intercept, and the neuron draws a single straight boundary through the feature space — one line, nothing fancier.
One neuron, one line. So how do networks draw curves? By combining several lines.
Layers compose simple boundaries into complex ones
Put a hidden layer of a few neurons between the inputs and the output. Each hidden neuron draws its own straight boundary; the output neuron then takes a weighted vote of those hidden signals. A vote across several lines can carve out regions no single line could — a corner, a stripe, an island. The classic proof is XOR: label a point 1 when exactly one of its two inputs is on. No straight line separates the two classes, so plain logistic regression is helpless. Give the network three hidden neurons and it stitches two lines together into the diagonal "pockets" XOR needs. Depth (more layers) just repeats the idea, letting later neurons reuse the features earlier ones discovered.
🧠 Train a 2-3-1 network on XOR
Two inputs → three hidden neurons → one output. Edges are weights: blue positive, orange negative, thicker = stronger. Press Train and watch the boundary bend from a useless line into the XOR pockets. Crank the learning rate too high and it overshoots; New network reseeds the weights (some starts get stuck in a poor local minimum — reseed to escape).
Training: rolling downhill
Where do the weights come from? They start random and are learned. Define a loss that measures how wrong the network's outputs are, and picture it as a landscape over all the weights: high where the network is bad, low where it's good. Gradient descent reads the local slope and takes a small step downhill, over and over, until the loss flattens out. The learning rate is the step size: too small and training crawls; too large and it leaps clean over the valley and overshoots, the loss bouncing or blowing up (drag the slider to 8+ above to see it thrash). The algorithm that computes those slopes efficiently through every layer is backpropagation — we're naming it, not deriving it. Because the landscape has dips and dead-ends, a run can settle in a local minimum and get XOR only half-right; reseeding the weights drops you somewhere new.
The words you meet in real code
Three pieces of vocabulary stand between this lesson and any tutorial you open next, and the widget above is already using two of them.
An activation is the function applied to the weighted sum, and the sigmoid is only the historical default. Most networks today use ReLU (rectified linear unit), which is about as simple as a function gets: positive values pass through unchanged, everything negative becomes zero. It has no upper bound, so "squashing" describes the sigmoid rather than activations in general. The hidden layer in the widget above uses a third option, tanh, which is a rescaled sigmoid running from −1 to 1.
ReLU took over on a numerical detail. The sigmoid's slope is steepest at its center, where it reaches exactly 0.25, and flattens toward zero at both ends. Gradient descent multiplies those slopes together layer by layer, so a chain of ten sigmoid layers shrinks the gradient by a factor of at most 0.2510, roughly one in a million, and the early layers stop learning at any useful rate. That is the vanishing gradient problem. ReLU's slope on the positive side is exactly 1, and multiplying by 1 changes nothing.
An epoch is one full pass through the training set, and the counter in the widget is measuring precisely that: 56 training points, one weight update per pass, which makes it full-batch gradient descent. Real training rarely works this way, because a full pass over a million images is far too slow to repeat thousands of times. The data is cut instead into mini-batches of perhaps 32 or 256 examples, each batch buying its own update, so a single epoch delivers thousands of small noisy steps rather than one exact one. That is stochastic gradient descent, and the noise turns out to be useful: it can jolt the search out of the shallow local minima you can watch trap the network above.
What this course does not cover
Everything above scales, but the machinery gets heavier. We're skipping the backprop derivation, and the specialized architectures (convolutional nets for images, transformers behind today's language models) that stack these ideas thousands of layers deep. The intuition survives the jump: still weighted sums, still squashes, still gradient descent rolling downhill. For a thesis on tabular research data you rarely need any of it — regression, random forests, and honest validation usually match a neural network and stay interpretable. Reach for deep learning when the data are images, audio, or text and the samples number in the tens of thousands.
Why it matters: Neural networks aren't magic. They're logistic regressions stacked so their straight boundaries combine into curved ones, with the weights fit by rolling downhill on an error surface. That single picture demystifies the whole field: hidden layers buy flexibility, depth buys reusable features, the learning rate sets the step size, and a bad start can strand you in a local minimum. Know when the flexibility is worth the loss of interpretability — and when a plain regression would have told you the same thing, legibly.
Common questions
Is a neural network really just stacked logistic regressions?
For the basic building block, essentially yes: a single neuron computes a weighted sum of its inputs plus a bias and passes it through an activation function, and with a sigmoid activation that is exactly logistic regression. What is genuinely new is composition. Stacking neurons into a hidden layer lets the network combine several straight boundaries into curved ones, and adding depth lets later layers reuse the features earlier ones discovered — which is why a network can learn patterns (like XOR, or the shapes in an image) that no single logistic regression can. So the atom is familiar; the power comes from wiring many of them together and fitting them jointly by gradient descent.
What is a learning rate?
The learning rate is the step size in gradient descent: how far the weights move in the downhill direction on each update. It is the single most important dial in training. Set it too small and learning crawls, taking thousands of tiny steps to reach a good solution; set it too large and each step overshoots the valley floor, so the loss oscillates or even diverges instead of settling (you can watch this happen by pushing the slider high in the widget above). In practice people try a few values on a log scale (say 0.001, 0.01, 0.1) and often decay the rate over time — large steps early to move fast, smaller steps later to fine-tune.
Do I need to learn deep learning for my thesis?
Usually not. For the tabular data most research produces (rows of participants with a few dozen variables), a neural network rarely beats well-chosen classical methods, and it costs you interpretability, tuning effort, and far more data to train honestly. Regression, regularized models, and random forests are typically as accurate and much easier to explain to a committee. Deep learning earns its keep when the inputs are images, audio, or raw text and you have tens of thousands of examples. Learn the intuition (it demystifies a lot of modern AI), but don't reach for a neural network just because it sounds impressive.