Neural Networks: The Intuition
"Neural network" sounds like something quite different from the regression you already know, but its building block is a model you met in Stats 2, logistic regression. A network stacks a handful of them so that straight decision boundaries can combine into any shape you need. This lesson covers the intuition only, without the calculus of backpropagation.
A neuron is just logistic regression
An artificial neuron takes some inputs, multiplies each by a weight, adds them up with a bias, and passes the total through an activation function. With a sigmoid activation, which squashes any total into the range 0 to 1, that is exactly logistic regression: output = σ(w₁x₁ + w₂x₂ + b). The weights are the slopes, the bias is the intercept, and the neuron draws a single straight boundary through the feature space.
One neuron draws one line. Networks draw curves by combining several lines.
Layers compose simple boundaries into complex ones
Put a hidden layer of a few neurons between the inputs and the output. Each hidden neuron draws its own straight boundary, and the output neuron takes a weighted combination of those hidden signals. Combining several lines can mark out regions that no single line can, such as a corner, a stripe or an island. The classic example is XOR: label a point 1 when exactly one of its two inputs is on. No straight line separates the two classes, so plain logistic regression cannot solve it. Give the network three hidden neurons and it combines two lines into the diagonal "pockets" XOR needs. Depth (more layers) repeats the idea, so later neurons can reuse the features that earlier ones found.
🧠 Train a 2-3-1 network on XOR
Two inputs → three hidden neurons → one output. Edges are weights: blue positive, orange negative, thicker = stronger. Press Train and the boundary bends from a useless line into the XOR pockets. Set the learning rate too high and training overshoots. New network reseeds the weights (some starting points get stuck in a poor local minimum, and reseeding gives a fresh start).
Training: rolling downhill
The weights start random and are learned. Define a loss that measures how wrong the network's outputs are, and picture it as a landscape over all the weights: high where the network is bad, low where it is good. Gradient descent computes the local slope and takes a small step downhill, again and again, until the loss stops falling. The learning rate is the step size. Too small and training is very slow. Too large and each step jumps past the bottom of the valley and overshoots, so the loss bounces around or grows (set the slider above to 8 or more to see this). The algorithm that computes the slopes efficiently through every layer is backpropagation, which this lesson names but does not derive. The landscape has several dips, so a run can settle in a local minimum and get XOR only half right. Reseeding the weights starts the descent from a different point.
The words you meet in real code
You will meet three more terms in any tutorial you open next, and the widget above already uses two of them.
An activation is the function applied to the weighted sum, and the sigmoid is only the historical default. Most networks today use ReLU (rectified linear unit), a very simple function: positive values pass through unchanged, and negative values become zero. It has no upper bound, so "squashing" describes the sigmoid rather than activations in general. The hidden layer in the widget above uses a third option, tanh, which is a rescaled sigmoid running from −1 to 1.
ReLU replaced the sigmoid because of a numerical problem. The sigmoid's slope is steepest at its center, where it is exactly 0.25, and flattens toward zero at both ends. Backpropagation multiplies these slopes together layer by layer. So a chain of ten sigmoid layers shrinks the gradient by a factor of at most 0.2510, roughly one in a million, and the early layers learn very slowly or not at all. That is the vanishing gradient problem. ReLU's slope on the positive side is exactly 1, and multiplying by 1 leaves the gradient unchanged.
An epoch is one full pass through the training set, and the widget's counter counts exactly that. The widget has 56 training points and makes one weight update per pass, which is full-batch gradient descent. Real training rarely works this way, because a full pass over a million images is far too slow to repeat thousands of times. In practice the data are split into mini-batches of perhaps 32 or 256 examples, and each batch gives its own update. So one epoch makes thousands of small, noisy steps rather than one exact one. That is stochastic gradient descent. The noise turns out to be useful, because it can move the search out of shallow local minima like the ones that sometimes trap the network above.
What this course does not cover
Everything above works at a much larger scale, with more complicated machinery. This course skips the backpropagation derivation and the specialized architectures (convolutional nets for images, the transformers behind today's language models) that stack these ideas many layers deep. The basic ideas stay the same: weighted sums, activation functions, and gradient descent moving downhill. For a thesis on tabular research data you rarely need any of it: regression, random forests and careful validation usually predict as well as a neural network and stay interpretable. Use deep learning when the data are images, audio or text and you have tens of thousands of examples.
Why it matters: a neural network is a stack of logistic regressions whose straight boundaries combine into curved ones, with the weights fitted by gradient descent on an error surface. That picture explains the main terms: hidden layers add flexibility, depth lets later layers reuse features, the learning rate sets the step size, and a bad start can leave you in a local minimum. Decide whether the extra flexibility is worth the loss of interpretability, or whether a plain regression would tell you the same thing in a form you can read.
Common questions
Is a neural network really just stacked logistic regressions?
For the basic building block, essentially yes: a single neuron computes a weighted sum of its inputs plus a bias and passes it through an activation function. With a sigmoid activation, that is exactly logistic regression. What is new is composition. Stacking neurons into a hidden layer lets the network combine several straight boundaries into curved ones, and adding depth lets later layers reuse the features that earlier ones found. So a network can learn patterns (like XOR, or the shapes in an image) that no single logistic regression can. The building block is familiar, and the network's flexibility comes from connecting many of them and fitting them together by gradient descent.
What is a learning rate?
The learning rate is the step size in gradient descent: how far the weights move in the downhill direction on each update. It is often the most important setting in training. Set it too small and learning is very slow, taking thousands of tiny steps to reach a good solution. Set it too large and each step overshoots the valley floor, so the loss oscillates or even diverges without settling (push the slider high in the widget above to see this). In practice people try a few values on a log scale (say 0.001, 0.01, 0.1) and often lower the rate over time: large steps early to move fast, smaller steps later to fine-tune.
Do I need to learn deep learning for my thesis?
Usually not. For the tabular data most research produces (rows of participants with a few dozen variables), a neural network rarely beats well-chosen classical methods. It also costs you interpretability and tuning effort, and it needs far more data to train and validate properly. Regression, regularized models, and random forests are typically as accurate and much easier to explain to a committee. Deep learning does better when the inputs are images, audio or raw text and you have tens of thousands of examples. Learn the intuition, because it explains a lot of modern AI, but don't use a neural network just because it sounds impressive.