Neural Networks. Inspiration from the Brain

Inspired by the brain, but not a copy

When you hear "neural network," you might picture nerve cells, synapses, blinking connections inside a head. That's not entirely wrong, the first researchers in the 1940s and 50s really did draw inspiration from the human brain. But: an artificial neural network is not a copy of the brain. It's a rough mathematical simplification that borrows a few ideas.

The brain has about 86 billion nerve cells, connected to each other in incredibly complex ways. A neural network in a computer has anywhere from a few million to a few hundred billion "artificial neurons," but these are extremely simple compared to real nerve cells. Still, this simplification goes surprisingly far.

What is an artificial neuron?

A neuron in AI is basically a small calculating machine. It receives numbers, either from other neurons or from outside, and outputs a number. That simple. But something interesting happens between input and output: the neuron weighs each input. Some inputs matter a lot to it, others it barely cares about.

Specifically, the neuron does three things:

1. It takes in all the input values.
2. It multiplies each input value by a weight. This weight is a number that says how important that input is.
3. It adds up the weighted values, and based on the result, decides what signal to pass on itself.

Here's the most striking part: a single neuron isn't particularly clever. It literally just does one multiplication, one addition, and one small decision. The magic only emerges once you wire thousands or millions of them together.

Layers, the structure of a network

Neurons are organized into layers. A typical structure looks like this:

• Input layer: This is where the raw data comes in. For an image, for example, the brightness values of each pixel. For text, the tokens we learned about in the last chapter.
• Hidden layers: The layers in between. They're called "hidden" because you can't directly see from the outside what they're doing. This is where the actual work happens, and there can be many of them.
• Output layer: This is where the result comes out. For a spam filter, for example, a number between 0 and 1: "How likely is it that this email is spam?"

Every neuron in a layer is usually connected to every neuron in the next layer. So if you have four neurons in one layer and six in the next, that's 4 × 6 = 24 connections, each with its own weight. In large networks, we're quickly talking about billions of connections.

Weights, what the network learns

We've already mentioned the word weight a few times. Let's quickly clarify what exactly that is and why it matters so much.

A weight is simply a number. One number per connection between two neurons. This number determines how strongly the signal from one neuron influences the next. If the weight is large, a lot of signal flows through; if it's small or zero, almost nothing flows.

And here's the real point: these weights are all a network "knows." When people say "the AI has learned," what that really means is: the billions of weights were adjusted so that the network gives the right answers for known examples. The intelligence, the knowledge, the abilities, all of that doesn't live in some text or database, it lives in these numbers.

What the network knows doesn't live in words. It lives in numbers.

Deep learning, when depth makes the difference

You've probably already heard the term deep learning. The "deep" stands for having many layers stacked on top of each other. Early neural networks in the 1980s often had only two or three layers. Today's networks have dozens, hundreds, sometimes thousands.

Why does depth help? Because each layer can assemble something more complex out of the patterns of the previous layer. A shallow architecture can only recognize simple relationships. A deep network can build increasingly abstract patterns step by step, from pixels to edges, from edges to shapes, from shapes to objects, from objects to meaning.

The breakthrough of deep learning came around 2012, when a deep network first beat all classical methods by a surprisingly wide margin in a famous image-recognition competition. Since then, deep learning has dominated almost every interesting AI problem.

Why deep networks are better, the idea behind it

Imagine you had to describe a building. If you only have one sentence, you have to stay very rough: "A big house." If you have five sentences, you can mention floors, windows, materials, and color. If you have a full page, you get into the details: architecture, history, atmosphere, neighborhood.

It's similar with layers in a network. Each additional layer allows a finer, more abstract description of the input. With language, for example: the lower layers "understand" letter sequences and individual words. Middle layers recognize sentence structure, grammar, simple meaning. Deeper layers capture style, tone, topics, even argumentative structure.

One important insight to close

We now have a picture of a neural network in mind: layers of neurons, connected by weights, that turn inputs into outputs step by step. What we haven't discussed yet is the most exciting question of all: how do the weights get their values? Who sets them? Nobody sets them by hand, that would be far too many numbers.

The answer is: training. The network learns its own weights by seeing millions or billions of examples and improving its weights by a tiny bit after each one. We'll look at exactly how that works in the next chapter.