The Machine You'll Learn On

Setting up a machine you can trust

This chapter walks through getting Python, a notebook, and git actually running and verified on your own computer, so that everything after this hands you code to run rather than code to read. It starts from one root cause of nearly all setup trouble: not knowing which Python is running or where it looks for packages. From there it covers checking your Python version, finding the exact path of the interpreter with which python3, and building an isolated virtual environment with venv so a project's packages can't collide with anything else on the machine. It installs the four libraries used throughout the course — NumPy, pandas, scikit-learn, and JupyterLab — and pins their exact versions into a requirements.txt file, a habit that pays off later when versions drift and old code stops matching new libraries. It explains what a notebook actually is: a document paired with a separate running process called the kernel, which holds all your variables in memory regardless of what your cells currently say. That mismatch between document and memory is the source of notebooks that look right but aren't, and the fix is a firm habit — restart the kernel and run everything from the top before trusting or sharing any result. The same "which Python" question resolves the classic case of a package installing fine but refusing to import, by checking sys.executable from inside the notebook itself. It closes by setting up a git repository as the container a portfolio ships in: initializing it, excluding the environment folder and notebook checkpoints, writing a short honest README, and making a first commit — building the habit of committing as you go rather than tidying up at the end.

What a leaderboard score is actually measuring

A preprint on sizing evaluations of long-horizon AI agents, by Vasundra Srinivasan, applies classical test-theory methods to benchmark scores and finds that an agent's own general capability accounts for under three percent of the variation in results, while the pairing of a specific agent with a specific task accounts for far more. Reliability collapses further on the hardest tasks, dropping to zero, and setups that looked most stable on visible data turned out to replicate worst on held-out tasks. The paper proposes a way to calculate how many tasks, trials, and judges an evaluation actually needs before a deployment decision based on it can be trusted.

Everything so far has been a map. You know what the words mean, you know how the field got here, and you know the four shapes a learning problem can take. None of that puts a working model on your screen. This chapter does the boring, necessary thing: it gets Python, a notebook and git running on your machine, confirmed working, so that from here on every episode can hand you code to run rather than code to admire.

Set aside twenty minutes and sit at a keyboard. We are going to type things.

Before we start, one claim, and it holds for the entire course. Almost every setup problem you will hit comes from a single source: ambiguity about which Python is running, and where that Python looks for its packages. Not a hard problem. A confusing one. Most computers have more than one Python on them. Your operating system may ship one. You may have installed another. A code editor may have quietly added a third. When you install a package and then can't import it, the package installed fine — it just went to a different Python than the one you're asking. Every step below is a way of removing that ambiguity, so that at any moment you can say with certainty which interpreter you are talking to.

Start with the plainest possible check. Open a terminal and ask for the Python version: type python3 space dash dash version and press return. You want to see a three-point-something. If you see it, you have an interpreter. If the command isn't found, install Python from the official site, or through your platform's usual package manager, and try again.

Now ask the more useful question. Type which python3 on macOS or Linux, or where python on Windows. This tells you the full path to the file that actually runs when you type that word. That path is the answer to "which Python", and getting into the habit of asking it will save you more time over the next ninety episodes than any other single habit in this chapter. When something is broken and you don't know why, ask which Python. Nine times in ten the answer is the bug.

Next, give the project a home. Make a folder — call it whatever you like, but something like ai-course works, and it will become your first repository. Move into it in the terminal.

Now the important part. Inside that folder we make a virtual environment. Type python3 space dash m space venv space dot venv. Read that as: run the venv module using this specific Python, and put the result in a hidden folder called dot venv. It takes a few seconds and prints nothing. That silence is normal.

Then activate it. On macOS or Linux, source the activate script inside that folder's bin directory — type source space dot venv slash bin slash activate. On Windows PowerShell it's the activate script inside the Scripts directory instead. You'll know it worked because your prompt changes; most shells stick the environment's name in parentheses at the start of the line.

Now re-ask the question. Type which python3 again. The answer has changed. It no longer points at some system location; it points inside your project folder, into dot venv. You have not installed a new Python. You have changed which one your shell reaches for first. That is the whole trick, and we'll take it apart in a minute.

With the environment active, install the core stack. One command: pip install numpy pandas scikit-learn jupyterlab. Four names. NumPy is the array library everything numeric in Python sits on. Pandas is the table library, the thing you'll use to load and clean data. Scikit-learn is the classical machine learning library, where your first model will come from. JupyterLab is the notebook server — the program that serves notebooks to your browser. Expect a wall of download output and maybe a minute of waiting, because these pull in dependencies of their own.

Now do one thing most tutorials skip, before you have any reason to care. Type pip freeze, redirecting the output into a file called requirements dot txt. What pip freeze prints is a list of every package currently installed in this environment, each one with its exact version number nailed down. That file is now a record of the precise state of this environment.

Why bother, when nothing has gone wrong yet? Because of what happens in six months. You come back to this project, install the same four package names on a new machine, and get newer versions of all four. Pandas has renamed an argument. Scikit-learn has changed a default. Your notebook, which worked perfectly, now throws an error on line three, and you have no idea whether the bug is in your code or in the gap between versions. Pinning versions means the next install reproduces this exact environment instead of approximating it. In the production phases this becomes a hard requirement, because a model's behavior depends on the versions of the libraries that trained it. Starting the habit now costs you one command.

Last step of the build. Type jupyter lab. Your terminal fills with log lines and your browser opens. Create a new Python notebook. In the first cell, type two lines: import pandas as pd, then print of pd dot version — the double-underscore version attribute. Press shift and return together.

A version number prints under the cell.

Stop there. That single printed line is what this whole chapter was for. It means a Python interpreter exists, a specific one you can name, with an isolated set of packages you control, being driven by a notebook in your browser. Every episode after this one starts from that line.

Now the rhythm turns, because you just used two things without knowing what they are, and you should not leave it there.

A virtual environment is not magic and it is not a container. It is a directory. Look inside dot venv and you'll find a place for executables and a place for installed packages, and that second place is where pip just put NumPy and pandas. When you activated it, the activate script did something almost embarrassingly simple: it put that environment's executable directory at the front of the list of places your shell searches for commands. That's why which python3 changed its answer. The shell finds the environment's Python first and stops looking. And that Python, when you ask it to import pandas, searches its own package directory rather than the system one. Two ordinary mechanisms — a search path and a folder of files — and between them they mean the messy pile of packages one project needs can't collide with the pile another project needs. If an environment ever gets hopelessly tangled, the fix is to delete the folder and rebuild it from your requirements file. Nothing precious lives in there.

A notebook is stranger, and worth understanding properly, because it gets asked about in interviews. A notebook file is not a program that runs. It is a document — structured text holding your cells, the code in them, and the output they last produced. The thing that actually runs your code is a separate, long-lived process called the kernel. When you opened JupyterLab and made a notebook, a Python process started up behind it. Your browser sends a cell's code to that process, the process runs it and sends back whatever came out, and the browser draws the result under the cell.

The consequence: the kernel holds state. Every variable you create stays alive in that process's memory until you restart it or shut it down. That's why notebooks feel so good for exploring data. You load a large file once, in one cell, and then poke at it for an hour without reloading. In a plain script, every run starts from nothing.

That same fact is the source of the classic notebook disaster, so let's name it plainly. Cells can be run in any order, and out of order, and more than once. The kernel remembers what happened, not what your document says. So consider this. You write a cell that defines a variable, run it, then edit the cell into something else, and never re-run it. The old value is still sitting in the kernel's memory. Everything downstream keeps working, using a value that no longer appears anywhere in your notebook. You now have a document showing correct-looking output for code that cannot produce it. Send that notebook to someone else, they run it top to bottom, and it fails — or worse, it succeeds with different numbers.

The fix is a habit, not a setting. Before you believe a result, and always before you show it to anyone, restart the kernel and run every cell from the top. In JupyterLab that's one menu item. If it runs clean, the document and the state agree, and the numbers you're looking at are numbers your code produced. If it breaks, better you than your reviewer. Get strict about this now, because by the end of Phase 1 you'll have a notebook you're showing to strangers.

The other classic is the one this chapter has been circling. You are in a notebook, you need a package, so you run pip install for it. It installs, you import it, and Python says it doesn't exist. What happened is that the pip you called belongs to a different environment than the kernel you're running — the system Python's pip, say, while the kernel is your project's Python. The package installed. It just installed somewhere your kernel never looks.

The one-line way to check, from inside the notebook: import sys and print sys dot executable. That prints the full path of the interpreter the kernel is actually running. If it doesn't point inside your project's dot venv, the kernel is not the environment you think it is. That's the same question — which Python — asked from inside the notebook instead of from the shell, and it resolves nearly every mysterious import error you will hit this year.

Which leaves git, and I want to frame it correctly.

We are not doing version control theory here, and I'm not going to pretend git is easy. What matters right now is what a git repository is for you: it is the container your portfolio ships in. When you apply for a job with no relevant degree and no relevant title, the thing an interviewer opens is a repository. Not a certificate. Not a course completion badge. A folder of code with a readable README that tells them what the project does and how to run it. That is the artifact. Everything you build over the next ninety episodes is going to live in one of these.

So, four commands, in your project folder, with the environment still active.

First, git init. That creates a hidden folder that will track the history of everything else in this directory.

Second, make a file called dot gitignore, and put two things in it: the name of your environment folder, dot venv, and the notebook checkpoint directory, which is a hidden folder called dot ipynb underscore checkpoints. Both lines matter. The environment folder holds thousands of files you did not write, weighs a lot, and is specific to your machine and operating system — that's exactly what requirements dot txt exists to replace. The checkpoint folder is autosave clutter Jupyter creates as you work. Neither belongs in a repository, and a repository with an environment folder committed into it looks, to anyone who opens it, like someone who has not done this before.

Third, write a README dot md. Three or four sentences is plenty at this stage: what this repository is, that it's your work through a machine learning course, and how to set it up — make an environment, install from requirements dot txt, start JupyterLab. Write it as though for a stranger, because eventually it is for one.

Fourth, git add with a dot to stage everything, then git commit with a message flag and something honest like "toolkit set up and verified". That's your first commit. The history starts here.

Do not put this off until you have something impressive. The habit is the point. A project committed as you go, with a README that explains itself, is a different object from a project you tried to tidy up on the last day. The first one reads like an engineer's work. The second reads like homework.

That closes the orientation. You have vocabulary, history, the four shapes a learning problem can take, and now a machine you can trust — a named interpreter, an isolated environment with pinned versions, a notebook whose kernel you understand, and a repository waiting to be filled.

Next we use it. We load a real dataset into this exact environment, look at it, and fit an actual model in a handful of lines, before opening up a single piece of the math underneath. The concrete first. The fundamental after.

What a leaderboard score is actually measuring

One paper this time, and it earns the slot because it is about something the whole back half of this course depends on: knowing whether a number you measured means anything.

The paper is a preprint that went up on arXiv on the eleventh of August, twenty twenty-six, by a researcher named Vasundra Srinivasan, on how to size evaluations of long-horizon AI agents so that a deployment decision based on them is reliable. Long-horizon means an agent working through a task with many steps, over time, the kind of thing Phase 8 of this course builds. And the way the paper attacks the problem is borrowed from an unexpected place: generalizability theory, a piece of classical psychometrics originally built for reasoning about whether a test score reflects the person or the testing conditions.

The finding in a sentence: when you break down where the variation in agent benchmark results actually comes from, the agent itself accounts for almost none of it.

Here are the numbers. Across three open long-horizon benchmark suites, the share of total variance attributable to the agent on its own — its general capability, the thing a leaderboard implies it is ranking — came in under three percent. The share attributable to the pairing of a particular agent with a particular task ran from about seven percent up to twenty-three percent. So the interaction is several times larger than the main effect. Read that plainly: the gap between two agents on a leaderboard is mostly telling you which one happens to suit those specific tasks, not which one is more capable in general.

It gets sharper on hard problems. The paper reports a reliability coefficient — roughly, how much of a score is signal rather than noise — of about zero point seven five across one full benchmark. Restrict to the hardest quarter of that same benchmark and the coefficient falls to zero. Not low. Zero. On the hardest tasks, which are the ones you actually care about before deploying, the measurement carries no reliable signal at all.

And one result that should make anyone uneasy. Configurations that looked most reliable on the benchmark's own training split were the ones that replicated worst on held-out tasks. The correlation between the two was strongly negative, close to minus zero point nine on one suite. Tuning your setup for apparent stability on the split you can see actively selected against stability on the tasks you cannot.

The paper does not stop at the complaint. It offers a framework it calls deployment decision reliability, which turns this variance decomposition into sample-size arithmetic: given the reliability you need for the decision you're making, how many tasks, how many trials and how many judges do you need. Ranking two agents against each other needs less than gating a production service-level agreement, and the framework tells you how much less. It also finds that single execution traces are nearly useless for characterizing failure modes — the error is large and idiosyncratic — but averaging over multiple trials on the same task-and-agent cell produces failure profiles that are stable and transfer.

Why this matters to you now, twenty minutes after installing pandas. Because the instinct it fights is the one you are about to develop. You will fit a model, print a score, and feel that you have learned something about the model. Sometimes you have. Often you have learned something about the particular split of data you happened to evaluate on. The discipline of asking how much of this number is the thing I am measuring, and how much is the conditions I measured it under, is what separates someone who reports scores from someone who can be trusted with a deployment.

That question has a name and a home in this course. It is variance decomposition, and it belongs to the probability and statistics core in Phase 2 — expectation and variance, estimators, and what it means for a measurement to be reliable rather than merely repeated. This paper is that machinery pointed at the most fashionable object in the field, and finding the object mostly hollow. When we get there, you will be able to derive why the numbers came out the way they did.