Same Prompt, Different Test Suite
Priya runs a prompt against the ENGAGE timesheet story and gets eleven test cases. Her colleague runs the same prompt, same model, same afternoon, and gets nine — with two she did not have and three of hers missing. Neither of them changed anything.
In a test tool, this is a defect. Here it is the specified behaviour, and a tester who does not know that will spend a week filing a bug against a model.
Why the output moves
A language model does not retrieve an answer. It predicts the next token, repeatedly, from a probability distribution — and then samples from that distribution rather than always taking the most likely token. Two runs walk two different paths through the same probabilities.
The word plausible is doing the work here. The model produces text that is statistically likely given its training data and your prompt. Likely is not the same as correct, and nothing in the mechanism checks the difference.
This has a consequence you will use for the rest of the course: an LLM's output on a test task is evidence, not a result. It is reviewed the way you would review a junior colleague's first draft — not because the model is junior, but because you cannot tell from the output alone which parts it knew and which parts it constructed.
What the model actually reads
Your prompt does not reach the model as words.
Tokenization splits the text into tokens — roughly, common words become one
token, rare words and identifiers break into several. login is likely one
token. ENGAGE_TIMESHEET_SUBMIT_BTN is likely six or more. This matters in one
very practical way: test artefacts are full of identifiers, and identifiers are
expensive.
Embeddings turn each token into a vector positioned so that tokens used in
similar ways sit near each other. This is why a model can connect "the submit
control" in your prompt with submitButton in your code without being told
they are the same thing — and why it will sometimes connect two things that
merely look similar.
The transformer is the architecture that lets the model weigh how every token relates to every other one across a long passage. It is the reason a model can hold a whole user story in mind rather than reading it a line at a time.
The window, and what falls out of it
The context window is how much text, measured in tokens, the model can consider at once — your prompt, your input data, and its own answer, all together.
Practical consequence, and it is the one that bites first: a nightly run log does not fit. Neither does a 400-case regression suite. When you paste more than the window holds, you do not get an error. You get an answer that quietly ignores the part that did not fit, phrased with the same confidence as one that read everything.
A larger window is not free either. More tokens means more computation, more latency, and more cost per call — a thread you will pick up again in the last unit when you estimate what a task costs to run every night.
Where GenAI sits
You will meet four terms and they are not synonyms.
- Symbolic AI encodes knowledge as explicit rules. Deterministic, inspectable, brittle.
- Classical machine learning learns from data you have prepared and features you have chosen. Defect triage models are usually this.
- Deep learning learns the features itself from large datasets.
- Generative AI uses deep learning to produce new content — text, code, images — by modelling patterns in what it was trained on.
The reason GenAI arrived in test teams and the others mostly did not: it is pre-trained. There is no dataset to assemble and no training phase before you get value from it on Monday. That convenience is exactly what makes the risk units worth a third of this course.
What you now know
Non-determinism is designed in, not broken. Tokens are what the model reads and what you pay for. The context window silently truncates. And "plausible" is the model's actual objective — correctness is yours.