# How Does a Text File Become a "Program"?
Take a look at this program:
factorial.mini — MiniLang
// compute 7!
let n = 7;
let result = 1;
while n > 1 {
result = result * n;
n = n - 1;
}
if result == 5040 {
print result;
} else {
print 0;
}
$ python minilang.py factorial.mini
5040
5040
The file above is just text. There are no Python commands or machine-code instructions inside it. The meaning of
let, the precedence of * over +, and the behavior of the while loop are all defined by us. If we can write a program that reads these rules and executes them, we own a programming language — even if ours is still small.
Key point: A programming language isn't just a set of keywords. Every language has two core contracts: Syntax determines which programs are well-formed, and Semantics determines what those valid programs mean.
# Are We Building an Interpreter or a Compiler?
These two words are often used interchangeably, but they aren't the same. A compiler usually transforms the program into another form — machine code, bytecode, or an intermediate language. An interpreter reads the program and executes its behavior. The boundary isn't always rigid; many modern languages first produce bytecode and then run it with a virtual machine or JIT.
Interpreter
Reads the program's structure and executes its meaning.
Source → AST → Result
Compiler
Transforms the program into another representation such as machine code or IR.
Source → Target Code
Our project is a Tree-Walk Interpreter: we convert the source code into a semantic tree, and the interpreter walks that tree. So in this article we won't build machine code, LLVM IR, bytecode, an optimizer, or a JIT. Still, our Lexer, Parser, AST, and execution rules are real, and they demonstrate the same ideas found in the implementation of larger languages.
Simplification warning: "small language" doesn't mean "fake toy" — it means we deliberately set aside features like functions, strings, scopes, modules, a type checker, a garbage collector, and machine code so the core of the work stays visible.
# The Roadmap of Our Language
Every MiniLang file passes through six stations:
1
Source — the
.mini file text2
Lexer — building tokens
3
Parser — checking the grammar
4
AST — building the semantic tree
5
Interpreter — executing the tree
✓
Output — result or error
If an unknown character appears, the Lexer stops. If the token order doesn't fit the grammar, the Parser raises an error. If the program uses a variable that doesn't exist or divides by zero, the error happens at runtime. This separation matters: not all errors come from the same stage.
# First, Define the Language's Contract
Before writing the Lexer we must decide what is valid in our language. MiniLang will have these capabilities:
01
Value
Integers, floats, and the Booleans
true and false02
Expression
Arithmetic, comparison, parentheses, and unary operators
03
State
Declaring variables with
let and assigning new values04
Control Flow
if/else conditionals and while loops05
I/O
Displaying values with the
print statement
Every simple statement ends with
;. Conditional and loop bodies go inside { }. Conditions don't require mandatory parentheses. In this version all variables live in a single global environment — meaning we have no block scope.
# Stage One — Lexer: Turn Text into Tokens
The Lexer (or Tokenizer) doesn't yet know what the program means. Its only job is to split the raw string into meaningful pieces. For example:
tokens.txt
Source:
let x = 10 + 2;
Tokens:
LET IDENT(x) = NUMBER(10) + NUMBER(2) ; EOF
Whitespace and comments mean nothing to the Parser, so the Lexer discards them. But the number
10 isn't just text — the Lexer also stores its numeric value. Likewise, let initially reads like an identifier, and then the keyword table reveals it's a reserved word.
This is the first part of
minilang.py. To keep the project short and readable, identifiers start with an English letter or _, comments start with //, and we keep each token's line number for error messages:
minilang.py — part 1: Token and Lexer
from __future__ import annotations
from dataclasses import dataclass, field
import re
import sys
from typing import Any
@dataclass(frozen=True)
class Token:
type: str
text: str
value: Any
line: int
TOKEN_PATTERN = re.compile(
r"""
(?P \d+(?:\.\d+)? )
| (?P [A-Za-z_]\w* )
| (?P //[^\n]* )
| (?P ==|!=|<=|>=|[+\-*/=!<>] )
| (?P [(){};] )
| (?P \n )
| (?P [ \t\r]+ )
| (?P . )
""",
re.VERBOSE,
)
KEYWORDS = {
"let": "LET",
"print": "PRINT",
"if": "IF",
"else": "ELSE",
"while": "WHILE",
"true": "TRUE",
"false": "FALSE",
}
def tokenize(source: str) -> list[Token]:
tokens: list[Token] = []
line = 1
for match in TOKEN_PATTERN.finditer(source):
kind = match.lastgroup
text = match.group()
if kind == "NEWLINE":
line += 1
elif kind in {"SPACE", "COMMENT"}:
continue
elif kind == "NUMBER":
value = float(text) if "." in text else int(text)
tokens.append(Token("NUMBER", text, value, line))
elif kind == "IDENT":
tokens.append(Token(KEYWORDS.get(text, "IDENT"), text, None, line))
elif kind in {"OP", "PUNCT"}:
tokens.append(Token(text, text, None, line))
else:
raise SyntaxError(f"line {line}: unexpected character {text!r}")
tokens.append(Token("EOF", "", None, line))
return tokens
The Lexer tells
= apart from == by matching the longer pattern first. Pattern order matters: if = matched first, the sequence == could be wrongly split into two separate tokens.
# Stage Two — Grammar: Which Orderings Are Valid?
Having tokens isn't enough. These two strings have nearly identical tokens, but only the first is a valid variable declaration:
Valid
let answer = 42;
Invalid
let = answer 42;
The grammar defines the allowed order of tokens. Our small grammar looks like this:
grammar.ebnf
program → statement* EOF ;
statement → "let" IDENT "=" expression ";"
| IDENT "=" expression ";"
| "print" expression ";"
| "if" expression block ("else" block)?
| "while" expression block ;
block → "{" statement* "}" ;
expression → equality ;
equality → comparison (("==" | "!=") comparison)* ;
comparison → term ((">" | ">=" | "<" | "<=") term)* ;
term → factor (("+" | "-") factor)* ;
factor → unary (("*" | "/") unary)* ;
unary → ("!" | "-") unary | primary ;
primary → NUMBER | "true" | "false" | IDENT
| "(" expression ")" ;
In this notation,
| means "one of these alternatives", ? means an optional part, and * means zero or more repetitions. That * is not our language's multiplication operator — it's just part of the grammar notation.
The more important point is the layering of expressions.
factor reads multiplication and division operators, while term builds addition and subtraction from factors. So in the expression 2 + 3 * 4, the sub-expression 3 * 4 becomes a subtree first and is then added to 2. Operator precedence isn't the result of a hidden if — it's written directly into the grammar.
# Stage Three — AST: Keep the Program's Shape, Not the Text's
The Parser could merely validate tokens, but for execution we need a structure that preserves the relationships between the program's parts. That structure is the Abstract Syntax Tree, or AST. "Abstract" means details like whitespace, newlines, comments, and even redundant parentheses are dropped — only the effective structure remains.
For the statement
let result = 2 + 3 * 4; the conceptual tree has this hierarchy:
ROOT
Let: result
Variable declaration holding the initializer expression
+
Binary: 2 + (...)
Left child: 2 — right child: the multiplication expression
*
Binary: 3 * 4
Multiplication sits lower than addition and is evaluated first
To keep the code simple, we represent every node type with a single generic
Node class. A large compiler usually has separate classes like BinaryExpression, VariableDeclaration, and WhileStatement; the generic model is enough for our project:
minilang.py — part 2: AST Node
@dataclass
class Node:
kind: str
value: Any = None
children: list[Node] = field(default_factory=list)
line: int = 0
# Stage Four — Parser: Build a Tree from Tokens
Our Parser uses Recursive Descent: roughly one function per grammar rule. The
statement function detects the statement type, and block collects statements until it reaches }:
minilang.py — part 3: Statement Parser
class Parser:
def __init__(self, tokens: list[Token]):
self.tokens = tokens
self.current = 0
def parse(self) -> list[Node]:
program: list[Node] = []
while not self.check("EOF"):
program.append(self.statement())
return program
def statement(self) -> Node:
if self.match("LET"):
name = self.consume("IDENT", "expected variable name")
self.consume("=", "expected '=' after variable name")
value = self.expression()
self.consume(";", "expected ';' after declaration")
return Node("let", name.text, [value], name.line)
if self.match("PRINT"):
value = self.expression()
self.consume(";", "expected ';' after value")
return Node("print", children=[value], line=self.previous().line)
if self.match("IF"):
line = self.previous().line
condition = self.expression()
then_branch = Node("block", children=self.block(), line=line)
else_branch = (
Node("block", children=self.block(), line=line)
if self.match("ELSE")
else Node("block", children=[], line=line)
)
return Node("if", children=[condition, then_branch, else_branch], line=line)
if self.match("WHILE"):
line = self.previous().line
condition = self.expression()
body = Node("block", children=self.block(), line=line)
return Node("while", children=[condition, body], line=line)
if self.check("IDENT") and self.peek_next().type == "=":
name = self.advance()
self.advance() # consume '='
value = self.expression()
self.consume(";", "expected ';' after assignment")
return Node("assign", name.text, [value], name.line)
token = self.peek()
raise SyntaxError(f"line {token.line}: expected a statement")
def block(self) -> list[Node]:
self.consume("{", "expected '{' before block")
statements: list[Node] = []
while not self.check("}") and not self.check("EOF"):
statements.append(self.statement())
self.consume("}", "expected '}' after block")
return statements
The expression part mirrors the grammar layers exactly. Each function consumes only its own level's operators and delegates operands to the next stronger level:
minilang.py — part 4: Expression Parser
def expression(self) -> Node:
return self.equality()
def equality(self) -> Node:
node = self.comparison()
while self.match("==", "!="):
operator = self.previous()
node = Node("binary", operator.text, [node, self.comparison()], operator.line)
return node
def comparison(self) -> Node:
node = self.term()
while self.match(">", ">=", "<", "<="):
operator = self.previous()
node = Node("binary", operator.text, [node, self.term()], operator.line)
return node
def term(self) -> Node:
node = self.factor()
while self.match("+", "-"):
operator = self.previous()
node = Node("binary", operator.text, [node, self.factor()], operator.line)
return node
def factor(self) -> Node:
node = self.unary()
while self.match("*", "/"):
operator = self.previous()
node = Node("binary", operator.text, [node, self.unary()], operator.line)
return node
def unary(self) -> Node:
if self.match("!", "-"):
operator = self.previous()
return Node("unary", operator.text, [self.unary()], operator.line)
return self.primary()
def primary(self) -> Node:
if self.match("NUMBER"):
token = self.previous()
return Node("literal", token.value, line=token.line)
if self.match("TRUE", "FALSE"):
token = self.previous()
return Node("literal", token.type == "TRUE", line=token.line)
if self.match("IDENT"):
token = self.previous()
return Node("variable", token.text, line=token.line)
if self.match("("):
node = self.expression()
self.consume(")", "expected ')' after expression")
return node
token = self.peek()
raise SyntaxError(f"line {token.line}: expected expression")
def match(self, *types: str) -> bool:
if self.peek().type in types:
self.advance()
return True
return False
def consume(self, token_type: str, message: str) -> Token:
if self.check(token_type):
return self.advance()
token = self.peek()
raise SyntaxError(f"line {token.line}: {message}; got {token.text!r}")
def check(self, token_type: str) -> bool:
return self.peek().type == token_type
def advance(self) -> Token:
token = self.peek()
if token.type != "EOF":
self.current += 1
return token
def peek(self) -> Token:
return self.tokens[self.current]
def peek_next(self) -> Token:
return self.tokens[min(self.current + 1, len(self.tokens) - 1)]
def previous(self) -> Token:
return self.tokens[self.current - 1]
Why the loop inside the Parser? The expression
10 - 3 - 2 must be built as (10 - 3) - 2. The loop in term makes the previous node the left child of each new node — and that's exactly what creates left-associativity of operators.
# Stage Five — Interpreter: Give the Tree Meaning
So far we've read the program, but nothing has happened yet. The AST only says what structure the program has. The interpreter must define a behavior for every node type:
let
Define — store the value in the environment
+
Evaluate — compute both children and add
if
Branch — execute only the matching branch
while
Repeat — re-evaluate the condition
In our version the environment is a simple dictionary. The statement
let x = 10; stores the value 10 under the key x. Every time a variable node x is evaluated, the interpreter reads the current value from that same dictionary. If the name doesn't exist, we raise a runtime error instead of guessing a value.
minilang.py — part 5: Tree-Walk Interpreter
class Interpreter:
def __init__(self):
self.environment: dict[str, Any] = {}
def run(self, program: list[Node]) -> None:
for statement in program:
self.execute(statement)
def execute(self, node: Node) -> None:
if node.kind == "let":
if node.value in self.environment:
self.fail(node, f"variable {node.value!r} is already defined")
self.environment[node.value] = self.evaluate(node.children[0])
elif node.kind == "assign":
if node.value not in self.environment:
self.fail(node, f"undefined variable {node.value!r}")
self.environment[node.value] = self.evaluate(node.children[0])
elif node.kind == "print":
print(self.stringify(self.evaluate(node.children[0])))
elif node.kind == "block":
for statement in node.children:
self.execute(statement)
elif node.kind == "if":
branch = node.children[1] if self.truthy(self.evaluate(node.children[0])) else node.children[2]
self.execute(branch)
elif node.kind == "while":
while self.truthy(self.evaluate(node.children[0])):
self.execute(node.children[1])
else:
self.fail(node, f"unknown statement {node.kind!r}")
def evaluate(self, node: Node) -> Any:
if node.kind == "literal":
return node.value
if node.kind == "variable":
if node.value not in self.environment:
self.fail(node, f"undefined variable {node.value!r}")
return self.environment[node.value]
if node.kind == "unary":
right = self.evaluate(node.children[0])
if node.value == "!":
return not self.truthy(right)
self.require_numbers(node, right)
return -right
if node.kind == "binary":
left = self.evaluate(node.children[0])
right = self.evaluate(node.children[1])
if node.value == "==":
return left == right
if node.value == "!=":
return left != right
self.require_numbers(node, left, right)
operations = {
"+": lambda: left + right,
"-": lambda: left - right,
"*": lambda: left * right,
"/": lambda: left / right,
">": lambda: left > right,
">=": lambda: left >= right,
"<": lambda: left < right,
"<=": lambda: left <= right,
}
if node.value == "/" and right == 0:
self.fail(node, "division by zero")
return operations[node.value]()
self.fail(node, f"unknown expression {node.kind!r}")
@staticmethod
def truthy(value: Any) -> bool:
return value if isinstance(value, bool) else value != 0
@staticmethod
def stringify(value: Any) -> str:
if isinstance(value, bool):
return "true" if value else "false"
if isinstance(value, float) and value.is_integer():
return str(int(value))
return str(value)
@staticmethod
def require_numbers(node: Node, *values: Any) -> None:
if any(isinstance(v, bool) or not isinstance(v, (int, float)) for v in values):
Interpreter.fail(node, "operand must be a number")
@staticmethod
def fail(node: Node, message: str) -> None:
raise RuntimeError(f"line {node.line}: {message}")
In conditions,
false and the number zero are falsy, while true and any nonzero number are truthy. Arithmetic operators only accept numbers; since Python's Boolean is a subclass of int, we deliberately reject Booleans separately so an expression like true + 1 doesn't accidentally become valid in MiniLang.
In a
while loop the condition must be re-evaluated on every iteration. If we computed the condition's value only once before the loop, changing the variable n inside the body would have no effect and the factorial loop would never terminate.
# Stage Six — Wire the Pieces Together
The
run function executes the article's pipeline in three lines: tokenize, parse, interpret. The command-line part reads the input file as UTF-8 and reports expected errors without dumping a long traceback:
minilang.py — part 6: Runner
def run(source: str) -> None:
tokens = tokenize(source)
program = Parser(tokens).parse()
Interpreter().run(program)
if __name__ == "__main__":
if len(sys.argv) != 2:
raise SystemExit("usage: python minilang.py program.mini")
try:
with open(sys.argv[1], encoding="utf-8") as source_file:
run(source_file.read())
except (OSError, SyntaxError, RuntimeError) as error:
print(f"MiniLang error: {error}", file=sys.stderr)
raise SystemExit(1) from error
Now place the six parts above in order into a file named
minilang.py and save the program from the start of the article as factorial.mini:
terminal — real run
$ python minilang.py factorial.mini
5040
The output 5040 didn't come from any ready-made factorial function. The Lexer tokenized the characters, the Parser built an AST from them, the Interpreter created the variables
n and result, tested the loop condition several times, performed the multiplications and assignments, and finally executed the correct branch of the if.
# What Happens When the Program Is Wrong?
A language isn't defined only by its success path; where and how it fails is part of its design. MiniLang has three main families of errors:
LEX
Unknown character
print @; — the Lexer can't tokenize @PARSE
Invalid structure
let x = 10 — the ; is missingRUN
Runtime error
print 1 / 0; — the structure is valid, but execution is impossible
broken.mini — output
$ python minilang.py broken.mini
MiniLang error: line 1: division by zero
Note that the Parser doesn't reject division by zero. The expression
1 / 0 is perfectly valid grammatically; the problem only surfaces when the Interpreter tries to run it. In contrast, a missing ; is caught before execution, during the parse stage.
# What Does This Language Still Lack?
MiniLang is now an executable language, but it's not yet suitable for building real software. We should know its limitations explicitly:
What we actually built
Lexer, grammar, recursive-descent parser, AST, variables, arithmetic, conditionals, loops, output, and errors.
What we deliberately skipped
Strings, functions, scopes, arrays, modules, a type checker, bytecode, an optimizer, a garbage collector, and native code.
Our environment doesn't even have block scope; a variable declared inside an
if survives outside it. We also have no separate semantic-analysis stage, so errors like undefined variables are detected at runtime. These aren't hidden bugs — they're the project's simplifying decisions.
To grow the language, each feature can be added at four points: the needed tokens in the Lexer, a new rule in the grammar, a suitable node in the AST, and its behavior in the Interpreter. For example, adding functions requires defining parameter and call syntax, building Function/Call nodes, creating a local environment, and handling
return.
# Why Is Building a Small Language Worth It?
When we only use a language, its keywords and operators seem obvious. We assume the computer "naturally" knows multiplication comes before addition, that
while means repetition, and where a variable name points. But none of that is natural — they're all conventions the language's creator defined and implemented.
Building MiniLang shows that a language isn't made of magic. The Lexer names the text. The Parser understands the relationships between tokens. The AST discards cosmetic details. The Interpreter walks the structure and gives each node meaning. If we want to take the next step, we can transform that AST into bytecode or LLVM IR instead of executing it directly — that's where our path shifts from a tree-walking interpreter toward a compiler.
Perhaps the most fascinating part is this: the computer never understood the word
while. We built a program that agreed on what behavior to perform when it sees one. A programming language is fundamentally a precise contract between a human and an implementation — a contract that, if it has the slightest ambiguity, makes programs unpredictable.
takeaway.txt
The computer never understood the word while;
only the program we wrote agreed on what to do when it sees one.
That's what a programming language is:
a precise contract that gives plain text its meaning.
only the program we wrote agreed on what to do when it sees one.
That's what a programming language is:
a precise contract that gives plain text its meaning.
