Mutant IR / ByteCode Reference

Mutant compiles to a flat, linear byte stream. There is no intermediate representation between the AST and the bytecode – the compiler walks the AST and emits bytes directly.

This document covers every layer of the bytecode pipeline: instruction encoding, the constant pool, compiler internals, the stack-machine VM, security machinery, and the polymorphic mutation engine.

Status note: this is a deep technical reference. For current feature enablement and rollout status (especially anti-tamper gates and polymorphic activation depth), use:

  1. ANTITAMPER_PROBE_ENABLEMENT_LLD
  2. PROCESS_INJECTION_DETECTION_LLD
  3. POLYMORPHIC_BYTECODE_LLD

Table of Contents#

  1. Architecture Overview
  2. Instruction Encoding
  3. Opcode Reference
  4. ByteCode Object
  5. Constant Pool
  6. Compiler Internals
  7. Symbol Table & Scoping
  8. VM: Stack Machine Design
  9. VM: Execution Loop
  10. Closures & Free Variables
  11. Structs at the Bytecode Level
  12. Enums at the Bytecode Level
  13. Control Flow
  14. Operand Encryption
  15. Runtime Security Features
  16. Polymorphic Mutation Engine
  17. Object Type System
  18. VM Constructor Variants
  19. Common Pitfalls

1. Architecture Overview#

Source text
    │
    ▼
  Lexer  (token/)
    │
    ▼
  Parser (parser/)  ──► AST (ast/)
    │
    ▼
 Compiler (compiler/)
    │  ├─ SymbolTable    resolves variable scopes
    │  ├─ CompilationScopes  one per function nesting level
    │  └─ PolymorphicEngine  optional bytecode mutation
    │
    ▼
 ByteCode struct
    │  ├─ Instructions   []byte  flat bytecode stream
    │  ├─ Constants      []object.Object  literal pool
    │  ├─ StructDefs     map[string][]*ast.Identifier
    │  └─ EnumDefs       map[string][]string
    │
    ▼
   VM  (vm/)
    │  ├─ Stack          []object.Object (grows dynamically)
    │  ├─ Globals        []object.Object (grows dynamically)
    │  ├─ Frames         []*Frame        (call stack)
    │  └─ Security subsystem

Mutant compiles to a flat, linear byte stream. There is no intermediate representation between the AST and the bytecode – the compiler walks the AST and emits bytes directly. The VM then executes those bytes in a register-less, expression-stack machine.


2. Instruction Encoding#

2.1 Types#

// code/code.go
type Instructions []byte   // the raw bytecode stream
type Opcode       byte     // a single opcode value (0-based iota)

2.2 Instruction Layout#

Every instruction is:

┌──────────┬──────────────────────────────────────────┐
│  1 byte  │  0, 1, 2, 3, or 4 bytes of operands      │
│  opcode  │  (widths defined per opcode in Definition) │
└──────────┴──────────────────────────────────────────┘

Multi-byte operands are always big-endian.

2.3 `Definition` Registry#

type Definition struct {
    Name          string
    OperandWidths []int   // widths of each operand in bytes (1 or 2)
}

The definitions map in code/code.go is the single source of truth for every opcode's operand layout. code.Lookup(byte) returns the definition or an error for unknown opcodes.

2.4 `Make` — Building Instructions#

func Make(op Opcode, operands ...int) []byte
  • Allocates 1 + sum(OperandWidths) bytes.
  • Writes the opcode at [0].
  • Writes each operand using binary.BigEndian.PutUint16 (2-byte) or a direct byte() cast (1-byte).
// Example: emit OpConstant with index 42
ins := code.Make(code.OpConstant, 42)
// Result: [0x00, 0x00, 0x2A]   (OpConstant=0, 42 big-endian)

2.5 Reading Operands#

Unencrypted (compiler / test use only):

func ReadOperands(def *Definition, ins Instructions) ([]int, int)

Returns a []int of decoded operand values and the total bytes consumed by those operands (not including the opcode byte itself).

Encrypted (VM runtime use — the only correct way during execution):

func ReadUint16(ins Instructions, length int64, password string, offset int64) (uint16, error)
func ReadUint8 (ins Instructions, length int64, password string, offset int64) (uint8,  error)

Both functions call security.SecureXOR* before decoding. length is the total byte length of the main function's instruction stream (used as part of the XOR key derivation). offset is the absolute byte position of the first operand byte within the instruction stream.

Critical rule (from project memory): Never read encrypted operands with ins[offset] or binary.BigEndian.Uint16(ins[offset:]) directly inside the VM. Always call ReadUint8 / ReadUint16. Reading raw bytes gives corrupted values when encryption is active.

2.6 Disassembling#

Instructions.String() produces a human-readable listing:

0000 OpConstant 0
0003 OpConstant 1
0006 OpAdd
0007 OpSetGlobal 0
0010 OpGetGlobal 0
...

It is not safe to call this on encrypted bytecode (it uses the unencrypted ReadOperands path). Use it only in tests or during non-password compilation.


3. Opcode Reference#

Opcodes are declared as an iota in code/code.go. Their numeric values are fixed by the declaration order below, so do not reorder them.

3.1 Complete Table#

# Name Operands (bytes) Stack effect Description
0 OpConstant idx (2) → val Push constants[idx]
1 OpPop — val → Discard top of stack
2 OpAdd — b, a → a+b Integer or float add; string concat
3 OpSub — b, a → a-b Subtract
4 OpMul — b, a → a*b Multiply
5 OpDiv — b, a → a/b Divide
6 OpMod — b, a → a%b Modulo
7 OpTrue — → true Push the singleton True object
8 OpFalse — → false Push the singleton False object
9 OpEqual — b, a → bool a == b
10 OpUnEqual — b, a → bool a != b
11 OpGreater — b, a → bool a > b
12 OpMinus — a → -a Unary negation
13 OpBang — a → !a Unary logical NOT
14 OpJumpFalse target (2) cond → Pop condition; if falsy jump to target
15 OpJump target (2) — Unconditional jump to target
16 OpNull — → null Push the singleton Null object
17 OpGetGlobal idx (2) → val Push globals[idx] (decrypted)
18 OpSetGlobal idx (2) val → Pop and store to globals[idx] (encrypted)
19 OpGetLocal idx (1) → val Push stack[bp+idx] (decrypted)
20 OpSetLocal idx (1) val → Pop and store to stack[bp+idx] (encrypted)
21 OpArray n (2) eN…e0 → arr Pop n elements, build Array, push it
22 OpHash n (2) vN,kN…v0,k0 → hash Pop n key+value pairs (n is even), build Hash, push it
23 OpIndex — idx, obj → val Pop index then object; resolve obj[idx]
24 OpCall argc (1) argN…arg0, fn → result Call function with argc arguments
25 OpReturnValue — val → (caller frame) Pop return value, restore frame, push value
26 OpReturn — (caller frame) Void return; restores frame, pushes Null
27 OpMultiValue count (2) vN…v0 → multi Pop count values into one MultiValue
28 OpDup — a → a, a Duplicate the top of the stack
29 OpDestructure count (2) multi → v0…vN Pop a MultiValue; push count values from it
30 OpGetBuiltin idx (2) → fn Push the builtin at BuiltinNames[idx & 0x7FFF] (see §4.1)
31 OpClosure fnIdx (2), numFree (1) fN…f0 → closure Pop numFree free vars; wrap constants[fnIdx] in a Closure
32 OpGetFree idx (1) → val Push currentClosure.Free[idx], through the cell if it is one (decrypted)
33 OpCurrentClosure — → closure Push the currently executing closure (for named recursion)
34 OpChkDbg — — Halt (or warn) if a debugger is detected
35 OpChkSnd — — Halt (or warn) if a sandbox environment is detected
36 OpBreak — → Break{} Push a Break sentinel onto the stack
37 OpContinue — → Continue{} Push a Continue sentinel onto the stack
38 OpMakeStruct typeIdx (2), fieldCount (1) fN…f0 → struct Pop fieldCount values, create a Struct
39 OpGetField nameIdx (2) struct → val Pop struct; push struct.Fields[constants[nameIdx]]
40 OpSetField nameIdx (2) val, struct → struct Pop value then struct; set field; push struct back
41 OpEnumValue typeIdx (2), tagIdx (2) → EnumValue Create EnumValue{TypeName, Tag, ordinal}
42 OpGreaterEqual — b, a → bool a >= b
43 OpSetIndex — val, idx, obj → obj Mutate obj[idx] in place; push the container back
44 OpBitAnd — b, a → a&b Bitwise AND; both operands must be INTEGER
45 OpBitOr — b, a → a|b Bitwise OR; both operands must be INTEGER
46 OpBitXor — b, a → a^b Bitwise XOR; both operands must be INTEGER
47 OpBitNot — a → ~a Bitwise complement; operand must be INTEGER
48 OpShiftLeft — b, a → a<<b Left shift; a negative count is a runtime error
49 OpShiftRight — b, a → a>>b Arithmetic right shift (sign-preserving); negative count errors
50 OpGetLocalCell slot (1) → val Push cell(bp+slot).Value (decrypted); the slot must be boxed
51 OpSetLocalCell slot (1) val → Pop and store into cell(bp+slot).Value (encrypted)
52 OpCaptureLocal slot (1) → cell Push the *Cell in bp+slot itself, for OpClosure's list
53 OpCaptureFree idx (1) → cell Push currentClosure.Free[idx] unread, for a nested capture
54 OpSetFree idx (1) val → Pop and store into Free[idx].Value (encrypted)
55 OpConcat n (2) pN…p0 → str Pop n pieces of an interpolated string; push the one string they spell
56 OpIterInit — iterable → iter Pop a collection; push a cursor over it (hash keys in Inspect order)
57 OpIterNext target (2), binds (1) iter → iter[, key], val Advance the cursor: push binds values on top of it, or jump to target when spent
58 OpMatchFail — subject → (error) Pop the unmatched subject and raise, naming the value that fell through
59 OpLess — b, a → bool a < b
60 OpLessEqual — b, a → bool a <= b

3.2 Stack Notation#

The "Stack effect" column uses the convention:

  • Values left of → are consumed (popped), listed bottom-first (so b, a means a is on top).
  • Values right of → are produced (pushed).
  • (caller frame) means the caller's frame is restored — no net stack value in the new frame's context.

3.3 Comparison Operands Run In Source Order#

< and <= have opcodes of their own. Like every other binary operator they compile the left operand first, so the operands' side effects run in the order they are written:

[compile a]         ← left operand first
[compile b]         ← right operand second
OpLess

There used to be no OpLess: the compiler emitted a < b as b > a, pushing the right operand first and reusing OpGreater. That saved two opcodes and cost the source-order guarantee — next() < next() called the right-hand next first, while every other operator, and the tree-walking evaluator macro expansion computes an unquote with, called the left-hand one first. The same expression then answered differently inline and inside a macro. OpLess and OpLessEqual were appended to the instruction set and the swap removed; OpGreater and OpGreaterEqual keep the values and the meanings they always had, so bytecode compiled before the change still runs.


4. ByteCode Object#

// compiler/compiler.go
type ByteCode struct {
    Instructions code.Instructions              // Main function byte stream
    Constants    []object.Object                // Constant pool (see §5)
    StructDefs   map[string][]*ast.Identifier   // Field name lists per struct type
    EnumDefs     map[string][]string            // Tag name lists per enum type
    LuaPatches   map[string]*object.LuaPatch    // Lua security hook patches
    Version      int                            // Container version (see §4.1)
    BuiltinNames []string                       // Builtins this program calls (§4.1)
    OpcodeMap    []byte                         // Undoes opcode permutation (§16)
    SourceFile   string                         // Path this was compiled from (§4.2)
    SourceText   string                         // The program's own source (§4.2)
    LineTable    code.LineTable                 // Offset -> start line/col (§4.2)
    EndTable     code.LineTable                 // Offset -> end line/col (§4.2)
    MacroTable   code.LineTable                 // Offset -> macro definition site (§4.2)
}

ByteCode is the artifact handed from the compiler to the VM. It is immutable after construction (the VM clones nothing; it reads the slices directly).

The LuaPatches field is populated by the Lua integration layer, not by the core compiler. See builtin/lua.go for context.

4.1 Container versions and builtin resolution#

Version Emitted by OpGetBuiltin operand
absent (decodes to 0, normalised to 1) up to v2.4.0 an ordinal into the global builtin.Builtins registry of the release that compiled it
2 v2.5.0 onward 0x8000 | i, where i indexes this program's own BuiltinNames

Version 1 made the registry append-only forever. An ordinal baked into an artifact means that entry can never be renamed, retired or reordered, because doing so would rebind every call in every .mu already written — which is why net_syn_scan stayed registered long after it was documented as deprecated.

Version 2 puts the names in the artifact. The compiler interns each builtin it references (SymbolTable.ReferenceBuiltin) and emits a position in that table; vm.resolveBuiltins binds every name to a function at construction, before a single instruction runs, and reports a name this runtime does not have rather than failing at whichever call site happens to be reached. Only referenced builtins are listed, so retiring an unrelated one cannot stop a program loading.

Version 1 artifacts from v2.3.0 to v2.4.x still run. Their operands resolve through builtin/legacy_ordinals.go, a frozen snapshot of the registry as it stood at v2.4.0 — frozen because those indices are baked into files that cannot be recompiled. A builtin that is renamed adds an entry to builtin.Aliases, which keeps the old name resolvable for old bytecode without keeping it callable from new source.

One ordinal meant two builtins before v2.5.0. help was appended to the registry at init, after every other entry, so it sat at whatever the end was: 398 in v2.3.x, and 408 in v2.4.x, whose ten new builtins put pmap at 398. legacy_ordinals.go therefore also records v2.3.x's registry, as the v2.4.x snapshot with help at 398: v2.3.x's 399 entries are its first 399, and v2.2.x's 211 entries are the first 211 of both tables. Neither v2.3.x nor v2.4.x wrote Version, so the runner reads which table a version 1 artifact needs from the container itself: gob writes ByteCode's type, naming every exported field, ahead of the value, and only v2.4.0 and later declare OpcodeMap there (compiler.LegacyRegistryOf). A stream whose type cannot be read that way is refused rather than guessed at.

Registries before v2.2.0 ordered their entries differently, and neither table describes them. A v2.2.x artifact that calls a builtin does not run correctly either, although its registry matches: v2.3.0 widened OpGetBuiltin's operand from one byte to two, so this runtime reads a v2.2.x builtin call's operand with the next instruction's first byte in it, binds whichever builtin the two bytes number (or stops when they number none), and reads every instruction after it out of step. v2.2.x wrote the same ByteCode type as v2.3.x, so nothing in the container tells such an artifact apart, and it is not refused.

Two development commits between the v2.3.0 and v2.4.0 tags, 821bece and 31182f3, had v2.4.0's registry but no OpcodeMap. Their artifacts read as v2.3.x, so a pmap call in one reaches help; the builtins v2.4.0 added after pmap still resolve, because the v2.3.x table keeps the snapshot's entries past 398.

Both stay for the whole 2.x line: retiring version 1, or the net_syn_scan alias of net_connect_scan, stops artifacts that already exist from running, which only a major release may do. 3.0 is the first release that may remove either, and its changelog will say so if it does.

The 0x8000 tag on version 2 operands (code.BuiltinNameTableFlag) is for the one case the version field cannot cover. A pre-v2.5 runtime does not know to read Version, so gob hands it a name-table index and it reads that as a registry ordinal — and a small index is a perfectly plausible ordinal, so it calls whatever sits there. The tag puts every version 2 operand far above the registry's length, so the old runtime trips its own bounds check (OpGetBuiltin: invalid builtin index=32768) instead of silently calling the wrong builtin. It is permanent: it cannot be retired without breaking the artifacts it protects.

4.2 Source positions#

LineTable maps an offset in an instruction stream back to the line and column that produced it. There is one per stream: ByteCode.LineTable covers Instructions, and every object.CompiledFunction carries its own alongside its Name. That is what makes a VM frame resolvable — a frame knows its closure and its offset, and the closure's function owns the table those are read against.

entry := uvarint(ipDelta) varint(lineDelta) varint(colDelta)

Deltas are against the previous entry; the line and column deltas are signed because positions move backwards routinely (a loop's jump is emitted after the body but belongs to the for above it). An entry is written only where the position changes.

There are two position tables per stream. LineTable gives where a construct starts, EndTable where it ends. A start alone names a line, which is where most languages stop; both ends let the reporter underline the span that failed, so total / count(xs) says which division. EndTable is a separate field in the same encoding, so it can be dropped on its own.

Statements, calls, infix expressions and index expressions anchor a position. Everything else inherits from the node enclosing it. Attributing a position to every expression node sounds more precise and is not: measured, it costs one entry per instruction, a table the size of the stream it describes. The four kinds that do anchor are the ones a reader lands on: a frame is a call, a statement is the unit a fault sits inside, and infix and index expressions are where a well-formed program actually fails at runtime -- division by zero, a type mismatch across an operator, an index past the end.

Debug info is not cheap and is not meant to be. The tables come to roughly a fifth of an encoded artifact, and SourceText roughly doubles what is left, so a local .mu runs about 1.7x the size of the same program built for release. That is the trade -g makes in a C toolchain, and it is paid only by artifacts that never leave the machine.

Parameter names travel too, on CompiledFunction.Params, so a traceback can print the arguments a frame received rather than only its name. They are gated on debug info for a second reason besides size: stack values are encrypted at rest, and rendering them decrypts them. A stripped build prints its frames without printing the program's own runtime values.

Macro expansions record two sites. MacroTable has the same encoding and is populated only over instructions a macro produced. LineTable gives the call the user wrote; MacroTable gives the macro's definition. Both are needed: when generated code is wrong, the call site contains none of the logic that failed. The parser's ast.Program.MacroExpansions side-table carries this from expansion to compilation.

An entry whose line is 0 is a reset. A lookup answers with the last entry at or before an offset, so "populated only over instructions a macro produced" needs an entry recording where an expansion stops as much as one recording where it starts -- without it the last macro in a program is reported as the origin of every instruction compiled after it. MacroTable and EndTable therefore carry entries whose accumulated line is 0, which mean the previous entry's coverage ends here: offsets from there on have no macro origin and no recorded end, and a reporter falls back to a caret rather than underlining a span it would be taking from another construct. LineTable never carries one -- an instruction with no position of its own belongs to the construct it was emitted for and inherits its line, which is what writing no entry achieves. The entry encoding does not change, so a decoder that knows nothing about resets reads line 0, and every consumer of a position already treats a zero line as no position at all.

Versioning. These fields needed no Version bump, unlike BuiltinNames. gob omits zero values and ignores fields it does not know, so a new runtime reading an old artifact sees empty tables — which is exactly true of it — and an old runtime reading a new artifact ignores them. Absence is already the correct reading in both directions. A program with no tables reports frames by name and no lines.

Stripping. ByteCode.StripDebugInfo() removes the file name, the embedded source, every table, and every function and parameter name. generator.compile calls it for release builds: those are what leave the machine, and a map from an artifact's bytecode back to its source is worth withholding from them. A .mu compiled to run locally keeps its positions.

Polymorphism does not strip. Mutation is on by default — mutant prog.mut compiles at level 5 — so dropping positions there would mean no ordinary run ever had them. Instead PolymorphicEngine.spliceFillers carries the tables through the same offset remap it already builds to repoint jumps (LineTable.Remap). A stream the engine declines to pad keeps its table unchanged, because nothing moved.


5. Constant Pool#

The constant pool is []object.Object stored in ByteCode.Constants. It is built incrementally by compiler.addConstant:

func (c *Compiler) addConstant(obj object.Object) int {
    c.constants = append(c.constants, obj)
    return len(c.constants) - 1  // index used as OpConstant operand
}

What lives in the pool:

Object type Example source Notes
*object.Integer 42, -7 int64
*object.Float 3.14 float64
*object.String "hello", field names, type names All string literals; also struct/enum type-name strings and field-name strings needed by struct/enum opcodes
*object.CompiledFunction Any fn(…){…} The function body's bytecode + metadata

Booleans and Null are NOT pooled. They use dedicated singleton opcodes (OpTrue, OpFalse, OpNull) that push global singleton objects.

Maximum pool size: 65 535 entries (uint16 operand range). In practice the limit is never reached in normal programs.

Constant indices are stable within a single compilation. If polymorphic constant-pool randomization is enabled (see §16), every operand that indexes the pool is rewritten to match the shuffled order before the ByteCode is returned. Which operands those are is declared by code.ConstantOperands, not assumed: OpConstant, OpClosure, OpMakeStruct, OpGetField, OpSetField and both operands of OpEnumValue.


6. Compiler Internals#

6.1 Compiler Struct#

type Compiler struct {
    constants         []object.Object        // Accumulates the constant pool
    symbolTable       *SymbolTable            // Current scope's symbol table
    scopes            []CompilationScope      // Stack of compilation scopes
    scopeIndex        int                     // Index into scopes[]
    structDefinitions map[string][]*ast.Identifier
    enumDefinitions   map[string][]string
    loopContexts      []LoopContext           // Stack of active for-loop contexts

    injectSecurityChecks bool                 // Enables OpChkDbg/OpChkSnd injection
    hasChkDbg            bool                 // Tracks whether DbgCheck was emitted
    hasChkSnd            bool                 // Tracks whether SndCheck was emitted

    polymorphicEngine *PolymorphicEngine      // nil = no mutation
}

6.2 Compilation Scope#

A CompilationScope holds:

type CompilationScope struct {
    instructions    code.Instructions    // byte buffer being built
    lastInstruction EmittedInstruction   // most recently emitted instruction
    prevInstruction EmittedInstruction   // the one before last (for pop-removal)

    unresolvedJumps map[int]unresolvedJump  // forward jumps awaiting a target
}

unresolvedJumps is per scope and not per compiler because a jump position is an offset into one stream and means something else in any other. The three position tables the scope also builds are omitted above; see section 11.

The main program compiles into scopes[0]. Each function literal pushes a new scope via enterScope() and pops it with leaveScope().

6.3 `emit` and Instruction Patching#

func (c *Compiler) emit(op code.Opcode, operands ...int) int

Returns the byte offset of the emitted instruction. This offset is saved so jumps can be back-patched:

// Emit jump with placeholder target
jumpPos := c.emitPlaceholder(code.OpJumpFalse)

// ... compile consequence ...

// Back-patch: replace the placeholder with the real destination
c.changeOperand(jumpPos, realTarget)

changeOperand calls replaceInstruction which overwrites the bytes at pos in the current instruction buffer. This works because Make produces the same byte length regardless of the operand value. Operands after the first are read back and re-emitted unchanged, so a wider instruction — OpIterNext carries a binding count after its target — keeps its tail instead of having it fall off the end of a shorter rebuilt instruction.

The placeholder is owed a target#

emitPlaceholder emits operand 0 as placeholderTarget (9999) and records the position in the current scope's unresolvedJumps; changeOperand removes it. Nothing else writes to that map, so between them it is exactly the set of jumps the compiler still owes a target.

assertJumpsResolved requires the set to be empty before a stream is finished, and there are two such moments: leaveScope, after which a function's stream becomes a constant in the pool and nobody holds offsets into it any more, and the end of the *ast.Program arm, which is the only ending the main stream has. Reaching either with an entry left is reported as an internal compiler error naming the opcode, the offset and the line it was emitted from — no program can ask for an unpatched jump, so it is a defect in the compiler rather than in the source.

The check is there because what an unpatched jump does instead is decided by the length of the program it is in, and all three outcomes were measured:

the stream what runs
shorter than 9999 bytes jumps past the end; the run is treated as finished and the program answers whatever was last popped — a boolean, for a program whose value is an integer
longer, byte 9999 mid-instruction decodes that byte as an opcode: "unknown opcode 225: the bytecode may be damaged or decrypted with the wrong key, or it may have been built by a newer version of mutant", none of which is true
longer, byte 9999 on a boundary resumes in unrelated code and does not terminate

assertJumpsResolved takes a mark — only jumps emitted at or after it are counted — because a program is not always the outermost thing being compiled, and an enclosing construct may legitimately be holding a forward jump of its own while everything inside it is compiled.

A backward jump does not go through emitPlaceholder: a loop's head is a position already known, so emit is given the real target and there is nothing to owe.

6.4 Pop-Removal Optimisation#

Expression statements emit OpPop after the expression so the stack stays balanced. But when an expression is the last statement of a function body, the OpPop must not be emitted (the value becomes the implicit return value).

The compiler detects this with lastInstructionIs(code.OpPop) and either:

  • Calls removeLastPop() — truncates the instruction buffer by one byte.
  • Calls replaceLastPopWithReturn() — overwrites the OpPop with OpReturnValue in-place (same size, no shift needed).

6.5 Function Literal Compilation#

1. enterScope()
2. If named function: symbolTable.DefineFunctionName(name)   → OpCurrentClosure support
3. For each parameter: symbolTable.Define(param)             → LocalScope
4. Compile body block
5. If last instruction is OpPop → replaceLastPopWithReturn()
6. If last instruction is not OpReturnValue → emit OpReturn
7.  Snapshot freeSymbols    = symbolTable.FreeSymbols
8.  Snapshot numLocals      = symbolTable.numDefinitions
9.  Snapshot capturedLocals = symbolTable.CapturedLocals()   (see §7.3.1)
10. leaveScope() → captures finished instruction bytes
11. boxCapturedLocals(insts, capturedLocals) → rewrites this function's own
    OpGetLocal/OpSetLocal for captured slots to their cell forms, in place
12. For each free symbol: emitCapture(sym) → pushes captured cells onto parent stack
13. compiledFn = &CompiledFunction{Instructions, NumLocals, NumParams, CapturedLocals}
14. fnIndex = addConstant(compiledFn)
15. emit(OpClosure, fnIndex, len(freeSymbols))

6.6 Security Opcode Injection#

When injectSecurityChecks = true (enabled by EnableSecurityOpcodeInjection()):

  • After each top-level statement, maybeEmitRandomSecurityCheckOpcodes() is called. It uses crypto/rand to decide with probability 1/3 whether to emit OpChkDbg and/or OpChkSnd.
  • When ByteCode() is called, ensureRequiredSecurityCheckOpcodes() appends whichever of the two opcodes were never randomly inserted, guaranteeing both are present at least once.

The VM enforces that both opcodes exist in the instruction stream when running in password/secure mode (validated before and after execution via validateSecurityCheckOpcodes).


7. Symbol Table & Scoping#

7.1 Symbol and Scope Types#

type SymbolScope string

const (
    GlobalScope   SymbolScope = "GLOBAL"    // top-level let bindings
    LocalScope    SymbolScope = "LOCAL"     // function-local let bindings
    BuiltinScope  SymbolScope = "BUILTIN"   // built-in functions
    FreeScope     SymbolScope = "FREE"      // captured from enclosing scope
    FunctionScope SymbolScope = "FUNCTION"  // self-reference for named functions
)

type Symbol struct {
    Name  string
    Scope SymbolScope
    Index int   // offset in its storage (globals[], stack[bp+i], Free[i], Builtins[i])
}

7.2 `Define`#

func (st *SymbolTable) Define(name string) Symbol
  • If st.Outer == nil → GlobalScope, index = numDefinitions++
  • Otherwise → LocalScope, index = numDefinitions++

numDefinitions also becomes the CompiledFunction.NumLocals (the VM reserves this many stack slots for the frame).

7.3 `Resolve` and Free Variable Promotion#

func (st *SymbolTable) Resolve(name string) (Symbol, bool)

Resolution order:

  1. Look up in st.store (current scope).
  2. If not found and st.Outer != nil, recurse into outer scope.
  3. If found in outer scope:
    • GlobalScope or BuiltinScope → return as-is (loaded via OpGetGlobal / OpGetBuiltin).
    • Any other scope → call defineFree(original):
      • Appends original to st.FreeSymbols.
      • Stores a new FreeScope symbol with Index = len(FreeSymbols)-1.
      • Returns the free symbol.
    • If the original was LocalScope, also mark that slot captured on the table that owns it. A FreeScope original needs no marking here: the recursive Resolve that produced it already marked whichever table owns the slot, so the marking propagates outward on its own.

FreeSymbols is consumed at the end of function compilation (step 10 in §6.5) to emit the instructions that push captured cells onto the stack before OpClosure.

7.3.1 Boxed captures#

A captured local does not live in its frame slot; the slot holds an *object.Cell and the value is inside it. Every closure that captures the variable holds that same pointer, so there is exactly one storage location per variable per call and a write through any path is a write all of them see. This is what object.Environment.Update has always done in the tree-walking evaluator, and before it the VM captured by value: a closure's writes went into its own copy, and writing free i actually landed on local i — usually a parameter.

The capture is discovered while the inner literal is compiled, by which time the enclosing function has already emitted plain OpGetLocal/OpSetLocal for the slot. So boxCapturedLocals rewrites those to OpGetLocalCell / OpSetLocalCell once the scope closes. Only the opcode byte changes — the cell forms take the same one-byte operand — so nothing moves and no jump target, line table offset or loop patch has to be recomputed. The alternative, boxing every local, needs no pass and puts a heap allocation and an indirection on the VM's hottest path.

CompiledFunction.CapturedLocals carries the boxed slot indices, ascending, and the VM allocates the cells in callClosure after the arguments are already in their slots — which is why a captured parameter needs no special case. CapturedLocals is semantic, not debug information, and is never stripped by ByteCode.StripDebugInfo: a release build that lost it would silently stop boxing.

Cells never reach gob and are never encrypted. cell.Value is encrypted on the way in and decrypted on the way out, so the at-rest guarantee locals already had is unchanged; encrypting the cell itself would return a new object, and a new cell is a second storage location — the by-value copy the design exists to remove.

Bytecode compiled before boxing existed has an empty CapturedLocals and no cell opcodes, so nothing is boxed and OpGetFree finds a plain value and passes it through. Such a program keeps its original behaviour exactly, including the by-value capture.

Workers get their own cells. pmap, peach and spawn run the callback on another VM and another goroutine, where a shared cell is a data race rather than a shared accumulator. Each worker is handed a closure whose cells have been copied, one level deep — the same rule as the globals snapshot beside it — so a worker's writes to a captured variable stay local.

7.4 `DefineFunctionName`#

func (st *SymbolTable) DefineFunctionName(name string) Symbol

Creates a FunctionScope symbol at index 0. When the compiler resolves the function's own name inside its body, loadSymbol emits OpCurrentClosure, which pushes the currently-running closure without going through the variable system. This allows direct recursion without a free-variable round-trip.

7.5 `loadSymbol` — Scope to Opcode Mapping#

func (c *Compiler) loadSymbol(s Symbol) {
    switch s.Scope {
    case GlobalScope:   c.emit(code.OpGetGlobal,  s.Index)
    case LocalScope:    c.emit(code.OpGetLocal,   s.Index)
    case BuiltinScope:  c.emit(code.OpGetBuiltin,
                            code.BuiltinNameTableFlag|c.symbolTable.ReferenceBuiltin(s.Name))
    case FreeScope:     c.emit(code.OpGetFree,    s.Index)
    case FunctionScope: c.emit(code.OpCurrentClosure)
    }
}

emitAssignStore is the write side of the same switch, and had been an if on GlobalScope while the enum grew to five values — the defect §7.3.1 describes. It writes globals, locals (OpSetLocal, rewritten to OpSetLocalCell if the slot turns out to be boxed) and captures (OpSetFree), and refuses builtins and the function's own name, neither of which is storage.

emitCapture is the third form: the capture list OpClosure consumes wants the storage itself rather than a read through it, so a LocalScope original becomes OpCaptureLocal, a FreeScope original OpCaptureFree, and the function's own name stays OpCurrentClosure — captured by value, because it is not storage.


8. VM: Stack Machine Design#

8.1 Constants#

// global/const.go
StackSize  = 2048   // initial stack capacity (doubles on overflow)
GlobalSize = 65536  // initial globals capacity (doubles on overflow)
MaxFrames  = 2048   // initial call-stack frame capacity (doubles on overflow)

All three slices are dynamically resized using growSize (doubles until large enough) and copy. There is no hard cap beyond memory limits.

8.2 VM Struct Layout#

type VM struct {
    constants    []object.Object     // constant pool from compilation
    stack        []object.Object     // value stack; TOS = stack[stackPointer-1]
    stackPointer int                 // points one past top-of-stack

    globals      []object.Object     // global variable storage

    frames       []*Frame            // call stack
    frameIndex   int                 // index of the *next* free slot (current = frames[frameIndex-1])

    inslen       int                 // byte length of the main bytecode stream (key derivation input)
    password     string              // XOR decryption key (empty = no encryption)

    stepCount    uint64              // instructions executed so far
    // ... integrity scheduling fields (see §15) ...

    secureMode   bool                // true = fatal errors on security violations
    structDefs   map[string]any      // struct field name lists
    enumDefs     map[string]any      // enum tag name lists
    enforceSecurityCheckOpcodes bool // true when running with a password
}

8.3 Frame#

type Frame struct {
    cl *object.Closure   // currently executing closure
    ip int               // instruction pointer; starts at -1
    bp int               // base pointer: stack[bp..bp+NumLocals-1] are locals
}

func (f *Frame) Instructions() code.Instructions { return f.cl.Fn.Instructions }

The IP is initialised to -1. On every iteration the main loop increments it before reading. This means the first instruction is read at ip=0 after the first increment.

8.4 Stack Layout During a Call#

Before OpCall:
  stack[...] = captured globals / earlier temps
  stack[sp-argc-1] = closure/builtin being called   ← fn
  stack[sp-argc]   = arg[0]
  stack[sp-argc+1] = arg[1]
  ...
  stack[sp-1]      = arg[argc-1]

execCall sets: newFrame.bp = sp - argc sp = bp + fn.NumLocals (reserves local slots)

Inside function: stack[bp + 0] = local variable 0 (first parameter fills this slot) stack[bp + 1] = local variable 1 ... stack[bp + NumLocals-1] = last local stack[sp..] = temporaries for expression evaluation

On return (OpReturnValue): returnValue = pop() sp = bp - 1 (unwinds frame including the fn slot below args) push(returnValue)

On return (OpReturn / void): sp = bp - 1 push(Null)

Important: bp - 1 rewinds past the closure slot that was on the stack before the call, effectively consuming the function object itself.


9. VM: Execution Loop#

9.1 `Run()` Structure#

vm.ensureFrameBoundaries()
vm.validateSecurityCheckOpcodes("before-execution")

loop: while currentFrame.ip < len(currentFrame.Instructions()) - 1: runIntegrityProbes() // SHA-256 + CFI checks currentFrame.ip++ stepCount++

    ip  = currentFrame.ip
    ins = currentFrame.Instructions()

    opcodeByte = SecureXOROneAt(ins[ip], inslen, password, ip)
    op = Opcode(opcodeByte)

    switch op:
        ... (one case per opcode) ...

vm.validateSecurityCheckOpcodes("after-execution")

9.2 IP Advancement Protocol#

Opcodes with operands must advance the IP by the number of operand bytes consumed. This is done explicitly inside each case, after reading the operand. Pattern:

case code.OpConstant:
    constIndex, _ := code.ReadUint16(ins[ip+1:], ...)
    vm.currentFrame().ip += 2     // skip 2-byte operand
    vm.push(vm.constants[constIndex])

Jump opcodes set IP to target - 1 so the main loop's ip++ at the top of the next iteration lands exactly at the target byte:

case code.OpJump:
    pos := int(readUint16(...))
    vm.currentFrame().ip = pos - 1   // loop will +1 before fetching

OpJumpFalse still advances by 2 (ip += 2) when the condition is truthy (i.e., the jump is not taken), to skip past the operand and continue linearly.

9.3 Per-Opcode Behaviour Summary#

Opcode IP advance Push/Pop details
OpConstant +2 push constants[idx]
OpPop 0 pop (discard)
OpAdd/Sub/Mul/Div 0 pop b, pop a, push result
OpMod 0 pop b, pop a, push a % b
OpTrue/False/Null 0 push singleton
OpEqual/UnEqual/Greater 0 pop b, pop a, push Boolean
OpMinus 0 pop a, push -a
OpBang 0 pop a, push !a
OpBitAnd/BitOr/BitXor 0 pop b, pop a, push a op b (INTEGER only)
OpShiftLeft/ShiftRight 0 pop b, pop a, push a op b; b < 0 errors
OpBitNot 0 pop a, push ~a (INTEGER only)
OpJump sets ip = target-1 —
OpJumpFalse +2 (not taken) or ip=target-1 pop condition
OpGetGlobal +2 push decrypted globals[idx]
OpSetGlobal +2 pop, encrypt, store to globals[idx]
OpGetLocal +1 push decrypted stack[bp+idx]
OpSetLocal +1 pop, encrypt, store stack[bp+idx]
OpArray +2 pop n, push Array
OpHash +2 pop n (must be even: k,v pairs), push Hash
OpIndex 0 pop index, pop obj, push result
OpCall +1 pop argc args + fn; push new frame
OpReturnValue — pop return val; pop frame; push val
OpReturn — pop frame; push Null
OpGetBuiltin +1 push the builtin resolved from BuiltinNames[idx & 0x7FFF]
OpClosure +3 pop numFree values; push Closure
OpGetFree +1 push Free[idx], through the cell if it is one
OpSetFree +1 pop, encrypt, store into Free[idx].Value
OpGetLocalCell +1 push decrypted cell(bp+idx).Value
OpSetLocalCell +1 pop, encrypt, store into cell(bp+idx).Value
OpCaptureLocal +1 push the *Cell in stack[bp+idx] itself
OpCaptureFree +1 push Free[idx] unread
OpCurrentClosure 0 push current Closure
OpChkDbg 0 detect debugger → error or warn
OpChkSnd 0 detect sandbox → error or warn
OpBreak 0 push Break{} sentinel
OpContinue 0 push Continue{} sentinel
OpMakeStruct +3 pop fieldCount values; push Struct
OpGetField +2 pop struct; push field value
OpSetField +2 pop value, pop struct; set field; push struct
OpEnumValue +4 push EnumValue

9.4 Global & Local Encryption at Runtime#

Values are encrypted on write and decrypted on read for globals and locals when a password is set:

// Write path
vm.globals[idx] = vm.encryptForStorage(vm.pop())
vm.stack[bp+idx] = vm.encryptForStorage(vm.pop())

// Read path
vm.push(vm.decryptForUse(vm.globals[idx]))
vm.push(vm.decryptForUse(vm.stack[bp+idx]))

mutil.EncryptObject / mutil.DecryptObject wrap the object in / unwrap from *object.Encrypted. If encryption fails (e.g., unsupported type), the object is stored/returned as-is.

A boxed local slot holds an *object.Cell, and the cell is deliberately one of the types EncryptObject passes through unchanged. Encryption returns a new object, and a new cell is a second storage location — the by-value capture boxing exists to remove. cell.Value goes through the same encrypt-on-write and decrypt-on-read as any other local, so nothing is left in the clear.


10. Closures & Free Variables#

10.1 Compilation#

Given:

let outer = fn() {
    let x = 5;
    let inner = fn() { x };
    inner
};

The compiler for inner:

  1. Sees x → resolves to outer's LocalScope.
  2. Calls defineFree(x_local) → FreeSymbols = [{Name:"x", Scope:LOCAL, Index:0}].
  3. Marks outer's slot 0 captured, so outer's CapturedLocals is [0].
  4. After body compilation, FreeSymbols has one entry.
  5. Outer's own stream is rewritten: the OpSetLocal 0 it emitted for let x = 5 becomes OpSetLocalCell 0. Same width, so nothing after it moves.
  6. In the parent scope, emitCapture for each free symbol emits OpCaptureLocal 0 — the cell itself, not a copy of the value.
  7. Emits OpClosure <innerFnIdx> 1.

Bytecode for outer (CapturedLocals: [0], so the VM boxes slot 0 at frame entry):

0000 OpConstant 0       ; integer 5
0003 OpSetLocalCell 0   ; let x = 5, written through the cell
0005 OpCaptureLocal 0   ; push slot 0's cell (the capture for inner)
0007 OpClosure 1 1      ; wrap constants[1] (inner fn) with 1 free var
0011 OpSetLocal 1       ; let inner = <closure>; nothing captures inner
0013 OpGetLocal 1       ; return inner
0015 OpReturnValue

10.2 Runtime: `pushClosure`#

func (vm *VM) pushClosure(constIndex, numFree int) error {
    fn   := vm.constants[constIndex].(*object.CompiledFunction)
    free := make([]object.Object, numFree)
    for i := 0; i < numFree; i++ {
        free[i] = vm.stack[vm.stackPointer-numFree+i]
    }
    vm.stackPointer -= numFree
    closure := &object.Closure{Fn: fn, Free: free}
    return vm.push(closure)
}

The captured values are read from the stack in forward order (they were pushed in forward order by the parent's emitCapture calls). What is copied into Free is the pointer each OpCaptureLocal / OpCaptureFree pushed, so the closure and the frame end up naming the same cell.

10.3 Accessing Free Variables#

case code.OpGetFree:
    freeIndex := ReadUint8(...)
    captured := vm.currentFrame().cl.Free[freeIndex]
    if cell, ok := captured.(*object.Cell); ok {
        captured = cell.Value
    }
    vm.push(vm.decryptForUse(captured))

Free holds cells, and OpSetFree writes back through them — so a closure's write is visible to the enclosing frame and to every other closure over the same variable, which is what makes an accumulator over a callback work.

The type test is not a guess. Nothing a program can compute is a *Cell, and the only two entries that are not cells are decidable: the enclosing function's own name, captured by value with OpCurrentClosure for recursion, and every entry in bytecode compiled before boxing existed.


11. Structs at the Bytecode Level#

11.1 Definition (compile-time only)#

struct Point { x, y }

No bytecode is emitted. The compiler records:

c.structDefinitions["Point"] = []*ast.Identifier{{Value:"x"}, {Value:"y"}}

This map is carried in ByteCode.StructDefs and loaded into vm.structDefs at VM construction.

11.2 Construction#

let p = Point { x: 1, y: 2 };

Compiled to:

OpConstant <idx of "Point">   ; push type name string
OpConstant <idx of 1>          ; push field value 1
OpConstant <idx of 2>          ; push field value 2
OpMakeStruct <typeIdx> 2       ; typeIdx = constant index of "Point", fieldCount = 2
OpSetGlobal <p_idx>

OpMakeStruct runtime behaviour:

  1. Reads typeIdx (2 bytes) and fieldCount (1 byte).
  2. Looks up constants[typeIdx] → must be *object.String containing the struct name.
  3. Looks up vm.structDefs[typeName] → ordered field name list.
  4. Verifies len(fieldNames) == fieldCount (error if not).
  5. Pops fieldCount values from stack in reverse (last field popped first), mapping them to fieldNames[fieldCount-1-i] so the first field value pushed maps to the first field name.
  6. Creates &object.Struct{TypeName, Fields: map[string]Object{...}}.

11.3 Field Access / Mutation#

p.x          ; OpGetField
p.x = 10    ; OpSetField

Field names are stored as string constants in the pool. The opcodes carry the constant index:

OpGetField <nameIdx>   ; pop struct, push struct.Fields[constants[nameIdx]]
OpSetField <nameIdx>   ; pop value, pop struct, set field, push struct back

Accessing a non-existent field pushes Null (no error). Setting a field on a non-struct object is a runtime error.


12. Enums at the Bytecode Level#

12.1 Definition (compile-time only)#

enum Color { Red, Green, Blue }

No bytecode emitted. Compiler records:

c.enumDefinitions["Color"] = []string{"Red", "Green", "Blue"}

Carried in ByteCode.EnumDefs → vm.enumDefs.

12.2 Tag Reference#

let c = Color.Red;

Compiled to:

OpEnumValue <typeIdx> <tagIdx>

Both operands are 2-byte constant indices pointing to string constants "Color" and "Red" respectively.

OpEnumValue runtime behaviour:

  1. Reads typeIdx (2 bytes) and tagIdx (2 bytes); advances IP by 4.
  2. Looks up both strings in the constant pool.
  3. Looks up vm.enumDefs[typeName] → []string.
  4. Scans the tag list for the matching string; ordinal = position index (0-based).
  5. Returns error if the tag is not found.
  6. Pushes &object.EnumValue{TypeName, Tag, Value: &Integer{int64(ordinal)}}.

13. Control Flow#

13.1 For Loop Structure#

for (let i = 0; i < 10; i = i + 1) { body }

Compiler emits (pseudocode):

[init: OpConstant 0, OpSetLocal 0]           ; let i = 0
[conditionStart: label A]
[condition: push (i < 10)]
OpJumpFalse → loopEnd                        ; if false, exit
[body: ...]
[OpPop if last statement is expression]
[post: i = i + 1]
OpJump → A                                   ; loop back
[loopEnd: label B]

A missing condition (for(;;)) emits OpTrue as the condition. A missing init or post simply emits nothing for those phases.

13.2 `LoopContext` and Back-Patching#

type LoopContext struct {
    breakPositions    []int   // byte offsets of OpJump instructions from break
    continuePositions []int   // byte offsets of OpJump instructions from continue
    pendingAtEntry    int     // operands the enclosing expressions held at entry
}

When break or continue is encountered:

  • Emit one OpPop for every operand the enclosing expressions have pushed since the loop was entered — c.pendingOperands - ctx.pendingAtEntry of them. Each target a break or a continue can reach reads the stack at the depth the loop was entered with, so a jump out of the middle of an expression has to drop that expression's operands on the way out. Without the pops, let t = [0, if (c) { break; } else { 1 }]; leaves the array's first element behind and the OpPop meant for a for-in's cursor takes it instead.
  • Emit OpJump 9999 (placeholder), after those pops.
  • Append the emitted position to ctx.breakPositions / ctx.continuePositions.

After the loop body and post-increment are fully compiled:

  • Patch all breakPositions → loopEnd address.
  • Patch all continuePositions → conditionStart address (re-runs the condition check, which then falls through to the post-increment).

At runtime, OpBreak / OpContinue push sentinel objects (&object.Break{} / &object.Continue{}). These are only meaningful in the evaluator (tree-walk) path; the compiled VM path uses only the jump instructions.

13.3 While Loop Structure#

while (c) { body }

The same shape with the init and post phases absent:

[conditionStart: label A]
[condition: push c]
OpJumpFalse → loopEnd
[body: ...]
OpJump → A
[loopEnd: label B]

continue back-patches to A, which for this loop is the condition rather than a post section — there is no post section to run.

13.4 For-In Loop Structure#

for (k, v in xs) { body }
[iterable: push xs]
OpIterInit                         ; [iter]
[head: label A]
OpIterNext → loopEnd, binds        ; [iter] → [iter, key, value], or jump
[bind value: OpSetLocal / OpSetGlobal]
[bind key, when the loop binds two names]
[body: ...]
OpJump → A
[loopEnd: label B]
OpPop                              ; drop the cursor

break back-patches to B — before the OpPop — so the cursor is dropped on every exit path. continue back-patches to A, so the advance is this loop's post section.

Two things about the body differ from §13.1 and are not stylistic. The body must be stack-neutral: the cursor lives underneath whatever the body leaves behind, so a body that leaks one value per iteration has the next OpIterNext reading that leftover as the cursor. And binds is read through the VM's operand reader rather than indexed out of the instruction stream directly — a raw byte index returns the obfuscated byte, which silently made a two-binding loop bind one.

13.5 Match Expressions#

match (subject) { p1 | p2 => a, p3 => b, _ => c }

A match is an expression, so every path through it leaves exactly one value:

[subject: push it once]
[per alternative]
  OpDup                            ; [subject, subject]
  [pattern]
  OpEqual                          ; [subject, bool]
  OpJumpFalse → next alternative
[on a match]
  OpPop                            ; drop the subject
  [arm body]
  [leave exactly one value]
  OpJump → end
[no arm matched, and no `_` arm]
  OpMatchFail                      ; pops the subject and raises
[end]

The subject lives on the stack behind OpDup rather than in a scratch local because SymbolTable.Define never reuses a slot: a temporary per match would spend one of the 256 one-byte local slots each time and eventually panic inside code.Make. A wildcard arm compiles as the body alone — no compare, no jump — and its presence is what omits the trailing OpMatchFail.

"Leave exactly one value" is decided from the syntax, not from the last instruction emitted: if the arm body's last statement is an expression statement, its OpPop is removed, and otherwise OpNull is emitted. The instruction-stream test that §6.4 describes cannot be used here, because a for (v in xs) also ends in an OpPop — the one dropping the cursor — and removing that one leaks the iterator as the value of the arm. if branches go through the same rule for the same reason.


14. Operand Encryption#

14.1 Key Derivation#

When a password string is provided to the VM, every opcode byte and every operand byte is XOR-encrypted using security.SecureXOR*. The key material is derived from:

  • The password string.
  • inslen — the total byte length of the main instruction stream.
  • offset — the absolute byte position of the byte being encrypted/decrypted.

This makes the key unique per byte position in the stream, so the same operand value at different offsets produces different encrypted bytes.

14.2 Encryption in the VM Loop#

Every single byte read from the instruction stream goes through decryption:

// Opcode byte
opcodeByte, _ = security.SecureXOROneAt(ins[ip], int64(vm.inslen), vm.password, int64(ip))

// 2-byte operand
value, _ = code.ReadUint16(ins[ip+1:], int64(vm.inslen), vm.password, int64(ip+1))

// 1-byte operand
value, _ = code.ReadUint8(ins[ip+1:], int64(vm.inslen), vm.password, int64(ip+1))

The length and offset arguments in ReadUint16 / ReadUint8 must match the actual position of those bytes in the stream. A mismatch produces a wrong value silently — there is no authentication tag.

14.3 No Encryption Path#

When password == "" (the default for vm.New):

  • security.SecureXOROneAt(b, ...) is a no-op that returns b unchanged.
  • ReadUint16 / ReadUint8 still call the XOR functions but effectively just do binary.BigEndian.Uint16.

15. Runtime Security Features#

15.1 Integrity Probes#

The VM maintains SHA-256 checksums of every CompiledFunction's instruction slice:

frameIntegrity map[*object.CompiledFunction][32]byte

The checksum is computed at VM construction for the main function. For closures, it is computed when the frame is first pushed.

Probe schedule (jittered to frustrate timing analysis):

integrityEvery  = 64 steps       (base interval for instruction-hash checks)
integrityJitter = XorShift64 PRNG seeded from SHA-256(mainInstructions)

nextProbeInterval = integrityEvery + (jitter % 31) nextSweepInterval = 251 + (jitter % 83)

At each probe:

  1. Recompute SHA-256(frame.Instructions()).
  2. Compare to the stored checksum.
  3. If mismatch → security.RecordIntegrityFailure + ApplyTamperResponse.

15.2 Control-Flow Integrity (CFI)#

frameBoundaries map[*object.CompiledFunction]map[int]struct{}

Before execution begins, buildInstructionBoundaries walks the encrypted instruction stream and records the byte offset of every opcode (i.e., every legal IP value). Jump targets are validated by checking whether the resolved ip (or ip+1 to account for the pre-decrement protocol) is in this set.

CFI is only active when password != "".

15.3 Security Opcodes#

Opcode Trigger Secure mode behaviour Non-secure mode
OpChkDbg security.IsDebuggerPresent() returns true Return ErrDebuggerDetected Log warning to stderr, continue
OpChkSnd security.IsSandboxed() returns true Return ErrSandboxDetected Log warning to stderr, continue

secureMode = true for any VM constructed with NewWithPassword*. secureMode = false for plain vm.New.

15.4 Tamper Response#

security.ApplyTamperResponse(event, stage, secureMode, err) is the central tamper handler:

  • If secureMode → return the error immediately (halts the VM).
  • If not secureMode → log a warning, return nil (execution continues).

15.5 Secure Memory Cleanup#

vm.CleanupRuntimeSensitiveData(clearGlobals bool, clearConstants bool)
vm.CleanupSensitiveData(clearGlobals bool)  // alias: always clears constants

These zero-out the stack, optionally the globals, and optionally zero and nil the CompiledFunction.Instructions byte slices using security.SecureZero (which prevents the compiler from optimising the wipe away). The password string is also cleared.

Intended to be called immediately after vm.Run() returns, before the VM object is GC'd.

15.6 Polymorphic Marker Stripping#

The marker never reaches the VM. generator.encode removes it before anything is written, using the level Compiler.PolymorphicLevel() reports, so what the runtime loads is the instruction stream and nothing else.

The VM used to strip it itself, by guessing: it read the last two bytes and treated [0xFF, n<=10] in either order as a marker. Ordinary bytecode produces that pattern on its own -- OpConstant 255 encodes as 00 00 FF, and a following OpPop makes the last two bytes FF 01 -- so real programs had two real bytes cut off and died on "not enough bytes for operand". There is nothing in the format that distinguishes a marker from instruction bytes that end the same way, which is why the decision belongs to the compiler that knows, not to the reader that has to guess.

If a marker ever did survive, 0xFF is not a defined opcode, so the VM reports an undecodable instruction rather than silently running a shortened program.


16. Polymorphic Mutation Engine#

All four stages run at any non-zero mutation level: InsertNOPs, InsertDeadCode, MutateOpcodes and RandomizeConstants. --mutation 0 is the only setting that leaves the program byte-identical.

ReorderInstructions used to be a fifth field with no implementation behind it. It has been removed: reordering needs a reverse mapping the VM has no way to carry, and a named stage that does nothing is how a stale opcode list once survived unnoticed.

Until this was fixed, RandomizeConstants was the only stage that ran and it was gated at level 6 while the CLI defaulted to 5 -- so the default ran the engine and emitted unmutated bytecode.

16.1 Engine Configuration#

type PolymorphicEngine struct {
    mutationLevel int          // 0 = disabled, 1–10 = intensity
    randomSeed    int64        // for reproducible builds
    rng           *mathrand.Rand
}

Constructed with compiler.EnablePolymorphism(level) (random seed via crypto/rand) or EnablePolymorphismWithSeed(level, seed) (deterministic).

16.2 Mutation Pipeline#

ByteCode.Mutate(bc):
    if InsertNOPs                     → spliceFillers(bc, config)     // every stream, jumps repointed
       or InsertDeadCode
    if RandomizeConstants             → randomizeConstantPool(bc)     // pool operands rewritten
    if MutateOpcodes                  → mutateOpcodes(bc)             // must be last
    append PolymorphicMarker

NOP insertion and dead-code insertion are two stages but one pass. Both splice a block at an instruction boundary and both need every jump in the stream repointed afterwards, so running them separately would decode the boundaries and rewrite the jumps twice to reach the same stream. spliceFillers picks a generator per insertion point from whichever stages are enabled.

The order is load-bearing. NOP insertion and constant-pool randomization both walk the instruction stream by operand width, so they have to run while the opcode bytes still mean what code.Lookup says. Opcode remapping destroys that and therefore runs last. It used to sit in the middle, which would have left the constant-pool pass parsing a remapped stream.

16.3 NOP Insertion#

Splices a stack-neutral pair in after a random subset of instruction boundaries:

OpNull  + OpPop
OpTrue  + OpPop
OpFalse + OpPop

Rate: level × 1.5 % of instruction boundaries, drawn from the seeded RNG, so --seed reproduces the build. Applied to the main stream and to every CompiledFunction in the constant pool.

Two properties make this safe, and the absence of either is why the stage was gated off rather than merely unfinished:

  • Jump targets are repointed. OpJump and OpJumpFalse carry absolute offsets (code.JumpOperands), so one inserted byte ahead of a target redirects it into the middle of an instruction. Every target is remapped to its instruction's new offset. The old implementation did not do this at all.
  • Every variant is stack-neutral. A bare OpPop used to be one of four choices, commented "safe if stack has something". At an arbitrary instruction boundary the stack holds the expression in progress, so that variant discarded a live value -- randomness decided whether the program was correct.

Nothing is inserted after the final instruction of a stream: the last value popped is what LastPoppedStackElement reports, and a trailing push/pop pair would make every program's result null.

If a stream cannot be decoded end to end, or a jump target is not an instruction boundary, or a new offset will not fit a uint16, padding is declined and the stream is returned untouched. A half-rewritten stream is not recoverable.

16.4 Opcode Remapping#

A Fisher-Yates shuffle of every opcode value using the deterministic RNG creates a bijective mapping original → shuffled. Every opcode byte in the instruction stream and in all CompiledFunction constants is rewritten through this mapping; operand bytes are left exactly as they were.

The rewrite walks by operand width. It used to walk one byte at a time and replace anything matching a defined opcode, which rewrote operand bytes as though they were opcodes -- OpConstant 5 encodes as 00 00 05, and all three of those bytes are defined opcode values.

The opcode set comes from code.AllOpcodes() rather than a list kept in the engine, which is what keeps the mapping bijective: a hand-written list that falls behind the code package produces a partial remap, where the opcodes it covers take new values while the ones it missed keep theirs, and the two collide. That drift had already happened (OpGreaterEqual, OpSetIndex) and stayed invisible because the stage was off.

Bytes that are not defined opcodes map to themselves in both directions, so an undecodable byte stays undecodable instead of being laundered into a valid instruction.

Reversal. The inverse table is shipped in ByteCode.OpcodeMap, 256 entries indexed by the byte found in the stream:

type ByteCode struct {
    ...
    OpcodeMap []byte   // mutated byte → real opcode; nil when not remapped
}

It travels with the program rather than being re-derived from the seed, because nothing that runs a .mu knows the seed. It is gob-encoded with the rest of ByteCode; an older file simply has no entry and decodes to nil, which is the unmutated case.

The VM applies it at every point a byte becomes an opcode (VM.decodeOpcode), which is three places, not one:

Site What breaks without it
execLoop fetch wrong instruction, wrong operand width
buildInstructionBoundaries control-flow integrity reports valid jumps as tampering
scanInstructionsForSecurityCheckOpcodes a program carrying OpChkDbg/OpChkSnd is reported as having lost them

A table that is not exactly 256 entries is treated as absent (normalizeOpcodeMap) rather than indexed past its end.

Remapping is all or nothing: if any stream cannot be decoded, no stream is rewritten and no table is shipped. A program with some streams remapped and some not cannot be described by one inverse table.

Compatibility: a remapped .mu requires a runtime that understands OpcodeMap. Standalone releases embed their runtime, so they are inherently matched; a .mu handed to an older CLI is not, and the file format carries no version negotiation.

16.5 Constant Pool Randomisation#

Fisher-Yates shuffle of the constant pool using the deterministic RNG, followed by a rewrite of every operand that indexes the pool -- in the main instruction stream and in each CompiledFunction constant.

The operands to rewrite are read from code.ConstantOperands, which names OpConstant, OpClosure, OpMakeStruct, OpGetField, OpSetField and both operands of OpEnumValue. The rewrite walks instructions by operand width rather than scanning for opcode bytes, because an operand byte can hold any value and would otherwise be misread as an opcode.

Missing an operand from ConstantOperands does not fail loudly: the operand still lands on a real pool entry, just the wrong one, so the program compiles clean and misbehaves inside the VM. TestWideOperandsAreClassified pins every two-byte operand as a constant-pool index, an instruction offset, or explicitly neither, so a new opcode cannot be added without that call being made for both ConstantOperands and JumpOperands.

16.6 Polymorphic Marker Format#

[... instructions ...][ 0xFF ][ level_byte ]

The marker is appended to Instructions after all mutations. It is encrypted together with the rest of the bytecode when a password is used.

Detection:

compiler.DetectPolymorphicLevel(instructions) int

Returns the level (0–10) or 0 if no valid marker is found.


17. Object Type System#

17.1 Core Interface#

type Object interface {
    Type()    ObjectType   // string constant, e.g. "INTEGER"
    Inspect() string       // human-readable value
}

17.2 Object Types#

ObjectType constant Go struct Notes
INTEGER_OBJ *Integer Value int64
FLOAT_OBJ *Float Value float64
BOOLEAN_OBJ *Boolean Value bool; singletons global.True, global.False
NULL_OBJ *Null singleton global.Null
STRING_OBJ *String Value string
BYTES_OBJ *Bytes Value []byte — a byte buffer; Inspect() is full hex
ARRAY_OBJ *Array Elements []Object
HASH_OBJ *Hash Pairs map[HashKey]HashPair
COMPILED_FN_OBJ *CompiledFunction Instructions, NumLocals, NumParams, CapturedLocals
CLOSURE_OBJ *Closure Fn *CompiledFunction, Free []Object — cells (§7.3.1)
CELL_OBJ *Cell Value Object — a captured variable's storage; runtime-only, never gob, never encrypted
BUILTIN_OBJ *Builtin Fn func(args ...Object) Object
FUNCTION_OBJ *Function AST-level function (evaluator path only)
RETURN_VALUE_OBJ *ReturnValue Sentinel (evaluator path only)
ERROR_OBJ *Error Message string
STRUCT_OBJ *Struct TypeName string, Fields map[string]Object
ENUM_VALUE_OBJ *EnumValue TypeName, Tag string, Value *Integer (ordinal)
ENCRYPTED_OBJ *Encrypted Value []byte — XOR-encrypted payload
BREAK_OBJ *Break Control-flow sentinel
CONTINUE_OBJ *Continue Control-flow sentinel
QUOTE_OBJ *Quote Macro / meta-programming AST wrapper
MACRO_OBJ *Macro Macro object (evaluator path only)
LUA_PATCH_OBJ *LuaPatch Lua security patch payload

17.3 `HashKey` Interface#

For types usable as hash keys, the object must implement:

type Hashable interface {
    HashKey() HashKey
}

type HashKey struct {
    Type  ObjectType
    Value uint64
    Text  string
}

Implemented by Integer, Float, Boolean, String and Bytes. Two keys are one entry only when every field is equal, and every field is exact: a boolean, an integer or a float's bits sit in Value, a string sits whole in Text, and a buffer's SHA-256 sits in Text -- never the buffer, which may be classified plaintext and would otherwise be copied into a string nothing can zero. Before 2.6.0 a string or a buffer was keyed by its FNV-1a-64 digest alone, so two values with one digest were one entry.

17.4 `CompiledFunction`#

type CompiledFunction struct {
    Instructions   code.Instructions // bytecode for this function's body
    NumLocals      int               // number of local variables (stack slots)
    NumParams      int               // number of parameters
    CapturedLocals []int             // slots an inner function closes over (§7.3.1)
}

NumLocals includes the parameters (parameters are the first NumParams locals). The VM advances stackPointer by NumLocals when entering a frame to pre-allocate the local variable slots, then boxes the slots named in CapturedLocals — after the arguments are already in place, so a captured parameter needs no special case.

CapturedLocals is semantic, not debug information, and is never stripped by ByteCode.StripDebugInfo: a release build that lost it would silently stop boxing and resume capturing by value. Empty means "this function captures nothing", which is also what every artifact compiled before boxing existed says, and correctly so.


18. VM Constructor Variants#

Constructor Password SecureMode Globals Notes
vm.New(bc) — true fresh Plain execution; no CFI
vm.NewWithPassword(bc, pw) ✓ true fresh Full security; enforces security opcodes
vm.NewWithPasswordMode(bc, pw, mode) ✓ configurable fresh
vm.NewWithGlobalStore(bc, globals) — true provided REPL: share globals across compilations
vm.NewWithGlobalStoreAndPassword(bc, globals, pw) ✓ true provided
vm.NewWithPasswordAndGlobalStore(bc, pw, globals) ✓ true provided
vm.NewWithPasswordAndGlobalStoreMode(bc, pw, globals, mode) ✓ configurable provided Maximum flexibility

"SecureMode = true" means security violations abort execution. "false" means they emit a warning to stderr and continue.


19. Common Pitfalls#

P1: Reading Encrypted Operands Without Decryption#

Wrong:

idx := binary.BigEndian.Uint16(ins[ip+1 : ip+3])

Correct:

idx, err := code.ReadUint16(ins[ip+1:], int64(vm.inslen), vm.password, int64(ip+1))

Applies to every operand read inside the VM's Run() loop, including opcode bytes themselves.

P2: Jump Off-by-One#

Jumps set ip = target - 1, not ip = target. The main loop does ip++ before fetching. If you set ip = target, the VM skips the first byte of the target instruction.

P3: `OpHash` Operand is 2× Pair Count#

OpHash n expects n/2 key-value pairs, so n is always even. The compiler emits len(node.Pairs) * 2. The VM pops exactly n objects to build the hash. Confusing pair count with element count causes a stack underflow.

P4: Struct Field Pop Order#

OpMakeStruct pops fields in reverse order (last field first). The compiler must push field values in forward declaration order. The mapping inside the VM is:

fieldName := fieldNames[fieldCount-1-i]  // i=0 → last field; i=fieldCount-1 → first field

P5: `OpSetField` Consumes and Re-Pushes the Struct#

stack: [..., structObj, value]
OpSetField: pop value, pop struct, mutate struct.Fields[name], push struct

The struct reference is consumed and re-pushed. Code after OpSetField has the (mutated) struct on top of the stack. If the mutation result is not needed, emit OpPop to discard it.

P6: `inslen` Must Match the Main Function's Byte Length#

The inslen field used in all XOR decryption calls is set once to len(bc.Instructions) (the main function's length) and never changes, even when executing inner closures. Inner functions' operands are decrypted with the same key parameter. This is by design — changing it for inner functions would break operand decryption.

P7: Do Not Disassemble Encrypted Bytecode with `Instructions.String()`#

Instructions.String() uses the unencrypted ReadOperands path. On encrypted bytecode it will produce garbage output or crash. Use it only in tests or non-password compilation paths.

Related reading

On this page

Open in GitHub ↗