CandleZip: Agentic Compression for Auditable Intelligence

# CandleZip: Agentic Compression for Auditable Intelligence

CandleZip exists to turn the well-known statement "compression = intelligence" into a falsifiable
engineering instrument. The project frames every tool call as a priced entropy sink:
an agent should only spend resources when those resources route chaos(entropy) out of
the data stream. The result is a research-grade benchmark directly measures intelligence, and emits an integer-bit
signal you can trust, cite, and extend. 

DOI: [10.5281/zenodo.17282860](https://doi.org/10.5281/zenodo.17282860)

---

## 1. Why CandleZip Exists

1. **Compression as intelligence, made concrete.** CandleZip operationalizes the
   long-standing view ([Mahoney](https://mattmahoney.net/dc/dce.html#Section_14), [Hutter](https://www.hutter1.net/ai/uaibook.htm#oneline), MDL) that predictive skill equals entropy
   reduction. We expand it agentically: an intelligent policy routes entropy into
   controllable sinks (tools, retrieval corpora, code interpreters) and earns credit
   only when the routed bits beat their price. 
2. **A small, falsifiable law.** The Unified Efficiency Principle defines intelligence
   as entropy reduced per priced resource. CandleZip instantiates that law with an
   entropy coder so that every gain is logged in whole bits, complete with
   shipped-overhead accounting. If the agent cheats—by leaking oracles or skipping
   decode-safety—the ledger will expose it.
3. **A bridge between theory and practice.** The same audit trail applies to LLMs,
   classical compressors, and humans (via a UI agent). Compression becomes a
   substrate-agnostic intelligence instrument rather than an abstract metaphor.
---

## 2. What is Intelligence, and how do you measure it?

The concept of intelligence = compression is not new, it extends work from early information theorists, and in particular Mahoney.
However, practical measures of intelligence **had** yet to be established on this principle. 

We use “definition” in an operational, information-theoretic sense: an agent is
intelligent to the extent that, under decode-safety and priced side-information, its
actions reduce the expected codelength of its own sensor/working history per unit
cost. This criterion is intended as necessary-for-competence and empirically useful
—not metaphysically sufficient—so non-adaptive dissipative systems are not by this
definition considered significantly Intelligent (example: Fridge).

Therefore, Intelligence can be measured by lossless compression, where reduced file
size is quite literally entropy reduction (more precisely, relocation).

This is substrate Agnostic--human intelligence can be measured using Candlezip, by
using a UI python script, rather than the LLM Agent script provided--this is provided and results are detailed. 
there is a working demonstration of this below.

**How CandleZip measures it.**
- We track **gross entropy reduction.** _ER_: the integer-bit drop between
  the baseline stream and the stream conditioned on accepted hints. (Entropy Reduction=ER)
- We subtract **shipped overhead** (gate records, hint payloads, headers) to obtain
  **net savings**. Gross and net are both published so audits can price resources
  differently.
- We report **ROI** as bits saved per priced resource (time, bytes, $), yielding an
  operational intelligence score consistent with the representation theorem in the
  accompanying preprint.

## Human-in-the-Loop (HITL) Usage

You can run CandleZip with a human-in-the-loop agent to have a person provide predictions (instead of an LLM). This is useful for benchmarking human vs. model entropy reduction per cost. The repository includes a drop-in HITL agent (`agent/agent_human.py`) that launches a web UI and returns the human's plaintext continuation to CandleZip.

Example:

```bash
# Self-test with human in the loop
./target/release/candlezip.exe --backend smollm --agent --scan \
  --scan-lookahead 512 --context 512 --reprime-interval 512 \
  --scan-agent-script agent/agent_human.py --scan_max_steps 12 \
  self-test benchmarks/your_file.txt
```

We also provide a screenshot of the HITL UI (`./imgs/hitl_image.png`) to demonstrate how MCP tools are made human-accessible inside the interface.

![HITL UI](./imgs/hitl_image.png)

## DOI:
[10.5281/zenodo.17282860](https://doi.org/10.5281/zenodo.17282860)

---

## 3. System Overview — From Philosophy to Implementation

CandleZip implements a sink-inclusive minimum description length workflow (SIMDL).
The system treats tools as priced observation channels (entropy sinks) and measures
the agent's effectiveness by how many bits the agent saves per unit cost. Key
concepts are:

- **Entropy routing.** Each chunk is encoded twice (baseline vs. tool-conditioned).
  The gate keeps a hint if its measured gross drop exceeds its priced cost.
- **Decode-safe accounting.** All information required for deterministic decode is
  either cached or shipped and debited explicitly; nothing is "free" just because a
 

... (truncated)