path: root/readme.md

# `trbldoc`

A really awful documentation generator.

Trbldoc scans source files for comments, extracts them as content chunks,
passes each chunk through a renderer, groups the results by namespace,
and assembles an HTML site.

### Dependencies

| Tool | Purpose |
|------|---------|
| Any POSIX shell (`ash`, `dash`, `sh`, `bash`, ...) | Runtime |
| `awk`, `sed` (with `-E`), `grep`, `find`, `mktemp`, `xargs` | Standard Unix tools |

That's it. Really. Runs under busybox systems, no GNU required!

## Installation

Drop `trbldoc.sh` into your `$PATH` and mark it executable:

```bash
sudo cp trbldoc.sh /usr/local/bin/trbldoc
sudo chmod +x /usr/local/bin/trbldoc
```

## Usage

```
trbldoc [-s <source1> -s <source2> ...] [-o <dest>] [-l <layout>] [-f ext;start;end] [-R prefix=script;opts] [-C] [source ...]
```

* `-s <source>` - Add a folder or file to scan. Repeatable.
* `-o <dest>` - Output directory (default: `.trblcache/built`).
* `-l <layout>` - Use an alternative `main.layout` template file instead of the built-in default.
* `-f ext;start;end` - Register a custom file extension with AWK-regex delimiters.
* `-R prefix=script;opts` - Set a resolver used for `@prefix/...` reference tags.
* `-C` - Force a clean rebuild — wipes cached chunks and timestamps before scanning.

### Environment variables

`trbldoc` allows you to use some environment variables to provide more information
while rendering.

| Variable | Description |
|----------|-------------|
| `TRBLDOC_SRC_REPO` | Repository URL substituted for `{{ src_repo }}` in doc comments. Falls back to `git remote get-url origin` if not provided. |
| `TRBLDOC_MD` | Override the Markdown renderer used for `.md` files and `@exec md` chunks. Defaults to `md2html --github --fpermissive-autolinks --ftables`. |
| `TRBLDOC_RST` | Override the ReStructured Text renderer used for `.rst` files and `@exec rst` chunks. Defaults to `%(body)s`. |
| `TRBLDOC_JOBS` | Number of parallel jobs used for extraction and rendering. Defaults to the detected processor count. |

### Quick start


Scan the `src/` folder and write output to `docs/`

```bash
trbldoc -s src -o docs
```

Include the source repository URL in generated pages

```
TRBLDOC_SRC_REPO=https://github.com/you/repo trbldoc -s src -o docs
```

## Writing doc comments

A doc comment is a block comment whose opening tag is followed _on the same
line_ by the renderer command to use for that chunk. `trbldoc` includes the
following markup format -> html mapping:

| alias | program |
| --- | --- |
| `md` | `md2html` |
| `rst` | `docutils` |

### C / C++ / SQL

```c
/* md
# My Function

Does something useful.
*/
```

### Lua

```lua
--[[ md
## Overview

Brief description here.
]]
```

### Python

```python
'''md
## Helper

Processes the input.
'''
```

`trbldoc` lets you call _arbitrary bash_(**!**) from source code comments.
Some users may be alarmed, or believe this is a security vulnerability.
The reality at the time of publishing is that it is almost all
software documentation is generated on computers controlled by
the same person or people that wrote the source code, and published on the internet.
If you can trust someone to write some source code that's running on your machine,
you can probably trust them to run some more code to generate documentation.
If you are interested in letting third parties generate documentation
without reading the code, or you want to generate documentation from untrusted input,
`trbldoc` is not right for your project.

With that said, allowing people to generate arbitrary html to explain their
code can be *extremely powerful*:


```lua
--[[ cat
<pre class="mermaid">
---
title: querying a resource
---
sequenceDiagram
    participant db@{"type": "database"}
    actor client
    client ->> host : REQ resource 123
    host ->> db : select * from resources where id = 123
    db ->> host : 0 rows
    host ->> client : 404 - File not Found
</pre>
]]
```

In addition, you can add new kinds of comment blocks, here's how you would
add comment blocks for [Nim](https://nim-lang.org/)

```bash
$ trbldoc -s src -f 'nim;#[;#]'
```

If you wanted to cite the [Nim standard library](https://nim-lang.org/docs/lib.html) in your comments like

```nim
#[ md
This function uses @nim/ropes under the hood...
#]
```

You need a way to turn `ropes` into the nim link, `https://nim-lang.org/docs/ropes.html`.
You might chose to use `sed`:

```bash
$ trbldoc -s src -f 'nim;#[;#]' -R 'nim=sed;-e s|^|https://nim-lang.org/docs/|g -e s|$|.html|g'
```

As your prefix to link resolution gets more complex, you may chose to use a
simple bash program to generate links. You may also chose to call `trbldoc`
from a bash script or makefile.

### Markdown (`.md`)

The whole file is treated as a single chunk and rendered with `md`.

---

Single-line comments (`/* ... */`, `--[[ ]]`, `''' '''`) on one line are
intentionally ignored, trbldoc only extracts multiline blocks.

Comment blocks that do not specify a program on it's first line are likewise ignored.

## In-chunk directives

Directives are lines inside a doc comment that control how the chunk is
processed. They are stripped before rendering.

| Directive | Effect |
|-----------|--------|
| `@name <name/space>` | Group this chunk under `<namespace>` using slash-separated paths (e.g. `lua/string`). Defaults to the relative source file path. Use slashes (`/`) as the separator. |
| `@priority <number>` | Controls chunk ordering when rendering pages. Higher values are rendered first. Falls back to alphabetical `@name` ordering. |
| `@exec <program>` | Override the program used to turn text into html for this chunk. |
| `@file <path>:<line>` | Override the displayed source file attribution. Useful if the file seen by `trbldoc` has been built from some other source file. |
| `@ref <name>` | Publish this chunk under a custom reference name for cross-linking. (e.g. if you want to shorten a long name like `@interface/composites/menus/dropdown` to just `@ui/dropdown`) |

## Template variables

The following `{{ }}` markers are expanded in chunk bodies before the renderer
runs (or, for `toc`, in a post-render pass):

| Variable | Expands to |
|----------|------------|
| `{{ src_repo }}` | Source repository URL (`TRBLDOC_SRC_REPO` env var or `git remote get-url origin`). |
| `{{ ref_string }}` | HTML link for this chunk's `@ref` value, resolved from the built-in references map. |
| `{{ toc }}` | Navigation list (`nav.partial` content) - substituted after all chunks are rendered. |
| `@prefix/path/to/resolve` | Create a link to another document in this or another project. |

The default page layout (`main.layout`) also supports:

| Variable | Expands to |
|----------|------------|
| `{{ title }}` | page title in `<title>`. |
| `{{ breadcrumb }}` | heading text for the current page. |
| `{{ nav }}` | navigation list. |
| `{{ yield }}` | rendered page body. |

Example:

```c
/* md
@name http/api/public/connect
@ref db/connect

Establishes a database connection.
*/
```

## Pipeline overview

`trbldoc` has several steps, in each step multiple actions are done in parallel,
as your `xargs` allows.

1. Finds all source files under the input folders whose extension has
   a registered comment form (`c`, `h`, `cpp`, `hpp`, `lua`, `py`, `sql`,
   `md`, `dot`, plus any `-f` additions).
2. Run each file through an AWK program that emits chunk files
   into `.trblcache/chunks/`.
3. For each chunk, strip directives, run the renderer, and append
   output to `.trblcache/namespace/<namespace>/index.html`.
4. Write `main.layout`, `nav.partial`, `global.meta`, and CSS
   into `.trblcache/namespace/`.
5. Apply the layout/template tokens to each namespace page and
   write the final site to `-o <dest>` (default `.trblcache/built`).

Chunk files in `.trblcache/chunks/` and last-processed timestamps in
`.trblcache/last_processed/` persist between runs. Only source files that have
changed since the last run are re-extracted. Use `-C` to force a full rebuild.
Rendered output (`index.html`, `nav.partial`, etc.) are always regenerated from
all chunks on every run. See [docs/trblcache](./docs/trblcache.md)
for why they have to be.

## Tests

The test suite uses [bats-core](https://github.com/bats-core/bats-core) ≥ 1.5.

```bash
npm install -g bats        # or: brew install bats-core
bats tests/unit tests/integration
```

Tests are organised as:

- `tests/unit/extraction.bats` AWK comment extraction per language
- `tests/unit/directives.bats` directive parsing and stripping
- `tests/unit/template_vars.bats` template variable pattern matching
- `tests/integration/pipeline.bats` end-to-end pipeline assertions
- `tests/integration/cli.bats` CLI option parsing

## Repository layout

```
trbldoc.sh             Main pipeline script
ref_resolvers/lua.sh   Reference Lua API prefix resolver
tests/
  helpers/             Shared test helpers, stubs, AWK program builders
    common.bash        Common test helpers
    profiler.bash      Helpers for profileing
  fixtures/            Small files used as test inputs
  unit/                Unit tests
  integration/         Integration tests
```