diff options
| author | Alexander M Pickering <alex@cogarr.net> | 2026-07-21 20:19:46 -0500 |
|---|---|---|
| committer | Alexander M Pickering <alex@cogarr.net> | 2026-07-21 20:19:46 -0500 |
| commit | 657fb0007f39f07cc0401e0c5d03e25df6234aa4 (patch) | |
| tree | f73fe23232dc80802f39caed738c38464a60ec6d | |
| download | trbldoc-657fb0007f39f07cc0401e0c5d03e25df6234aa4.tar.gz trbldoc-657fb0007f39f07cc0401e0c5d03e25df6234aa4.tar.bz2 trbldoc-657fb0007f39f07cc0401e0c5d03e25df6234aa4.zip | |
Inital commit.
47 files changed, 5273 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..cbdc0e0 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +reports/* diff --git a/docs/coverage.md b/docs/coverage.md new file mode 100644 index 0000000..c6371d0 --- /dev/null +++ b/docs/coverage.md @@ -0,0 +1,55 @@ +# Test-suite coverage + +`scripts/coverage-tests.sh` reports which lines of trbldoc's shell sources the +test suite actually executes. + +## Usage + +```sh +# Coverage of trbldoc.sh from the integration suite (default) +sh scripts/coverage-tests.sh + +# Whole suite, also cover a resolver, fail under 70% +sh scripts/coverage-tests.sh -s all -t "trbldoc.sh ref_resolvers/lua.sh" -m 70 +``` + +Options: `-s all|unit|integration`, `-o OUTDIR`, `-t "file.sh ..."` (targets), +`-m MIN_PERCENT` (exit non-zero below this; default `0` = no gate). + +## Output + +Written to `reports/coverage-<timestamp>/` (or `-o`): + +- `index.html` — per-file coverage with bars and an overall percentage. +- `html/<file>.html` — annotated source; green = executed, red = missed, + grey = non-executable. +- `summary.csv` — `file,executable,covered,percent`. +- `covered.tsv` — raw union of executed `file,line` pairs. + +## How it works + +The bats tests run the tool as `bash "$TRBLDOC"`. Coverage exports +`BASH_ENV=scripts/lib/trace-init.sh`, so each target process enables `set -x` +with a `PS4` that stamps `BASH_SOURCE:LINENO` to a per-PID trace file. After +the run, the union of executed lines is compared against the executable lines +in each target source to compute coverage. No changes to `trbldoc.sh` or the +tests are needed. + +Only scripts whose basename is listed in `-t` are traced, so bats internals +and the `md2html` stub stay out of the report. + +## Determining "executable" lines + +The denominator is a heuristic: a line counts as executable when, after +trimming, it is non-blank, not a comment, and not a pure structural token +(`then`, `do`, `done`, `else`, `fi`, `esac`, `{`, `}`, `(`, `)`, `;;`). + +## Limitations + +- Coverage is collected under `bash` (the shell the suite already uses). + Lines only reachable under a different shell are not distinguished. +- xtrace reports the line where a command *starts*. Interior lines of + heredocs and multi-line strings are not individually reported and may show + as missed; exclude such files or read the numbers with that caveat. +- Code in subprocesses spawned via `ash`/`xargs` is not traced (not bash), + so it is not counted toward coverage of the main script. diff --git a/docs/profiling.md b/docs/profiling.md new file mode 100644 index 0000000..9978347 --- /dev/null +++ b/docs/profiling.md @@ -0,0 +1,81 @@ +# Profiling the test suite + +`scripts/profile-tests.sh` reports where time goes when running the bats +suite, at two levels of detail. + +## What it produces + +1. **Per-test timing** (always). Runs bats with `--timing` and the JUnit + report formatter, then ranks every test slowest-first. Answers *which + tests take a long time*. +2. **Per-line hotspots** (with `-p`). Re-runs the suite with `trbldoc.sh` + under `bash` xtrace and attributes elapsed wall-time to individual source + lines. Answers *which code takes a long time*. + +## Usage + +```sh +# Per-test timing for the integration suite (default) +sh scripts/profile-tests.sh + +# Include per-line hotspots inside trbldoc.sh +sh scripts/profile-tests.sh -p + +# Profile the whole suite, show the 30 slowest tests +sh scripts/profile-tests.sh -s all -n 30 +``` + +Options: `-s all|unit|integration`, `-o OUTDIR`, `-n TOPN`, `-p` (hotspots), +`-t "file.sh ..."` (hotspot target scripts). + +## Output + +Written to `reports/profile-<timestamp>/` (or `-o`): + +- `timings.html` — ranked per-test bar chart. +- `timings.csv` / `timings.sorted.tsv` — raw per-test data. +- `hotspots.html` / `hotspots.csv` — per-line self-time (only with `-p`). +- `junit/`, `bats-output.txt` — raw bats artifacts. + +## Profiling real trbldoc runs (outside tests) + +Use `scripts/profile-run.sh` when you want hotspot data for real workloads +rather than test execution. + +```sh +# Single direct run +sh scripts/profile-run.sh -- -s src -o docs + +# Repeat 3 clean runs to stabilize hotspot ranking +sh scripts/profile-run.sh -n 3 -C -- -s src -o docs +``` + +Outputs under `reports/profile-run-<timestamp>/`: + +- `run-summary.csv` — wall-clock runtime per run. +- `hotspots.html` — per-line self-time aggregated from xtrace markers. +- `phase-hotspots.html` — coarse pipeline phase self-time (discovery, + extraction, render ordering, rendering, nav, ref resolution, toc, assemble), + inferred from traced line ranges in `trbldoc.sh`. + +## How the hotspot profiler works + +The bats tests invoke the tool as `bash "$TRBLDOC"`. The profiler exports +`BASH_ENV=scripts/lib/trace-init.sh`, which every non-interactive bash sources +at startup. For scripts whose basename is in `TRBLDOC_TRACE_TARGETS` +(default `trbldoc.sh`) it enables `set -x` with a `PS4` that stamps +`BASH_SOURCE:LINENO` and `EPOCHREALTIME` to a per-PID trace file. The +aggregator computes the delta between consecutive traced lines *within each +process*, so trbldoc's parallel `xargs -P` workers are not cross-attributed. + +## Limitations + +- Hotspot timing needs `bash` ≥ 5 (`EPOCHREALTIME`). The report scripts + themselves are POSIX sh and run under mingw bash, WSL bash and busybox ash. +- Subprocesses trbldoc spawns via `ash`/`xargs` are not bash and are not + traced; hotspots reflect the main-script control flow, not the render + helper's internals. +- Per-line self-time is derived from trace timestamps and includes tracing + overhead, so treat the numbers as relative, not absolute. +- Phase hotspots are inferred from source-line boundaries and are intended for + prioritisation, not exact accounting. diff --git a/docs/trblcache.md b/docs/trblcache.md new file mode 100644 index 0000000..7351fb0 --- /dev/null +++ b/docs/trblcache.md @@ -0,0 +1,83 @@ +# Cacheing + +Nearly all caching happens in `$(pwd)/.trblcache` by default (hereafter `$CACHE`), +here's a reminder of the process `trblcache` uses when building documentation: + +0. Scan: find all source files under the input folders whose extension has + a registered comment form (`c`, `h`, `cpp`, `hpp`, `lua`, `py`, `sql`, + `md`, `dot`, plus any `-f` additions). +0. Extract: run each file through an AWK program that emits chunk files + into `$CACHE/chunks/`. +0. Render: for each chunk, strip directives, run the renderer, and append + output to `$CACHE/namespace/<namespace>/index.html`. +0. Assemble: write `main.layout`, `nav.partial`, `global.meta`, and CSS + into `$CACHE/namespace/`. +0. Build: apply the layout/template tokens to each namespace page and + write the final site to `-o <dest>` (default `$CACHE/built`). + +Of these, only chunk extraction can be cached. Cacheing works by first +creating a file under `$CACHE/last_processed/<path>` when a file is processed, +and then on subsequent processing, ignore files if their `$CACHE/last_processed/<path>` +file timestamp is newer than the `<path>` timestamp. If your filesystem does +not support file timestamps, `trbldoc` will never assume the cache is invalid, +and will serve stale data. If you are on such a system, always run with `-C` to +force `trbldoc` to generate new chunks. + +## Notes + +### No cached chunks +The rendered chunks, generally, cannot be cached. If the user provided their +own tooling (i.e. bash scripts) for `trbldoc`, `trbldoc` cannot check for updates +to the scripts in order to invalidate the cached rendered chunk. Checking +file last updated times won't work, if a script invokes another script or relies +on any system environment the user's updates won't take. Example: + +``` +project +| src/ +| | file1.c - last updated 1d ago +| tools/ +| | checker.sh - last updated 1d ago +| | graph.sh - last updated 30s ago +``` + +file1.c + +```c +... +/* tools/checker.sh tools/graph.sh +strict graph { + a -- b + a -- b + b -- a [color=blue] +} +*/ +... +``` + +checker.sh + +```sh +#!/bin/sh + +if [ -z "$(command $1 -V)" ]; then + echo "Could not find $1" + exit -1 +fi + +exec $1 +``` + +graph.sh + +```sh +#!/bin/sh + +exec dot +``` + +It would be impossible for `trbldoc` to determine that `tools/checker.sh` uses +`tools/graph.sh` without forcing it to run in a hermetically sealed sandbox that +carefully controls what environment, filesystem, and other programs may be run. +Such heavy sandboxing is against the vision of `trbldoc` as a simple, single-file +documentation generator. diff --git a/readme.md b/readme.md new file mode 100644 index 0000000..29cd403 --- /dev/null +++ b/readme.md @@ -0,0 +1,275 @@ +# `trbldoc` + +A really awful documentation generator. + +Trbldoc scans source files for comments, extracts them as content chunks, +passes each chunk through a renderer, groups the results by namespace, +and assembles an HTML site. + +### Dependencies + +| Tool | Purpose | +|------|---------| +| Any POSIX shell (`ash`, `dash`, `sh`, `bash`, ...) | Runtime | +| `awk`, `sed` (with `-E`), `grep`, `find`, `mktemp`, `xargs` | Standard Unix tools | + +That's it. Really. Runs under busybox systems, no GNU required! + +## Installation + +Drop `trbldoc.sh` into your `$PATH` and mark it executable: + +```bash +sudo cp trbldoc.sh /usr/local/bin/trbldoc +sudo chmod +x /usr/local/bin/trbldoc +``` + +## Usage + +``` +trbldoc [-s <source1> -s <source2> ...] [-o <dest>] [-l <layout>] [-f ext;start;end] [-R prefix=script;opts] [-C] [source ...] +``` + +* `-s <source>` - Add a folder or file to scan. Repeatable. +* `-o <dest>` - Output directory (default: `.trblcache/built`). +* `-l <layout>` - Use an alternative `main.layout` template file instead of the built-in default. +* `-f ext;start;end` - Register a custom file extension with AWK-regex delimiters. +* `-R prefix=script;opts` - Set a resolver used for `@prefix/...` reference tags. +* `-C` - Force a clean rebuild — wipes cached chunks and timestamps before scanning. + +### Environment variables + +`trbldoc` allows you to use some environment variables to provide more information +while rendering. + +| Variable | Description | +|----------|-------------| +| `TRBLDOC_SRC_REPO` | Repository URL substituted for `{{ src_repo }}` in doc comments. Falls back to `git remote get-url origin` if not provided. | +| `TRBLDOC_MD` | Override the Markdown renderer used for `.md` files and `@exec md` chunks. Defaults to `md2html --github --fpermissive-autolinks --ftables`. | +| `TRBLDOC_RST` | Override the ReStructured Text renderer used for `.rst` files and `@exec rst` chunks. Defaults to `%(body)s`. | +| `TRBLDOC_JOBS` | Number of parallel jobs used for extraction and rendering. Defaults to the detected processor count. | + +### Quick start + + +Scan the `src/` folder and write output to `docs/` + +```bash +trbldoc -s src -o docs +``` + +Include the source repository URL in generated pages + +``` +TRBLDOC_SRC_REPO=https://github.com/you/repo trbldoc -s src -o docs +``` + +## Writing doc comments + +A doc comment is a block comment whose opening tag is followed _on the same +line_ by the renderer command to use for that chunk. `trbldoc` includes the +following markup format -> html mapping: + +| alias | program | +| --- | --- | +| `md` | `md2html` | +| `rst` | `docutils` | + +### C / C++ / SQL + +```c +/* md +# My Function + +Does something useful. +*/ +``` + +### Lua + +```lua +--[[ md +## Overview + +Brief description here. +]] +``` + +### Python + +```python +'''md +## Helper + +Processes the input. +''' +``` + +`trbldoc` lets you call _arbitrary bash_(**!**) from source code comments. +Some users may be alarmed, or believe this is a security vulnerability. +The reality at the time of publishing is that it is almost all +software documentation is generated on computers controlled by +the same person or people that wrote the source code, and published on the internet. +If you can trust someone to write some source code that's running on your machine, +you can probably trust them to run some more code to generate documentation. +If you are interested in letting third parties generate documentation +without reading the code, or you want to generate documentation from untrusted input, +`trbldoc` is not right for your project. + +With that said, allowing people to generate arbitrary html to explain their +code can be *extremely powerful*: + + +```lua +--[[ cat +<pre class="mermaid"> +--- +title: querying a resource +--- +sequenceDiagram + participant db@{"type": "database"} + actor client + client ->> host : REQ resource 123 + host ->> db : select * from resources where id = 123 + db ->> host : 0 rows + host ->> client : 404 - File not Found +</pre> +]] +``` + +In addition, you can add new kinds of comment blocks, here's how you would +add comment blocks for [Nim](https://nim-lang.org/) + +```bash +$ trbldoc -s src -f 'nim;#[;#]' +``` + +If you wanted to cite the [Nim standard library](https://nim-lang.org/docs/lib.html) in your comments like + +```nim +#[ md +This function uses @nim/ropes under the hood... +#] +``` + +You need a way to turn `ropes` into the nim link, `https://nim-lang.org/docs/ropes.html`. +You might chose to use `sed`: + +```bash +$ trbldoc -s src -f 'nim;#[;#]' -R 'nim=sed;-e s|^|https://nim-lang.org/docs/|g -e s|$|.html|g' +``` + +As your prefix to link resolution gets more complex, you may chose to use a +simple bash program to generate links. You may also chose to call `trbldoc` +from a bash script or makefile. + +### Markdown (`.md`) + +The whole file is treated as a single chunk and rendered with `md`. + +--- + +Single-line comments (`/* ... */`, `--[[ ]]`, `''' '''`) on one line are +intentionally ignored, trbldoc only extracts multiline blocks. + +Comment blocks that do not specify a program on it's first line are likewise ignored. + +## In-chunk directives + +Directives are lines inside a doc comment that control how the chunk is +processed. They are stripped before rendering. + +| Directive | Effect | +|-----------|--------| +| `@name <name/space>` | Group this chunk under `<namespace>` using slash-separated paths (e.g. `lua/string`). Defaults to the relative source file path. Use slashes (`/`) as the separator. | +| `@priority <number>` | Controls chunk ordering when rendering pages. Higher values are rendered first. Falls back to alphabetical `@name` ordering. | +| `@exec <program>` | Override the program used to turn text into html for this chunk. | +| `@file <path>:<line>` | Override the displayed source file attribution. Useful if the file seen by `trbldoc` has been built from some other source file. | +| `@ref <name>` | Publish this chunk under a custom reference name for cross-linking. (e.g. if you want to shorten a long name like `@interface/composites/menus/dropdown` to just `@ui/dropdown`) | + +## Template variables + +The following `{{ }}` markers are expanded in chunk bodies before the renderer +runs (or, for `toc`, in a post-render pass): + +| Variable | Expands to | +|----------|------------| +| `{{ src_repo }}` | Source repository URL (`TRBLDOC_SRC_REPO` env var or `git remote get-url origin`). | +| `{{ ref_string }}` | HTML link for this chunk's `@ref` value, resolved from the built-in references map. | +| `{{ toc }}` | Navigation list (`nav.partial` content) - substituted after all chunks are rendered. | +| `@prefix/path/to/resolve` | Create a link to another document in this or another project. | + +The default page layout (`main.layout`) also supports: + +| Variable | Expands to | +|----------|------------| +| `{{ title }}` | page title in `<title>`. | +| `{{ breadcrumb }}` | heading text for the current page. | +| `{{ nav }}` | navigation list. | +| `{{ yield }}` | rendered page body. | + +Example: + +```c +/* md +@name http/api/public/connect +@ref db/connect + +Establishes a database connection. +*/ +``` + +## Pipeline overview + +`trbldoc` has several steps, in each step multiple actions are done in parallel, +as your `xargs` allows. + +1. Finds all source files under the input folders whose extension has + a registered comment form (`c`, `h`, `cpp`, `hpp`, `lua`, `py`, `sql`, + `md`, `dot`, plus any `-f` additions). +2. Run each file through an AWK program that emits chunk files + into `.trblcache/chunks/`. +3. For each chunk, strip directives, run the renderer, and append + output to `.trblcache/namespace/<namespace>/index.html`. +4. Write `main.layout`, `nav.partial`, `global.meta`, and CSS + into `.trblcache/namespace/`. +5. Apply the layout/template tokens to each namespace page and + write the final site to `-o <dest>` (default `.trblcache/built`). + +Chunk files in `.trblcache/chunks/` and last-processed timestamps in +`.trblcache/last_processed/` persist between runs. Only source files that have +changed since the last run are re-extracted. Use `-C` to force a full rebuild. +Rendered output (`index.html`, `nav.partial`, etc.) are always regenerated from +all chunks on every run. See [docs/trblcache](./docs/trblcache.md) +for why they have to be. + +## Tests + +The test suite uses [bats-core](https://github.com/bats-core/bats-core) ≥ 1.5. + +```bash +npm install -g bats # or: brew install bats-core +bats tests/unit tests/integration +``` + +Tests are organised as: + +- `tests/unit/extraction.bats` AWK comment extraction per language +- `tests/unit/directives.bats` directive parsing and stripping +- `tests/unit/template_vars.bats` template variable pattern matching +- `tests/integration/pipeline.bats` end-to-end pipeline assertions +- `tests/integration/cli.bats` CLI option parsing + +## Repository layout + +``` +trbldoc.sh Main pipeline script +ref_resolvers/lua.sh Reference Lua API prefix resolver +tests/ + helpers/ Shared test helpers, stubs, AWK program builders + common.bash Common test helpers + profiler.bash Helpers for profileing + fixtures/ Small files used as test inputs + unit/ Unit tests + integration/ Integration tests +``` diff --git a/ref_resolvers/lua.sh b/ref_resolvers/lua.sh new file mode 100644 index 0000000..ec65080 --- /dev/null +++ b/ref_resolvers/lua.sh @@ -0,0 +1,342 @@ +#!/bin/bash +# ref_resolvers/lua.sh +# Resolves a Lua manual section reference to its full URL. +# The reference name is read from stdin. +# Outputs the URL to stdout. +reference="$(cat)" +webroot="https://www.lua.org/manual/5.2/manual.html" + +case $(echo $reference | tr '[:upper:]' '[:lower:]') in + + "introduction") + printf "%s#%s" "$webroot" "1" + exit 0 + ;; + + "basic concepts") + printf "%s#%s" "$webroot" "2" + exit 0 + ;; + + "values and types") + printf "%s#%s" "$webroot" "2.1" + exit 0 + ;; + + "environments and the global environment") + printf "%s#%s" "$webroot" "2.2" + exit 0 + ;; + + "error handling") + printf "%s#%s" "$webroot" "2.3" + exit 0 + ;; + + "metatables and metamethods") + printf "%s#%s" "$webroot" "2.4" + exit 0 + ;; + + "garbage collection") + printf "%s#%s" "$webroot" "2.5" + exit 0 + ;; + + "garbage-collection metamethods") + printf "%s#%s" "$webroot" "2.5.1" + exit 0 + ;; + + "weak tables") + printf "%s#%s" "$webroot" "2.5.2" + exit 0 + ;; + + "coroutines") + printf "%s#%s" "$webroot" "2.6" + exit 0 + ;; + + "the language") + printf "%s#%s" "$webroot" "3" + exit 0 + ;; + + "lexical conventions") + printf "%s#%s" "$webroot" "3.1" + exit 0 + ;; + + "variables") + printf "%s#%s" "$webroot" "3.2" + exit 0 + ;; + + "statements") + printf "%s#%s" "$webroot" "3.3" + exit 0 + ;; + + "blocks") + printf "%s#%s" "$webroot" "3.3.1" + exit 0 + ;; + + "chunks") + printf "%s#%s" "$webroot" "3.3.2" + exit 0 + ;; + + "assignment") + printf "%s#%s" "$webroot" "3.3.3" + exit 0 + ;; + + "control structures") + printf "%s#%s" "$webroot" "3.3.4" + exit 0 + ;; + + "for statement") + printf "%s#%s" "$webroot" "3.3.5" + exit 0 + ;; + + "function calls as statements") + printf "%s#%s" "$webroot" "3.3.6" + exit 0 + ;; + + "local declarations") + printf "%s#%s" "$webroot" "3.3.7" + exit 0 + ;; + + "expressions") + printf "%s#%s" "$webroot" "3.4" + exit 0 + ;; + + "arithmetic operators") + printf "%s#%s" "$webroot" "3.4.1" + exit 0 + ;; + + "coercion") + printf "%s#%s" "$webroot" "3.4.2" + exit 0 + ;; + + "relational operators") + printf "%s#%s" "$webroot" "3.4.3" + exit 0 + ;; + + "logical operators") + printf "%s#%s" "$webroot" "3.4.4" + exit 0 + ;; + + "concatenation") + printf "%s#%s" "$webroot" "3.4.5" + exit 0 + ;; + + "the length operator") + printf "%s#%s" "$webroot" "3.4.6" + exit 0 + ;; + + "precedence") + printf "%s#%s" "$webroot" "3.4.7" + exit 0 + ;; + + "table constructors") + printf "%s#%s" "$webroot" "3.4.8" + exit 0 + ;; + + "function calls") + printf "%s#%s" "$webroot" "3.4.9" + exit 0 + ;; + + "function definitions") + printf "%s#%s" "$webroot" "3.4.10" + exit 0 + ;; + + "visibility rules") + printf "%s#%s" "$webroot" "3.5" + exit 0 + ;; + + "the application program interface") + printf "%s#%s" "$webroot" "4" + exit 0 + ;; + + "the stack") + printf "%s#%s" "$webroot" "4.1" + exit 0 + ;; + + "stack size") + printf "%s#%s" "$webroot" "4.2" + exit 0 + ;; + + "valid and acceptable indices") + printf "%s#%s" "$webroot" "4.3" + exit 0 + ;; + + "c closures") + printf "%s#%s" "$webroot" "4.4" + exit 0 + ;; + + "registry") + printf "%s#%s" "$webroot" "4.5" + exit 0 + ;; + + "error handling in c") + printf "%s#%s" "$webroot" "4.6" + exit 0 + ;; + + "handling yields in c") + printf "%s#%s" "$webroot" "4.7" + exit 0 + ;; + + "functions and types") + # Matches section 4.8 (C API). Section 5.1 (Auxiliary Library) has the + # same title; this entry resolves to the first occurrence. + printf "%s#%s" "$webroot" "4.8" + exit 0 + ;; + + "the debug interface") + printf "%s#%s" "$webroot" "4.9" + exit 0 + ;; + + "the auxiliary library") + printf "%s#%s" "$webroot" "5" + exit 0 + ;; + + "standard libraries") + printf "%s#%s" "$webroot" "6" + exit 0 + ;; + + "basic functions") + printf "%s#%s" "$webroot" "6.1" + exit 0 + ;; + + "coroutine manipulation") + printf "%s#%s" "$webroot" "6.2" + exit 0 + ;; + + "modules") + printf "%s#%s" "$webroot" "6.3" + exit 0 + ;; + + "string manipulation") + printf "%s#%s" "$webroot" "6.4" + exit 0 + ;; + + "patterns") + printf "%s#%s" "$webroot" "6.4.1" + exit 0 + ;; + + "table manipulation") + printf "%s#%s" "$webroot" "6.5" + exit 0 + ;; + + "mathematical functions") + printf "%s#%s" "$webroot" "6.6" + exit 0 + ;; + + "bitwise operations") + printf "%s#%s" "$webroot" "6.7" + exit 0 + ;; + + "input and output facilities") + printf "%s#%s" "$webroot" "6.8" + exit 0 + ;; + + "operating system facilities") + printf "%s#%s" "$webroot" "6.9" + exit 0 + ;; + + "the debug library") + printf "%s#%s" "$webroot" "6.10" + exit 0 + ;; + + "lua standalone") + printf "%s#%s" "$webroot" "7" + exit 0 + ;; + + "incompatibilities with the previous version") + printf "%s#%s" "$webroot" "8" + exit 0 + ;; + + "changes in the language") + printf "%s#%s" "$webroot" "8.1" + exit 0 + ;; + + "changes in the libraries") + printf "%s#%s" "$webroot" "8.2" + exit 0 + ;; + + "changes in the api") + printf "%s#%s" "$webroot" "8.3" + exit 0 + ;; + + "the complete syntax of lua") + printf "%s#%s" "$webroot" "9" + exit 0 + ;; + + # Lua type names — resolve to the Values and Types section (2.1) + "nil" \ + | "boolean" \ + | "number" \ + | "string" \ + | "function" \ + | "userdata" \ + | "thread" \ + | "table") + printf "%s#%s" "$webroot" "2.1" + exit 0 + ;; + + # Alternative lowercase spelling for coroutines + "coroutines") + printf "%s#%s" "$webroot" "2.6" + exit 0 + ;; + +esac diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..91adbd3 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,50 @@ +# Report scripts + +Portable POSIX-sh tooling to generate reports from the bats test suite. All +entrypoints run under mingw bash, WSL bash and Alpine/busybox ash, and depend +only on `bats`, `awk`, `sort` and coreutils. + +## `profile-tests.sh` + +Per-test timing (and optional per-line hotspots). See `doc/profiling.md`. +## `profile-run.sh` + +Direct profiling for real `trbldoc.sh` invocations (outside bats). Produces: + +- per-run wall-clock summary (`run-summary.csv`) +- line hotspots (`hotspots.html`) +- pipeline phase hotspots inferred from trace line ranges (`phase-hotspots.html`) + +Example: + +```sh +sh scripts/profile-run.sh -n 3 -C -- -s tests/fixtures/pipeline -o .trblcache/built +``` +## `hotspot-tests.sh` + +Repeatable hotspot analysis for the bats suite without relying on JUnit output. +It captures: + +- per-test timing from raw `bats --timing` output +- line hotspots from xtrace logs +- optional phase hotspots for `trbldoc.sh` (when helper scripts are present) + +```sh +sh scripts/hotspot-tests.sh -s all -n 30 +``` + +## `coverage-tests.sh` + +Shell line coverage of `trbldoc.sh`. See `doc/coverage.md`. + +## `generate-reports.sh` + +Run both into one timestamped `reports/` directory. Puts reports into a +`reports/run-<timestamp>/` directory. + +```sh +sh scripts/generate-reports.sh -p +``` + +`lib/` holds shared helpers: `common.sh` (sh helpers), `trace-init.sh` (the +`BASH_ENV` xtrace hook), and the `*.awk` extractors/renderers. diff --git a/scripts/coverage-tests.sh b/scripts/coverage-tests.sh new file mode 100644 index 0000000..cf03825 --- /dev/null +++ b/scripts/coverage-tests.sh @@ -0,0 +1,143 @@ +#!/bin/sh +# scripts/coverage-tests.sh +# Generate line-coverage reports for trbldoc's shell sources from the test +# suite. Answers "what is the test suite not exercising". +# +# Mechanism: the suite runs the tool via `bash "$TRBLDOC"`, so exporting +# BASH_ENV=scripts/lib/trace-init.sh makes each target process emit an xtrace +# of every executed line. We union the executed lines across the whole run and +# compare against the executable lines in each target source. No modification +# of trbldoc.sh or the tests is required. +# +# POSIX sh; works under mingw bash, WSL bash and Alpine/busybox ash. The traced +# run itself uses bash (the suite already invokes bash), which is one of the +# supported target shells. +# +# Usage: +# scripts/coverage-tests.sh [-s all|unit|integration] [-o OUTDIR] +# [-t "file.sh ..."] [-m MIN_PERCENT] +# +# -s SUITE suite to run (default: integration) +# -o OUTDIR output directory (default: reports/coverage-<timestamp>) +# -t LIST space-separated target source files (default: trbldoc.sh) +# -m MIN_PERCENT fail (exit 2) if overall coverage is below this (default: 0) + +set -u + +SELF="$0" +. "$(dirname -- "$0")/lib/common.sh" +tb_resolve_paths "$SELF" + +SUITE=integration +OUTDIR="" +TARGETS="trbldoc.sh" +MIN_PERCENT=0 + +while getopts ":s:o:t:m:h" opt; do + case "$opt" in + s) SUITE="$OPTARG" ;; + o) OUTDIR="$OPTARG" ;; + t) TARGETS="$OPTARG" ;; + m) MIN_PERCENT="$OPTARG" ;; + h) sed -n '2,30p' "$SELF"; exit 0 ;; + :) tb_die "option -$OPTARG requires an argument" ;; + \?) tb_die "unknown option -$OPTARG" ;; + esac +done + +tb_require awk +tb_require sort + +[ -n "$OUTDIR" ] || OUTDIR="$REPO_ROOT/reports/coverage-$(tb_timestamp)" +case "$OUTDIR" in + /*|[A-Za-z]:[\\/]*) + ;; + *) + OUTDIR="$REPO_ROOT/$OUTDIR" + ;; +esac +mkdir -p "$OUTDIR/html" || tb_die "cannot create output dir: $OUTDIR" +JUNIT_DIR="$OUTDIR/junit"; mkdir -p "$JUNIT_DIR" +TRACE_DIR="$OUTDIR/trace"; rm -rf "$TRACE_DIR"; mkdir -p "$TRACE_DIR" + +SUITE_DIRS="$(tb_suite_dirs "$SUITE")" +SUITE_DIRS_REL="$(tb_suite_dirs_rel "$SUITE")" + +# Basename filter for the trace hook + absolute source paths for reporting. +TRACE_TARGETS="" +SRC_FILES="" +for tf in $TARGETS; do + TRACE_TARGETS="$TRACE_TARGETS ${tf##*/}" + case "$tf" in + /*) p="$tf" ;; + *) p="$REPO_ROOT/$tf" ;; + esac + [ -f "$p" ] || tb_die "target source not found: $p" + SRC_FILES="$SRC_FILES $p" +done + +# ── run suite under xtrace ─────────────────────────────────────────────────── +tb_info "running bats ($SUITE) under coverage instrumentation..." +if tb_bats_usable "$SUITE"; then + BASH_ENV="$LIB_DIR/trace-init.sh" + TRBLDOC_TRACE_DIR="$TRACE_DIR" + TRBLDOC_TRACE_MODE="cov" + TRBLDOC_TRACE_TARGETS="$TRACE_TARGETS" + export BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS + # shellcheck disable=SC2086 + bats --report-formatter junit --output "$JUNIT_DIR" $SUITE_DIRS \ + > "$OUTDIR/bats-output.txt" 2>&1 || tb_info "bats reported failures (see bats-output.txt)" + unset BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS +elif command -v wsl >/dev/null 2>&1; then + tb_info "local bats appears unusable; retrying via WSL bats..." + WSL_REPO="$(tb_to_wsl_path "$REPO_ROOT")" + WSL_LIB="$(tb_to_wsl_path "$LIB_DIR")" + WSL_TRACE="$(tb_to_wsl_path "$TRACE_DIR")" + WSL_JUNIT="$(tb_to_wsl_path "$JUNIT_DIR")" + wsl -- bash -lc \ + "cd '$WSL_REPO' && export BASH_ENV='$WSL_LIB/trace-init.sh' TRBLDOC_TRACE_DIR='$WSL_TRACE' TRBLDOC_TRACE_MODE='cov' TRBLDOC_TRACE_TARGETS='$TRACE_TARGETS'; bats --report-formatter junit --output '$WSL_JUNIT' $SUITE_DIRS_REL" \ + > "$OUTDIR/bats-output.txt" 2>&1 || tb_info "bats (WSL) reported failures (see bats-output.txt)" +else + tb_die "local bats command appears unusable and no WSL fallback is available" +fi + +if ! ls "$TRACE_DIR"/trace.*.log >/dev/null 2>&1; then + if grep -qi "bash\\\\r" "$OUTDIR/bats-output.txt" 2>/dev/null; then + tb_die "bats appears broken (CRLF shebang: 'bash\\r'). Reinstall bats or run dos2unix on the bats scripts, then retry." + fi + tb_die "no trace data captured; ensure bats actually runs tests (see $OUTDIR/bats-output.txt)" +fi + +# ── union executed lines ───────────────────────────────────────────────────── +awk -f "$LIB_DIR/trace-extract.awk" -v MODE=cov "$TRACE_DIR"/trace.*.log \ + | sort -u > "$OUTDIR/covered.tsv" + +# ── per-file coverage report + summary ─────────────────────────────────────── +# shellcheck disable=SC2086 +awk -f "$LIB_DIR/coverage-report.awk" \ + -v COVERED="$OUTDIR/covered.tsv" \ + -v HTMLDIR="$OUTDIR/html" \ + $SRC_FILES > "$OUTDIR/summary.rows" + +{ + printf 'file,executable,covered,percent\n' + cat "$OUTDIR/summary.rows" +} > "$OUTDIR/summary.csv" + +awk -f "$LIB_DIR/render-coverage-index.awk" \ + -v TITLE="trbldoc coverage ($SUITE)" \ + "$OUTDIR/summary.rows" > "$OUTDIR/index.html" + +# ── overall percent + threshold gate ───────────────────────────────────────── +OVERALL="$(awk -F',' '{t+=$2; c+=$3} END{ if(t>0) printf "%.1f", c*100.0/t; else print "0.0" }' "$OUTDIR/summary.rows")" + +printf '\n==== coverage summary (%s) ====\n' "$SUITE" +awk -F',' '{ printf "%6.1f%% %-20s (%d/%d)\n", $4, $1, $3, $2 }' "$OUTDIR/summary.rows" +printf ' overall: %s%%\n' "$OVERALL" +printf '\nReports written to: %s\n' "$OUTDIR" +printf ' coverage index: %s\n' "$OUTDIR/index.html" + +# Threshold enforcement (integer or decimal compare via awk). +if awk -v o="$OVERALL" -v m="$MIN_PERCENT" 'BEGIN{ exit !(m>0 && o+0 < m+0) }'; then + tb_die "coverage ${OVERALL}% is below the required minimum ${MIN_PERCENT}%" +fi diff --git a/scripts/generate-reports.sh b/scripts/generate-reports.sh new file mode 100644 index 0000000..77c4b0f --- /dev/null +++ b/scripts/generate-reports.sh @@ -0,0 +1,58 @@ +#!/bin/sh +# scripts/generate-reports.sh +# Convenience wrapper: run both the profiler and the coverage report for a +# suite into a single timestamped directory under reports/. +# +# POSIX sh; works under mingw bash, WSL bash and Alpine/busybox ash. +# +# Usage: +# scripts/generate-reports.sh [-s all|unit|integration] [-o OUTDIR] +# [-p] [-m MIN_PERCENT] [-t "file.sh ..."] +# +# -s SUITE suite to run (default: integration) +# -o OUTDIR base output directory (default: reports/run-<timestamp>) +# -p include per-line hotspots in the profile report +# -m MIN_PERCENT coverage threshold to enforce (default: 0 = no gate) +# -t LIST space-separated target source files (default: trbldoc.sh) + +set -u + +SELF="$0" +. "$(dirname -- "$0")/lib/common.sh" +tb_resolve_paths "$SELF" + +SUITE=integration +OUTDIR="" +HOTSPOTS=0 +MIN_PERCENT=0 +TARGETS="trbldoc.sh" + +while getopts ":s:o:pm:t:h" opt; do + case "$opt" in + s) SUITE="$OPTARG" ;; + o) OUTDIR="$OPTARG" ;; + p) HOTSPOTS=1 ;; + m) MIN_PERCENT="$OPTARG" ;; + t) TARGETS="$OPTARG" ;; + h) sed -n '2,20p' "$SELF"; exit 0 ;; + :) tb_die "option -$OPTARG requires an argument" ;; + \?) tb_die "unknown option -$OPTARG" ;; + esac +done + +[ -n "$OUTDIR" ] || OUTDIR="$REPO_ROOT/reports/run-$(tb_timestamp)" +mkdir -p "$OUTDIR" || tb_die "cannot create output dir: $OUTDIR" + +PROFILE_ARGS="-s $SUITE -o $OUTDIR/profile -t $TARGETS" +[ "$HOTSPOTS" -eq 1 ] && PROFILE_ARGS="$PROFILE_ARGS -p" + +tb_info "=== profiling ===" +# shellcheck disable=SC2086 +sh "$SCRIPTS_DIR/profile-tests.sh" $PROFILE_ARGS + +tb_info "=== coverage ===" +# shellcheck disable=SC2086 +sh "$SCRIPTS_DIR/coverage-tests.sh" -s "$SUITE" -o "$OUTDIR/coverage" \ + -t "$TARGETS" -m "$MIN_PERCENT" + +printf '\nAll reports under: %s\n' "$OUTDIR" diff --git a/scripts/hotspot-tests.sh b/scripts/hotspot-tests.sh new file mode 100644 index 0000000..8973e6b --- /dev/null +++ b/scripts/hotspot-tests.sh @@ -0,0 +1,204 @@ +#!/bin/sh +# scripts/hotspot-tests.sh +# Repeatable hotspot analysis for the trbldoc test suite. +# +# Produces: +# - per-test timings (parsed from `bats --timing`) +# - line-level hotspots for target scripts from xtrace logs +# - optional phase hotspots for trbldoc.sh +# +# Usage: +# scripts/hotspot-tests.sh [-s all|unit|integration] [-o OUTDIR] +# [-n TOPN] [-t "file.sh ..."] +# +# -s SUITE suite to run (default: all) +# -o OUTDIR output directory (default: reports/hotspots-<timestamp>) +# -n TOPN number of slowest tests to print (default: 25) +# -t LIST space-separated target scripts for trace hotspots (default: trbldoc.sh) + +set -u + +SELF="$0" +. "$(dirname -- "$0")/lib/common.sh" +tb_resolve_paths "$SELF" + +SUITE=all +OUTDIR="" +TOPN=25 +TARGETS="trbldoc.sh" + +while getopts ":s:o:n:t:h" opt; do + case "$opt" in + s) SUITE="$OPTARG" ;; + o) OUTDIR="$OPTARG" ;; + n) TOPN="$OPTARG" ;; + t) TARGETS="$OPTARG" ;; + h) sed -n '2,24p' "$SELF"; exit 0 ;; + :) tb_die "option -$OPTARG requires an argument" ;; + \?) tb_die "unknown option -$OPTARG" ;; + esac +done + +tb_require awk +tb_require sort + +case "$TOPN" in + ''|*[!0-9]*|0) tb_die "-n TOPN must be a positive integer" ;; +esac + +[ -n "$OUTDIR" ] || OUTDIR="$REPO_ROOT/reports/hotspots-$(tb_timestamp)" +case "$OUTDIR" in + /*|[A-Za-z]:[\\/]*) ;; + *) OUTDIR="$REPO_ROOT/$OUTDIR" ;; +esac +mkdir -p "$OUTDIR" || tb_die "cannot create output dir: $OUTDIR" + +SUITE_DIRS="$(tb_suite_dirs "$SUITE")" +SUITE_DIRS_REL="$(tb_suite_dirs_rel "$SUITE")" +TIMING_RAW="$OUTDIR/bats-timing-output.txt" +TRACE_DIR="$OUTDIR/trace" +rm -rf "$TRACE_DIR"; mkdir -p "$TRACE_DIR" + +# Build trace target filter and resolve first target source for HTML context. +TRACE_TARGETS="" +FIRST_TARGET="" +for tf in $TARGETS; do + TRACE_TARGETS="$TRACE_TARGETS ${tf##*/}" + if [ -z "$FIRST_TARGET" ]; then + FIRST_TARGET="$tf" + fi +done + +tb_info "running bats ($SUITE) with --timing output..." +if tb_bats_usable "$SUITE"; then + # shellcheck disable=SC2086 + bats --timing $SUITE_DIRS > "$TIMING_RAW" 2>&1 || tb_info "bats reported failures (timing output still collected)" + BATS_MODE="local" +elif command -v wsl >/dev/null 2>&1; then + tb_info "local bats appears unusable; retrying timing run via WSL bats..." + WSL_REPO="$(tb_to_wsl_path "$REPO_ROOT")" + wsl -- bash -lc \ + "cd '$WSL_REPO' && bats --timing $SUITE_DIRS_REL" \ + > "$TIMING_RAW" 2>&1 || tb_info "bats (WSL) reported failures (timing output still collected)" + BATS_MODE="wsl" +else + tb_die "local bats command appears unusable and no WSL fallback is available" +fi + +# Parse bats timing lines: "ok N <name> in <ms>ms" +awk ' +match($0, /^ok [0-9]+ (.*) in ([0-9]+)ms$/, m) { + name = m[1] + ms = m[2] + 0 + suite = name + sub(/:.*/, "", suite) + printf "%.3f\t%s\t%s\t%d\n", ms / 1000.0, suite, name, ms +} +' "$TIMING_RAW" > "$OUTDIR/timings.tsv" + +if [ ! -s "$OUTDIR/timings.tsv" ]; then + tb_die "no per-test timing rows parsed (check $TIMING_RAW)" +fi + +sort -t "$(printf '\t')" -k1,1nr "$OUTDIR/timings.tsv" > "$OUTDIR/timings.sorted.tsv" +{ + printf 'seconds,suite,test,milliseconds\n' + awk -F '\t' '{ gsub(/"/,"\"\"",$3); printf "%s,%s,\"%s\",%s\n",$1,$2,$3,$4 }' "$OUTDIR/timings.sorted.tsv" +} > "$OUTDIR/timings.csv" + +awk -f "$LIB_DIR/render-timing.awk" \ + -v TITLE="trbldoc test timings ($SUITE)" \ + "$OUTDIR/timings.sorted.tsv" > "$OUTDIR/timings.html" + +tb_info "running bats ($SUITE) under xtrace timing hook..." +if [ "$BATS_MODE" = "local" ]; then + BASH_ENV="$LIB_DIR/trace-init.sh" + TRBLDOC_TRACE_DIR="$TRACE_DIR" + TRBLDOC_TRACE_MODE="time" + TRBLDOC_TRACE_TARGETS="$TRACE_TARGETS" + export BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS + # shellcheck disable=SC2086 + bats $SUITE_DIRS > "$OUTDIR/bats-trace-output.txt" 2>&1 || tb_info "bats reported failures during trace run" + unset BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS +else + WSL_REPO="$(tb_to_wsl_path "$REPO_ROOT")" + WSL_LIB="$(tb_to_wsl_path "$LIB_DIR")" + WSL_TRACE="$(tb_to_wsl_path "$TRACE_DIR")" + wsl -- bash -lc \ + "cd '$WSL_REPO' && export BASH_ENV='$WSL_LIB/trace-init.sh' TRBLDOC_TRACE_DIR='$WSL_TRACE' TRBLDOC_TRACE_MODE='time' TRBLDOC_TRACE_TARGETS='$TRACE_TARGETS'; bats $SUITE_DIRS_REL" \ + > "$OUTDIR/bats-trace-output.txt" 2>&1 || tb_info "bats (WSL) reported failures during trace run" +fi + +if ls "$TRACE_DIR"/trace.*.log >/dev/null 2>&1; then + awk -f "$LIB_DIR/trace-extract.awk" -v MODE=time "$TRACE_DIR"/trace.*.log \ + | sort -t "$(printf '\t')" -k1 -rn > "$OUTDIR/hotspots.tsv" + { + printf 'total_ms,hits,file,line\n' + awk -F '\t' '{ printf "%s,%s,%s,%s\n",$1,$2,$3,$4 }' "$OUTDIR/hotspots.tsv" + } > "$OUTDIR/hotspots.csv" + + case "$FIRST_TARGET" in + /*|[A-Za-z]:[\\/]*) SRC_PATH="$FIRST_TARGET" ;; + *) SRC_PATH="$REPO_ROOT/$FIRST_TARGET" ;; + esac + awk -f "$LIB_DIR/render-hotspots.awk" \ + -v TITLE="trbldoc line hotspots ($SUITE)" \ + -v SRC="$SRC_PATH" \ + "$OUTDIR/hotspots.tsv" > "$OUTDIR/hotspots.html" +else + tb_info "no trace logs captured for target(s):$TRACE_TARGETS" +fi + +# Optional phase hotspots when trbldoc.sh + helper scripts are available. +if [ -f "$OUTDIR/hotspots.tsv" ] && [ -f "$LIB_DIR/trace-phase-extract.awk" ] && [ -f "$LIB_DIR/render-phases.awk" ] && [ -f "$REPO_ROOT/trbldoc.sh" ]; then + _find_line() { + awk -v p="$1" 'index($0, p) { print NR; exit }' "$REPO_ROOT/trbldoc.sh" + } + L_DISCOVER_END="$(_find_line '> "$source_files_list"')" + L_EXTRACT_END="$(_find_line '# Resolve src_repo once before the chunk loop.')" + L_ORDER_END="$(_find_line 'done < "$sorted_chunk_order_file"')" + L_RENDER_END="$(_find_line 'done < "$chunk_render_records_file"')" + L_NAV_END="$(_find_line '# Second pass: resolve inline references now that global.meta is complete.')" + L_REFS_END="$(_find_line 'rm -f "$_ref_find_tmp"')" + L_TOC_END="$(_find_line '# Post-processing pass: substitute {{ toc }} in all rendered index.html files')" + L_ASSEMBLE_END="$(_find_line 'assemble_site "$cache_dir/namespace" "$out_dir"')" + + if [ -n "$L_DISCOVER_END" ] && [ -n "$L_EXTRACT_END" ] && [ -n "$L_ORDER_END" ] \ + && [ -n "$L_RENDER_END" ] && [ -n "$L_NAV_END" ] && [ -n "$L_REFS_END" ] \ + && [ -n "$L_TOC_END" ] && [ -n "$L_ASSEMBLE_END" ]; then + awk -f "$LIB_DIR/trace-phase-extract.awk" \ + -v SRC_BASE="trbldoc.sh" \ + -v L_DISCOVER_END="$L_DISCOVER_END" \ + -v L_EXTRACT_END="$L_EXTRACT_END" \ + -v L_ORDER_END="$L_ORDER_END" \ + -v L_RENDER_END="$L_RENDER_END" \ + -v L_NAV_END="$L_NAV_END" \ + -v L_REFS_END="$L_REFS_END" \ + -v L_TOC_END="$L_TOC_END" \ + -v L_ASSEMBLE_END="$L_ASSEMBLE_END" \ + "$TRACE_DIR"/trace.*.log \ + | sort -t "$(printf '\t')" -k1 -rn > "$OUTDIR/phase-hotspots.tsv" + + { + printf 'total_ms,hits,phase\n' + awk -F '\t' '{ printf "%s,%s,%s\n",$1,$2,$3 }' "$OUTDIR/phase-hotspots.tsv" + } > "$OUTDIR/phase-hotspots.csv" + + awk -f "$LIB_DIR/render-phases.awk" \ + -v TITLE="trbldoc phase hotspots ($SUITE)" \ + "$OUTDIR/phase-hotspots.tsv" > "$OUTDIR/phase-hotspots.html" + fi +fi + +printf '\n==== slowest %s tests ====\n' "$TOPN" +awk -F '\t' -v n="$TOPN" 'NR<=n { printf "%8.3fs %s\n", $1, $3 }' "$OUTDIR/timings.sorted.tsv" +printf '\nReports written to: %s\n' "$OUTDIR" +printf ' per-test timing : %s\n' "$OUTDIR/timings.html" +if [ -f "$OUTDIR/hotspots.html" ]; then + printf ' line hotspots : %s\n' "$OUTDIR/hotspots.html" +else + printf ' line hotspots : not generated (target not exercised)\n' +fi +if [ -f "$OUTDIR/phase-hotspots.html" ]; then + printf ' phase hotspots : %s\n' "$OUTDIR/phase-hotspots.html" +fi diff --git a/scripts/lib/common.sh b/scripts/lib/common.sh new file mode 100644 index 0000000..0fdf98a --- /dev/null +++ b/scripts/lib/common.sh @@ -0,0 +1,105 @@ +# scripts/lib/common.sh +# Shared helpers for the trbldoc report scripts. +# +# POSIX sh compatible. Verified to work under mingw bash, WSL bash and +# Alpine/busybox ash. Sourced by the entrypoint scripts; do not execute +# directly. + +# Resolve the repository root from the location of the scripts/ directory. +# Sets: REPO_ROOT, SCRIPTS_DIR, LIB_DIR +tb_resolve_paths() { + # $1: path to the calling script ($0) + script_path="$1" + # Directory containing the calling script (scripts/). + SCRIPTS_DIR="$(CDPATH= cd -- "$(dirname -- "$script_path")" && pwd)" + LIB_DIR="$SCRIPTS_DIR/lib" + REPO_ROOT="$(CDPATH= cd -- "$SCRIPTS_DIR/.." && pwd)" + export REPO_ROOT SCRIPTS_DIR LIB_DIR +} + +# Print an ISO-8601 UTC timestamp suitable for directory names. +# Falls back to epoch seconds when the platform date(1) cannot format UTC. +tb_timestamp() { + if date -u +%Y%m%dT%H%M%SZ 2>/dev/null; then + return 0 + fi + printf '%s\n' "$(date +%s)" +} + +# Emit an informational message to stderr. +tb_info() { + printf '[trbldoc-report] %s\n' "$*" >&2 +} + +# Emit an error message to stderr and return non-zero. +tb_die() { + printf '[trbldoc-report] ERROR: %s\n' "$*" >&2 + exit 1 +} + +# Verify a command exists on PATH, otherwise abort. +tb_require() { + command -v "$1" >/dev/null 2>&1 || tb_die "required tool not found on PATH: $1" +} + +# Resolve the bats suite directories for a target keyword. +# $1: all|unit|integration ; echoes space-separated dirs relative to REPO_ROOT. +tb_suite_dirs() { + case "$1" in + unit) printf '%s\n' "$REPO_ROOT/tests/unit" ;; + integration) printf '%s\n' "$REPO_ROOT/tests/integration" ;; + all) printf '%s\n' "$REPO_ROOT/tests/unit $REPO_ROOT/tests/integration" ;; + *) tb_die "unknown suite '$1' (expected: all|unit|integration)" ;; + esac +} + +# Resolve bats suite directories relative to repository root. +# $1: all|unit|integration ; echoes a space-separated relative path list. +tb_suite_dirs_rel() { + case "$1" in + unit) printf '%s\n' "tests/unit" ;; + integration) printf '%s\n' "tests/integration" ;; + all) printf '%s\n' "tests/unit tests/integration" ;; + *) tb_die "unknown suite '$1' (expected: all|unit|integration)" ;; + esac +} + +# Convert a POSIX-style MSYS path (/d/foo/bar) to a WSL Linux path +# (/mnt/d/foo/bar). Other paths are returned unchanged. +tb_to_wsl_path() { + p="$1" + case "$p" in + [A-Za-z]:\\*|[A-Za-z]:/*) + d="$(printf '%s' "$p" | cut -c1 | tr 'A-Z' 'a-z')" + rest="$(printf '%s' "$p" | cut -c3- | sed 's#\\\\#/#g')" + rest="$(printf '%s' "$rest" | sed 's#^/*##')" + printf '/mnt/%s/%s\n' "$d" "$rest" + ;; + /[A-Za-z]/*) + d="$(printf '%s' "$p" | cut -c2 | tr 'A-Z' 'a-z')" + rest="$(printf '%s' "$p" | cut -c4-)" + printf '/mnt/%s/%s\n' "$d" "$rest" + ;; + *) + printf '%s\n' "$p" + ;; + esac +} + +# Returns success if the local `bats` command appears usable for this repo. +# Some mixed Windows/MSYS setups expose a broken bats wrapper that exits 0 +# without running tests or producing output; this guard catches that case. +tb_bats_usable() { + suite="${1:-integration}" + dirs="$(tb_suite_dirs "$suite")" + # shellcheck disable=SC2086 + count="$(bats --count $dirs 2>/dev/null | tr -d '[:space:]')" + case "$count" in + ''|*[!0-9]*) + return 1 + ;; + *) + return 0 + ;; + esac +} diff --git a/scripts/lib/coverage-report.awk b/scripts/lib/coverage-report.awk new file mode 100644 index 0000000..982eb45 --- /dev/null +++ b/scripts/lib/coverage-report.awk @@ -0,0 +1,84 @@ +# scripts/lib/coverage-report.awk +# Produce line-coverage reports for shell sources from an executed-line set. +# +# Inputs: +# -v COVERED=<file> TSV of "<base>\t<line>" executed lines (sorted -u). +# -v HTMLDIR=<dir> directory to write per-file <base>.html reports into. +# ARGV the source files to report on (absolute paths). +# +# Output (stdout): one CSV row per source file: +# file,total_executable,covered,percent +# +# "Executable" lines are determined heuristically: non-blank, non-comment +# lines that are not pure structural tokens (then/do/done/else/fi/esac/{/}). +# Heredoc/multiline string bodies are approximated and may read as missed; +# see docs/coverage.md for the documented limitations. + +function trim(s) { sub(/^[ \t]+/, "", s); sub(/[ \t]+$/, "", s); return s } + +function is_exec(line, t) { + t = trim(line) + if (t == "") return 0 + if (t ~ /^#/) return 0 + if (t ~ /^(then|do|done|else|fi|esac)([ \t].*)?$/) return 0 + if (t ~ /^[{}]([ \t;].*)?$/) return 0 + if (t ~ /^[()]([ \t].*)?$/) return 0 + if (t ~ /^;;/) return 0 + return 1 +} + +function esc(s) { + gsub(/&/, "\\&", s) + gsub(/</, "\\<", s) + gsub(/>/, "\\>", s) + return s +} + +function basename(p) { sub(/.*\//, "", p); sub(/.*\\/, "", p); return p } + +BEGIN { + FS = "\t" + # Load the executed-line set. + while ((getline ln < COVERED) > 0) { + nf = split(ln, a, "\t") + if (nf >= 2) covered[a[1] SUBSEP a[2]] = 1 + } + close(COVERED) +} + +# New source file: open its HTML report and reset counters. +FNR == 1 { + if (cur != "") finish_file() + cur = FILENAME + curbase = basename(FILENAME) + total = 0; hit = 0 + html = HTMLDIR "/" curbase ".html" + print "<!DOCTYPE html><html lang=\"en\"><head><meta charset=\"utf-8\">" > html + print "<title>coverage: " esc(curbase) "</title>" > html + print "<style>body{font:13px/1.5 monospace;margin:0;padding:1rem;background:#fff;color:#111}" > html + print "h1{font:600 16px sans-serif}" > html + print "table{border-collapse:collapse;width:100%}" > html + print "td{padding:0 .5rem;white-space:pre;vertical-align:top}" > html + print "td.n{color:#999;text-align:right;user-select:none;border-right:1px solid #ddd}" > html + print ".hit{background:#e6ffed}.miss{background:#ffeef0}.na{color:#999}" > html + print "</style></head><body>" > html + print "<h1>coverage: " esc(curbase) "</h1><table>" > html +} + +{ + cls = "na" + if (is_exec($0)) { + total++ + if ((curbase SUBSEP FNR) in covered) { hit++; cls = "hit" } else { cls = "miss" } + } + print "<tr class=\"" cls "\"><td class=\"n\">" FNR "</td><td>" esc($0) "</td></tr>" > html +} + +function finish_file( pct) { + print "</table></body></html>" > html + close(html) + pct = (total > 0 ? (hit * 100.0 / total) : 0) + printf "%s,%d,%d,%.1f\n", curbase, total, hit, pct +} + +END { if (cur != "") finish_file() } diff --git a/scripts/lib/render-coverage-index.awk b/scripts/lib/render-coverage-index.awk new file mode 100644 index 0000000..cc5af1b --- /dev/null +++ b/scripts/lib/render-coverage-index.awk @@ -0,0 +1,48 @@ +# scripts/lib/render-coverage-index.awk +# Render the coverage index page from summary rows. +# Input CSV rows (no header): "<file>,<executable>,<covered>,<percent>". +# Per-file detail pages are expected at html/<file>.html. +# Variables: -v TITLE="..." + +function esc(s) { + gsub(/&/, "\\&", s); gsub(/</, "\\<", s); gsub(/>/, "\\>", s) + return s +} + +function color(p) { + if (p + 0 >= 80) return "#2da44e" + if (p + 0 >= 50) return "#bf8700" + return "#cf222e" +} + +BEGIN { FS = ","; n = 0; tot = 0; cov = 0 } + +{ + n++ + f[n] = $1; ex[n] = $2 + 0; cv[n] = $3 + 0; pc[n] = $4 + 0 + tot += $2 + 0; cov += $3 + 0 +} + +END { + overall = (tot > 0 ? cov * 100.0 / tot : 0) + print "<!DOCTYPE html><html lang=\"en\"><head><meta charset=\"utf-8\">" + print "<title>" esc(TITLE) "</title>" + print "<style>body{font:14px/1.5 sans-serif;margin:0;padding:1.5rem;color:#111}" + print "h1{font-size:18px}.overall{font-size:15px;margin:.5rem 0 1rem}" + print "table{border-collapse:collapse;width:100%;max-width:820px}" + print "th,td{padding:.35rem .5rem;text-align:left;border-bottom:1px solid #eee}" + print "td.num{text-align:right;font-variant-numeric:tabular-nums}" + print ".track{background:#eee;border-radius:3px;height:.7em;width:160px}" + print ".fill{height:.7em;border-radius:3px}" + print "a{color:#0969da;text-decoration:none}a:hover{text-decoration:underline}" + print "</style></head><body>" + print "<h1>" esc(TITLE) "</h1>" + printf "<div class=\"overall\">Overall: <strong style=\"color:%s\">%.1f%%</strong> (%d/%d executable lines)</div>\n", color(overall), overall, cov, tot + print "<table><thead><tr><th>file</th><th>coverage</th><th>%</th><th>covered</th><th>executable</th></tr></thead><tbody>" + for (i = 1; i <= n; i++) { + printf "<tr><td><a href=\"html/%s.html\">%s</a></td>", esc(f[i]), esc(f[i]) + printf "<td><div class=\"track\"><div class=\"fill\" style=\"width:%.0f%%;background:%s\"></div></div></td>", pc[i], color(pc[i]) + printf "<td class=\"num\">%.1f%%</td><td class=\"num\">%d</td><td class=\"num\">%d</td></tr>\n", pc[i], cv[i], ex[i] + } + print "</tbody></table></body></html>" +} diff --git a/scripts/lib/render-hotspots.awk b/scripts/lib/render-hotspots.awk new file mode 100644 index 0000000..0eb2b6d --- /dev/null +++ b/scripts/lib/render-hotspots.awk @@ -0,0 +1,56 @@ +# scripts/lib/render-hotspots.awk +# Render a per-line hotspot HTML report from sorted TSV rows. +# Input TSV (sorted slowest-first): "<total_ms>\t<hits>\t<file>\t<line>". +# Variables: +# -v TITLE="..." +# -v SRC=<path> source file for the primary target, used to show line text. + +function esc(s) { + gsub(/&/, "\\&", s); gsub(/</, "\\<", s); gsub(/>/, "\\>", s) + return s +} + +BEGIN { + FS = "\t"; n = 0 + # Load source lines for text display (best effort). + if (SRC != "") { + i = 0 + while ((getline ln < SRC) > 0) { i++; srctext[i] = ln } + close(SRC) + srcbase = SRC; sub(/.*\//, "", srcbase); sub(/.*\\/, "", srcbase) + } +} + +{ + n++ + ms[n] = $1 + 0 + hits[n] = $2 + 0 + file[n] = $3 + line[n] = $4 + 0 + if (n == 1) maxv = $1 + 0 +} + +END { + print "<!DOCTYPE html><html lang=\"en\"><head><meta charset=\"utf-8\">" + print "<title>" esc(TITLE) "</title>" + print "<style>body{font:14px/1.5 sans-serif;margin:0;padding:1.5rem;color:#111}" + print "h1{font-size:18px}.meta{color:#555;margin-bottom:1rem}" + print "table{border-collapse:collapse;width:100%}" + print "th,td{padding:.3rem .5rem;text-align:left;border-bottom:1px solid #eee}" + print "td.num{text-align:right;font-variant-numeric:tabular-nums}" + print "code{font:12px/1.4 monospace;white-space:pre}" + print ".bar{height:.7em;background:#e8703a;border-radius:2px}" + print "</style></head><body>" + print "<h1>" esc(TITLE) "</h1>" + print "<div class=\"meta\">cumulative self-time per source line, slowest first</div>" + print "<table><thead><tr><th>#</th><th>ms</th><th>hits</th><th>file:line</th><th>source</th><th></th></tr></thead><tbody>" + for (i = 1; i <= n; i++) { + w = (maxv > 0 ? (ms[i] * 100.0 / maxv) : 0) + txt = "" + if (file[i] == srcbase && (line[i] in srctext)) txt = srctext[line[i]] + printf "<tr><td class=\"num\">%d</td><td class=\"num\">%.1f</td><td class=\"num\">%d</td>", i, ms[i], hits[i] + printf "<td>%s:%d</td><td><code>%s</code></td>", esc(file[i]), line[i], esc(txt) + printf "<td><div class=\"bar\" style=\"width:%.1f%%\"></div></td></tr>\n", w + } + print "</tbody></table></body></html>" +} diff --git a/scripts/lib/render-phases.awk b/scripts/lib/render-phases.awk new file mode 100644 index 0000000..fcb11ec --- /dev/null +++ b/scripts/lib/render-phases.awk @@ -0,0 +1,41 @@ +# scripts/lib/render-phases.awk +# Render per-phase hotspot HTML from sorted TSV rows. +# Input TSV: "<total_ms>\t<hits>\t<phase>". +# Variables: +# -v TITLE="..." + +function esc(s) { + gsub(/&/, "\\&", s); gsub(/</, "\\<", s); gsub(/>/, "\\>", s) + return s +} + +BEGIN { FS = "\t"; n = 0 } + +{ + n++ + ms[n] = $1 + 0 + hits[n] = $2 + 0 + phase[n] = $3 + if (n == 1) maxv = ms[n] +} + +END { + print "<!DOCTYPE html><html lang=\"en\"><head><meta charset=\"utf-8\">" + print "<title>" esc(TITLE) "</title>" + print "<style>body{font:14px/1.5 sans-serif;margin:0;padding:1.5rem;color:#111}" + print "h1{font-size:18px}.meta{color:#555;margin-bottom:1rem}" + print "table{border-collapse:collapse;width:100%}" + print "th,td{padding:.3rem .5rem;text-align:left;border-bottom:1px solid #eee}" + print "td.num{text-align:right;font-variant-numeric:tabular-nums}" + print ".bar{height:.7em;background:#4cbf8f;border-radius:2px}" + print "</style></head><body>" + print "<h1>" esc(TITLE) "</h1>" + print "<div class=\"meta\">cumulative self-time by pipeline phase, slowest first</div>" + print "<table><thead><tr><th>#</th><th>ms</th><th>hits</th><th>phase</th><th></th></tr></thead><tbody>" + for (i = 1; i <= n; i++) { + w = (maxv > 0 ? (ms[i] * 100.0 / maxv) : 0) + printf "<tr><td class=\"num\">%d</td><td class=\"num\">%.1f</td><td class=\"num\">%d</td><td>%s</td>", i, ms[i], hits[i], esc(phase[i]) + printf "<td><div class=\"bar\" style=\"width:%.1f%%\"></div></td></tr>\n", w + } + print "</tbody></table></body></html>" +} diff --git a/scripts/lib/render-timing.awk b/scripts/lib/render-timing.awk new file mode 100644 index 0000000..ffd807f --- /dev/null +++ b/scripts/lib/render-timing.awk @@ -0,0 +1,41 @@ +# scripts/lib/render-timing.awk +# Render a per-test timing HTML report from sorted TSV rows. +# Input TSV (already sorted slowest-first): "<seconds>\t<suite>\t<test>". +# Variables: -v TITLE="..." + +function esc(s) { + gsub(/&/, "\\&", s); gsub(/</, "\\<", s); gsub(/>/, "\\>", s) + return s +} + +BEGIN { FS = "\t"; n = 0; total = 0 } + +{ + n++ + sec[n] = $1 + 0 + suite[n] = $2 + name[n] = $3 + total += $1 + 0 + if (n == 1) maxv = $1 + 0 +} + +END { + print "<!DOCTYPE html><html lang=\"en\"><head><meta charset=\"utf-8\">" + print "<title>" esc(TITLE) "</title>" + print "<style>body{font:14px/1.5 sans-serif;margin:0;padding:1.5rem;color:#111}" + print "h1{font-size:18px}.meta{color:#555;margin-bottom:1rem}" + print "table{border-collapse:collapse;width:100%}" + print "th,td{padding:.3rem .5rem;text-align:left;border-bottom:1px solid #eee}" + print "td.num{text-align:right;font-variant-numeric:tabular-nums}" + print ".bar{height:.7em;background:#4c8bf5;border-radius:2px}" + print "</style></head><body>" + print "<h1>" esc(TITLE) "</h1>" + printf "<div class=\"meta\">%d tests · %.3f s total</div>\n", n, total + print "<table><thead><tr><th>#</th><th>seconds</th><th>test</th><th></th></tr></thead><tbody>" + for (i = 1; i <= n; i++) { + w = (maxv > 0 ? (sec[i] * 100.0 / maxv) : 0) + printf "<tr><td class=\"num\">%d</td><td class=\"num\">%.3f</td><td>%s</td>", i, sec[i], esc(name[i]) + printf "<td><div class=\"bar\" style=\"width:%.1f%%\"></div></td></tr>\n", w + } + print "</tbody></table></body></html>" +} diff --git a/scripts/lib/timing-extract.awk b/scripts/lib/timing-extract.awk new file mode 100644 index 0000000..9cb15d7 --- /dev/null +++ b/scripts/lib/timing-extract.awk @@ -0,0 +1,32 @@ +# scripts/lib/timing-extract.awk +# Extract per-test durations from bats JUnit report XML. +# +# Emits TSV: "<seconds>\t<suite>\t<test name>" for each <testcase>. +# The caller sorts and renders. Handles testcase elements that span the line +# regardless of whether they are self-closing or contain <failure> children. + +function attr(line, key, re, s) { + re = key "=\"" + if (index(line, re) == 0) return "" + s = line + sub(".*" re, "", s) # drop up to opening quote + sub(/".*/, "", s) # drop from closing quote + return s +} + +function unesc(s) { + gsub(/"/, "\"", s) + gsub(/'/, "'", s) + gsub(/</, "<", s) + gsub(/>/, ">", s) + gsub(/&/, "\\&", s) + return s +} + +/<testcase[ \t]/ { + name = unesc(attr($0, "name")) + cls = unesc(attr($0, "classname")) + t = attr($0, "time") + if (t == "") t = "0" + printf "%s\t%s\t%s\n", t, cls, name +} diff --git a/scripts/lib/trace-extract.awk b/scripts/lib/trace-extract.awk new file mode 100644 index 0000000..7b8682a --- /dev/null +++ b/scripts/lib/trace-extract.awk @@ -0,0 +1,71 @@ +# scripts/lib/trace-extract.awk +# Extract instrumented source locations from xtrace logs produced via +# scripts/lib/trace-init.sh. +# +# Marker lines look like (bash repeats the leading '@' once per nesting level): +# @@TBL@@ /path/to/trbldoc.sh:1234 1712345678.123456 +# +# Variables: +# MODE=cov (default) -> prints "<base>\t<line>" per marker line +# MODE=time -> prints "<total_ms>\t<hits>\t<base>\t<line>" at END, +# computing per-line durations from consecutive markers +# WITHIN each trace file (one file == one process, so +# parallel xargs workers are not cross-attributed). + +function basename(p) { + sub(/.*\//, "", p) # strip up to last forward slash + sub(/.*\\/, "", p) # strip up to last backslash (defensive) + return p +} + +# Parse a marker line into globals g_base and g_line (and g_epoch for time). +# Returns 1 on success, 0 if the line is not a usable marker. +function parse(line, s, n, f, loc, rest) { + if (line !~ /@+TBL@@ /) return 0 + s = line + sub(/^.*TBL@@ /, "", s) # drop marker and everything before it + n = split(s, f, " ") + if (n < 1) return 0 + loc = f[1] # <source>:<lineno> + if (!match(loc, /:[0-9]+$/)) return 0 + g_line = substr(loc, RSTART + 1) + g_base = basename(substr(loc, 1, RSTART - 1)) + if (MODE == "time") { + g_epoch = (n >= 2 ? f[2] : "") + } + return 1 +} + +BEGIN { if (MODE == "") MODE = "cov" } + +# Reset per-process state at the start of each trace file. +FNR == 1 { have_prev = 0 } + +{ + if (!parse($0)) next + if (MODE == "cov") { + print g_base "\t" g_line + next + } + # MODE == time: attribute elapsed time to the PREVIOUS marker. + if (have_prev && g_epoch != "" && prev_epoch != "") { + d = g_epoch - prev_epoch + if (d < 0) d = 0 + key = prev_base SUBSEP prev_line + ms[key] += d * 1000.0 + hits[key] += 1 + base_of[key] = prev_base + line_of[key] = prev_line + } + prev_base = g_base + prev_line = g_line + prev_epoch = g_epoch + have_prev = 1 +} + +END { + if (MODE != "time") exit + for (k in ms) { + printf "%.3f\t%d\t%s\t%s\n", ms[k], hits[k], base_of[k], line_of[k] + } +} diff --git a/scripts/lib/trace-init.sh b/scripts/lib/trace-init.sh new file mode 100644 index 0000000..80f1ab0 --- /dev/null +++ b/scripts/lib/trace-init.sh @@ -0,0 +1,42 @@ +# scripts/lib/trace-init.sh +# Instrumentation hook sourced automatically by every non-interactive bash +# invocation via the BASH_ENV environment variable. +# +# The bats suite runs the tool with `bash "$TRBLDOC"`, so exporting +# BASH_ENV=<this file> causes each trbldoc.sh process to enable xtrace at +# startup WITHOUT modifying trbldoc.sh or the test files. The emitted trace is +# consumed by the coverage and hotspot aggregators. +# +# Activated only when TRBLDOC_TRACE_DIR is set, and only for scripts whose +# basename appears in TRBLDOC_TRACE_TARGETS (default: trbldoc.sh). This keeps +# unrelated bash helpers (md2html stub, bats internals) out of the trace. +# +# TRBLDOC_TRACE_MODE selects the PS4 format: +# cov -> "<marker> <source>:<lineno>" (line coverage) +# time -> "<marker> <source>:<lineno> <EPOCHREALTIME>" (line timing; bash>=5) +# +# Every traced line is prefixed with the marker token @@TBL@@. bash repeats the +# FIRST PS4 character once per call-nesting level, so the aggregators match the +# marker with a tolerant pattern rather than a fixed prefix. + +if [ -n "${TRBLDOC_TRACE_DIR:-}" ]; then + __tb_self="$0" + __tb_base="${__tb_self##*/}" + case " ${TRBLDOC_TRACE_TARGETS:-trbldoc.sh} " in + *" ${__tb_base} "*) + # Dedicated append fd per process; PID keeps parallel xargs workers apart. + __tb_trace_file="${TRBLDOC_TRACE_DIR}/trace.$$.log" + # shellcheck disable=SC3023 + exec 7>>"$__tb_trace_file" + export BASH_XTRACEFD=7 + if [ "${TRBLDOC_TRACE_MODE:-cov}" = "time" ]; then + # EPOCHREALTIME is bash>=5; used only for the hotspot profiler. + PS4='@@TBL@@ ${BASH_SOURCE}:${LINENO} ${EPOCHREALTIME} ' + else + PS4='@@TBL@@ ${BASH_SOURCE}:${LINENO} ' + fi + export PS4 + set -x + ;; + esac +fi diff --git a/scripts/lib/trace-phase-extract.awk b/scripts/lib/trace-phase-extract.awk new file mode 100644 index 0000000..429f8cf --- /dev/null +++ b/scripts/lib/trace-phase-extract.awk @@ -0,0 +1,75 @@ +# scripts/lib/trace-phase-extract.awk +# Aggregate xtrace timing markers into coarse trbldoc pipeline phases. +# +# Input: trace.*.log files emitted by scripts/lib/trace-init.sh in time mode. +# Output rows: "<total_ms>\t<hits>\t<phase>" at END. +# +# Required -v vars: +# SRC_BASE +# L_DISCOVER_END +# L_EXTRACT_END +# L_ORDER_END +# L_RENDER_END +# L_NAV_END +# L_REFS_END +# L_TOC_END +# L_ASSEMBLE_END + +function basename(p) { + sub(/.*\//, "", p) + sub(/.*\\/, "", p) + return p +} + +function parse(line, s, n, f, loc) { + if (line !~ /@+TBL@@ /) return 0 + s = line + sub(/^.*TBL@@ /, "", s) + n = split(s, f, " ") + if (n < 2) return 0 + loc = f[1] + if (!match(loc, /:[0-9]+$/)) return 0 + g_line = substr(loc, RSTART + 1) + 0 + g_base = basename(substr(loc, 1, RSTART - 1)) + g_epoch = f[2] + return 1 +} + +function phase_for(line) { + if (line <= L_DISCOVER_END) return "discover_sources" + if (line <= L_EXTRACT_END) return "extract_chunks" + if (line <= L_ORDER_END) return "build_render_order" + if (line <= L_RENDER_END) return "render_chunks" + if (line <= L_NAV_END) return "build_nav" + if (line <= L_REFS_END) return "resolve_refs" + if (line <= L_TOC_END) return "apply_toc" + if (line <= L_ASSEMBLE_END) return "assemble_site" + return "other" +} + +BEGIN { + if (SRC_BASE == "") SRC_BASE = "trbldoc.sh" +} + +FNR == 1 { have_prev = 0 } + +{ + if (!parse($0)) next + if (g_base != SRC_BASE) next + if (have_prev && g_epoch != "" && prev_epoch != "") { + d = g_epoch - prev_epoch + if (d < 0) d = 0 + ph = phase_for(prev_line) + ms[ph] += d * 1000.0 + hits[ph] += 1 + } + prev_line = g_line + prev_epoch = g_epoch + have_prev = 1 +} + +END { + for (ph in ms) { + printf "%.3f\t%d\t%s\n", ms[ph], hits[ph], ph + } +} diff --git a/scripts/profile-run.sh b/scripts/profile-run.sh new file mode 100644 index 0000000..74ab425 --- /dev/null +++ b/scripts/profile-run.sh @@ -0,0 +1,180 @@ +#!/bin/sh +# scripts/profile-run.sh +# Profile direct trbldoc invocations (outside bats) using the existing xtrace +# hook and report pipeline. +# +# Outputs: +# - run-summary.tsv/csv wall-clock per run +# - hotspots.tsv/csv/html per-line cumulative self-time +# - phase-hotspots.tsv/csv/html per-phase cumulative self-time +# +# Usage: +# scripts/profile-run.sh [-o OUTDIR] [-n RUNS] [-C] [-t "file.sh ..."] -- [trbldoc args...] +# +# -o OUTDIR output directory (default: reports/profile-run-<timestamp>) +# -n RUNS number of repeated runs (default: 1) +# -C force clean rebuild (-C) on every run +# -t LIST trace target script basenames (default: trbldoc.sh) + +set -u + +SELF="$0" +. "$(dirname -- "$0")/lib/common.sh" +tb_resolve_paths "$SELF" + +OUTDIR="" +RUNS=1 +CLEAN=0 +TARGETS="trbldoc.sh" + +while getopts ":o:n:Ct:h" opt; do + case "$opt" in + o) OUTDIR="$OPTARG" ;; + n) RUNS="$OPTARG" ;; + C) CLEAN=1 ;; + t) TARGETS="$OPTARG" ;; + h) sed -n '2,24p' "$SELF"; exit 0 ;; + :) tb_die "option -$OPTARG requires an argument" ;; + \?) tb_die "unknown option -$OPTARG" ;; + esac +done +shift $((OPTIND - 1)) + +[ "${1:-}" = "--" ] && shift + +tb_require awk +tb_require sort +tb_require bash + +case "$RUNS" in + ''|*[!0-9]*|0) tb_die "-n RUNS must be a positive integer" ;; +esac + +[ -n "$OUTDIR" ] || OUTDIR="$REPO_ROOT/reports/profile-run-$(tb_timestamp)" +case "$OUTDIR" in + /*|[A-Za-z]:[\\/]*) ;; + *) OUTDIR="$REPO_ROOT/$OUTDIR" ;; +esac +mkdir -p "$OUTDIR" || tb_die "cannot create output dir: $OUTDIR" + +TRACE_DIR="$OUTDIR/trace" +RUN_DIR="$OUTDIR/runs" +mkdir -p "$TRACE_DIR" "$RUN_DIR" + +TRBLDOC_PATH="$REPO_ROOT/trbldoc.sh" +[ -f "$TRBLDOC_PATH" ] || tb_die "trbldoc source not found: $TRBLDOC_PATH" + +TRACE_TARGETS="" +for tf in $TARGETS; do + TRACE_TARGETS="$TRACE_TARGETS ${tf##*/}" +done + +# Resolve line anchors once for phase aggregation. If any anchor is missing, +# skip phase aggregation rather than failing the whole profiling run. +tb_find_line() { + _pat="$1" + awk -v p="$_pat" 'index($0, p) { print NR; exit }' "$TRBLDOC_PATH" +} +L_DISCOVER_END="$(tb_find_line '> "$source_files_list"')" +L_EXTRACT_END="$(tb_find_line '# Resolve src_repo once before the chunk loop.')" +L_ORDER_END="$(tb_find_line 'done < "$sorted_chunk_order_file"')" +L_RENDER_END="$(tb_find_line 'done < "$chunk_render_records_file"')" +L_NAV_END="$(tb_find_line '# Second pass: resolve inline references now that global.meta is complete.')" +L_REFS_END="$(tb_find_line 'rm -f "$_ref_find_tmp"')" +L_TOC_END="$(tb_find_line '# Post-processing pass: substitute {{ toc }} in all rendered index.html files')" +L_ASSEMBLE_END="$(tb_find_line 'assemble_site "$cache_dir/namespace" "$out_dir"')" + +: > "$OUTDIR/run-summary.tsv" + +run_idx=1 +while [ "$run_idx" -le "$RUNS" ]; do + tb_info "profiling run $run_idx/$RUNS..." + run_out="$RUN_DIR/run-$run_idx" + mkdir -p "$run_out" + run_stdout="$run_out/stdout.txt" + run_stderr="$run_out/stderr.txt" + run_start_s="$(date +%s)" + + BASH_ENV="$LIB_DIR/trace-init.sh" + TRBLDOC_TRACE_DIR="$TRACE_DIR" + TRBLDOC_TRACE_MODE="time" + TRBLDOC_TRACE_TARGETS="$TRACE_TARGETS" + export BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS + + if [ "$CLEAN" -eq 1 ]; then + (cd "$REPO_ROOT" && bash "$TRBLDOC_PATH" -C "$@") >"$run_stdout" 2>"$run_stderr" + else + (cd "$REPO_ROOT" && bash "$TRBLDOC_PATH" "$@") >"$run_stdout" 2>"$run_stderr" + fi + run_exit=$? + unset BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS + + run_end_s="$(date +%s)" + run_ms=$(( (run_end_s - run_start_s) * 1000 )) + printf '%s\t%s\t%s\n' "$run_idx" "$run_ms" "$run_exit" >> "$OUTDIR/run-summary.tsv" + run_idx=$((run_idx + 1)) +done + +sort -t "$(printf '\t')" -k2 -rn "$OUTDIR/run-summary.tsv" > "$OUTDIR/run-summary.sorted.tsv" +{ + printf 'run,total_ms,exit_code\n' + awk -F '\t' '{ printf "%s,%s,%s\n",$1,$2,$3 }' "$OUTDIR/run-summary.sorted.tsv" +} > "$OUTDIR/run-summary.csv" + +if ls "$TRACE_DIR"/trace.*.log >/dev/null 2>&1; then + awk -f "$LIB_DIR/trace-extract.awk" -v MODE=time "$TRACE_DIR"/trace.*.log \ + | sort -t "$(printf '\t')" -k1 -rn > "$OUTDIR/hotspots.tsv" + { + printf 'total_ms,hits,file,line\n' + awk -F '\t' '{ printf "%s,%s,%s,%s\n",$1,$2,$3,$4 }' "$OUTDIR/hotspots.tsv" + } > "$OUTDIR/hotspots.csv" + + FIRST_TARGET="${TARGETS%% *}" + case "$FIRST_TARGET" in + /*|[A-Za-z]:[\\/]*) SRC_PATH="$FIRST_TARGET" ;; + *) SRC_PATH="$REPO_ROOT/$FIRST_TARGET" ;; + esac + awk -f "$LIB_DIR/render-hotspots.awk" \ + -v TITLE="trbldoc line hotspots (direct runs)" \ + -v SRC="$SRC_PATH" \ + "$OUTDIR/hotspots.tsv" > "$OUTDIR/hotspots.html" +else + tb_die "no trace logs captured (check run stderr logs under $RUN_DIR)" +fi + +if [ -n "$L_DISCOVER_END" ] && [ -n "$L_EXTRACT_END" ] && [ -n "$L_ORDER_END" ] \ + && [ -n "$L_RENDER_END" ] && [ -n "$L_NAV_END" ] && [ -n "$L_REFS_END" ] \ + && [ -n "$L_TOC_END" ] && [ -n "$L_ASSEMBLE_END" ]; then + awk -f "$LIB_DIR/trace-phase-extract.awk" \ + -v SRC_BASE="trbldoc.sh" \ + -v L_DISCOVER_END="$L_DISCOVER_END" \ + -v L_EXTRACT_END="$L_EXTRACT_END" \ + -v L_ORDER_END="$L_ORDER_END" \ + -v L_RENDER_END="$L_RENDER_END" \ + -v L_NAV_END="$L_NAV_END" \ + -v L_REFS_END="$L_REFS_END" \ + -v L_TOC_END="$L_TOC_END" \ + -v L_ASSEMBLE_END="$L_ASSEMBLE_END" \ + "$TRACE_DIR"/trace.*.log \ + | sort -t "$(printf '\t')" -k1 -rn > "$OUTDIR/phase-hotspots.tsv" + + { + printf 'total_ms,hits,phase\n' + awk -F '\t' '{ printf "%s,%s,%s\n",$1,$2,$3 }' "$OUTDIR/phase-hotspots.tsv" + } > "$OUTDIR/phase-hotspots.csv" + + awk -f "$LIB_DIR/render-phases.awk" \ + -v TITLE="trbldoc phase hotspots (direct runs)" \ + "$OUTDIR/phase-hotspots.tsv" > "$OUTDIR/phase-hotspots.html" +else + tb_info "phase anchor discovery incomplete; skipping phase-hotspots report" +fi + +printf '\n==== slowest runs ====\n' +awk -F '\t' '{ printf "run %s %8.1f ms exit=%s\n",$1,$2,$3 }' "$OUTDIR/run-summary.sorted.tsv" +printf '\nReports written to: %s\n' "$OUTDIR" +printf ' run summary : %s\n' "$OUTDIR/run-summary.csv" +printf ' line hotspots : %s\n' "$OUTDIR/hotspots.html" +if [ -f "$OUTDIR/phase-hotspots.html" ]; then + printf ' phase hotspots : %s\n' "$OUTDIR/phase-hotspots.html" +fi diff --git a/scripts/profile-tests.sh b/scripts/profile-tests.sh new file mode 100644 index 0000000..498ae25 --- /dev/null +++ b/scripts/profile-tests.sh @@ -0,0 +1,184 @@ +#!/bin/sh +# scripts/profile-tests.sh +# Generate timing reports for the trbldoc test suite. +# +# Two levels of detail: +# 1. Per-test wall-clock timings from bats (--timing + JUnit report). Always +# produced. Answers "which tests take a long time". +# 2. Per-line hotspots inside trbldoc.sh (optional, --hotspots). Uses the +# BASH_ENV xtrace hook to time individual source lines across the whole +# run. Answers "which code takes a long time". +# +# POSIX sh; works under mingw bash, WSL bash and Alpine/busybox ash. +# +# Usage: +# scripts/profile-tests.sh [-s all|unit|integration] [-o OUTDIR] +# [-n TOPN] [-p] [-t "file.sh ..."] +# +# -s SUITE suite to run (default: integration) +# -o OUTDIR output directory (default: reports/profile-<timestamp>) +# -n TOPN number of rows in the printed "slowest" summary (default: 20) +# -p also collect per-line hotspots inside the target script(s) +# -t LIST space-separated target source files for -p (default: trbldoc.sh) + +set -u + +SELF="$0" +. "$(dirname -- "$0")/lib/common.sh" +tb_resolve_paths "$SELF" + +SUITE=integration +OUTDIR="" +TOPN=20 +HOTSPOTS=0 +TARGETS="trbldoc.sh" + +while getopts ":s:o:n:pt:h" opt; do + case "$opt" in + s) SUITE="$OPTARG" ;; + o) OUTDIR="$OPTARG" ;; + n) TOPN="$OPTARG" ;; + p) HOTSPOTS=1 ;; + t) TARGETS="$OPTARG" ;; + h) sed -n '2,30p' "$SELF"; exit 0 ;; + :) tb_die "option -$OPTARG requires an argument" ;; + \?) tb_die "unknown option -$OPTARG" ;; + esac +done + +tb_require awk +tb_require sort + +[ -n "$OUTDIR" ] || OUTDIR="$REPO_ROOT/reports/profile-$(tb_timestamp)" +case "$OUTDIR" in + /*|[A-Za-z]:[\\/]*) + ;; + *) + OUTDIR="$REPO_ROOT/$OUTDIR" + ;; +esac +mkdir -p "$OUTDIR" || tb_die "cannot create output dir: $OUTDIR" +JUNIT_DIR="$OUTDIR/junit" +mkdir -p "$JUNIT_DIR" + +SUITE_DIRS="$(tb_suite_dirs "$SUITE")" +SUITE_DIRS_REL="$(tb_suite_dirs_rel "$SUITE")" + +# ── 1. per-test timing via bats JUnit ─────────────────────────────────────── +tb_info "running bats ($SUITE) with timing..." +# --timing adds per-test durations; the junit report captures them in XML. +# We do not abort on test failures: timing is still useful. +if tb_bats_usable "$SUITE"; then + # shellcheck disable=SC2086 + bats --timing --report-formatter junit --output "$JUNIT_DIR" $SUITE_DIRS \ + > "$OUTDIR/bats-output.txt" 2>&1 || tb_info "bats reported failures (see bats-output.txt)" + BATS_MODE="local" +elif command -v wsl >/dev/null 2>&1; then + tb_info "local bats appears unusable; retrying via WSL bats..." + WSL_REPO="$(tb_to_wsl_path "$REPO_ROOT")" + WSL_JUNIT="$(tb_to_wsl_path "$JUNIT_DIR")" + wsl -- bash -lc \ + "cd '$WSL_REPO' && bats --timing --report-formatter junit --output '$WSL_JUNIT' $SUITE_DIRS_REL" \ + > "$OUTDIR/bats-output.txt" 2>&1 || tb_info "bats (WSL) reported failures (see bats-output.txt)" + BATS_MODE="wsl" +else + tb_die "local bats command appears unusable and no WSL fallback is available" +fi + +# Collect all JUnit XML the formatter produced. +: > "$OUTDIR/timings.tsv" +for xml in "$JUNIT_DIR"/*.xml; do + [ -f "$xml" ] || continue + awk -f "$LIB_DIR/timing-extract.awk" "$xml" >> "$OUTDIR/timings.tsv" +done + +if [ ! -s "$OUTDIR/timings.tsv" ]; then + if grep -qi "bash\\\\r" "$OUTDIR/bats-output.txt" 2>/dev/null; then + tb_die "bats appears broken (CRLF shebang: 'bash\\r'). Reinstall bats or run dos2unix on the bats scripts, then retry." + fi + tb_die "no per-test timings parsed (check $OUTDIR/bats-output.txt)" +fi + +# Sort slowest first. +sort -t "$(printf '\t')" -k1 -rn "$OUTDIR/timings.tsv" > "$OUTDIR/timings.sorted.tsv" + +# CSV summary. +{ + printf 'seconds,suite,test\n' + awk -F '\t' '{ gsub(/"/,"\"\"",$3); printf "%s,%s,\"%s\"\n",$1,$2,$3 }' \ + "$OUTDIR/timings.sorted.tsv" +} > "$OUTDIR/timings.csv" + +# HTML report. +awk -f "$LIB_DIR/render-timing.awk" \ + -v TITLE="trbldoc test timings ($SUITE)" \ + "$OUTDIR/timings.sorted.tsv" > "$OUTDIR/timings.html" + +# ── 2. optional per-line hotspots ──────────────────────────────────────────── +if [ "$HOTSPOTS" -eq 1 ]; then + tb_info "collecting per-line hotspots (this reruns the suite under xtrace)..." + TRACE_DIR="$OUTDIR/trace" + rm -rf "$TRACE_DIR"; mkdir -p "$TRACE_DIR" + + # Build the trace-target basename filter. + TRACE_TARGETS="" + for tf in $TARGETS; do + TRACE_TARGETS="$TRACE_TARGETS ${tf##*/}" + done + + BASH_ENV="$LIB_DIR/trace-init.sh" + TRBLDOC_TRACE_DIR="$TRACE_DIR" + TRBLDOC_TRACE_MODE="time" + TRBLDOC_TRACE_TARGETS="$TRACE_TARGETS" + if [ "$BATS_MODE" = "local" ]; then + export BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS + # shellcheck disable=SC2086 + bats --report-formatter junit --output "$JUNIT_DIR" $SUITE_DIRS \ + > "$OUTDIR/bats-hotspots-output.txt" 2>&1 \ + || tb_info "bats reported failures during hotspot run" + unset BASH_ENV TRBLDOC_TRACE_DIR TRBLDOC_TRACE_MODE TRBLDOC_TRACE_TARGETS + else + WSL_REPO="$(tb_to_wsl_path "$REPO_ROOT")" + WSL_LIB="$(tb_to_wsl_path "$LIB_DIR")" + WSL_TRACE="$(tb_to_wsl_path "$TRACE_DIR")" + WSL_JUNIT="$(tb_to_wsl_path "$JUNIT_DIR")" + wsl -- bash -lc \ + "cd '$WSL_REPO' && export BASH_ENV='$WSL_LIB/trace-init.sh' TRBLDOC_TRACE_DIR='$WSL_TRACE' TRBLDOC_TRACE_MODE='time' TRBLDOC_TRACE_TARGETS='$TRACE_TARGETS'; bats --report-formatter junit --output '$WSL_JUNIT' $SUITE_DIRS_REL" \ + > "$OUTDIR/bats-hotspots-output.txt" 2>&1 \ + || tb_info "bats (WSL) reported failures during hotspot run" + fi + + if ls "$TRACE_DIR"/trace.*.log >/dev/null 2>&1; then + awk -f "$LIB_DIR/trace-extract.awk" -v MODE=time "$TRACE_DIR"/trace.*.log \ + | sort -t "$(printf '\t')" -k1 -rn > "$OUTDIR/hotspots.tsv" + { + printf 'total_ms,hits,file,line\n' + awk -F '\t' '{ printf "%s,%s,%s,%s\n",$1,$2,$3,$4 }' "$OUTDIR/hotspots.tsv" + } > "$OUTDIR/hotspots.csv" + # Resolve first target path for source display in the HTML. + FIRST_TARGET="${TARGETS%% *}" + case "$FIRST_TARGET" in + /*) SRC_PATH="$FIRST_TARGET" ;; + *) SRC_PATH="$REPO_ROOT/$FIRST_TARGET" ;; + esac + awk -f "$LIB_DIR/render-hotspots.awk" \ + -v TITLE="trbldoc line hotspots ($SUITE)" \ + -v SRC="$SRC_PATH" \ + "$OUTDIR/hotspots.tsv" > "$OUTDIR/hotspots.html" + else + if grep -qi "bash\\\\r" "$OUTDIR/bats-hotspots-output.txt" 2>/dev/null; then + tb_die "bats appears broken (CRLF shebang: 'bash\\r') during hotspot run. Reinstall bats or run dos2unix on the bats scripts." + fi + tb_info "no trace data captured; skipping hotspots report" + HOTSPOTS=0 + fi +fi + +# ── printed summary ────────────────────────────────────────────────────────── +printf '\n==== slowest %s tests ====\n' "$TOPN" +awk -F '\t' -v n="$TOPN" 'NR<=n { printf "%8.3fs %s\n", $1, $3 }' \ + "$OUTDIR/timings.sorted.tsv" + +printf '\nReports written to: %s\n' "$OUTDIR" +printf ' per-test timing : %s\n' "$OUTDIR/timings.html" +[ "$HOTSPOTS" -eq 1 ] && printf ' line hotspots : %s\n' "$OUTDIR/hotspots.html" diff --git a/tests/fixtures/c/directives.c b/tests/fixtures/c/directives.c new file mode 100644 index 0000000..4becc88 --- /dev/null +++ b/tests/fixtures/c/directives.c @@ -0,0 +1,7 @@ +/* md +@name test/directives +@file generated/source.c +@ref test/directives-ref +Directive test body content. +*/ +void directives_demo() {} diff --git a/tests/fixtures/c/inline_ref.c b/tests/fixtures/c/inline_ref.c new file mode 100644 index 0000000..72ef395 --- /dev/null +++ b/tests/fixtures/c/inline_ref.c @@ -0,0 +1,5 @@ +/* md +@name test/inline-ref +See @lua/Coroutines for details on cooperative multitasking. +*/ +void inline_ref_func() {} diff --git a/tests/fixtures/c/multi_chunk.c b/tests/fixtures/c/multi_chunk.c new file mode 100644 index 0000000..57fd463 --- /dev/null +++ b/tests/fixtures/c/multi_chunk.c @@ -0,0 +1,11 @@ +/* md +@name test/alpha +First chunk content. +*/ +void foo() {} + +/* md +@name test/beta +Second chunk content. +*/ +void bar() {} diff --git a/tests/fixtures/c/no_comments.c b/tests/fixtures/c/no_comments.c new file mode 100644 index 0000000..a667cef --- /dev/null +++ b/tests/fixtures/c/no_comments.c @@ -0,0 +1,3 @@ +/* This is a regular comment, not a doc comment */ +// Another non-doc comment +int noop() { return 42; } diff --git a/tests/fixtures/c/ref_string.c b/tests/fixtures/c/ref_string.c new file mode 100644 index 0000000..711ac20 --- /dev/null +++ b/tests/fixtures/c/ref_string.c @@ -0,0 +1,6 @@ +/* md +@name test/ref-string +@ref lua/Coroutines +Return type: {{ ref_string }} +*/ +void ref_string_func() {} diff --git a/tests/fixtures/c/rst_exec.c b/tests/fixtures/c/rst_exec.c new file mode 100644 index 0000000..3aabf9f --- /dev/null +++ b/tests/fixtures/c/rst_exec.c @@ -0,0 +1,8 @@ +/* rst +@name test/rst-exec +RST Heading +=========== + +This chunk should be rendered by the rst renderer. +*/ +int noop() { return 0; } diff --git a/tests/fixtures/c/shared_namespace.c b/tests/fixtures/c/shared_namespace.c new file mode 100644 index 0000000..25f064a --- /dev/null +++ b/tests/fixtures/c/shared_namespace.c @@ -0,0 +1,11 @@ +/* md +@name test/shared +Part one of the shared page. +*/ +void part_one() {} + +/* md +@name test/shared +Part two of the shared page. +*/ +void part_two() {} diff --git a/tests/fixtures/c/simple.c b/tests/fixtures/c/simple.c new file mode 100644 index 0000000..2595929 --- /dev/null +++ b/tests/fixtures/c/simple.c @@ -0,0 +1,5 @@ +/* md +@name test/simple +Hello from a C comment. +*/ +int main() { return 0; } diff --git a/tests/fixtures/c/src_repo.c b/tests/fixtures/c/src_repo.c new file mode 100644 index 0000000..fb0d8b6 --- /dev/null +++ b/tests/fixtures/c/src_repo.c @@ -0,0 +1,5 @@ +/* md +@name test/src-repo +Generated from {{ src_repo }}/src.c +*/ +void src_repo_func() {} diff --git a/tests/fixtures/c/toc.c b/tests/fixtures/c/toc.c new file mode 100644 index 0000000..bb0782b --- /dev/null +++ b/tests/fixtures/c/toc.c @@ -0,0 +1,5 @@ +/* md +@name test/toc-chunk +Navigation: {{ toc }} +*/ +void toc_func() {} diff --git a/tests/fixtures/lua/simple.lua b/tests/fixtures/lua/simple.lua new file mode 100644 index 0000000..de527a9 --- /dev/null +++ b/tests/fixtures/lua/simple.lua @@ -0,0 +1,5 @@ +--[[ md +@name test/lua-simple +Hello from a Lua comment. +]] +local x = 1 diff --git a/tests/fixtures/lua/single_line.lua b/tests/fixtures/lua/single_line.lua new file mode 100644 index 0000000..f541825 --- /dev/null +++ b/tests/fixtures/lua/single_line.lua @@ -0,0 +1,2 @@ +--[[ md This comment opens and closes on one line — no chunk. ]] +local x = 1 diff --git a/tests/fixtures/md/readme.md b/tests/fixtures/md/readme.md new file mode 100644 index 0000000..5833cab --- /dev/null +++ b/tests/fixtures/md/readme.md @@ -0,0 +1,2 @@ +# Project Overview +Hello from the readme. diff --git a/tests/fixtures/md/simple.md b/tests/fixtures/md/simple.md new file mode 100644 index 0000000..494ade7 --- /dev/null +++ b/tests/fixtures/md/simple.md @@ -0,0 +1,3 @@ +@name test/md-simple +# Markdown Page +Hello from a markdown file. diff --git a/tests/fixtures/py/simple.py b/tests/fixtures/py/simple.py new file mode 100644 index 0000000..b8e2ce9 --- /dev/null +++ b/tests/fixtures/py/simple.py @@ -0,0 +1,6 @@ +'''md +@name test/py-simple +Hello from a Python doc comment. +''' +def noop(): + pass diff --git a/tests/fixtures/py/single_line.py b/tests/fixtures/py/single_line.py new file mode 100644 index 0000000..baafa10 --- /dev/null +++ b/tests/fixtures/py/single_line.py @@ -0,0 +1,2 @@ +'''This comment opens and closes on one line — no chunk.''' +x = 1 diff --git a/tests/helpers/common.bash b/tests/helpers/common.bash new file mode 100644 index 0000000..779cc96 --- /dev/null +++ b/tests/helpers/common.bash @@ -0,0 +1,263 @@ +# tests/helpers/common.bash +# Sourced by every bats test file via `load`. +# Provides: workspace isolation, PATH stubs, assertion helpers. + +# ── workspace setup/teardown ──────────────────────────────────────────────── + +# Call in setup(): creates an isolated temp directory and cds into it. +# A stub for md2html is pre-installed in a fake bin/ on PATH. +setup_workspace() { + TEST_TMPDIR="$(mktemp -d)" + # trap 'rm -rf $TEST_TMPDIR' EXIT + # Fake bin injected before the real PATH so stubs take priority + STUB_BIN="$TEST_TMPDIR/bin" + mkdir -p "$STUB_BIN" + install_stub_md2html "$STUB_BIN" + ORIG_DIR="$(pwd)" + export PATH="$STUB_BIN:$PATH" + # Work inside the temp dir so .trblcache is isolated + pushd "$TEST_TMPDIR" >/dev/null +} + +# Call in teardown(): removes the temp directory. +teardown_workspace() { + popd >/dev/null 2>&1 || true + rm -rf "$TEST_TMPDIR" +} + +# ── stubs ─────────────────────────────────────────────────────────────────── + +# md2html stub: echoes a predictable HTML marker for easy assertion. +# Accepts any flags; always succeeds. +install_stub_md2html() { + local bin_dir="$1" + cat > "$bin_dir/md2html" <<'EOF' +#!/usr/bin/env bash +# Stub: find the last non-flag argument as the input file, or read stdin. +# Flags (args starting with -) from the md alias are ignored. +input="" +for arg in "$@"; do + [[ "$arg" == -* ]] || input="$arg" +done +if [[ -n "$input" ]]; then + content="$(cat "$input")" +else + content="$(cat)" +fi +echo "<p class=\"stub-md\">$content</p>" +EOF + chmod +x "$bin_dir/md2html" +} + + +# ── AWK program builders ───────────────────────────────────────────────────── +# These mirror the comment_form patterns from trbldoc.sh so extraction +# logic can be unit-tested without sourcing the whole script. + +CACHE_DIR=".trblcache" + +# Returns the AWK program string for C-style /* */ comments. +awk_prog_c() { + local cache="$1" + cat <<EOF +BEGIN{} +/\*\// { + if(in_comment) {print "EOF"} + in_comment=0 +} +in_comment {print} +/\/\* .+/ && !/\*\// { + print "chunkfile=\$(mktemp -p ${cache}/chunks)" + print "cat > \$chunkfile << \"EOF\"\\n@file " name ":" NR ; + if(\$2) {\$1=""; print "@exec " \$0} + in_comment=1; +} +EOF +} + +# Returns the AWK program string for Lua +awk_prog_lua() { + local cache="$1" + cat <<EOF +BEGIN{} +/\]\]/ { + if(in_comment) {print "EOF"} + in_comment=0 +} +in_comment {print} +/--\[\[ .+/ && !/\]\]$/ { + print "chunkfile=\$(mktemp -p ${cache}/chunks)" + print "cat > \$chunkfile << \"EOF\"\\n@file " name ":" NR ; + if(\$2) {\$1=""; print "@exec " \$0} + in_comment=1; +} +EOF +} + +# Returns the AWK program string for Python. +# Handles '''renderer format (renderer glued directly to ''', no space required). +awk_prog_py() { + local cache="$1" + sed "s|__PY_CACHE__|${cache}|g" << 'PYEOF' +BEGIN{} +/'''/ { + if(in_comment) {print "EOF"} + in_comment=0 +} +in_comment {print} +/'''.+/ && !/'''$/ { + print "chunkfile=$(mktemp -p __PY_CACHE__/chunks)" + print "cat > $chunkfile << \"EOF\"\n@file " name ":" NR ; + renderer = $0 + sub(/'''/, "", renderer) + sub(/^[[:space:]]+/, "", renderer) + sub(/[[:space:]]+$/, "", renderer) + if (renderer != "") {print "@exec " renderer} + in_comment=1; +} +PYEOF +} + +# Returns the AWK program string for .md files +awk_prog_md() { + local cache="$1" + cat <<EOF +BEGIN{ + print "chunkfile=\$(mktemp -p ${cache}/chunks)" + print "cat > \$chunkfile << \"EOF\"\\n@file " name ":1\\n@exec md" ; +} +{print \$0} +END{print "EOF"} +EOF +} + +# ── extraction helper ──────────────────────────────────────────────────────── + +# run_extraction FILE AWK_PROG CACHE_DIR +# Runs AWK against FILE, evals the output shell script, and populates CACHE_DIR/chunks. +run_extraction() { + local file="$1" + local awk_prog="$2" + local cache="$3" + mkdir -p "$cache/chunks" + local chunk_script + chunk_script=$(awk -v name="$file" "$awk_prog" < "$file") + eval "$chunk_script" +} + +# chunk_count CACHE_DIR +# Prints the number of chunk files produced. +chunk_count() { + find "$1/chunks" -type f 2>/dev/null | wc -l | tr -d ' ' +} + +# chunk_contains CACHE_DIR PATTERN +# Succeeds if any chunk file contains a line matching PATTERN (grep -q). +chunk_contains() { + grep -rl "$2" "$1/chunks" >/dev/null 2>&1 +} + +# ── assertion helpers ──────────────────────────────────────────────────────── + +# assert_file_exists PATH +assert_file_exists() { + if [[ ! -f "$1" ]]; then + echo "ASSERTION FAILED: expected file to exist: $1" >&2 + return 1 + fi +} + +# assert_dir_exists PATH +assert_dir_exists() { + if [[ ! -d "$1" ]]; then + echo "ASSERTION FAILED: expected directory to exist: $1" >&2 + return 1 + fi +} + +# assert_contains FILE PATTERN +assert_contains() { + if ! grep -q "$2" "$1" 2>/dev/null; then + echo "ASSERTION FAILED: '$2' not found in $1" >&2 + echo "--- file contents ---" >&2 + cat "$1" >&2 + return 1 + fi +} + +# assert_not_contains FILE PATTERN +assert_not_contains() { + if grep -q "$2" "$1" 2>/dev/null; then + echo "ASSERTION FAILED: '$2' unexpectedly found in $1" >&2 + return 1 + fi +} + +# assert_equal ACTUAL EXPECTED +assert_equal() { + if [[ "$1" != "$2" ]]; then + echo "ASSERTION FAILED: expected '$2', got '$1'" >&2 + return 1 + fi +} + +# ── HTML validation helpers ────────────────────────────────────────────────── + +# assert_valid_html FILE +# Validates FILE as a complete HTML5 document using tidy. +# Mustache template markers ({{...}}, {{{...}}}) are substituted with +# structurally valid HTML before linting to avoid false positives from +# template syntax that tidy cannot parse. +# Skips automatically if tidy is not installed. +# tidy exit codes: 0 = clean, 1 = warnings only, 2 = errors. +# This helper fails only on exit code 2 (actual errors). +assert_valid_html() { + local file="$1" + if ! command -v tidy >/dev/null 2>&1; then + skip "tidy not installed (apt install tidy)" + fi + local tmp + tmp=$(mktemp --suffix=.html) + # Replace {{ var }} template markers with valid HTML equivalents so tidy + # sees a structurally correct document. + sed \ + -e 's/{{[^}]*}}/placeholder/g' \ + "$file" > "$tmp" + local tidy_out rc + tidy_out=$(tidy -errors -quiet --show-warnings no -utf8 "$tmp" 2>&1) + rc=$? + rm -f "$tmp" + if [ "$rc" -ge 2 ]; then + echo "HTML VALIDATION FAILED: $file" >&2 + echo "$tidy_out" >&2 + return 1 + fi +} + +# assert_valid_html_fragment FILE +# Wraps FILE in a minimal HTML5 document shell, then validates with tidy. +# Use for HTML fragment files (e.g. rendered chunk index.html output) that +# are not complete documents on their own. +# Skips automatically if tidy is not installed. +assert_valid_html_fragment() { + local file="$1" + if ! command -v tidy >/dev/null 2>&1; then + skip "tidy not installed (apt install tidy)" + fi + local tmp + tmp=$(mktemp --suffix=.html) + { + echo '<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><title>fragment</title></head><body>' + cat "$file" + echo '</body></html>' + } > "$tmp" + local tidy_out rc + tidy_out=$(tidy -errors -quiet --show-warnings no -utf8 "$tmp" 2>&1) + rc=$? + rm -f "$tmp" + if [ "$rc" -ge 2 ]; then + echo "HTML VALIDATION FAILED: $file" >&2 + echo "$tidy_out" >&2 + return 1 + fi +} diff --git a/tests/helpers/profiler.bash b/tests/helpers/profiler.bash new file mode 100644 index 0000000..c3d5207 --- /dev/null +++ b/tests/helpers/profiler.bash @@ -0,0 +1,53 @@ +# Profiler implementation +profile() { + # Set default options + local file=profiler.log + + # Open a file descriptor for writing to $file and save it in $tracefd + exec {tracefd}>"$file" + # Send trace output to $tracefd + export BASH_XTRACEFD="$tracefd" + # Print microsecond time in trace output + export PS4='+ $EPOCHREALTIME ' + # Enable tracing, run script, and disable tracing + set -x + bash -x -- "$@" + set +x + # Un-redirect the trace output. This also closes the file descriptor. + unset BASH_XTRACEFD + export -n BASH_XTRACEFD PS4 + + # Remove "source" line from output and change last line to only include the + # timestamp with no name. + sed -i -e 1d -e '$s/\(+\+ [0-9\.]\+\) .*$/\1/' "$file" +} + +analyze() { + # Set defaults + local file=profiler.log + local -a sortcmd=(cat) tablecmd=(cat) + + # Declare variables + local timestamp nestlvl cmd next_timestamp next_nestlvl next_cmd duration + # Open file as a file descriptor so we can re-use the same stream + exec {fd}<"$file"; + # Read first line + read -r nestlvl timestamp cmd <&"$fd" + # Process each line + while read -r next_nestlvl next_timestamp next_cmd + do + duration="$(echo "scale=6; $next_timestamp" - "$timestamp" | bc)" + # Prepend leading zero + if [ "${duration:0:1}" = . ] + then + duration="0$duration" + fi + echo "$duration" "$nestlvl" "$cmd" + + timestamp="$next_timestamp" + nestlvl="$next_nestlvl" + cmd="$next_cmd" + done <&"$fd" | "${sortcmd[@]}" | "${tablecmd[@]}" + # Close file descriptor + exec {fd}<&- +} diff --git a/tests/integration/cli.bats b/tests/integration/cli.bats new file mode 100644 index 0000000..5ffc55c --- /dev/null +++ b/tests/integration/cli.bats @@ -0,0 +1,215 @@ +#!/usr/bin/env bats +# tests/integration/cli.bats +# Tests for trbldoc.sh command-line option parsing. +# +# Each test runs the full script (with md2html stubs) and asserts that +# the correct sources are scanned and outputs produced, based on the options +# passed. + +load "../helpers/common.bash" +load "../helpers/profiler.bash" + +FIXTURES="$BATS_TEST_DIRNAME/../fixtures" +_REPO_ROOT="${REPO_ROOT:-$(cd "$BATS_TEST_DIRNAME/../.." && pwd)}" +TRBLDOC="$_REPO_ROOT/trbldoc.sh" + +setup() { + setup_workspace + CACHE=".trblcache" + NS="$CACHE/namespace" +} + +teardown() { + teardown_workspace +} + +# ── helpers ────────────────────────────────────────────────────────────────── + +run_pipeline() { + bash "$TRBLDOC" "$@" # 2>/dev/null + #profile "$TRBLDOC" "$@" 2>/dev/null + #analyze >> profile.txt +} + +stage_fixture() { + local src="$1" + local dest="$2" + mkdir -p "$(dirname "$dest")" + cp -r "$src" "$dest" +} + +# ── error cases ────────────────────────────────────────────────────────────── + +@test "cli: no arguments exits non-zero with an error message" { + run bash "$TRBLDOC" 2>&1 + [ "$status" -ne 0 ] + [[ "$output" == *"trbldoc:"* ]] +} + +@test "cli: unknown option exits non-zero" { + run bash "$TRBLDOC" -z 2>&1 + [ "$status" -ne 0 ] +} + +@test "cli: option missing its required argument exits non-zero" { + run bash "$TRBLDOC" -s 2>&1 + [ "$status" -ne 0 ] +} + +# ── -s (source folder) ─────────────────────────────────────────────────────── + +@test "cli: -s flag specifies the source folder to scan" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_dir_exists "$NS/test/simple" +} + +@test "cli: multiple -s flags scan all specified folders" { + stage_fixture "$FIXTURES/c/simple.c" "src1/simple.c" + stage_fixture "$FIXTURES/c/multi_chunk.c" "src2/multi_chunk.c" + run_pipeline -s src1 -s src2 + assert_dir_exists "$NS/test/simple" + assert_dir_exists "$NS/test/alpha" +} + +# ── -o (output directory) ──────────────────────────────────────────────────── + +@test "cli: -o flag creates the specified output directory" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src -o my_output + assert_dir_exists "my_output" + assert_file_exists "my_output/test/simple/index.html" +} + +@test "cli: without -o the default output directory .trblcache/built is created" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_dir_exists "$CACHE/built" +} + +# ── -f (custom file extension) ─────────────────────────────────────────────── + +@test "cli: -f flag registers a custom file extension for scanning" { + mkdir -p src + # Create a .myext file with C-style doc comments. + # The -f option takes ext;start_regex;end_regex (AWK patterns without slashes). + cat > src/custom.myext <<'SRCEOF' +/* md +@name test/custom-ext +Hello from a custom extension. +*/ +int noop() {} +SRCEOF + run_pipeline -s src -f 'myext;\/\*;\*\/' + assert_dir_exists "$NS/test/custom-ext" +} + +@test "cli: -f registered extension produces correct chunk content" { + mkdir -p src + cat > src/custom.myext <<'SRCEOF' +/* md +@name test/custom-content +Content from custom ext. +*/ +int noop() {} +SRCEOF + run_pipeline -s src -f 'myext;\/\*;\*\/' + assert_contains "$NS/test/custom-content/index.html" "Content from custom ext" +} + +# ── TRBLDOC_MD (markdown renderer override) ────────────────────────────────── + +@test "cli: TRBLDOC_MD overrides the markdown renderer for .md files" { + mkdir -p src + cat > src/doc.md <<'EOF' +@name test/md-override +# Heading +Some content. +EOF + cat > custom_md.sh <<'EOF' +#!/usr/bin/env bash +# Stub: emit a recognisable marker instead of real HTML. +cat "$@" > /dev/null +echo "<p class=\"custom-md-renderer\">custom renderer was here</p>" +EOF + chmod +x custom_md.sh + TRBLDOC_MD="./custom_md.sh" run_pipeline -s src + assert_contains "$NS/test/md-override/index.html" "custom-md-renderer" +} + +@test "cli: TRBLDOC_MD override is not used when unset (default md2html stub runs)" { + stage_fixture "$FIXTURES/md/simple.md" "src/simple.md" + run_pipeline -s src + # The default stub wraps content in class=stub-md; confirm it ran. + assert_contains "$NS/test/md-simple/index.html" "stub-md" +} + +# ── built-in site assembly ─────────────────────────────────────────────────── + +@test "cli: built output is assembled by default without external zod" { + mkdir -p src + cat > src/simple.c <<'EOF' +/* md +@name test/simple +Hello from a C comment. +*/ +int main() { return 0; } +EOF + run_pipeline -s src + assert_file_exists "$CACHE/built/test/simple/index.html" + assert_contains "$CACHE/built/test/simple/index.html" "<!DOCTYPE html>" + assert_contains "$CACHE/built/test/simple/index.html" "Hello from a C comment" + assert_not_contains "$CACHE/built/test/simple/index.html" "{{{yield}}}" +} + +@test "cli: default main.layout includes stylesheet and excludes highlightjs mermaid and mathjax" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/main.layout" '<link href="/stylesheets/style.css" rel="stylesheet">' + assert_not_contains "$NS/main.layout" "highlight.min.js" + assert_not_contains "$NS/main.layout" "highlightAll()" + assert_not_contains "$NS/main.layout" "mermaid.min.js" + assert_not_contains "$NS/main.layout" "mermaid.initialize" + assert_not_contains "$NS/main.layout" "mathjax" +} + +@test "cli: -l uses the provided main.layout template" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + cat > alt.layout <<'EOF' +<!DOCTYPE html> +<html lang="en"> +<head><meta charset="utf-8"><title>{{ title }}</title></head> +<body> +<header>ALT {{ breadcrumb }}</header> +<nav><ul>{{ nav }}</ul></nav> +<main>{{ yield }}</main> +</body> +</html> +EOF + run_pipeline -s src -l alt.layout + assert_contains "$NS/main.layout" "ALT {{ breadcrumb }}" + assert_not_contains "$NS/main.layout" "Generated by trbldoc" +} + +@test "cli: -l fails when layout template file does not exist" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run bash "$TRBLDOC" -s src -l does-not-exist.layout 2>&1 + [ "$status" -ne 0 ] + [[ "$output" == *"layout template not found"* ]] +} + +@test "cli: files without a registered extension are not scanned" { + mkdir -p src + # .xyz has no registered comment form; no output namespace should appear. + cat > src/ignored.xyz <<'SRCEOF' +/* md +@name test/should-not-exist +This should not be processed. +*/ +SRCEOF + run_pipeline -s src + if [[ -d "$NS/test/should-not-exist" ]]; then + echo "FAILED: .xyz file was unexpectedly processed" >&2 + return 1 + fi +} diff --git a/tests/integration/pipeline.bats b/tests/integration/pipeline.bats new file mode 100644 index 0000000..2f43304 --- /dev/null +++ b/tests/integration/pipeline.bats @@ -0,0 +1,842 @@ +#!/usr/bin/env bats +# tests/integration/pipeline.bats +# End-to-end tests for trbldoc.sh. +# +# External markdown rendering is stubbed via helpers/common.bash. +# Tests inspect both .trblcache/namespace intermediate output and the assembled +# .trblcache/built pages produced by trbldoc itself. +# +# Tests marked `skip` document known bugs from todo.md. Remove `skip` after +# the corresponding bug is fixed. + +load "../helpers/common.bash" + +FIXTURES="$BATS_TEST_DIRNAME/../fixtures" +# REPO_ROOT is exported by run_tests.sh; fall back to relative path when bats +# is invoked directly on this file. +_REPO_ROOT="${REPO_ROOT:-$(cd "$BATS_TEST_DIRNAME/../.." && pwd)}" +TRBLDOC="$_REPO_ROOT/trbldoc.sh" + +setup() { + setup_workspace + CACHE=".trblcache" + NS="$CACHE/namespace" +} + +teardown() { + teardown_workspace +} + +# ── helpers ───────────────────────────────────────────────────────────────── + +# Copy a fixture file into a temporary source tree and run the pipeline. +# Usage: run_pipeline SRC_DIR [additional source dirs...] +run_pipeline() { + bash "$TRBLDOC" "$@" +} + +# Copy a fixture into a local source dir for a clean run. +stage_fixture() { + local src="$1" # path to fixture file or dir + local dest="$2" # destination relative path in tmp workspace + mkdir -p "$(dirname "$dest")" + cp -r "$src" "$dest" +} + +install_busybox_tool_wrapper() { + local tool="$1" + local busybox_bin="$2" + cat > "$STUB_BIN/$tool" <<EOF +#!/bin/sh +if [ -n "\${TRBLDOC_TEST_TOOL_LOG:-}" ]; then + printf '%s\n' "$tool" >> "\$TRBLDOC_TEST_TOOL_LOG" +fi +exec "$busybox_bin" $tool "\$@" +EOF + chmod +x "$STUB_BIN/$tool" +} + +# ── cache structure ────────────────────────────────────────────────────────── + +@test "pipeline: .trblcache directory is created on run" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" +run_pipeline -s src + assert_dir_exists ".trblcache" +} + +@test "pipeline: namespace subdirectory is created on run" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_dir_exists ".trblcache/namespace" +} + +@test "pipeline: chunks subdirectory is created on run" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_dir_exists ".trblcache/chunks" +} + +@test "pipeline: global.meta is created" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_file_exists "$NS/global.meta" +} + +@test "pipeline: main.layout is created" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_file_exists "$NS/main.layout" +} + +@test "pipeline: main.layout passes HTML validation" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_valid_html "$NS/main.layout" +} + +@test "pipeline: main.layout has well-formed HTML structure" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + local layout="$NS/main.layout" + # <html> must appear before <head> + local html_line head_line body_line footer_line close_body_line + html_line=$(grep -n '<html' "$layout" | head -1 | cut -d: -f1) + head_line=$(grep -n '<head' "$layout" | head -1 | cut -d: -f1) + body_line=$(grep -n '<body' "$layout" | head -1 | cut -d: -f1) + footer_line=$(grep -n '<footer' "$layout" | head -1 | cut -d: -f1) + close_body_line=$(grep -n '</body>' "$layout" | head -1 | cut -d: -f1) + # <html> before <head> + [ "$html_line" -lt "$head_line" ] + # exactly one <head> + local head_count + head_count=$(grep -c '<head>' "$layout") + assert_equal "$head_count" "1" + # <footer> inside <body>: footer before </body> + [ "$footer_line" -lt "$close_body_line" ] +} + +@test "pipeline: nav.partial is created" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_file_exists "$NS/nav.partial" +} + +# ── namespace output ───────────────────────────────────────────────────────── + +@test "pipeline: named chunk creates namespace output folder" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_dir_exists "$NS/test/simple" +} + +@test "pipeline: namespace output folder contains index.html" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_file_exists "$NS/test/simple/index.html" +} + +@test "pipeline: built page exists for rendered namespace" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_file_exists "$CACHE/built/test/simple/index.html" +} + +@test "pipeline: built page applies layout tokens and contains rendered body" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$CACHE/built/test/simple/index.html" "<!DOCTYPE html>" + assert_contains "$CACHE/built/test/simple/index.html" "<title>test/simple</title>" + assert_contains "$CACHE/built/test/simple/index.html" "Hello from a C comment" + assert_contains "$CACHE/built/test/simple/index.html" "href='/test/simple'" +} + +@test "pipeline: index.html contains rendered chunk body" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/test/simple/index.html" "Hello from a C comment" +} + +@test "pipeline: runs with BusyBox awk and xargs from PATH" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + + if ! command -v busybox >/dev/null 2>&1; then + skip "busybox is not installed" + fi + + local busybox_bin + busybox_bin="$(command -v busybox)" + export TRBLDOC_TEST_TOOL_LOG="$TEST_TMPDIR/busybox-tools.log" + : > "$TRBLDOC_TEST_TOOL_LOG" + + install_busybox_tool_wrapper awk "$busybox_bin" + install_busybox_tool_wrapper xargs "$busybox_bin" + + run bash "$TRBLDOC" -s src 2>&1 + [ "$status" -eq 0 ] + + assert_file_exists "$NS/test/simple/index.html" + assert_contains "$NS/test/simple/index.html" "Hello from a C comment" + + local awk_count xargs_count + awk_count=$(grep -c '^awk$' "$TRBLDOC_TEST_TOOL_LOG" || true) + xargs_count=$(grep -c '^xargs$' "$TRBLDOC_TEST_TOOL_LOG" || true) + [ "$awk_count" -gt 0 ] + [ "$xargs_count" -gt 0 ] +} + +@test "pipeline: rendered namespace index.html passes HTML validation" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_valid_html_fragment "$NS/test/simple/index.html" +} + +@test "pipeline: multi-chunk file creates two separate namespace folders" { + stage_fixture "$FIXTURES/c/multi_chunk.c" "src/multi_chunk.c" + run_pipeline -s src + assert_dir_exists "$NS/test/alpha" + assert_dir_exists "$NS/test/beta" +} + +@test "pipeline: each namespace folder has its own index.html" { + stage_fixture "$FIXTURES/c/multi_chunk.c" "src/multi_chunk.c" + run_pipeline -s src + assert_file_exists "$NS/test/alpha/index.html" + assert_file_exists "$NS/test/beta/index.html" +} + +@test "pipeline: each namespace index.html contains the correct body" { + stage_fixture "$FIXTURES/c/multi_chunk.c" "src/multi_chunk.c" + run_pipeline -s src + assert_contains "$NS/test/alpha/index.html" "First chunk content" + assert_contains "$NS/test/beta/index.html" "Second chunk content" +} + +@test "pipeline: two chunks with same @name both appear in the shared index.html" { + stage_fixture "$FIXTURES/c/shared_namespace.c" "src/shared_namespace.c" + run_pipeline -s src + assert_contains "$NS/test/shared/index.html" "Part one" + assert_contains "$NS/test/shared/index.html" "Part two" +} + +@test "pipeline: @priority orders chunks before namespace/source ordering" { + mkdir -p src + cat > src/priority_high.c <<'EOF' +/* md +@name test/priority-order +@priority 100 +Priority high +*/ +int high() { return 0; } +EOF + cat > src/priority_low.c <<'EOF' +/* md +@name test/priority-order +@priority 1 +Priority low +*/ +int low() { return 0; } +EOF + cat > src/priority_default.c <<'EOF' +/* md +@name test/priority-order +Priority default +*/ +int def() { return 0; } +EOF + run_pipeline -s src + local out="$NS/test/priority-order/index.html" + local line_high line_low line_default first_render second_render + line_high=$(grep -n "Priority high" "$out" | head -1 | cut -d: -f1) + line_low=$(grep -n "Priority low" "$out" | head -1 | cut -d: -f1) + line_default=$(grep -n "Priority default" "$out" | head -1 | cut -d: -f1) + [ "$line_high" -lt "$line_low" ] + [ "$line_low" -lt "$line_default" ] + first_render="$(cat "$out")" + run_pipeline -s src + second_render="$(cat "$out")" + assert_equal "$first_render" "$second_render" +} + +@test "pipeline: Lua file chunk creates correct namespace output" { + stage_fixture "$FIXTURES/lua/simple.lua" "src/simple.lua" + run_pipeline -s src + assert_file_exists "$NS/test/lua-simple/index.html" +} + +@test "pipeline: Python file chunk creates correct namespace output" { + stage_fixture "$FIXTURES/py/simple.py" "src/simple.py" + run_pipeline -s src + assert_file_exists "$NS/test/py-simple/index.html" +} + +@test "pipeline: Python chunk body appears in namespace output" { + stage_fixture "$FIXTURES/py/simple.py" "src/simple.py" + run_pipeline -s src + assert_contains "$NS/test/py-simple/index.html" "Hello from a Python doc comment" +} + +@test "pipeline: Markdown file chunk creates correct namespace output" { + stage_fixture "$FIXTURES/md/simple.md" "src/simple.md" + run_pipeline -s src + assert_file_exists "$NS/test/md-simple/index.html" +} + +@test "pipeline: @exec rst chunk is rendered using TRBLDOC_RST override" { + stage_fixture "$FIXTURES/c/rst_exec.c" "src/rst_exec.c" + cat > custom_rst.sh <<'EOF' +#!/usr/bin/env bash +if [[ $# -gt 0 ]]; then + content="$(cat "$1")" +else + content="$(cat)" +fi +echo "<section class=\"stub-rst\">$content</section>" +EOF + chmod +x custom_rst.sh + TRBLDOC_RST="./custom_rst.sh" run_pipeline -s src + assert_file_exists "$NS/test/rst-exec/index.html" + assert_contains "$NS/test/rst-exec/index.html" "stub-rst" + assert_contains "$NS/test/rst-exec/index.html" "RST Heading" +} + +@test "pipeline: file with no doc comments creates no namespace folders" { + stage_fixture "$FIXTURES/c/no_comments.c" "src/no_comments.c" + run_pipeline -s src + # Only the fixed namespace files (global.meta, nav.partial, etc.) should exist; + # no test/* content namespace folder should be present. + if [[ -d "$NS/test" ]]; then + echo "FAILED: unexpected namespace folder created for file with no doc comments" >&2 + ls "$NS/test" >&2 + return 1 + fi +} + +# ── global.meta ───────────────────────────────────────────────────────────── + +@test "pipeline: global.meta contains entry for the processed namespace" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/global.meta" "test/simple" +} + +# ── nav.partial ────────────────────────────────────────────────────────────── + +@test "pipeline: nav.partial contains link for processed namespace" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/nav.partial" "test/simple" +} + +@test "pipeline: nav.partial does not contain duplicate entries" { + stage_fixture "$FIXTURES/c/shared_namespace.c" "src/shared_namespace.c" + run_pipeline -s src + # Count occurrences of the shared namespace link; must be exactly 1. + local count + count=$(grep -c "test/shared" "$NS/nav.partial" || true) + assert_equal "$count" "1" +} + +@test "pipeline: nav.partial groups related namespaces as a tree" { + mkdir -p doc + cat > doc/doc.md <<'EOF' +# Doc +EOF + cat > doc/index.md <<'EOF' +# Index +EOF + cat > doc/love_loader.md <<'EOF' +# Love Loader +EOF + cat > doc/ui_notes.md <<'EOF' +# UI Notes +EOF + run_pipeline -s doc + assert_contains "$NS/nav.partial" "<li>doc/<ul>" + assert_contains "$NS/nav.partial" "href='/doc/doc.md'>doc.md</a>" + assert_contains "$NS/nav.partial" "href='/doc/index.md'>index.md</a>" + assert_contains "$NS/nav.partial" "href='/doc/love_loader.md'>love_loader.md</a>" + assert_contains "$NS/nav.partial" "href='/doc/ui_notes.md'>ui_notes.md</a>" +} + +@test "pipeline: dotted @name values are allowed" { + mkdir -p src + cat > src/dotted.c <<'EOF' +/* md +@name ui.element.Element +Dotted namespace. +*/ +int dotted() { return 0; } +EOF + run_pipeline -s src + assert_dir_exists "$NS/ui.element.Element" + assert_contains "$NS/nav.partial" "href='/ui.element.Element'" +} + +@test "pipeline: explicit slash-separated @name builds correct nav tree" { + mkdir -p src + cat > src/ui_parent.c <<'EOF' +/* md +@name ui +UI root documentation. +*/ +int ui_root() { return 0; } +EOF + cat > src/ui_element.c <<'EOF' +/* md +@name ui/element/Element +Element details. +*/ +int ui_element() { return 0; } +EOF + run_pipeline -s src + assert_contains "$NS/nav.partial" "href='/ui'>ui</a>/<ul>" + assert_contains "$NS/nav.partial" "href='/ui/element/Element'>Element</a>" + assert_dir_exists "$NS/ui/element/Element" +} + +# ── inline reference resolution ───────────────────────────────────────────── + +@test "pipeline: @lua/<path> tokens are resolved to Markdown links" { + stage_fixture "$FIXTURES/c/inline_ref.c" "src/inline_ref.c" + run_pipeline -s src + # ref_resolvers/lua.sh "Coroutines" returns https://www.lua.org/manual/5.2/manual.html#2.6 + assert_contains "$NS/test/inline-ref/index.html" "lua.org/manual" + assert_not_contains "$NS/test/inline-ref/index.html" "@lua/Coroutines" +} + +@test "pipeline: -R adds a custom resolver prefix" { + mkdir -p src + cat > src/custom_ref.c <<'EOF' +/* md +@name test/custom-ref +Link: @custom/topic +*/ +int noop() {} +EOF + cat > custom_resolver.sh <<'EOF' +#!/usr/bin/env bash +opt="$1" +path="$(cat)" +printf "https://example.test/%s/%s" "$opt" "$path" +EOF + chmod +x custom_resolver.sh + run_pipeline -s src -R 'custom=./custom_resolver.sh;modeA' + assert_contains "$NS/test/custom-ref/index.html" "https://example.test/modeA/topic" + assert_not_contains "$NS/test/custom-ref/index.html" "@custom/topic" +} + +@test "pipeline: -R overrides an existing resolver prefix" { + mkdir -p src + cat > src/lua_override.c <<'EOF' +/* md +@name test/lua-override +Link: @lua/Coroutines +*/ +int noop() {} +EOF + cat > lua_override.sh <<'EOF' +#!/usr/bin/env bash +path="$(cat)" +printf "https://override.test/%s" "$path" +EOF + chmod +x lua_override.sh + run_pipeline -s src -R 'lua=./lua_override.sh' + assert_contains "$NS/test/lua-override/index.html" "https://override.test/Coroutines" + assert_not_contains "$NS/test/lua-override/index.html" "lua.org/manual" +} + +@test "pipeline: @exec renderer receives TRBLCACHE environment variable (repro)" { + mkdir -p src + cat > src/renderer_cache_env.c <<'EOF' +/* ./require_trblcache.sh +@name test/renderer-cache-env +Renderer env test. +*/ +int noop() { return 0; } +EOF + cat > require_trblcache.sh <<'EOF' +#!/usr/bin/env bash +if [[ -z "${TRBLCACHE:-}" ]]; then + echo "TRBLCACHE is missing in renderer" >&2 + exit 23 +fi +echo "<p>cache=$TRBLCACHE</p>" +EOF + chmod +x require_trblcache.sh + run bash "$TRBLDOC" -s src 2>&1 + [ "$status" -eq 0 ] + assert_contains "$NS/test/renderer-cache-env/index.html" "cache=./.trblcache" + [[ "$output" != *"TRBLCACHE is missing in renderer"* ]] + [[ "$output" != *"renderer failed (exit 23)"* ]] +} + +@test "pipeline: @exec renderer receives FILEPATH environment variable (repro)" { + mkdir -p src + cat > src/renderer_filepath_env.c <<'EOF' +/* ./require_filepath.sh +@name test/renderer-filepath-env +Renderer FILEPATH env test. +*/ +int noop() { return 0; } +EOF + cat > require_filepath.sh <<'EOF' +#!/usr/bin/env bash +if [[ -z "${FILEPATH:-}" ]]; then + echo "FILEPATH is missing in renderer" >&2 + exit 24 +fi +if [[ "${FILEPATH}" != "src/renderer_filepath_env.c" ]]; then + echo "FILEPATH is incorrect in renderer: $FILEPATH" >&2 + exit 25 +fi +echo "<p>filepath=$FILEPATH</p>" +EOF + chmod +x require_filepath.sh + run bash "$TRBLDOC" -s src 2>&1 + [ "$status" -eq 0 ] + assert_contains "$NS/test/renderer-filepath-env/index.html" "filepath=src/renderer_filepath_env.c" + [[ "$output" != *"FILEPATH is missing in renderer"* ]] + [[ "$output" != *"FILEPATH is incorrect in renderer"* ]] + [[ "$output" != *"renderer failed (exit 24)"* ]] + [[ "$output" != *"renderer failed (exit 25)"* ]] +} + +@test "pipeline: @exec renderer receives LINENUM environment variable (repro)" { + mkdir -p src + cat > src/renderer_linenum_env.c <<'EOF' +int prelude = 0; + +/* ./require_linenum.sh +@name test/renderer-linenum-env +Renderer LINENUM env test. +*/ +int noop() { return 0; } +EOF + cat > require_linenum.sh <<'EOF' +#!/usr/bin/env bash +if [[ -z "${LINENUM:-}" ]]; then + echo "LINENUM is missing in renderer" >&2 + exit 26 +fi +if [[ "${LINENUM}" != "3" ]]; then + echo "LINENUM is incorrect in renderer: $LINENUM" >&2 + exit 27 +fi +echo "<p>linenum=$LINENUM</p>" +EOF + chmod +x require_linenum.sh + run bash "$TRBLDOC" -s src 2>&1 + [ "$status" -eq 0 ] + assert_contains "$NS/test/renderer-linenum-env/index.html" "linenum=3" + [[ "$output" != *"LINENUM is missing in renderer"* ]] + [[ "$output" != *"LINENUM is incorrect in renderer"* ]] + [[ "$output" != *"renderer failed (exit 26)"* ]] + [[ "$output" != *"renderer failed (exit 27)"* ]] +} +@test "pipeline: inline @exec function receives TRBLCACHE FILEPATH and LINENUM (repro)" { + mkdir -p src + cat > src/renderer_inline_function_env.c <<'EOF' +/* render_inline(){ if [[ -z "${TRBLCACHE:-}" || -z "${FILEPATH:-}" || -z "${LINENUM:-}" ]]; then echo "missing env in inline renderer function" >&2; return 28; fi; printf "<p>cache=%s filepath=%s linenum=%s</p>\n" "$TRBLCACHE" "$FILEPATH" "$LINENUM"; }; render_inline +@name test/renderer-inline-function-env +Renderer inline function env test. +*/ +int noop() { return 0; } +EOF + run bash "$TRBLDOC" -s src 2>&1 + [ "$status" -eq 0 ] + assert_contains "$NS/test/renderer-inline-function-env/index.html" "cache=./.trblcache" + assert_contains "$NS/test/renderer-inline-function-env/index.html" "filepath=src/renderer_inline_function_env.c" + assert_contains "$NS/test/renderer-inline-function-env/index.html" "linenum=1" + [[ "$output" != *"missing env in inline renderer function"* ]] + [[ "$output" != *"renderer failed (exit 28)"* ]] +} + +@test "pipeline: @name tokens resolve to internal refs before external resolvers" { + # Internal refs: @<name> resolves to the first chunk with @ref <name> + mkdir -p src + cat > src/target.c <<'EOF' +/* md +@name api/target +@ref my-target +Target documentation. +*/ +int target() {} +EOF + cat > src/source.c <<'EOF' +/* md +@name api/source +See also: @my-target for details. +*/ +int source() {} +EOF + run_pipeline -s src + # @my-target should resolve to the internal link from global.meta + assert_contains "$NS/api/source/index.html" "href='/api/target#my-target'" + assert_not_contains "$NS/api/source/index.html" "@my-target" +} + +# ── template variable substitution ───────────────────────────────────────── + +@test "pipeline: {{ src_repo }} is substituted with TRBLDOC_SRC_REPO value" { + stage_fixture "$FIXTURES/c/src_repo.c" "src/src_repo.c" + TRBLDOC_SRC_REPO="https://example.com/repo" run_pipeline -s src + assert_contains "$NS/test/src-repo/index.html" "https://example.com/repo" + assert_not_contains "$NS/test/src-repo/index.html" '{{ src_repo }}' +} + +@test "pipeline: {{ ref_string }} is substituted with the resolved reference link" { + stage_fixture "$FIXTURES/c/ref_string.c" "src/ref_string.c" + run_pipeline -s src + # @ref lua/Coroutines is resolved via ref_resolvers/lua.sh; the output + # contains the full ref name as link text and the resolved URL. + assert_contains "$NS/test/ref-string/index.html" "lua/Coroutines" + assert_contains "$NS/test/ref-string/index.html" "lua.org/manual" + assert_not_contains "$NS/test/ref-string/index.html" '{{ ref_string }}' +} +@test "pipeline: From attribution is linked to source when TRBLDOC_SRC_REPO is set" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + TRBLDOC_SRC_REPO="https://example.com/repo/blob/main" run_pipeline -s src + assert_contains "$NS/test/simple/index.html" "From <a href='https://example.com/repo/blob/main/src/simple.c#L1'>src/simple.c:1</a>" +} + +@test "pipeline: {{ toc }} is substituted with nav links after rendering" { + stage_fixture "$FIXTURES/c/toc.c" "src/toc.c" + run_pipeline -s src + # After substitution, the chunk output should contain the nav link for the + # namespace itself, and the literal marker should be gone. + assert_contains "$NS/test/toc-chunk/index.html" "/test/toc-chunk" + assert_not_contains "$NS/test/toc-chunk/index.html" '{{ toc }}' +} + +@test "pipeline: main.layout uses {{ title }} two-bracket format" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/main.layout" '{{ title }}' + assert_not_contains "$NS/main.layout" '{{title}}' +} + +@test "pipeline: main.layout uses {{ nav }} format instead of {{> nav}}" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/main.layout" '{{ nav }}' + assert_not_contains "$NS/main.layout" '{{> nav}}' +} + +@test "pipeline: main.layout uses {{ yield }} format instead of {{{yield}}}" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/main.layout" '{{ yield }}' + assert_not_contains "$NS/main.layout" '{{{yield}}}' +} + +@test "pipeline: main.layout uses {{ breadcrumb }} format" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_contains "$NS/main.layout" '{{ breadcrumb }}' + assert_not_contains "$NS/main.layout" '{{breadcrumb}}' +} + +@test "pipeline: old-style {{src_repo}} without spaces is not substituted" { + mkdir -p src + cat > src/old_fmt.c <<'EOF' +/* md +@name test/old-fmt-src-repo +Old format: {{src_repo}} +*/ +void f() {} +EOF + TRBLDOC_SRC_REPO="https://example.com" run_pipeline -s src + assert_contains "$NS/test/old-fmt-src-repo/index.html" '{{src_repo}}' +} + +@test "pipeline: old-style {{toc}} without spaces is not substituted" { + mkdir -p src + cat > src/old_fmt_toc.c <<'EOF' +/* md +@name test/old-fmt-toc +Old format: {{toc}} +*/ +void f() {} +EOF + run_pipeline -s src + assert_contains "$NS/test/old-fmt-toc/index.html" '{{toc}}' +} + +# ── known defects (skipped – convert to active tests after bug fixes) ──────── + +@test "pipeline: chunk without @name uses source file path as namespace" { + # A chunk with no @name should use its own @file value (the source file path), + # not whatever file was last iterated in the scan loop. + mkdir -p src + cat > src/anon.c <<'EOF' +/* md +Chunk with no @name directive. +*/ +void anon() {} +EOF + run_pipeline -s src + # Expect a namespace folder derived from src/anon.c. + assert_dir_exists "$NS/src/anon.c" +} + +@test "pipeline: second run does not duplicate nav entries" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + run_pipeline -s src # second run + local count + count=$(grep -c "test/simple" "$NS/nav.partial" || true) + assert_equal "$count" "1" +} + +# ── logic bug regression tests ──────────────────────────────────────────── + +@test "pipeline: source folder with spaces in name is discovered via -s" { + # Bug: infolders is a space-separated string; folder names with spaces break discovery. + mkdir -p "src folder" + stage_fixture "$FIXTURES/c/simple.c" "src folder/simple.c" + run_pipeline -s "src folder" + assert_dir_exists "$NS/test/simple" +} + +@test "pipeline: source filename with spaces is processed correctly" { + # Bug: source_files and chunk_files use unquoted iteration which breaks on spaces. + # This fixture has no @name directive, so namespace is derived from source path. + mkdir -p src + cat > 'src/simple file.c' <<'EOF' +/* md +Hello from a C comment. +*/ +int main() { return 0; } +EOF + run_pipeline -s src + # Namespace is derived from source file path (no @name means use file path) + assert_file_exists "$NS/src/simple file.c/index.html" + assert_contains "$NS/src/simple file.c/index.html" "Hello from a C comment" +} + +@test "pipeline: nav entries with ampersands in {{ toc }} are substituted correctly" { + # Bug: toc_oneline sed substitution treats & as matched text. + mkdir -p src + cat > 'src/a.c' <<'EOF' +/* md +@name test/a +Navigation: {{ toc }} +*/ +int a() {} +EOF + cat > 'src/b.c' <<'EOF' +/* md +@name test/b&c +Chunk with ampersand. +*/ +int b() {} +EOF + run_pipeline -s src + # Verify the nav.partial contains the entries + assert_contains "$NS/nav.partial" "test/b&c" + # Verify the rendered output preserves this (not garbled by sed) + assert_contains "$NS/test/a/index.html" "test/b&c" + assert_not_contains "$NS/test/a/index.html" '{{ toc }}' +} + +# ── readme.md → top-level index ───────────────────────────────────────────── + +@test "pipeline: readme.md creates root-level index.html" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + assert_file_exists "$NS/index.html" +} + +@test "pipeline: readme.md content appears in root index.html" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + assert_contains "$NS/index.html" "Hello from the readme" +} + +@test "pipeline: README.md (uppercase) creates root-level index.html" { + mkdir -p src + cat > src/README.md <<'EOF' +# Overview +Uppercase README content. +EOF + run_pipeline -s src + assert_file_exists "$NS/index.html" + assert_contains "$NS/index.html" "Uppercase README content" +} + +@test "pipeline: readme.md with explicit @name uses custom namespace, not root" { + mkdir -p src + cat > src/readme.md <<'EOF' +@name custom/readme +# Custom Name +Content with custom name. +EOF + run_pipeline -s src + assert_file_exists "$NS/custom/readme/index.html" + if [[ -f "$NS/index.html" ]]; then + echo "ASSERTION FAILED: unexpected $NS/index.html created for readme with @name" >&2 + return 1 + fi +} + +@test "pipeline: root index.html from readme.md contains rendered HTML tags" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + assert_contains "$NS/index.html" "<p" +} + +@test "pipeline: root index.html from readme.md passes HTML fragment validation" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + assert_valid_html_fragment "$NS/index.html" +} + +@test "pipeline: nav.partial contains root href when readme.md is present" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + assert_contains "$NS/nav.partial" "href='/'" +} + +@test "pipeline: global.meta contains entry for index when readme.md is present" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + assert_contains "$NS/global.meta" "index:" +} + +@test "pipeline: readme.md alongside other source files produces both root index and other pages" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + assert_file_exists "$NS/index.html" + assert_file_exists "$NS/test/simple/index.html" + assert_contains "$NS/index.html" "Hello from the readme" + assert_contains "$NS/test/simple/index.html" "Hello from a C comment" +} + +@test "pipeline: root index.html source attribution points to readme.md" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + assert_contains "$NS/index.html" "readme.md" +} + +@test "pipeline: readme.md is assembled to the built root, not a subdirectory" { + stage_fixture "$FIXTURES/md/readme.md" "src/readme.md" + run_pipeline -s src + # Must land at the built root, not inside an "index" subdirectory. + assert_file_exists ".trblcache/built/index.html" + if [[ -f ".trblcache/built/index/index.html" ]]; then + echo "ASSERTION FAILED: readme was built to built/index/index.html instead of built/index.html" >&2 + return 1 + fi +} + +# ── known defects + +@test "pipeline: second run does not duplicate index.html content" { + stage_fixture "$FIXTURES/c/simple.c" "src/simple.c" + run_pipeline -s src + run_pipeline -s src + local count + count=$(grep -c "Hello from a C comment" "$NS/test/simple/index.html" || true) + assert_equal "$count" "1" +} diff --git a/tests/run_tests.sh b/tests/run_tests.sh new file mode 100644 index 0000000..4569be3 --- /dev/null +++ b/tests/run_tests.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# tests/run_tests.sh +# Entry point for the trbldoc test suite. +# Requires bats-core: https://github.com/bats-core/bats-core +# +# Quick install (pick one): +# npm install -g bats +# brew install bats-core +# git clone https://github.com/bats-core/bats-core ~/.bats && ~/.bats/install.sh /usr/local +# +# Usage: +# bash tests/run_tests.sh # run all tests +# bash tests/run_tests.sh unit # run only unit tests +# bash tests/run_tests.sh integration # run only integration tests + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-${BASH_SOURCE[0]}}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + +# ── bats check ────────────────────────────────────────────────────────────── +if ! command -v bats &>/dev/null; then + echo "ERROR: bats is not installed or not on PATH." >&2 + echo "Install with one of:" >&2 + echo " npm install -g bats" >&2 + echo " brew install bats-core" >&2 + echo " git clone https://github.com/bats-core/bats-core ~/.bats && ~/.bats/install.sh /usr/local" >&2 + exit 1 +fi + +# ── suite selection ────────────────────────────────────────────────────────── +TARGET="${1:-all}" + +case "$TARGET" in + unit) + SUITE_DIRS=("$SCRIPT_DIR/unit") + ;; + integration) + SUITE_DIRS=("$SCRIPT_DIR/integration") + ;; + all) + SUITE_DIRS=("$SCRIPT_DIR/unit" "$SCRIPT_DIR/integration") + ;; + *) + echo "Usage: $0 [all|unit|integration]" >&2 + exit 1 + ;; +esac + +# Export so bats test files can locate project files +export REPO_ROOT + +echo "Running trbldoc test suite (target: $TARGET)" +echo "bats $(bats --version)" +echo "" + +bats --print-output-on-failure "${SUITE_DIRS[@]}" diff --git a/tests/unit/directives.bats b/tests/unit/directives.bats new file mode 100644 index 0000000..3a57951 --- /dev/null +++ b/tests/unit/directives.bats @@ -0,0 +1,235 @@ +#!/usr/bin/env bats +# tests/unit/directives.bats +# Verifies that chunk directive parsing (the awk one-liners used in the chunk +# loop of trbldoc.sh) correctly reads @name, @exec, @file, and @ref, and that +# the sed strip pass removes directives from chunk bodies without touching +# non-directive body lines. + +load "../helpers/common.bash" + +FIXTURES="$BATS_TEST_DIRNAME/../fixtures" + +setup() { + setup_workspace + CACHE=".trblcache" + mkdir -p "$CACHE/chunks" +} + +teardown() { + teardown_workspace +} + +# ── helpers used in this file ─────────────────────────────────────────────── + +# Write a chunk file directly (bypassing extraction) so directive tests +# are independent of extraction correctness. +make_chunk() { + local path="$1" + shift + mkdir -p "$(dirname "$path")" + printf '%s\n' "$@" > "$path" +} + +# Read @name from a chunk file (mirrors trbldoc.sh logic). +parse_name() { + awk '/@name .+/ {print $2}' "$1" +} + +# Read @exec from a chunk file. +parse_exec() { + awk '/@exec .+/ {$1=""; print}' "$1" | sed 's/^ //' +} + +# Read @file from a chunk file. +parse_file() { + awk '/@file .+/ {print $2}' "$1" +} + +# Read @ref from a chunk file. +parse_ref() { + awk '/@ref .+/ {print $2}' "$1" +} +# Parse chunk metadata using the same awk program as trbldoc.sh's +# parse_chunk_metadata() helper. +parse_chunk_metadata_record() { + awk ' +BEGIN { + us = sprintf("%c", 31) + from_p = "" + namespace_p = "" + priority_p = "" + program_p = "" + ref_p = "" +} +/^[ \t]*@file / && from_p == "" { + from_p = $0 + sub(/^[ \t]*@file /, "", from_p) +} +/^[ \t]*@name / && namespace_p == "" { + namespace_p = $2 +} +/^[ \t]*@priority / && priority_p == "" { + priority_p = $2 +} +/^[ \t]*@exec / && program_p == "" { + program_p = $0 + sub(/^[ \t]*@exec /, "", program_p) +} +/^[ \t]*@ref / && ref_p == "" { + ref_p = $2 +} +END { + printf "%s%s%s%s%s%s%s%s%s\n", from_p, us, namespace_p, us, priority_p, us, program_p, us, ref_p +} +' "$1" +} + +# Apply the directive strip (mirrors trbldoc.sh line 231). +strip_directives() { + sed -i "s/^@.*//g" "$1" +} + +# ── @name ─────────────────────────────────────────────────────────────────── + +@test "@name: parsed correctly from chunk" { + local chunk="$CACHE/chunks/test_name" + make_chunk "$chunk" "@file src.c:1" "@exec md" "@name my/namespace" "body text" + local result + result="$(parse_name "$chunk")" + assert_equal "$result" "my/namespace" +} + +@test "@name: absent when directive is missing" { + local chunk="$CACHE/chunks/test_no_name" + make_chunk "$chunk" "@file src.c:1" "@exec md" "body text" + local result + result="$(parse_name "$chunk")" + assert_equal "$result" "" +} + +@test "@name: only the first token after @name is captured" { + local chunk="$CACHE/chunks/test_name_token" + make_chunk "$chunk" "@name some/path extra ignored" + local result + result="$(parse_name "$chunk")" + assert_equal "$result" "some/path" +} + +# ── @exec ─────────────────────────────────────────────────────────────────── + +@test "@exec: renderer name parsed correctly" { + local chunk="$CACHE/chunks/test_exec" + make_chunk "$chunk" "@file src.c:1" "@exec md2html --github" "@name ns" + local result + result="$(parse_exec "$chunk")" + assert_equal "$result" "md2html --github" +} + +@test "@exec: absent when directive is missing" { + local chunk="$CACHE/chunks/test_no_exec" + make_chunk "$chunk" "@file src.c:1" "@name ns" "body" + local result + result="$(parse_exec "$chunk")" + assert_equal "$result" "" +} + +# ── @file ─────────────────────────────────────────────────────────────────── + +@test "@file: source path parsed correctly" { + local chunk="$CACHE/chunks/test_file" + make_chunk "$chunk" "@file src/lib.c:42" "@exec md" "@name ns" + local result + result="$(parse_file "$chunk")" + assert_equal "$result" "src/lib.c:42" +} + +# ── @ref ──────────────────────────────────────────────────────────────────── + +@test "@ref: reference name parsed correctly" { + local chunk="$CACHE/chunks/test_ref" + make_chunk "$chunk" "@file src.c:1" "@exec md" "@name ns" "@ref my/ref-name" + local result + result="$(parse_ref "$chunk")" + assert_equal "$result" "my/ref-name" +} + +@test "@ref: absent when directive is missing" { + local chunk="$CACHE/chunks/test_no_ref" + make_chunk "$chunk" "@file src.c:1" "@exec md" "@name ns" + local result + result="$(parse_ref "$chunk")" + assert_equal "$result" "" +} + +@test "metadata parser: directives with leading spaces or tabs are parsed" { + local chunk="$CACHE/chunks/test_directive_leading_whitespace" + local sep expected result + make_chunk \ + "$chunk" \ + " @file src/lib.c:42" \ + $'\t@name docs/indented' \ + " @priority 99" \ + $' \t@exec md2html --github' \ + " @ref refs/anchor" + sep="$(printf '\037')" + expected="src/lib.c:42${sep}docs/indented${sep}99${sep}md2html --github${sep}refs/anchor" + result="$(parse_chunk_metadata_record "$chunk")" + assert_equal "$result" "$expected" +} + +# ── directive stripping (sed pass) ────────────────────────────────────────── + +@test "strip: @name line is removed from chunk body" { + local chunk="$CACHE/chunks/test_strip_name" + make_chunk "$chunk" "@file src.c:1" "@name my/ns" "real body content" + strip_directives "$chunk" + assert_not_contains "$chunk" "^@name" +} + +@test "strip: @exec line is removed from chunk body" { + local chunk="$CACHE/chunks/test_strip_exec" + make_chunk "$chunk" "@exec md" "real body content" + strip_directives "$chunk" + assert_not_contains "$chunk" "^@exec" +} + +@test "strip: @file line is removed from chunk body" { + local chunk="$CACHE/chunks/test_strip_file" + make_chunk "$chunk" "@file src.c:1" "real body content" + strip_directives "$chunk" + assert_not_contains "$chunk" "^@file" +} + +@test "strip: @ref line is removed from chunk body" { + local chunk="$CACHE/chunks/test_strip_ref" + make_chunk "$chunk" "@ref some/ref" "real body content" + strip_directives "$chunk" + assert_not_contains "$chunk" "^@ref" +} + +@test "strip: non-directive body lines are preserved after stripping" { + local chunk="$CACHE/chunks/test_strip_body" + make_chunk "$chunk" "@name ns" "@exec md" "keep this line" "and this one" + strip_directives "$chunk" + assert_contains "$chunk" "keep this line" + assert_contains "$chunk" "and this one" +} + +@test "strip: directives from all-directives fixture are present before strip" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/directives.c" "$prog" "$CACHE" + # Before stripping, @ref should be present + chunk_contains "$CACHE" "@ref" +} + +@test "strip: body content from all-directives fixture survives strip" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/directives.c" "$prog" "$CACHE" + # Strip all chunks + for f in "$CACHE/chunks/"*; do + strip_directives "$f" + done + chunk_contains "$CACHE" "Directive test body content" +} diff --git a/tests/unit/extraction.bats b/tests/unit/extraction.bats new file mode 100644 index 0000000..21c6784 --- /dev/null +++ b/tests/unit/extraction.bats @@ -0,0 +1,171 @@ +#!/usr/bin/env bats +# tests/unit/extraction.bats +# Verifies that the AWK extraction programs correctly identify doc comment +# boundaries and produce chunk files for each supported language. +# +# Tests operate directly on the AWK programs (via helpers/common.bash) without +# running the full trbldoc.sh pipeline. + +load "../helpers/common.bash" + +FIXTURES="$BATS_TEST_DIRNAME/../fixtures" + +setup() { + setup_workspace + CACHE=".trblcache" + mkdir -p "$CACHE/chunks" +} + +teardown() { + teardown_workspace +} + +# ── C (/* ... */) ────────────────────────────────────────────────────────── + +@test "C: single doc comment produces one chunk" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/simple.c" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "1" +} + +@test "C: two doc comments produce two chunks" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/multi_chunk.c" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "2" +} + +@test "C: file with no doc comments produces no chunks" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/no_comments.c" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "0" +} + +@test "C: chunk contains @exec directive for renderer" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/simple.c" "$prog" "$CACHE" + chunk_contains "$CACHE" "@exec" +} + +@test "C: chunk contains @file directive with source file name" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/simple.c" "$prog" "$CACHE" + chunk_contains "$CACHE" "@file" +} + +@test "C: chunk body lines are captured" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/simple.c" "$prog" "$CACHE" + chunk_contains "$CACHE" "Hello from a C comment" +} + +@test "C: non-comment source lines are not captured in chunk" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/simple.c" "$prog" "$CACHE" + # 'int main' is code, not doc + if grep -rl "int main" "$CACHE/chunks" >/dev/null 2>&1; then + echo "FAILED: code line 'int main' was captured in a chunk" >&2 + return 1 + fi +} + +@test "C: single-line /* */ comment produces no chunks" { + local prog + prog="$(awk_prog_c "$CACHE")" + # no_comments.c contains a single-line /* ... */ comment on the first line + run_extraction "$FIXTURES/c/no_comments.c" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "0" +} + +@test "C: two-chunk file produces chunks with distinct content" { + local prog + prog="$(awk_prog_c "$CACHE")" + run_extraction "$FIXTURES/c/multi_chunk.c" "$prog" "$CACHE" + chunk_contains "$CACHE" "First chunk content" + chunk_contains "$CACHE" "Second chunk content" +} + +# ── Lua (--[[ ... ]]) ────────────────────────────────────────────────────── + +@test "Lua: single doc comment produces one chunk" { + local prog + prog="$(awk_prog_lua "$CACHE")" + run_extraction "$FIXTURES/lua/simple.lua" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "1" +} + +@test "Lua: chunk body lines are captured" { + local prog + prog="$(awk_prog_lua "$CACHE")" + run_extraction "$FIXTURES/lua/simple.lua" "$prog" "$CACHE" + chunk_contains "$CACHE" "Hello from a Lua comment" +} + +@test "Lua: non-comment source lines are not captured" { + local prog + prog="$(awk_prog_lua "$CACHE")" + run_extraction "$FIXTURES/lua/simple.lua" "$prog" "$CACHE" + if grep -rl "local x" "$CACHE/chunks" >/dev/null 2>&1; then + echo "FAILED: code line 'local x' was captured in a chunk" >&2 + return 1 + fi +} + +@test "Lua: single-line --[[ ]] comment produces no chunks" { + local prog + prog="$(awk_prog_lua "$CACHE")" + run_extraction "$FIXTURES/lua/single_line.lua" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "0" +} + +# ── Python ('''renderer ... ''') ─────────────────────────────────────────── + +@test "Python: single doc comment produces one chunk" { + local prog + prog="$(awk_prog_py "$CACHE")" + run_extraction "$FIXTURES/py/simple.py" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "1" +} + +@test "Python: chunk body lines are captured" { + local prog + prog="$(awk_prog_py "$CACHE")" + run_extraction "$FIXTURES/py/simple.py" "$prog" "$CACHE" + chunk_contains "$CACHE" "Hello from a Python doc comment" +} + +@test "Python: single-line '''content''' on one line produces no chunks" { + local prog + prog="$(awk_prog_py "$CACHE")" + run_extraction "$FIXTURES/py/single_line.py" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "0" +} + +# ── Markdown (whole file = one chunk) ────────────────────────────────────── + +@test "Markdown: whole file produces exactly one chunk" { + local prog + prog="$(awk_prog_md "$CACHE")" + run_extraction "$FIXTURES/md/simple.md" "$prog" "$CACHE" + assert_equal "$(chunk_count "$CACHE")" "1" +} + +@test "Markdown: chunk contains @exec md directive" { + local prog + prog="$(awk_prog_md "$CACHE")" + run_extraction "$FIXTURES/md/simple.md" "$prog" "$CACHE" + chunk_contains "$CACHE" "@exec md" +} + +@test "Markdown: chunk includes file body content" { + local prog + prog="$(awk_prog_md "$CACHE")" + run_extraction "$FIXTURES/md/simple.md" "$prog" "$CACHE" + chunk_contains "$CACHE" "Hello from a markdown file" +} diff --git a/tests/unit/template_vars.bats b/tests/unit/template_vars.bats new file mode 100644 index 0000000..5d95667 --- /dev/null +++ b/tests/unit/template_vars.bats @@ -0,0 +1,160 @@ +#!/usr/bin/env bats +# tests/unit/template_vars.bats +# Unit tests for template variable substitution patterns. +# +# These tests replicate the escape_sed_pattern_literal logic from trbldoc.sh +# and verify that: +# - the canonical {{ var }} format (two brackets, one space each side) is matched +# - the old {{varname}} format (two brackets, no spaces) is NOT matched +# - the layout-specific patterns ({{ title }}, {{ nav }}, {{ yield }}, +# {{ breadcrumb }}) work correctly in sed address and substitution position + +load "../helpers/common.bash" + +setup() { + setup_workspace +} + +teardown() { + teardown_workspace +} + +# Replicate escape_sed_pattern_literal from trbldoc.sh. +escape_pat() { + printf '%s' "$1" | sed -e 's/[][\\.^$*+?(){}|]/\\&/g' +} + +# ── chunk template variables ────────────────────────────────────────────────── + +@test "template vars: {{ src_repo }} is matched and substituted" { + local pat result + pat=$(escape_pat '{{ src_repo }}') + result=$(printf '%s' 'Link: {{ src_repo }}/file.c' | sed -E "s|$pat|https://example.com|g") + assert_equal "$result" 'Link: https://example.com/file.c' +} + +@test "template vars: {{src_repo}} without spaces is NOT matched" { + local pat result + pat=$(escape_pat '{{ src_repo }}') + result=$(printf '%s' 'Old: {{src_repo}}' | sed -E "s|$pat|https://example.com|g") + assert_equal "$result" 'Old: {{src_repo}}' +} + +@test "template vars: {{ toc }} is matched and substituted" { + local pat result + pat=$(escape_pat '{{ toc }}') + result=$(printf '%s' 'Nav: {{ toc }}' | sed -E "s|$pat|<ul><li>nav</li></ul>|g") + assert_equal "$result" 'Nav: <ul><li>nav</li></ul>' +} + +@test "template vars: {{toc}} without spaces is NOT matched" { + local pat result + pat=$(escape_pat '{{ toc }}') + result=$(printf '%s' 'Old: {{toc}}' | sed -E "s|$pat|<ul><li>nav</li></ul>|g") + assert_equal "$result" 'Old: {{toc}}' +} + +@test "template vars: {{ ref_string }} is matched and substituted" { + local pat result + pat=$(escape_pat '{{ ref_string }}') + result=$(printf '%s' "See: {{ ref_string }}." | sed -E "s|$pat|<a href='/x'>x</a>|g") + assert_equal "$result" "See: <a href='/x'>x</a>." +} + +@test "template vars: {{ref_string}} without spaces is NOT matched" { + local pat result + pat=$(escape_pat '{{ ref_string }}') + result=$(printf '%s' 'Old: {{ref_string}}' | sed -E "s|$pat|<a href='/x'>x</a>|g") + assert_equal "$result" 'Old: {{ref_string}}' +} + +@test "template vars: multiple {{ src_repo }} occurrences are all substituted" { + local pat result + pat=$(escape_pat '{{ src_repo }}') + result=$(printf '%s' '{{ src_repo }}/a and {{ src_repo }}/b' | sed -E "s|$pat|https://r|g") + assert_equal "$result" 'https://r/a and https://r/b' +} + +# ── layout template variables ───────────────────────────────────────────────── + +@test "template vars: layout {{ title }} is matched and substituted" { + local result + result=$(printf '%s' '<title>{{ title }}</title>' \ + | sed -E "s|\\{\\{ title \\}\\}|My Page|g") + assert_equal "$result" '<title>My Page</title>' +} + +@test "template vars: layout {{title}} without spaces is NOT matched" { + local result + result=$(printf '%s' '<title>{{title}}</title>' \ + | sed -E "s|\\{\\{ title \\}\\}|My Page|g") + assert_equal "$result" '<title>{{title}}</title>' +} + +@test "template vars: layout {{ breadcrumb }} is matched and substituted" { + local result + result=$(printf '%s' '<h1>{{ breadcrumb }}</h1>' \ + | sed -E "s|\\{\\{ breadcrumb \\}\\}|Section|g") + assert_equal "$result" '<h1>Section</h1>' +} + +@test "template vars: layout {{breadcrumb}} without spaces is NOT matched" { + local result + result=$(printf '%s' '{{breadcrumb}}' \ + | sed -E "s|\\{\\{ breadcrumb \\}\\}|Section|g") + assert_equal "$result" '{{breadcrumb}}' +} + +@test "template vars: layout {{ nav }} is matched and substituted" { + local result + result=$(printf '%s' '<ul>{{ nav }}</ul>' \ + | sed -E "s|\\{\\{ nav \\}\\}|<li>item</li>|g") + assert_equal "$result" '<ul><li>item</li></ul>' +} + +@test "template vars: layout old {{> nav}} partial syntax is NOT matched by {{ nav }} pattern" { + local result + result=$(printf '%s' '<ul>{{> nav}}</ul>' \ + | sed -E "s|\\{\\{ nav \\}\\}|<li>item</li>|g") + assert_equal "$result" '<ul>{{> nav}}</ul>' +} + +@test "template vars: layout {{ yield }} works as sed address to include a file" { + local tmpbody tmplayout output + tmpbody=$(mktemp) + tmplayout=$(mktemp) + printf '<p>body content</p>\n' > "$tmpbody" + printf '{{ yield }}\n' > "$tmplayout" + output=$(sed -E \ + -e "/\\{\\{ yield \\}\\}/r $tmpbody" \ + -e "/\\{\\{ yield \\}\\}/d" \ + "$tmplayout") + rm -f "$tmpbody" "$tmplayout" + assert_equal "$output" '<p>body content</p>' +} + +@test "template vars: layout old {{{yield}}} triple-brace is NOT matched by {{ yield }} pattern" { + local tmpbody tmplayout output + tmpbody=$(mktemp) + tmplayout=$(mktemp) + printf '<p>body content</p>\n' > "$tmpbody" + printf '{{{yield}}}\n' > "$tmplayout" + output=$(sed -E \ + -e "/\\{\\{ yield \\}\\}/r $tmpbody" \ + -e "/\\{\\{ yield \\}\\}/d" \ + "$tmplayout") + rm -f "$tmpbody" "$tmplayout" + # {{{yield}}} should remain unchanged — the new pattern only matches {{ yield }} + assert_equal "$output" '{{{yield}}}' +} + +@test "template vars: substitution value containing special sed chars is safe" { + # Verify that replacement values with & and \ don't corrupt the output + # (this exercises the escape_sed_replacement_literal path) + local pat esc_repl result + pat=$(escape_pat '{{ src_repo }}') + # Simulate escape_sed_replacement_literal for a URL containing & + esc_repl=$(printf '%s' 'https://example.com/a&b' | sed -e 's/[\\&|]/\\&/g') + result=$(printf '%s' '{{ src_repo }}/path' | sed -E "s|$pat|$esc_repl|g") + assert_equal "$result" 'https://example.com/a&b/path' +} diff --git a/trbldoc.sh b/trbldoc.sh new file mode 100644 index 0000000..99e89be --- /dev/null +++ b/trbldoc.sh @@ -0,0 +1,940 @@ +#!/bin/sh + +# trbldoc.sh +# A really awful documentation tool +# Installation +# Simply drop into /usr/local/bin/trbldoc and mark executable +# $ sudo cp trbldoc.sh /usr/local/bin/trbldoc +# $ sudo chmod +x /usr/local/bin/trbldoc +# Usage: +# trbldoc [options] +# Description: +# trbldoc works in several passes, first, it looks at each file in the +# <source> folders, and for each file it looks for comments +# and extracts them as a series of "chunks" - +# unbroken lines of comments. Each chunk should declare +# a program. The text of the chunk will be fed into this program +# on standard in, and trbldoc expects html for this chunk on standard out. +# +# Then, it groups chunks according to their "@name <doc_name>" line, +# and orders them according to their "@priority n" line. +# If a comment doesn't have a name, <doc_name> is the file path the +# comment was found in. If a comment doesn't have a priority, it's priority +# is the line number the chunk started on. +# +# Chunks may additionaly have the following "documentation identifiers": +# @name <name/with/slashes> - Override the file path this chunk was found +# in +# @exec <program> <arg1> <arg2> ... - Override the program we use to +# process this chunk +# @file <name> - Pretend this chunk came from another file +# e.g. if the file being documented is built from another +# (source) file. +# @ref <name> - This chunk should be referenced by this name (instead of +# of it's @name). +# @priority <number> - Higher values render first when assembling pages. +# helps keep important information at the top. +# +# The following template variables may be used anywhere in chunk body text +# and are substituted before the chunk's program is run: +# {{src_repo}} - source repository URL (see TRBLDOC_SRC_REPO). Useful +# for building links back to source files. +# {{ref_string}} - HTML link for this chunk's @ref directive, resolved +# via the matching ref_resolvers script. Empty if @ref is absent +# or the resolver produces no output. +# {{toc}} - full site navigation rendered as HTML list items. +# +# Then, it resolves refrences, replacing text of "@<name>" with a link +# to the first chunk declaring "@ref <name>" (by priority/source order). +# If no internal ref matches, it falls back to external resolvers: +# "@<script>/<path>" runs ref_resolvers/<script>.sh with <path> as input. +# +# Options: +# -R '<prefix>=<script>;<options>' - When a document tag (@<prefix>/ref/path) +# contains the prefix <prefix>, strip it, and pass the rest as +# stdin to <script>, run with <options>. The following environment +# variables are available to <script>: +# $TRBLCACHE - a cache folder for this project. +# $FILEPATH - The file the reference came from, extension included +# $LINENUM - The line number the reference was found on +# -s <source> - Add <source> to files or folders scanned +# -o <dest> - Set the destination path that documentation is output to. +# -l <layout> - Use an alternative main.layout template file instead of +# the built-in default. +# -f 'file_ext;start_regex;end_regex' - Add regexes for file extensions to +# extract comments. Comments begin with "start_regex", followed +# by a newline-terminated command to run the body with. +# -C - Force a clean rebuild: clears all cached chunks and last-processed +# timestamps before scanning. Equivalent to manually deleting +# .trblcache/chunks/ and .trblcache/last_processed/. +# Environment variables: +# TRBLDOC_MD - Override the markdown renderer used for .md files and +# "@exec md" chunks. Defaults to: +# "md2html --github --fpermissive-autolinks --ftables" +# TRBLDOC_SRC_REPO - Repository URL substituted for {{src_repo}} in +# doc comments. Falls back to "git remote get-url origin". +# Parse command-line options +# -s <source> : source folder to scan (repeatable) +# -o <dest> : final output directory (default: .trblcache/built) +# -l <layout> : alternative main.layout template path +# -f ext;start;end : register a custom file extension with AWK-regex delimiters +# -R prefix=script;opts : reference-resolver override +NL=' +' +US=$(printf '\037') +ref_resolver_overrides="" +extra_formats="" +infolders="" +out_dir="" +layout_template="" +clean_build=0 +while getopts ":s:o:l:R:f:C" opt; do + case "$opt" in + s) infolders="${infolders}${OPTARG}${NL}" ;; + o) out_dir="$OPTARG" ;; + l) layout_template="$OPTARG" ;; + R) ref_resolver_overrides="${ref_resolver_overrides}${OPTARG}${NL}" ;; + f) extra_formats="${extra_formats}${OPTARG}${NL}" ;; + C) clean_build=1 ;; + :) echo "trbldoc: option -$OPTARG requires an argument" >&2; exit 1 ;; + ?) echo "trbldoc: unknown option: -$OPTARG" >&2; exit 1 ;; + esac +done +shift $((OPTIND - 1)) +if [ -z "$infolders" ]; then + echo "trbldoc: no source folders specified; use -s <folder>" >&2 + exit 1 +fi +resolver_overrides_dir=$(mktemp -d) +# Parse -R overrides: prefix=script;options +printf '%s' "$ref_resolver_overrides" | while IFS= read -r override_spec; do + [ -z "$override_spec" ] && continue + override_prefix="${override_spec%%=*}" + override_rhs="${override_spec#*=}" + override_script="${override_rhs%%;*}" + override_opts="" + if [ "$override_rhs" != "$override_script" ]; then + override_opts="${override_rhs#*;}" + fi + if [ -n "$override_prefix" ] && [ -n "$override_script" ]; then + printf '%s' "$override_script" > "$resolver_overrides_dir/${override_prefix}.script" + printf '%s' "$override_opts" > "$resolver_overrides_dir/${override_prefix}.opts" + fi +done +TRBLDOC_RESOLVER_OVERRIDES_RAW="$ref_resolver_overrides" +export TRBLDOC_RESOLVER_OVERRIDES_RAW +configure_renderer_aliases() { :; } + +SCRIPT_DIR="${SCRIPT_DIR:-$(cd "$(dirname "$0")" && pwd)}" + +sed_pattern_literal='s/[][\\.^$*+?(){}|]/\\&/g' +sed_replacement_literal='s/[\\&|]/\\&/g' +validate_namespace_path() { + local namespace="$1" + if ! pathchk -P "$namespace" >/dev/null 2>&1; then + echo "trbldoc: Invalid @name, got $namespace" + return 1 + fi + + local part _old_ifs + set -f + _old_ifs="$IFS" + IFS='/' + set -- $namespace + IFS="$_old_ifs" + set +f + for part do + if [ -z "$part" ]; then + echo "trbldoc: @name must not contain empty path segments, got '$namespace'" >&2 + return 1 + fi + if [ "$part" = "." ] || [ "$part" = ".." ]; then + echo "trbldoc: @name must not contain '.' or '..' path segments, got '$namespace'" >&2 + return 1 + fi + done +} +namespace_for_chunk() { + local source_path="$1" + local explicit_namespace="$2" + local source_basename="${source_path##*/}" + local namespace="$explicit_namespace" + + if [ -z "$namespace" ]; then + case $(printf '%s' "$source_basename" | tr '[:upper:]' '[:lower:]') in + readme.md) return 0 ;; + esac + fi + if [ -z "$namespace" ]; then + namespace="$source_path" + fi + validate_namespace_path "$namespace" || return 1 + printf '%s' "$namespace" +} + +parse_chunk_metadata() { + local chunk_file="$1" + awk ' +BEGIN { + us = sprintf("%c", 31) + from_p = "" + namespace_p = "" + priority_p = "" + program_p = "" + ref_p = "" +} +{sub(/\r$/, "", $0)} +/^[[:blank:]]*@file / && from_p == "" { + from_p = $0 + sub(/^[[:blank:]]*@file /, "", from_p) +} +/^[[:blank:]]*@name / && namespace_p == "" { namespace_p = $2 } +/^[[:blank:]]*@priority / && priority_p == "" { priority_p = $2 } +/^[[:blank:]]*@exec / && program_p == "" { + program_p = $0 + sub(/^[[:blank:]]*@exec /, "", program_p) +} +/^[[:blank:]]*@ref / && ref_p == "" { ref_p = $2 } +END { + printf "%s%s%s%s%s%s%s%s%s\n", from_p, us, namespace_p, us, priority_p, us, program_p, us, ref_p +} +' "$chunk_file" +} + +get_parallel_jobs() { + nproc 2>/dev/null || \ + getconf _NPROCESSORS_ONLN 2>/dev/null || \ + grep -c '^processor' /proc/cpuinfo 2>/dev/null || \ + echo 4 +} + +# Apply many literal replacements by streaming file content through +# chained sed processes and writing back once at the end. +# replacements_file format: <search><US><replacement> one pair per line. +replace_literal_pairs_in_file() { + local target_file="$1" + local replacements_file="$2" + local token_sep + local tmp_file + local sed_chain + + [ ! -s "$replacements_file" ] && return 0 + token_sep=$(printf '\037') + tmp_file=$(mktemp) || return 1 + + # Single-quote a string for safe eval embedding. + _quote_sq() { + printf "'%s'" "$(printf '%s' "$1" | sed "s/'/'\\\\''/g")" + } + + local cmd="cat $(_quote_sq "$target_file")" + sed_chain=$(awk -v us="$token_sep" ' +function escape_pattern(v) { gsub(/[][\\.^$*+?(){}|]/, "\\\\&", v); return v } +function escape_replacement(v) { gsub(/[\\&|]/, "\\\\&", v); return v } +function escape_dquote(v) { gsub(/["\\`$]/, "\\\\&", v); return v } +{ + sep = index($0, us) + if (sep == 0) next + search = substr($0, 1, sep - 1) + repl = substr($0, sep + 1) + search = escape_pattern(search) + repl = escape_replacement(repl) + expr = "s|" search "|" repl "|g" + expr = escape_dquote(expr) + printf " | sed -r \"%s\"", expr +} +' "$replacements_file") + + cmd="$cmd$sed_chain > $(_quote_sq "$tmp_file")" + eval "$cmd" || { + rm -f "$tmp_file" + return 1 + } + mv "$tmp_file" "$target_file" +} + +# Extract first "From ..." citation source (without HTML tags) for resolver context. +extract_source_from_cite() { + local html_file="$1" + awk ' +match($0, /<cite>From .*<\/cite>/) { + source = substr($0, RSTART + 11, RLENGTH - 18) + gsub(/<[^>]*>/, "", source) + print source + exit +} +' "$html_file" +} + +# Emit unique reference tokens from HTML: +# I<US>@token for internal refs, E<US>@prefix/path for external refs. +collect_html_ref_tokens() { + local html_file="$1" + awk ' +{ + internal_line = $0 + while (match(internal_line, /@[a-zA-Z0-9_.\/-]+/)) { + token = substr(internal_line, RSTART, RLENGTH) + if (index(token, "|") == 0 && !seen_internal[token]++) { + printf "I%c%s\n", 31, token + } + internal_line = substr(internal_line, RSTART + RLENGTH) + } + + external_line = $0 + while (match(external_line, /@[a-zA-Z0-9_-]+\/[^[:space:]<>"]+/)) { + token = substr(external_line, RSTART, RLENGTH) + if (!seen_external[token]++) { + printf "E%c%s\n", 31, token + } + external_line = substr(external_line, RSTART + RLENGTH) + } +} +' "$html_file" +} + +# render_layout_page $layout_file $nav_file $body_file $page_title $page_breadcrub $output_file +render_layout_page() { + # Read nav content (may contain newlines, so flatten for sed) + local nav_html + local escaped_title + local escaped_breadcrumb + local escaped_nav + nav_html=$(tr '\n' ' ' < "$2") + escaped_title=$(escape_sed_replacement_literal "$4") + escaped_breadcrumb=$(escape_sed_replacement_literal "$5") + escaped_nav=$(escape_sed_replacement_literal "$nav_html") + + # Render layout to output in a single sed pass. + # {{ yield }} is expected to be on its own line. + sed -E \ + -e "s|\\{\\{ title \\}\\}|$escaped_title|g" \ + -e "s|\\{\\{ breadcrumb \\}\\}|$escaped_breadcrumb|g" \ + -e "s|\\{\\{ nav \\}\\}|$escaped_nav|g" \ + -e "/\\{\\{ yield \\}\\}/r $3" \ + -e "/\\{\\{ yield \\}\\}/d" \ + "$1" > "$6" +} + +assemble_site() { + local namespace_root="$1" + local destination_root="$2" + local layout_file="$namespace_root/main.layout" + local nav_file="$namespace_root/nav.partial" + local rendered_page + local rel_path + local namespace_path + local page_title="index" + local page_breadcrumb="index" + local target_dir + local output_file + + rm -rf "$destination_root" + mkdir -p "$destination_root" + + if [ -d "$namespace_root/stylesheets" ]; then + mkdir -p "$destination_root/stylesheets" + cp -R "$namespace_root/stylesheets/." "$destination_root/stylesheets/" + fi + + _asm_find_tmp=$(mktemp) + find "$namespace_root" -name "index.html" > "$_asm_find_tmp" + while IFS= read -r rendered_page; do + rel_path="${rendered_page#$namespace_root/}" + namespace_path="${rel_path%/index.html}" + if [ "$rel_path" = "index.html" ]; then + namespace_path="" + fi + if [ -n "$namespace_path" ]; then + target_dir="$destination_root/$namespace_path" + page_title="$namespace_path" + page_breadcrumb="$namespace_path" + else + target_dir="$destination_root" + fi + mkdir -p "$target_dir" + output_file="$target_dir/index.html" + render_layout_page "$layout_file" "$nav_file" "$rendered_page" "$page_title" "$page_breadcrumb" "$output_file" + done < "$_asm_find_tmp" + rm -f "$_asm_find_tmp" +} + +# Cache directory +cache_dir=./.trblcache +export TRBLCACHE="$cache_dir" +# Resolve output directory now that cache_dir is known +out_dir="${out_dir:-$cache_dir/built}" +mkdir -p "$cache_dir" +common_functions_sh="$cache_dir/_common_functions.sh" +cat > "$common_functions_sh" << 'COMMON_SH' +escape_sed_pattern_literal() { + local value="$1" + printf '%s' "$value" | sed -e 's/[][\\.^$*+?(){}|]/\\&/g' +} + +escape_sed_replacement_literal() { + local value="$1" + printf '%s' "$value" | sed -e 's/[\\&|]/\\&/g' +} + +md() { ${TRBLDOC_MD:-md2html --github --fpermissive-autolinks --ftables} "$@"; } +rst() { ${TRBLDOC_RST:-docutils --template ./.trblcache/rsttemplate.txt} "$@"; } + +# resolve_ref PREFIX PATH FILEPATH LINENUM +# Resolves a reference by looking up the resolver script, +# exports TRBLCACHE/FILEPATH/LINENUM env vars, and runs the script with PATH as stdin. +# Prints the resolved URL to stdout, or nothing if resolution fails. +resolve_ref() { + local prefix="$1" + local resolver_script="$SCRIPT_DIR/ref_resolvers/${prefix}.sh" + local resolver_opts="" + + if [ -f "$resolver_overrides_dir/${prefix}.script" ]; then + resolver_script=$(cat "$resolver_overrides_dir/${prefix}.script") + resolver_opts=$(cat "$resolver_overrides_dir/${prefix}.opts" 2>/dev/null || true) + elif [ -n "${TRBLDOC_RESOLVER_OVERRIDES_RAW:-}" ]; then + printf '%s' "$TRBLDOC_RESOLVER_OVERRIDES_RAW" | while IFS= read -r override_spec; do + [ -z "$override_spec" ] && continue + override_prefix="${override_spec%%=*}" + [ "$override_prefix" != "$prefix" ] && continue + override_rhs="${override_spec#*=}" + resolver_script="${override_rhs%%;*}" + resolver_opts="" + if [ "$override_rhs" != "$resolver_script" ]; then + resolver_opts="${override_rhs#*;}" + fi + break + done + fi + + if [ ! -f "$resolver_script" ]; then + return 1 + fi + + export TRBLCACHE="$cache_dir" + export FILEPATH="$3" + export LINENUM="$4" + + if [ -n "$resolver_opts" ]; then + printf '%s\n' "$2" | eval "\"$resolver_script\" $resolver_opts" 2>/dev/null || true + else + printf '%s\n' "$2" | "$resolver_script" 2>/dev/null || true + fi +} +COMMON_SH +. "$common_functions_sh" +export common_functions_sh + +parallel_jobs="${TRBLDOC_JOBS:-$(get_parallel_jobs)}" + +# `$cache/last_processed` stores empty files to represent the last updated date (so we don't reprocess files if they haven't been updated) +# `$cache/namespace` is the output of documentation execution, to be aggregated and copied into build/ +# `$cache/ref` references, things that should be resolved after all the pages are generated +# `$cache/chunks` # Chunks extracted from files, kept seperate so we can just delete the chunks for one file if it's changed. +# `$cache/comment_forms` stores awk programs that extract chunks from files based on their extension +# Weirdly, this is one of the slower parts of trbldoc. +xargs -n 1 -P $parallel_jobs mkdir -p <<EOF +$cache_dir/last_processed +$cache_dir/namespace +$cache_dir/ref +$cache_dir/chunks +$cache_dir/comment_forms +$out_dir +EOF + + +# Various opening and closing tags for different languages (based on file extension) +# Each entry: ext,end_regex,start_regex (AWK extended regex patterns, no slashes) +comment_forms=" +c,\*\/,\/\* +lua,\]\],\-\-\[\[ +" + +# Helper: write comment-extraction awk script from start/end regex pair. +# Uses single-quote splicing to inject shell variables into awk source. +_write_comment_form() { + printf '%s\n' ' +BEGIN{} +/'"$2"'/ { + if(in_comment) {print "EOF"} + in_comment=0 +} +in_comment {print} +/'"$3"' .+/ && !/'"$2"'$/ { + print "chunkfile=$(mktemp -p '"${cache_dir}"'/chunks)" + print "cat > $chunkfile << \"EOF\"\n@file " name ":" NR ; + if($2) {$1=""; print "@exec " $0} + in_comment=1; +}' > "$1" +} + +comment_form_dir="$cache_dir/comment_forms" + +# Built-in comment forms from the table +printf '%s' "$comment_forms" | while IFS= read -r _cf_line; do + case "$_cf_line" in *[!" "]*) ;; *) continue ;; esac + _cf_lang="${_cf_line%%,*}" + _cf_rest="${_cf_line#*,}" + _cf_end="${_cf_rest%%,*}" + _cf_start="${_cf_rest#*,}" + _write_comment_form "$comment_form_dir/${_cf_lang}.awk" "$_cf_end" "$_cf_start" +done + +# Aliases for same-syntax languages +for _alias_ext in h cpp hpp sql dot ts js; do + cp "$comment_form_dir/c.awk" "$comment_form_dir/${_alias_ext}.awk" 2>/dev/null || true +done +for _alias_ext in moon tl; do + cp "$comment_form_dir/lua.awk" "$comment_form_dir/${_alias_ext}.awk" 2>/dev/null || true +done + +# Python: ''' is both the open and close delimiter. Renderer name is +# glued directly after the opening ''' with no space required (e.g. '''md). +# Custom AWK uses sub() for renderer extraction instead of field splitting. +sed "s|__PY_CACHE__|${cache_dir}|g" << 'PYAWK' > "$comment_form_dir/py.awk" +BEGIN{} +/'''/ { + if(in_comment) {print "EOF"} + in_comment=0 +} +in_comment {print} +/'''.+/ && !/'''$/ { + print "chunkfile=$(mktemp -p __PY_CACHE__/chunks)" + print "cat > $chunkfile << \"EOF\"\n@file " name ":" NR ; + renderer = $0 + sub(/'''/, "", renderer) + sub(/^[[:space:]]+/, "", renderer) + sub(/[[:space:]]+$/, "", renderer) + if (renderer != "") {print "@exec " renderer} + in_comment=1; +} +PYAWK + +# .md files only have 1 chunk +printf '%s\n' ' +BEGIN{ + print "chunkfile=$(mktemp -p '"${cache_dir}"'/chunks)" + print "cat > $chunkfile << \"EOF\"\n@file " name ":" NR "\n@exec md" ; +} +{print $0} +END{print "EOF"}' > "$comment_form_dir/md.awk" +echo "%(body)s" > "$cache_dir/rsttemplate.txt" + +# Apply -f extra file format definitions from command line. +# Format: ext;start_regex;end_regex (AWK regex patterns without slashes). +printf '%s' "$extra_formats" | while IFS= read -r fmt_spec; do + [ -z "$fmt_spec" ] && continue + _ef_ext="${fmt_spec%%;*}" + _ef_rest="${fmt_spec#*;}" + _ef_start="${_ef_rest%%;*}" + _ef_end="${_ef_rest#*;}" + if [ -n "$_ef_ext" ] && [ -n "$_ef_start" ] && [ -n "$_ef_end" ]; then + _write_comment_form "$comment_form_dir/${_ef_ext}.awk" "$_ef_end" "$_ef_start" + fi +done + + +echo "" > "$cache_dir/namespace/global.meta" +if [ -n "$layout_template" ]; then + if [ ! -f "$layout_template" ]; then + echo "trbldoc: layout template not found: $layout_template" >&2 + exit 1 + fi + cp "$layout_template" "$cache_dir/namespace/main.layout" +else +cat > $cache_dir/namespace/main.layout << EOF +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>{{ title }}</title> + <link href="/stylesheets/style.css" rel="stylesheet"> + </head> + <body> + <header> + <h1><a href="#">{{ breadcrumb }}</a></h1> + </header> + <nav><ul>{{ nav }}</ul></nav> + <article> + {{ yield }} + </article> + <footer> + <p>Generated by trbldoc</p> + </footer> + </body> +</html> +EOF +fi +mkdir -p "$cache_dir/namespace/stylesheets" +cat > "$cache_dir/namespace/stylesheets/style.css" << EOF +body{ + margin:40px auto; + max-width:80ch; + line-height:1.6; + font-size:18px; + color:#444; + padding:0 10px +} +h1,h2,h3{line-height:1.2} +caption,td{border-bottom:1px solid black;} +nav{float:left;position:absolute;left:10%;} +table{width:100%;} +EOF + +# Find all the source files +# Build extension filter from awk files in comment_form_dir +ext_filter="" +for _awk_file in "$comment_form_dir"/*.awk; do + [ -f "$_awk_file" ] || continue + _ext=$(basename "$_awk_file" .awk) + ext_filter="${ext_filter}${ext_filter:+|}${_ext}" +done +# Populate source files list (one per line) +source_files_list="$cache_dir/source_files.list" +printf '%s' "$infolders" \ +| xargs --no-run-if-empty -I '{}' find "{}" -type f \ +| grep -E "\.(${ext_filter})$" \ +> "$source_files_list" + +# Analyze source files for chunks +# -C wipes the chunk cache and all timestamps for a full re-scan. +# Otherwise only source files newer than their last-processed marker are +# re-extracted; chunks for unchanged files are reused as-is. +if [ "$clean_build" -eq 1 ]; then + rm -rf "$cache_dir/chunks" "$cache_dir/last_processed" +fi +mkdir -p "$cache_dir/chunks" +mkdir -p "$cache_dir/last_processed" + +# Rendered output is always rebuilt from the full set of chunks. +# This has to be the case because the `@exec` may change. +# `@exec` is just a program, we can't garentee it'll do the same stuff next run. +# See `docs/trblcache.md for more info +find "$cache_dir/namespace" -name "index.html" -delete 2>/dev/null || true +> "$cache_dir/namespace/nav.partial" +export cache_dir comment_form_dir +changed_sources_list="$cache_dir/changed_sources.list" +tracked_sources_list="$cache_dir/tracked_sources.list" +removed_sources_list="$cache_dir/removed_sources.list" +purge_sources_list="$cache_dir/purge_sources.list" +> "$changed_sources_list" + +# Build changed source list (only files newer than their marker, or missing marker). +if [ -s "$source_files_list" ]; then + while IFS= read -r source_file; do + last="$cache_dir/last_processed/$source_file" + if [ -f "$last" ] && [ ! "$source_file" -nt "$last" ]; then + continue + fi + printf '%s\n' "$source_file" >> "$changed_sources_list" + done < "$source_files_list" +fi + +# Compute removed sources from stale markers, then remove those markers. +find "$cache_dir/last_processed" -type f 2>/dev/null \ + | sed "s|^$cache_dir/last_processed/||" > "$tracked_sources_list" +grep -Fvx -f "$source_files_list" "$tracked_sources_list" 2>/dev/null > "$removed_sources_list" || true +if [ -s "$removed_sources_list" ]; then + awk -v base="$cache_dir/last_processed/" '{ print base $0 }' "$removed_sources_list" \ + | xargs rm -f 2>/dev/null || true +fi + +# Purge stale chunks once (changed + removed), instead of per-worker scans. +{ + cat "$changed_sources_list" 2>/dev/null + cat "$removed_sources_list" 2>/dev/null +} | awk 'NF && !seen[$0]++' > "$purge_sources_list" +if [ -s "$purge_sources_list" ]; then + _purge_patterns=$(mktemp) + awk '{ printf "@file %s:\n", $0 }' "$purge_sources_list" > "$_purge_patterns" + grep -rlFf "$_purge_patterns" "$cache_dir/chunks/" 2>/dev/null \ + | xargs rm -f 2>/dev/null || true + rm -f "$_purge_patterns" +fi +# Create helper script for parallel extraction +_extract_helper="$cache_dir/_extract_helper.sh" +cat > "$_extract_helper" << 'EXTRACT_SH' +#!/bin/sh +source_file="$1" +file_name=$(basename -- "$source_file") +file_ext=${file_name##*.} +awk_prog_file="$comment_form_dir/$file_ext.awk" +if [ ! -f "$awk_prog_file" ]; then + exit 0 +fi +chunk_script=$(awk -v name="$source_file" -f "$awk_prog_file" "$source_file") +eval "$chunk_script" +last="$cache_dir/last_processed/$source_file" +mkdir -p "${last%/*}" && touch "$last" +EXTRACT_SH +chmod +x "$_extract_helper" +if [ -s "$changed_sources_list" ]; then + tr '\n' '\0' < "$changed_sources_list" \ + | xargs -0 -n 1 -P "$parallel_jobs" "$_extract_helper" +fi + +# Resolve src_repo once before the chunk loop. +# Prefers the TRBLDOC_SRC_REPO environment variable; falls back to the git +# remote origin URL; defaults to empty string if neither is available. +src_repo_val="${TRBLDOC_SRC_REPO:-$(git remote get-url origin 2>/dev/null || echo '')}" + +_chunk_files_list=$(mktemp) +find "$cache_dir/chunks" -type f > "$_chunk_files_list" +chunk_order_file="$cache_dir/chunk.render_order" +> "$chunk_order_file" +chunk_order_delim="$US" +while IFS= read -r chunk_file; do + IFS="$US" read -r chunk_from_p chunk_namespace_p chunk_priority_p chunk_program_p chunk_ref_p <<_USEOF_ +$(parse_chunk_metadata "$chunk_file") +_USEOF_ + chunk_source_path="${chunk_from_p%:*}" + chunk_source_line="${chunk_from_p##*:}" + if ! printf '%s' "$chunk_source_line" | grep -qE '^-?[0-9]+$'; then + chunk_source_line=0 + fi + chunk_namespace="$(namespace_for_chunk "$chunk_source_path" "$chunk_namespace_p")" || exit 1 + chunk_priority=0 + if printf '%s' "$chunk_priority_p" | grep -qE '^-?[0-9]+$'; then + chunk_priority="$chunk_priority_p" + fi + printf "%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s\n" \ + "$chunk_priority" \ + "$chunk_namespace" \ + "$chunk_source_path" \ + "$chunk_source_line" \ + "$chunk_file" \ + "$chunk_from_p" \ + "$chunk_program_p" \ + "$chunk_ref_p" >> "$chunk_order_file" +done < "$_chunk_files_list" +rm -f "$_chunk_files_list" +# Deterministic render order: +# 1 @priority (higher numbers first) +# 2 namespace +# 3 source path +# 4 source line +# 5 chunk temp file path (stable tie-breaker) +sorted_chunk_order_file="$cache_dir/chunk.render_order.sorted" +LC_ALL=C sort -t "$US" -k1,1nr -k2,2 -k3,3 -k4,4n -k5,5 "$chunk_order_file" > "$sorted_chunk_order_file" +chunk_render_records_file="$cache_dir/chunk.render_records" +> "$chunk_render_records_file" +chunk_seq=0 +while IFS="$US" read -r _chunk_priority namespace _chunk_source_path _chunk_source_line chunk_file from_p program_p ref_p; do + chunk_seq=$((chunk_seq + 1)) + printf "%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s${chunk_order_delim}%s\n" \ + "$chunk_seq" \ + "$_chunk_priority" \ + "$namespace" \ + "$_chunk_source_path" \ + "$_chunk_source_line" \ + "$chunk_file" \ + "$from_p" \ + "$program_p" \ + "$ref_p" >> "$chunk_render_records_file" +done < "$sorted_chunk_order_file" +render_artifacts_dir="$cache_dir/rendered_chunks" +rm -rf "$render_artifacts_dir" +mkdir -p "$render_artifacts_dir" +export SCRIPT_DIR src_repo_val render_artifacts_dir US +export TRBLDOC_RESOLVER_OVERRIDES_RAW resolver_overrides_dir +# Pre-compute the escaped sed patterns/replacements for template variables +# once in the parent so child renderers avoid repeated escape subshells. +# {{ref_string}} pattern is constant; {{src_repo}} pattern+replacement are constant. +_esc_ref_pat=$(escape_sed_pattern_literal '{{ ref_string }}') +_esc_src_pat=$(escape_sed_pattern_literal '{{ src_repo }}') +_esc_src_repl=$(escape_sed_replacement_literal "$src_repo_val") +export _esc_ref_pat _esc_src_pat _esc_src_repl +_render_helper="$cache_dir/_render_helper.sh" +cat > "$_render_helper" << 'RENDER_SH' +#!/bin/sh +render_record="$1" +IFS="$US" read -r chunk_seq _chunk_priority namespace _chunk_source_path _chunk_source_line chunk_file from_p program_p ref_p <<_USEOF_ +$render_record +_USEOF_ +program="${program_p:-cat}" +. "$common_functions_sh" +# Compute the {{ref_string}} replacement. Only chunks with @ref need the +# full resolve_ref call. +_esc_ref_repl="" +if [ -n "$ref_p" ]; then + _rs_prefix="${ref_p%%/*}" + _rs_path="${ref_p#*/}" + _rs_url=$(resolve_ref "$_rs_prefix" "$_rs_path" "$_chunk_source_path" "$_chunk_source_line") + if [ -n "$_rs_url" ]; then + _esc_ref_repl=$(escape_sed_replacement_literal "<a href='$_rs_url'>$ref_p</a>") + fi +fi +export TRBLCACHE="$cache_dir" FILEPATH="$_chunk_source_path" LINENUM="$_chunk_source_line" +artifact_prefix=$(printf '%s/%08d' "$render_artifacts_dir" "$chunk_seq") +# One sed pass: strip directives + substitute both template variables, piped +# directly into the renderer. The escaped patterns and the {{src_repo}} +# replacement are pre-computed in the parent; only {{ref_string}}'s +# replacement (when @ref is present) is computed per-chunk. +render_rc=0 +sed -E \ + -e "s/^@.*//g" \ + -e "s|${_esc_ref_pat}|${_esc_ref_repl}|g" \ + -e "s|${_esc_src_pat}|${_esc_src_repl}|g" \ + "$chunk_file" | eval "$program" > "${artifact_prefix}.body" || render_rc=$? +if [ "$render_rc" -ne 0 ]; then + echo "trbldoc: renderer failed (exit $render_rc): $program" >&2 + echo "trbldoc: chunk source: $from_p" >&2 +fi +RENDER_SH +chmod +x "$_render_helper" +if [ -s "$chunk_render_records_file" ]; then + while IFS= read -r render_record; do + printf '%s\0' "$render_record" + done < "$chunk_render_records_file" \ + | xargs -0 -n 1 -P "$parallel_jobs" "$_render_helper" +fi +while IFS="$US" read -r chunk_seq _chunk_priority namespace _chunk_source_path _chunk_source_line _chunk_file from_p _program_p ref_p; do + ref="${ref_p:-$namespace}" + artifact_prefix=$(printf '%s/%08d' "$render_artifacts_dir" "$chunk_seq") + output="" + if [ -f "${artifact_prefix}.body" ]; then + output=$(cat "${artifact_prefix}.body") + fi + if [ -n "$output" ]; then + from_html="<cite>From $from_p</cite>" + if [ -n "$src_repo_val" ]; then + src_file_path="$_chunk_source_path" + src_file_line="" + if printf '%s' "$_chunk_source_line" | grep -qE '^[1-9][0-9]*$'; then + src_file_line="$_chunk_source_line" + fi + src_file_url="${src_repo_val%/}/$src_file_path" + if [ -n "$src_file_line" ]; then + src_file_url="${src_file_url}#L${src_file_line}" + fi + from_html="<cite>From <a href='$src_file_url'>$from_p</a></cite>" + fi + if [ -n "$namespace" ]; then + outfolder="$cache_dir/namespace/$namespace" + else + outfolder="$cache_dir/namespace" + fi + if [ ! -d "$outfolder" ]; then + mkdir -p "$outfolder" + fi + echo "$output" "$from_html" >> "$outfolder/index.html" + fi + # internal links for generated cross references + if [ "$ref" = "$namespace" ]; then + if [ -n "$ref" ]; then + echo "$ref: <a href='/$ref'>$ref</a>" >> "$cache_dir/namespace/global.meta" + else + echo "index: <a href='/'>index</a>" >> "$cache_dir/namespace/global.meta" + fi + else + echo "$ref: <a href='/$namespace#$ref'>$ref</a>" >> "$cache_dir/namespace/global.meta" + fi +done < "$chunk_render_records_file" + +> "$cache_dir/namespace/nav.partial" +find "$cache_dir/namespace" -name "index.html" \ +| sed \ + -e 's;/index\.html$;;' \ + -e "s;^$cache_dir/namespace/*;;" \ +| sort -u \ +| awk ' +function emit(path, nxt, n, c, i) { + if (path == "") { print "<li><a href=\047/\047>index</a></li>"; return } + n = split(path, p, "/") + c = 0 + while (c < depth && c < n - 1 && q[c+1] == p[c+1]) c++ + for (; depth > c; depth--) print "</ul></li>" + for (i = c + 1; i < n; i++) print "<li>" p[i] "/<ul>" + if (index(nxt, path "/") == 1) { print "<li><a href=\047/" path "\047>" p[n] "</a>/<ul>"; depth = n } + else { print "<li><a href=\047/" path "\047>" p[n] "</a></li>"; depth = n - 1 } + split(path, q, "/") +} +NR > 1 { emit(prev, $0) } +{ prev = $0 } +END { + if (NR) emit(prev, "") + for (; depth > 0; depth--) print "</ul></li>" +} +' > "$cache_dir/namespace/nav.partial" + +# Second pass: resolve inline references now that global.meta is complete. +# Deduplicate global.meta (keep first occurrence per ref name, which has highest priority) +_deduped_meta="$cache_dir/namespace/global.meta.deduped" +awk -F': ' '!seen[$1]++' "$cache_dir/namespace/global.meta" > "$_deduped_meta" + +# Lookup ref link HTML from the deduplicated file +_ref_link_lookup() { + awk -v name="$1" '{ + p = index($0, ": ") + if (p > 0 && substr($0, 1, p - 1) == name) { + print substr($0, p + 2) + exit + } + }' "$_deduped_meta" +} + +token_sep="$US" +_ref_find_tmp=$(mktemp) +find "$cache_dir/namespace" -name "index.html" > "$_ref_find_tmp" +while IFS= read -r htmlfile; do + replacement_pairs_file=$(mktemp) + # Extract source file info for external resolver env vars + _html_from=$(extract_source_from_cite "$htmlfile") + _html_filepath="${_html_from%:*}" + _html_linenum="${_html_from##*:}" + if ! printf '%s' "$_html_linenum" | grep -qE '^[0-9]+$'; then + _html_linenum=0 + fi + collect_html_ref_tokens "$htmlfile" | while IFS="$token_sep" read -r ref_token_type ref_token; do + [ -z "$ref_token" ] && continue + if [ "$ref_token_type" = "I" ]; then + ref_name="${ref_token#@}" # strip leading @ + _link_html=$(_ref_link_lookup "$ref_name") + if [ -n "$_link_html" ]; then + # Found in global.meta - use the pre-built HTML link + printf '%s%s%s\n' "$ref_token" "$token_sep" "$_link_html" >> "$replacement_pairs_file" + fi + continue + fi + if [ "$ref_token_type" != "E" ]; then + continue + fi + + # Resolve @script/path tokens (external refs) - only if not matched above + ref_name="${ref_token#@}" + _link_html=$(_ref_link_lookup "$ref_name") + if [ -n "$_link_html" ]; then + continue + fi + script_name="${ref_name%%/*}" # get script prefix before / + ref_path="${ref_name#*/}" # get path after @prefix/ + resolved_url=$(resolve_ref "$script_name" "$ref_path" "$_html_filepath" "$_html_linenum") + if [ -n "$resolved_url" ]; then + replacement_link="<a href='${resolved_url}'>$script_name🔗${ref_path}</a>" + printf '%s%s%s\n' "$ref_token" "$token_sep" "$replacement_link" >> "$replacement_pairs_file" + else + echo "trbldoc: unresolved ref: $ref_token (at $_html_filepath:$_html_linenum)" >&2 + fi + done + + replace_literal_pairs_in_file "$htmlfile" "$replacement_pairs_file" + rm -f "$replacement_pairs_file" + +done < "$_ref_find_tmp" +rm -f "$_ref_find_tmp" + +# Post-processing pass: substitute {{ toc }} in all rendered index.html files +# now that `nav.partial` is finalised. Lines are joined into one so the sed +# substitution works without needing multi-line mode. +toc_oneline=$( \ + cat "$cache_dir/namespace/nav.partial" \ + | tr '\n' ' ' \ + | sed -e "$sed_replacement_literal" \ +) +toc_repl=$( \ + printf "%s" "{{ toc }}" \ + | sed -e "$sed_pattern_literal" \ +) +find "$cache_dir/namespace" -name "index.html" \ +| xargs -n 1 -P "$parallel_jobs" sed -E -i -e "s|$toc_repl|$toc_oneline|g" +assemble_site "$cache_dir/namespace" "$out_dir" |
