diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/coverage.md | 55 | ||||
| -rw-r--r-- | docs/profiling.md | 81 | ||||
| -rw-r--r-- | docs/trblcache.md | 83 |
3 files changed, 219 insertions, 0 deletions
diff --git a/docs/coverage.md b/docs/coverage.md new file mode 100644 index 0000000..c6371d0 --- /dev/null +++ b/docs/coverage.md @@ -0,0 +1,55 @@ +# Test-suite coverage + +`scripts/coverage-tests.sh` reports which lines of trbldoc's shell sources the +test suite actually executes. + +## Usage + +```sh +# Coverage of trbldoc.sh from the integration suite (default) +sh scripts/coverage-tests.sh + +# Whole suite, also cover a resolver, fail under 70% +sh scripts/coverage-tests.sh -s all -t "trbldoc.sh ref_resolvers/lua.sh" -m 70 +``` + +Options: `-s all|unit|integration`, `-o OUTDIR`, `-t "file.sh ..."` (targets), +`-m MIN_PERCENT` (exit non-zero below this; default `0` = no gate). + +## Output + +Written to `reports/coverage-<timestamp>/` (or `-o`): + +- `index.html` — per-file coverage with bars and an overall percentage. +- `html/<file>.html` — annotated source; green = executed, red = missed, + grey = non-executable. +- `summary.csv` — `file,executable,covered,percent`. +- `covered.tsv` — raw union of executed `file,line` pairs. + +## How it works + +The bats tests run the tool as `bash "$TRBLDOC"`. Coverage exports +`BASH_ENV=scripts/lib/trace-init.sh`, so each target process enables `set -x` +with a `PS4` that stamps `BASH_SOURCE:LINENO` to a per-PID trace file. After +the run, the union of executed lines is compared against the executable lines +in each target source to compute coverage. No changes to `trbldoc.sh` or the +tests are needed. + +Only scripts whose basename is listed in `-t` are traced, so bats internals +and the `md2html` stub stay out of the report. + +## Determining "executable" lines + +The denominator is a heuristic: a line counts as executable when, after +trimming, it is non-blank, not a comment, and not a pure structural token +(`then`, `do`, `done`, `else`, `fi`, `esac`, `{`, `}`, `(`, `)`, `;;`). + +## Limitations + +- Coverage is collected under `bash` (the shell the suite already uses). + Lines only reachable under a different shell are not distinguished. +- xtrace reports the line where a command *starts*. Interior lines of + heredocs and multi-line strings are not individually reported and may show + as missed; exclude such files or read the numbers with that caveat. +- Code in subprocesses spawned via `ash`/`xargs` is not traced (not bash), + so it is not counted toward coverage of the main script. diff --git a/docs/profiling.md b/docs/profiling.md new file mode 100644 index 0000000..9978347 --- /dev/null +++ b/docs/profiling.md @@ -0,0 +1,81 @@ +# Profiling the test suite + +`scripts/profile-tests.sh` reports where time goes when running the bats +suite, at two levels of detail. + +## What it produces + +1. **Per-test timing** (always). Runs bats with `--timing` and the JUnit + report formatter, then ranks every test slowest-first. Answers *which + tests take a long time*. +2. **Per-line hotspots** (with `-p`). Re-runs the suite with `trbldoc.sh` + under `bash` xtrace and attributes elapsed wall-time to individual source + lines. Answers *which code takes a long time*. + +## Usage + +```sh +# Per-test timing for the integration suite (default) +sh scripts/profile-tests.sh + +# Include per-line hotspots inside trbldoc.sh +sh scripts/profile-tests.sh -p + +# Profile the whole suite, show the 30 slowest tests +sh scripts/profile-tests.sh -s all -n 30 +``` + +Options: `-s all|unit|integration`, `-o OUTDIR`, `-n TOPN`, `-p` (hotspots), +`-t "file.sh ..."` (hotspot target scripts). + +## Output + +Written to `reports/profile-<timestamp>/` (or `-o`): + +- `timings.html` — ranked per-test bar chart. +- `timings.csv` / `timings.sorted.tsv` — raw per-test data. +- `hotspots.html` / `hotspots.csv` — per-line self-time (only with `-p`). +- `junit/`, `bats-output.txt` — raw bats artifacts. + +## Profiling real trbldoc runs (outside tests) + +Use `scripts/profile-run.sh` when you want hotspot data for real workloads +rather than test execution. + +```sh +# Single direct run +sh scripts/profile-run.sh -- -s src -o docs + +# Repeat 3 clean runs to stabilize hotspot ranking +sh scripts/profile-run.sh -n 3 -C -- -s src -o docs +``` + +Outputs under `reports/profile-run-<timestamp>/`: + +- `run-summary.csv` — wall-clock runtime per run. +- `hotspots.html` — per-line self-time aggregated from xtrace markers. +- `phase-hotspots.html` — coarse pipeline phase self-time (discovery, + extraction, render ordering, rendering, nav, ref resolution, toc, assemble), + inferred from traced line ranges in `trbldoc.sh`. + +## How the hotspot profiler works + +The bats tests invoke the tool as `bash "$TRBLDOC"`. The profiler exports +`BASH_ENV=scripts/lib/trace-init.sh`, which every non-interactive bash sources +at startup. For scripts whose basename is in `TRBLDOC_TRACE_TARGETS` +(default `trbldoc.sh`) it enables `set -x` with a `PS4` that stamps +`BASH_SOURCE:LINENO` and `EPOCHREALTIME` to a per-PID trace file. The +aggregator computes the delta between consecutive traced lines *within each +process*, so trbldoc's parallel `xargs -P` workers are not cross-attributed. + +## Limitations + +- Hotspot timing needs `bash` ≥ 5 (`EPOCHREALTIME`). The report scripts + themselves are POSIX sh and run under mingw bash, WSL bash and busybox ash. +- Subprocesses trbldoc spawns via `ash`/`xargs` are not bash and are not + traced; hotspots reflect the main-script control flow, not the render + helper's internals. +- Per-line self-time is derived from trace timestamps and includes tracing + overhead, so treat the numbers as relative, not absolute. +- Phase hotspots are inferred from source-line boundaries and are intended for + prioritisation, not exact accounting. diff --git a/docs/trblcache.md b/docs/trblcache.md new file mode 100644 index 0000000..7351fb0 --- /dev/null +++ b/docs/trblcache.md @@ -0,0 +1,83 @@ +# Cacheing + +Nearly all caching happens in `$(pwd)/.trblcache` by default (hereafter `$CACHE`), +here's a reminder of the process `trblcache` uses when building documentation: + +0. Scan: find all source files under the input folders whose extension has + a registered comment form (`c`, `h`, `cpp`, `hpp`, `lua`, `py`, `sql`, + `md`, `dot`, plus any `-f` additions). +0. Extract: run each file through an AWK program that emits chunk files + into `$CACHE/chunks/`. +0. Render: for each chunk, strip directives, run the renderer, and append + output to `$CACHE/namespace/<namespace>/index.html`. +0. Assemble: write `main.layout`, `nav.partial`, `global.meta`, and CSS + into `$CACHE/namespace/`. +0. Build: apply the layout/template tokens to each namespace page and + write the final site to `-o <dest>` (default `$CACHE/built`). + +Of these, only chunk extraction can be cached. Cacheing works by first +creating a file under `$CACHE/last_processed/<path>` when a file is processed, +and then on subsequent processing, ignore files if their `$CACHE/last_processed/<path>` +file timestamp is newer than the `<path>` timestamp. If your filesystem does +not support file timestamps, `trbldoc` will never assume the cache is invalid, +and will serve stale data. If you are on such a system, always run with `-C` to +force `trbldoc` to generate new chunks. + +## Notes + +### No cached chunks +The rendered chunks, generally, cannot be cached. If the user provided their +own tooling (i.e. bash scripts) for `trbldoc`, `trbldoc` cannot check for updates +to the scripts in order to invalidate the cached rendered chunk. Checking +file last updated times won't work, if a script invokes another script or relies +on any system environment the user's updates won't take. Example: + +``` +project +| src/ +| | file1.c - last updated 1d ago +| tools/ +| | checker.sh - last updated 1d ago +| | graph.sh - last updated 30s ago +``` + +file1.c + +```c +... +/* tools/checker.sh tools/graph.sh +strict graph { + a -- b + a -- b + b -- a [color=blue] +} +*/ +... +``` + +checker.sh + +```sh +#!/bin/sh + +if [ -z "$(command $1 -V)" ]; then + echo "Could not find $1" + exit -1 +fi + +exec $1 +``` + +graph.sh + +```sh +#!/bin/sh + +exec dot +``` + +It would be impossible for `trbldoc` to determine that `tools/checker.sh` uses +`tools/graph.sh` without forcing it to run in a hermetically sealed sandbox that +carefully controls what environment, filesystem, and other programs may be run. +Such heavy sandboxing is against the vision of `trbldoc` as a simple, single-file +documentation generator. |