Know what your application did.

Every request and job gets a stable id, one log line and, when something fails, a report that stays out of your logs.

This guide teaches each task once and names one example. Every field, variable and limit has its home in the reference.

The examples use the Bookshelf application of the Learn path. ./bookshelf is its binary, APP on the commands page, run on the machine where it serves.

Stage 1 · Follow requests

Follow one request

Every routed request carries X-Request-ID: a well-formed inbound one is kept, any other is replaced by a UUID, and a W3C traceparent continues the caller’s trace. The response, the log line and the error report all name the same ids, and a job enqueued by the request carries them on.

Read the log line

Each request, job and schedule run writes one wide line from the crema log source: its name, its ids, its duration and outcome, and counts of queries, their time, view time, outbound calls, cache reads and jobs enqueued. JSON is the default outside development and test; CARAMEL_LOG_FORMAT chooses the format and LOG_LEVEL the level. Outside development, no line carries the request path or an exception’s message, and no line ever carries a query’s bind values.

Report errors

An unhandled error in a request is reported once, with its class, a fingerprint and the first frame of your own code. The fingerprint ignores lines and messages, so one defect stays one group across deploys. Call Caramel::Crema.report(error) for an error you handle, and register a hook to forward reports, redacted of secrets, to an error tracker.

Caramel::Crema.on_error do |report|
  Tracker.capture(report.error_class, report.fingerprint, report.message)
end

The message and backtrace reach the hook, the ops socket and the development tools, and nothing else.

Set what counts as slow

A slow request or job is kept for ops traces --slow and flagged in the development toolbar; a slow query adds to the log line’s slow_queries. Change the three thresholds in your application’s configuration. A statement that runs past PostgreSQL’s statement_timeout, which CARAMEL_STATEMENT_TIMEOUT sets, is cancelled and reported as an error instead.

Caramel::Crema.slow_request = 500.milliseconds
Caramel::Crema.slow_job = 30.seconds
Caramel::Crema.slow_query = 50.milliseconds

See where a query came from

Every statement a request runs starts with a comment that names the action, so pg_stat_activity and PostgreSQL’s slow-query log say which code ran it; jobs and schedules are tagged the same way, and each pool names itself in application_name. A job continues the trace that enqueued it, Caramel::Outbound calls send traceparent on, and SweetTalk commands, factories and services are timed as domain spans. When the trace records spans, a statement that repeats five or more times in one request is reported as a repeated query.

Stage 2 · Develop

Dump a value while developing

dump prints a value with its file and line to standard error and returns the value, so it fits inside an expression. It works in actions, views and jobs while frappe dev runs, and does nothing in a production build. frappe lint reports a dump left in the code.

book = dump(App::Book.find!(contract.id))

Read an error page

In development an unhandled error shows its class and message, every frame of your own code with an Open in editor link, the source around the failing line, the request, the queries that ran before it and its causes. Copy as Markdown copies all of it for a bug report or an agent. A client that asks for JSON gets the same as JSON. Compile errors link to the file in the same way; CARAMEL_EDITOR names the editor.

Inspect requests and jobs

Every full HTML page frappe dev serves carries a toolbar with the request’s route, status, time and query count, flags for errors, slow requests and repeated queries, and the recent requests. Each part links to its place in the inspector at /__caramel/dev/inspector, which lists recent requests, jobs and schedule runs with their timeline, queries and logs, groups errors by fingerprint and lists the builds. Copy for an agent puts a request on the clipboard as Markdown, and each query has Copy SQL and Copy with values for psql. The browser’s developer tools show database and view time through Server-Timing.

Watch from the menu bar

Latte’s menu bar app shows each site’s error count beside its state, the newest error in its submenu, and an Open inspector item. A failed build or a new error fingerprint counts beside the cup and posts a macOS notification that opens the inspector; allow notifications for Latte when macOS asks. A notification carries the error’s class and location, never its message.

Read the proxy’s view of a request

Latte’s HTTPS proxy writes one JSON line per request to the site’s own access.log, with the application’s X-Request-Id, beside app.log and compiler.log. Caddy redacts Cookie and Authorization.

frappe logs access
frappe logs access --follow

Expect: Prints the last 200 lines of the site’s access.log, one JSON line per request with status, duration, size, host, URI and response headers; --follow keeps printing. Before the site has served a request it exits 1 with No access log for NAME yet. Run frappe dev.

Follow a trace across sites

Latte runs a local OTLP collector on 127.0.0.1:4318, and frappe dev sends its application’s traces there. A site that calls another through Caramel::Outbound sends a traceparent, so both sites’ spans share one trace, and the inspector’s trace page and frappe trace REF --md add an Across services section. Any other service joins by exporting OTLP/HTTP JSON to the same address; a Caramel service outside frappe dev needs require "caramel/crema/otlp" for that.

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
OTEL_EXPORTER_OTLP_PROTOCOL=http/json \
OTEL_SERVICE_NAME=billing ./billing serve

Expect: billing serves as usual and prints nothing about the exporter. After the bookshelf calls it, frappe trace last --md in the bookshelf ends with an ## Across services section that lists billing’s spans.

Ask from the terminal

These commands read the events frappe dev kept, whether or not it still runs: frappe traces lists recent requests and jobs, frappe trace REF shows one, and frappe errors groups runtime errors and repeated queries since the newest successful build, so a rebuild clears the errors it fixed. With --agent, or when stdout is not a terminal, they print MRDP.

frappe errors --agent
frappe trace last-error --md
frappe traces --errors --limit=5

Expect: frappe errors --agent prints OK errors 0 and exits 0 when nothing failed since the newest successful build; otherwise one ERR RUNTIME:500 or ERR REPEATED_QUERY block per group, and exits 1. frappe trace last-error --md prints the failed trace as Markdown with a ## Backtrace section, and exits 1 when no trace failed. frappe traces --errors --limit=5 lists the newest five failed traces.

Stage 3 · Operate

Look inside a running application

A serve process opens a private ops socket next to its application socket; only its owner can use it, and nothing is added to the public listener. CARAMEL_OPS_SOCKET moves it, turns it off, or gives a work process one. The application binary is its own client.

./bookshelf ops status
./bookshelf ops tail --errors --slow=500
./bookshelf ops errors
./bookshelf ops error 9f2c4e1a7b3d
./bookshelf ops traces --slow
./bookshelf ops trace 8b1f6c2a --md
./bookshelf ops metrics

Expect: ops status prints seven lines, App, Requests, Jobs, Database, Memory, Fibers and Crema; the first names the application, its Caramel version, role, pid and uptime. ops tail streams finished traces and errors until you stop it. ops errors prints a table with FINGERPRINT COUNT LAST_SEEN CLASS LOCATION SOURCE, the errors seen since the process started; ops error FINGERPRINT prints one with its redacted message and backtrace. ops traces --slow lists the kept slow traces and ops trace REF --md prints one as Markdown. ops metrics prints Prometheus text. Without a running process each exits 1 with No ops socket at PATH. Is the application running with CARAMEL_OPS_SOCKET?

Errors live in memory on the socket and nowhere else, so forward Caramel::Crema.on_error reports to an error tracker for history. A scraper that can reach the socket can read ops metrics.

Trace one person’s requests

ops debug-token --minutes=15 prints a signed token. A request that sends it as X-Caramel-Debug, or in the __Host-caramel_debug cookie, is traced in full, answered with X-Caramel-Trace, kept for ops traces --debug and passed to the jobs it enqueues.

./bookshelf ops debug-token --minutes=15

Expect: Prints the token and its expiry, a curl line that sends the header and a browser line that sets the cookie.

Open the console

The console is the same data as HTML, read-only: in-flight work, a live tail, queues, database health, errors, traces and insights. Its trace pages carry no bind values.

./bookshelf ops console

Expect: Prints ssh -N -L 8765:SOCKET HOST and then open http://localhost:8765/. It does not connect: run the ssh line on your laptop, then open the address.

Inspect queues and the database

The jobs and db diagnose commands need the database, not a running application: queue counts, failures grouped by class and error, one job with its stored error, a retry, and PostgreSQL’s own view of connections, locks, cache hits, indexes, vacuum and slow statements. frappe db diagnose runs the last one against the development database.

./bookshelf jobs
./bookshelf jobs failed
./bookshelf db diagnose

Expect: jobs prints one row per queue with running, retrying, queued, scheduled, failed, finished_1h and oldest_due_s. jobs failed prints the newest 20 failure groups with their class, error class, count, last time and newest id. db diagnose prints one section per check and exits 0 when any section was readable.

Stage 4 · Keep history

Keep latency history

Logs say what happened; insights says how it has been going. The recorder keeps one row per minute for each route, job class, schedule, SQL statement and outbound host in caramel_metrics, aggregates only: never a path, a bind value or a message. New applications already require it; add it to an older one and run frappe migrate once, because its table comes with Cold Brew’s framework tables.

require "caramel"
require "caramel/crema/recorder"

./bookshelf insights --since=24h
./bookshelf insights --kind=sql --since=1h

Expect: Each prints a table with KEY COUNT ERR P50MS P95MS MAXMS TOTAL_S, the 20 rows with the most total time, or No aggregates in the last 24h. Is the recorder running? when the window is empty. Without the recorder it prints The Crema recorder is off. Add require "caramel/crema/recorder" to config/application.cr. and exits 1. The console’s Insights page shows the same tables.

Send traces to a tracing backend

Add require "caramel/crema/otlp" and set the standard OpenTelemetry variables; the application then exports each sampled request, job and schedule run, with a child span for every query and outbound call, as OTLP over HTTP with JSON bodies. Spans name routes and parameterized SQL, never paths, bind values, log text or error messages.

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com
OTEL_EXPORTER_OTLP_HEADERS=x-api-key=SECRET
OTEL_SERVICE_NAME=bookshelf
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Expect: The process prints nothing about the exporter; sampled traces appear in the backend under bookshelf. Without OTEL_EXPORTER_OTLP_ENDPOINT nothing is sent, and a protocol other than http/json logs a warning and leaves export off.

Read the tracing design ↗ · Read the production access design ↗

Stage 5 · Limits

What does not compile

Crema is on in every application and adds no word to the language, so it rarely stops a build.

Not covered yet

• The collector and the exporter take OTLP over HTTP with JSON bodies only; protobuf is not supported.
• Errors on the ops socket live in memory only; a restart clears them.
• An error tracker is yours to connect through Caramel::Crema.on_error; Crema ships no client.

module App::Probes
  struct Show < App::ApplicationAction
    contract do
    end

    def handle(contract : Contract)
      raise "probe: do-not-log"
    end
  end
end

    get "/probe", App::Probes::Show

curl -s -o /dev/null -w "%{http_code}\n" https://bookshelf.caramel/probe
frappe errors --agent
frappe trace last-error --md

./bookshelf ops status
./bookshelf ops errors
./bookshelf ops metrics