Reference / Crema
See what your application did.
Crema is Caramel’s observability: request ids, one log line for every request, job and schedule run, error reports, an owner-only ops socket, an opt-in recorder and exporter, and the development tools.
| Contract | Value |
|---|
| Release | 0.15.0 |
|---|
| Needs | A Caramel application; the recorder and exporter are opt-in requires |
|---|
| Covers | Request ids, the log line, error reports, the ops socket, metrics, the recorder, the OTLP exporter and the development tools |
|---|
| Proved by | frappe traces --agent · frappe trace last --md · frappe errors --agent |
|---|
Learning it for the first time? The guide Know what your application did walks through it in order. This page lists every name, field, variable and limit.
Limits at a glance
| Limit | Value |
|---|
Inbound X-Request-ID | 8–128 characters |
| Slow request, default | 1 second |
| Slow job or schedule run, default | 5 seconds |
| Slow query, default | 100 milliseconds |
| Runs that make a repeated statement | 5 |
| Spans in a trace (more are counted as dropped) | 200 |
| Distinct statements tracked for repeats in a trace | 500 |
| Span detail kept in the trace ring | 2,048 bytes |
| Log entry kept as a span | 1,000 bytes |
| Binds kept per statement, each cut at | 20, 200 bytes |
Copy with values literal; longer stays $n | 2,000 bytes |
| Error report backtrace | 50 frames of 2,048 bytes |
| Error report message | 8,192 bytes |
| Causes listed | 5 |
| Error ring (least recently seen is evicted) | 500 fingerprints |
| Trace ring | 200 traces |
/v1/traces and /v2/traces limit | default 50, at most 200 |
| Concurrent tails (events buffered for each) | 8 (1,000) |
Tail keep-alive : ping | every 15 seconds |
| Ops connection read and write timeout | 10 seconds |
POST /v1/debug-tokens minutes | 1–120, default 15 |
POST /v1/debug-tokens body | 1 KiB |
| Debug token lifetime | 2 hours |
Distinct request series (the rest fold into (other)) | 1,000 |
| Recorder keys of each kind per flush | 500 |
| Recorder SQL key | 500 bytes |
| Recorder flush interval | 15 seconds |
| Recorder retention (pruned hourly) | 7 days |
| Recorder histogram | 12 buckets: 5, 10, 25, 50, 100, 250, 500, 1,000, 2,500, 5,000, 10,000 ms and overflow |
APP insights rows | 20 |
| Exporter queue | 2,048 traces |
| Exporter batch | every 5 seconds or at 512 spans |
| Exporter timeouts | 5 seconds connect, 10 seconds read |
| Error page frames | 80 |
| Source file shown in the error page | 1 MiB |
dump span | 8,192 bytes |
| Inspector traces listed | 500 |
| Toolbar request list | last 10 |
frappe traces --limit default | 20 |
| Frappé keeps | 500 traces, 200 standalone errors, 50 builds |
events.jsonl rolls at | 8 MiB |
| Trace or request id prefix | at least 6 characters |
| Database statement and lock timeout, default | 30 s, 5 s (migrate: none, 5 s) |
auto_explain threshold | 250 ms |
access.log rolls at | 1 MB |
| Collector body rejected over | 4 MiB |
| Collector memory | 2,000 traces, 200 spans each, 32 attributes per span |
Crema runs in every application
Crema is part of every application. What it does depends on how the process runs and which parts you opt into.
| Mode | What runs |
|---|
| Always on | A request id and trace for every routed request, job and schedule run; the canonical log line; error reports; the in-memory error and trace rings; the metrics behind ops metrics. A serve process also opens the ops socket. |
| Opt in | require "caramel/crema/recorder" keeps latency history in the database; new applications already have it. require "caramel/crema/otlp" exports traces to a tracing backend. |
| Development only | Code compiled with -D caramel_development, which frappe dev, frappe check and one-shot development builds pass, and run with CARAMEL_ENV=development: the error page, dump, Server-Timing, query binds and source lines, and the event sink behind the inspector. |
Settings are set before the application starts
Set these in config/application.cr, before the application starts.
| Setting | Meaning |
|---|
Caramel::Crema.slow_request | Time::Span, default 1 second. A request at least this slow is kept in the trace ring as slow and flagged in the toolbar. |
Caramel::Crema.slow_job | Default 5 seconds. Applies to jobs and schedule runs. |
Caramel::Crema.slow_query | Default 100 milliseconds. Each statement at least this slow counts in the log line’s slow_queries. |
Caramel::Crema.on_error { |report| … } | Called for every error report, handled or not, with the redacted message and backtrace. |
Caramel::Crema.report(error, handled: true, source: nil) | Reports an error you rescued. It never raises. A handled report writes a WARN line and leaves the trace’s outcome ok; handled: false marks the trace failed. |
Caramel::Crema.measure(kind, name, detail = nil) { … } | Times a block as a span of the current trace. Kinds: sql, http, view, enqueue, log, dump and domain. |
Caramel::Crema.command(name, syntax) { |arguments| … } | Adds a command to the application binary and returns its exit code. The first word of the syntax is the name; registering a name again replaces the earlier command. The binary’s usage text lists the syntax. |
Caramel::Crema.console_page(segment) { |request| … } | Adds a page at /segment to the ops console. The block returns HTML. |
Caramel::Crema.subscribe(sink) | Adds a sink that receives finished traces and error reports. Remove it with unsubscribe. |
Caramel.dump(value) | Development only. See below. |
Environment variables choose mode, format and socket
| Variable | Meaning |
|---|
CARAMEL_ENV | development adds request paths and error messages to log lines. In a development build it also turns on the error page, dump, Server-Timing, query binds and the literals behind Copy with values. test and development default the log format to text. Unset counts as production. |
CARAMEL_LOG_FORMAT | json or text. Any other value falls back to text in development and test, JSON elsewhere. |
LOG_LEVEL | Default Info. Applies to every log source, not only crema. |
CARAMEL_OPS_SOCKET | A path for the ops socket, or off. Unset, a serve process uses its CARAMEL_SOCKET path with .sock replaced by .ops.sock. A work process opens a socket only when this names a path. The ops client reads the same variables. |
CARAMEL_EDITOR | An editor preset or a link template. See the development section. |
CARAMEL_PROJECT_ROOT | The project root, default the current directory. Error locations are shown relative to it, and its app, config, src and db folders count as your code. |
APP_SECRET | The application secret. Debug tokens are signed with a key derived from it, so rotating it ends every token. |
CARAMEL_DEV_EVENTS | Set by frappe dev for development builds: the socket the application sends its events to. Do not set it yourself. |
CARAMEL_STATEMENT_TIMEOUT, CARAMEL_LOCK_TIMEOUT | PostgreSQL’s statement_timeout and lock_timeout on every pooled connection of every command but migrate: whole seconds, 0 to 86400, default 30 and 5. migrate reads CARAMEL_MIGRATION_STATEMENT_TIMEOUT and CARAMEL_MIGRATION_LOCK_TIMEOUT, default 0 and 5. 0 means none. Cold Brew’s pool reads the runtime pair. |
CARAMEL_POOL_SIZE, CARAMEL_POOL_TIMEOUT | Connections a command’s pool holds, 1 to 32, default 4, and seconds a checkout waits for a free one, 1 to 300, default 5. An invalid value exits 1 with one line naming it. |
OTEL_* | Seven variables read by the exporter; see below. |
| Condition | Outcome |
|---|
CARAMEL_ENV set to anything but development, test or production | stops every database command with CARAMEL_ENV must be development, test or production |
| A timeout variable that is not a whole number of seconds from 0 to 86400 | stops the command with KEY must be a whole number of seconds from 0 to 86400 |
A statement past statement_timeout | is cancelled by PostgreSQL, which reports it before the pool’s own read timeout |
Every traced response carries a request id
| Condition | Outcome |
|---|
Inbound X-Request-ID of 8 to 128 characters from A-Z a-z 0-9 . _ : - | kept |
Any other inbound X-Request-ID | replaced by a UUID |
| Static files, and the 400 and 421 refusals | not traced, and carry no id |
traceparent with version 00, lowercase hex, a 32-digit trace id, a 16-digit parent id and flags | continues the caller’s trace and supplies the log line’s trace_id and parent_id; the first flag bit says whether the caller sampled it |
A traceparent with all-zero ids | ignored |
enqueue | stores the traceparent, the request id and the debug flag in caramel_jobs.context, never a message; the job continues that trace |
| A schedule run, or a missing or malformed job context | starts a new trace |
Each Caramel::Outbound call | is an http span named METHOD host, and sends a traceparent unless you set one |
| A debug trace | sends X-Caramel-Trace with the trace id |
Every finished trace writes one log line
The crema log source writes one INFO line for each finished trace, named request, job or schedule.
| Field | Present when |
|---|
name | Always. GET /books/:id, the job class or the schedule name. An unrouted request is GET (none); a method outside GET, HEAD, POST, PUT, PATCH, DELETE and OPTIONS is OTHER. |
request_id, trace_id, parent_id | The ids above. parent_id only with a valid inbound or stored traceparent. |
duration_ms, outcome | Always. outcome is error after an unhandled error or a 500, or when a job’s perform raised, even if it will retry. |
method, route, action, status, bytes, streamed | Requests. route and action only when a route matched; bytes is left out for a streamed response. |
path | Requests, only when CARAMEL_ENV=development. |
job_id, queue, attempt, queue_lag_ms | Jobs. The lag is the time past run_at, never negative. |
db_count, db_ms, db_wait_ms | Statements, their time, and the wait for a pooled connection. |
view_ms, outbound_count, outbound_ms | The outermost view’s time and the Caramel::Outbound calls. |
cache_hits, cache_misses, enqueued | Caramel::Cache reads and jobs enqueued. |
slow_queries, repeated_queries | Statements past slow_query, and distinct statements that ran five or more times. Repeats are counted only when the trace records spans: the ops socket, the recorder or the exporter is on, or the build is for development. A trace keeps at most 200 spans (more are counted as dropped) and tracks at most 500 distinct statements for repeats. |
debug | true for a debug trace. |
error_class, fingerprint | Only when the trace ended in an unhandled error. |
| Condition | Outcome |
|---|
| A field with no value | is left out |
| Any log entry written inside a trace | also carries trace_id and request_id |
| A log entry written inside a trace that records spans | is kept as a log span of up to 1,000 bytes |
| A slow request or job | shows in the trace ring, the toolbar and the inspector; there is no slow field. A slow statement counts in slow_queries |
| Form | Shape |
|---|
| error line | One error entry for each report: WARN when handled, ERROR when not. Fields: error_class, fingerprint, location (relative file:line), source (the trace name, or stream or cold_brew.worker, .maintenance, .scheduler, .hooks NAME), handled, request_id and trace_id. The message is added only when CARAMEL_ENV=development. |
| JSON | One object per line: ts (UTC, milliseconds), lowercase level, source, msg, then the fields. A field named like one of those four is written as data_NAME, so an error’s source becomes data_source. The default outside development and test. |
| text | HH:MM:SS.mmm LEVEL and a one-line summary with one decimal per time: 12:00:00.123 INFO request GET /books/1 200 12.4ms db=3/4.1ms view=2.0ms App::Books::Show request_id=8b1f; outside development the route /books/:id stands where the path is. Values with spaces are quoted. The default in development and test. |
Error reports are redacted and stay in memory
| Part | Content |
|---|
| Fields | error_class, message, backtrace (at most 50 frames of 2,048 bytes), fingerprint, location, handled, source, request_id, trace_id, occurred_at and the classes of up to five causes. A message is cut to 8,192 bytes. |
| Fingerprint | The first 12 hex digits of a SHA-256 over the error class, the path of the first frame in your own code and its method name. With no such frame, the first parseable frame; with none, the class alone. Lines, columns and messages are never hashed. |
| Redaction | Applied to every message and backtrace frame: values of environment variables whose names contain SECRET, PASSWORD, TOKEN, API_KEY or DATABASE_URL become [redacted]; PostgreSQL URLs become [database URL redacted]; name=value, name: value and JSON pairs whose name holds password, secret, token, api key, access key, private key, credential or authorization, and bare Bearer tokens, become [credential redacted]. If redaction fails the report keeps its class and the message [unavailable]. |
| Where it lives | A report’s message and backtrace exist in memory: the ring behind ops errors, and development surfaces. Crema’s logs, tail events, metric labels, exports and caramel_metrics carry the class and fingerprint, never the message. The ring holds 500 fingerprints; the least recently seen is evicted. |
| Condition | Outcome |
|---|
| A job’s failed run | is written unredacted by Cold Brew to caramel_jobs.last_error: the class, the message and the first 12 backtrace lines; APP jobs show ID prints it |
| A process restart | clears the error and trace rings; forward Crema.on_error reports to an error tracker for history |
Not stored Messages and backtraces never reach logs, exports or the recorder.They stay in memory and in development surfaces. Only caramel_jobs.last_error, which Cold Brew writes, keeps a job’s message on disk.
Every statement in a trace starts with a tag
Every statement run inside a trace starts with a comment: /*action='App%3A%3ABooks%3A%3AShow'*/ for a request, /*job='…'*/ for a job and /*schedule='…'*/ for a schedule. The value is the action or job class or the schedule name, percent-encoded. Pools name themselves in application_name: caramel-web for serve, caramel-cold-brew for workers and caramel-listen for PubSub.
| Condition | Outcome |
|---|
| A statement run outside a trace | carries no tag |
| A statement’s binds in a development build | are kept for the inspector: at most 20, each cut at 200 bytes, NULL for nil |
| A span’s detail in the trace ring | is cut at 2,048 bytes |
A statement over slow_query | counts in the log line’s slow_queries |
The ops socket answers its owner only
An HTTP/1.1 server on a Unix socket, for the application’s owner.
| Condition | Outcome |
|---|
| A directory that is not a real directory you own with no group or other permissions | stays off: the application keeps serving, and the log says why |
| The socket file | is mode 0600 |
| A live socket | is never replaced; a dead one is |
| Each connection | has a 10-second read and write timeout |
| A tail with nothing to send for 15 seconds | sends : ping to keep the connection open |
A request whose Host is not ops, localhost, 127.0.0.1 or [::1] | answers 421 |
A write with an Origin header | answers 403 |
| A wrong method | answers 404 |
| A ninth concurrent tail | answers 429 |
POST /v1/debug-tokens without an application secret, which a work process lacks | answers 404 |
| Every answer | is JSON with a version and carries Cache-Control: no-store |
| Endpoint | Answers |
|---|
GET /v1/status | App, role, uptime, environment, in-flight work, request counts and p95, memory, database pools, workers, scheduler, sinks and dropped events. |
GET /v1/requests, /v1/fibers | In-flight requests and jobs; fibers grouped by name. |
GET /v1/metrics | Prometheus text. |
GET /v1/tail | Server-sent events of finished traces and errors. Query: errors=1, slow=MS, logs=1. At most 8 at once, 1,000 events buffered for each. A debug trace always passes the filters. With logs=1 the events include log lines exactly as written to stdout. |
GET /v1/errors, /v1/errors/FINGERPRINT | Errors since the process started, grouped, newest first. The second route is the only one that returns a message and backtrace. |
GET /v1/traces?limit=N&reason=error|slow|debug | The trace ring, 50 by default, at most 200. A trace is kept when it failed, was slow or carried a debug token. The ring holds 200 traces. |
GET /v1/traces/REF | One trace with its spans. REF is a prefix of at least 6 characters of a trace id or request id. |
POST /v1/debug-tokens | JSON {"minutes": N}, 1 to 120, default 15, in a body of at most 1 KiB. |
The read-only console is the same data as HTML at /, /requests, /tail, /jobs, /database, /errors, /traces and /insights. It can neither retry a job nor issue a token, and its content security policy allows only its own scripts and styles.
Binary commands exit 0, 1 or 2
The commands page lists each one’s syntax; these are the details. Run ops where the process runs; --socket=PATH works on every subcommand.
See every command →| Command | Does |
|---|
ops status | Seven lines: App, Requests, Jobs, Database, Memory, Fibers and Crema. --json prints the status document. |
ops requests, ops fibers | In-flight work; fibers by name. |
ops metrics | Prometheus text, as served. |
ops tail | Streams until you stop it. --slow takes a value here, and MS must be an integer. |
ops errors | Fingerprint, count, last seen, class, location and source. |
ops error FINGERPRINT | One error with its redacted message and backtrace. A miss says that restarts clear the rings. |
ops traces | One reason flag; errors win over slow, slow over debug. The reason flags take no value. |
ops trace REF | One trace as text, or Markdown with --md. |
ops debug-token | Prints the token, its expiry, a curl line with the header and a browser line that sets the cookie. |
ops console | Prints ssh -N -L 8765:SOCKET HOST and http://localhost:8765/. It does not connect. |
jobs, jobs stats | Each queue’s running, retrying, queued, scheduled and failed jobs, the finished in the last hour and the age of the oldest due job. |
jobs failed | Failures grouped by job class and error class, newest first; --limit defaults to 20. |
jobs show ID | Every column of one job, including last_error and context. It prints to your terminal only. |
jobs retry ID, jobs retry --class=NAME | Writes to the database: clears the failure and runs the job once more, an abandoned job included. It keeps attempts. |
jobs unknown | The queued job classes this binary does not compile, one per line, for a deploy to check before the binary takes over. |
db diagnose | Sections connections, long_running (over 1 second), blocking, cache_hit, seq_scans, unused_indexes, vacuum, table_sizes and outliers (the slowest statements of your role, from pg_stat_statements). |
db roles DATABASE MIGRATION_ROLE RUNTIME_ROLE | Prints a psql script that creates the two login roles, the database the migration role owns and the grants Latte gives a site: the runtime role reads and writes rows through default privileges. It sets no passwords, grants both roles to the role that runs it, and needs no database. Names match [a-z_][a-z0-9_]*, up to 63 characters. |
insights | The recorder’s top 20 rows by count under the headers KEY COUNT ERR P50MS P95MS MAXMS TOTAL_S. --since defaults to 1h, --kind to request. |
| Condition | Outcome |
|---|
| Exit status | 0 is success, 1 a failure or nothing found, 2 a usage error |
ops with no socket at the path | exits 1 with No ops socket at PATH. Is the application running with CARAMEL_OPS_SOCKET? |
ops errors | shows UTC times; ops traces and ops tail show local times |
ops traces --slow=500, a reason flag with a value | exits 2, a usage error |
ops tail --slow without a value | exits 2, a usage error |
jobs, db diagnose and insights | need DATABASE_URL and no running application |
jobs with CARAMEL_ENV=test | refuses |
insights without the recorder | exits 1 with The Crema recorder is off. Add require "caramel/crema/recorder" to config/application.cr. |
insights with no rows in the window | prints No aggregates in the last 1h. Is the recorder running? and exits 0 |
jobs unknown when any class is unknown | exits 1; with no caramel_jobs table yet, it prints nothing and exits 0 |
db diagnose with an unreadable section | says so and the rest continue; exits 0 if any section was readable |
db roles with a name that does not match [a-z_][a-z0-9_]* up to 63 characters | exits 2 |
Metrics count from the process start
| Condition | Outcome |
|---|
| A request route | is a route template; an unmatched route is (none) |
| More than 1,000 distinct request series | fold into (other) |
| Values | count from the process start and are not stored |
| Durations | are histograms in seconds with bounds from 0.005 to 10 and +Inf |
| Metric | Type and labels |
|---|
caramel_requests_total | counter: method, route, status. |
caramel_request_duration_seconds | histogram: method, route. |
caramel_jobs_total | counter: job, outcome. |
caramel_job_duration_seconds | histogram: job. |
caramel_job_queue_lag_seconds | histogram: queue. |
caramel_schedules_total | counter: schedule, outcome. |
caramel_errors_total | counter: error_class. |
caramel_inflight | gauge: kind is request, job or schedule. |
caramel_gc_heap_bytes, caramel_gc_free_bytes | gauges for the Crystal heap. |
caramel_gc_allocated_bytes_total | counter of bytes allocated. |
caramel_fibers | gauge: fibers alive. |
caramel_db_pool_connections | gauge: pool, state is open, idle or in_flight. |
caramel_db_pool_max | gauge: pool, the pool’s limit. |
caramel_crema_dropped_total | counter: sink, events a sink lost. |
caramel_build_info | gauge, always 1: app, caramel. |
caramel_process_start_time_seconds | gauge: when the process started. |
Debug tokens expire within two hours
A token is EXPIRY.SIGNATURE: the expiry as Unix seconds and an HMAC-SHA-256 of it, keyed from the application secret.
| Condition | Outcome |
|---|
| A token | is valid until its expiry and never longer than two hours, whatever it claims |
The X-Caramel-Debug header and the __Host-caramel_debug cookie both sent | the header wins |
An invalid X-Caramel-Debug header | is not replaced by the cookie |
| A valid token | traces the request in full, answers with X-Caramel-Trace, keeps it in the trace ring with the reason debug, always exports it when the exporter is on, and passes it to the jobs it enqueues |
| A token that leaks | stays valid until it expires, because the signature covers only the expiry |
The recorder stores aggregates, never a path or a message
| Condition | Outcome |
|---|
require "caramel/crema/recorder" | aggregates each finished trace in memory and writes to caramel_metrics |
| Upgrading | run frappe migrate, since caramel_metrics comes with Cold Brew’s framework tables |
| What is stored | route templates, job classes, schedule names, parameterized SQL and outbound hosts, never a path, a bind value, a message or a trace |
| More than 500 keys of one kind per flush | fold into (other) |
| A failed flush | loses its batch, writes one warning and counts toward caramel_crema_dropped_total |
| Name | Meaning |
|---|
caramel_metrics | bucket (the start of the minute), kind, key, count, errors, total_ms, max_ms and histogram, a 12-bucket latency count with bounds 5, 10, 25, 50, 100, 250, 500, 1,000, 2,500, 5,000 and 10,000 ms, and overflow. The primary key is (bucket, kind, key). Rows merge by adding counts and keeping the larger maximum. |
kind | request, job, schedule, sql or outbound. |
SQL keys | Quoted literals and bare numbers become ?, IN (…) lists become IN (?), whitespace collapses, and the key is cut to 500 bytes. |
Caramel::Crema::Recorder.flush_interval | Default 15 seconds. Each flush upserts into the current minute. |
Caramel::Crema::Recorder.retention | Default 7 days. Pruned hourly. |
The exporter sends nothing until an endpoint is set
require "caramel/crema/otlp" exports each sampled request, job and schedule run as OTLP over HTTP with JSON bodies, with a child span for every query and outbound call.
| Condition | Outcome |
|---|
| No endpoint set | nothing is sent |
| Protobuf | not supported |
An OTEL_* variable other than the seven listed | not read |
OTEL_EXPORTER_OTLP_PROTOCOL other than unset or http/json | logs a warning and leaves export off |
An unknown OTEL_TRACES_SAMPLER | warns and uses traceidratio |
| A parent-based sampler | follows the caller’s flag when there is one |
| A failed trace that sampling skipped | is exported as its root span |
| A debug trace | is always sampled |
| A failed batch | is dropped, counted, and logged at most once a minute |
| Variable | Meaning |
|---|
OTEL_EXPORTER_OTLP_ENDPOINT | The base URL; /v1/traces is appended. |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | Used as given, and wins over the base. |
OTEL_EXPORTER_OTLP_PROTOCOL | Unset or http/json. |
OTEL_EXPORTER_OTLP_HEADERS | Comma-separated name=value pairs, percent-decoded. |
OTEL_SERVICE_NAME | Default: the application’s title. |
OTEL_TRACES_SAMPLER | always_on, always_off, traceidratio (the default), parentbased_always_on, parentbased_always_off or parentbased_traceidratio. |
OTEL_TRACES_SAMPLER_ARG | The ratio, 0 to 1; default 1. |
| Part | Behaviour |
|---|
| Spans | A request is a server span, a job a consumer span and a schedule an internal span; queries and outbound calls are client spans. log and dump spans are never exported. Attributes are route templates, db.query.text with parameterized SQL, the HTTP method and status, the job’s queue, id and attempt, and caramel.request_id. A failed trace adds the exception’s class only. |
| Sampling | Decided once per trace. |
| Delivery | Up to 2,048 traces wait in memory and go out every 5 seconds or at 512 spans, with a 5-second connect and 10-second read timeout. |
Development tools need a development build
| Condition | Outcome |
|---|
A binary built without -D caramel_development, or run without CARAMEL_ENV=development | holds none of these tools; a check confirms a production binary holds none of it |
An unrecognised CARAMEL_EDITOR | falls back to zed |
Server-Timing on a streamed response | is not sent |
| Tool | Behaviour |
|---|
| Error page | An unhandled error answers 500 with the class, the redacted message, up to 80 frames with Open in editor links, four lines of source either side of your first frame (files under app, config, src or db up to 1 MiB), the request, the queries before the error and the causes. A client that asks for JSON first gets an error object with the class, message, location, request and trace ids and backtrace instead. |
dump(value) | Prints dump file:line: value to standard error and returns the value; inside a trace it adds a dump span of up to 8,192 bytes. In actions, views and jobs, and as Caramel.dump. In other builds it returns the value untouched. |
Server-Timing | db;dur=…;desc="N queries", view;dur=…, total;dur=…. |
| Toolbar | frappe dev adds it to full HTML responses that carry an X-Request-ID, not to htmx partials, the build-error page or the inspector. It shows the route, status, time and query count, flags for errors, slow requests and repeated queries, and the last ten requests. Each part links to #timeline, #queries or #error on the request’s page (the Timeline and Queries sections are on the page even when empty and say so; the Error section is there for a failed request). The minimise button shrinks it to a dot, and the dot restores it. ` opens and closes the list, j and k move in it, Esc closes it; shortcuts are ignored in fields and with a modifier key. Copy for an agent fetches /__caramel/dev/trace.md?id=ID (a hexadecimal id of 6 to 32 characters, development session only). Theme cycles auto, light and dark. It polls every 400 ms and reloads when the build changes. |
| Inspector | /__caramel/dev/inspector lists up to 500 traces, filtered by ?only=errors, slow or jobs. Beside it: /inspector/traces/ID (a prefix of at least 6 characters), /inspector/errors and /inspector/builds. Each query on a trace page has Copy SQL and Copy with values; the values come from literals built when the query ran (numbers and booleans bare, text, times and bytes quoted, nil as NULL, arrays as ARRAY[…]) and never leave development. Placeholders without a value stay as $n. The page follows the system’s light or dark appearance; the Theme button sets it for the toolbar and the inspector together (localStorage key caramel.dev.theme). Its endpoints answer GET only and check the host. |
CARAMEL_EDITOR | A preset or a template with {path}, {line} and {column}. Presets: zed (the default), vscode, cursor, sublime, textmate and idea. frappe dev also reads it from the project’s .env. |
Caramel/Dump | frappe lint reports any call to dump without a receiver or on Caramel, including your own method of that name. The rule is on in new applications’ .ameba.yml. |
Frappé commands replay kept events
All of these run inside an application and need no running frappe dev: they replay the events it kept. A value needs =: --limit=5, not --limit 5.
| Command | Does |
|---|
frappe traces [--errors] [--slow=MS] [--limit=N] [--agent|--human] | Recent requests, jobs and schedule runs, newest first. --slow=MS keeps traces of at least that many milliseconds; --errors keeps failed ones. --limit defaults to 20. --agent ends with OK traces N. |
frappe trace REF [--md] | One trace, or an error outside any trace. REF is last, last-error or a prefix of at least 6 characters of a trace id, a request id or the fingerprint of the error the trace ended with, or of an error outside a trace. --md adds sections for the error, backtrace, queries, logs, dumps and, when Latte holds spans from other services, Across services. |
frappe errors [--agent|--human] | Errors and repeated queries since the newest successful build, grouped. |
frappe logs [app|compiler|access] [--follow] | A site log through tail; access is the proxy’s. The Local environment page gives the line count and the folder. |
frappe db diagnose | Builds a development binary and runs its db diagnose against the development database. |
| Condition | Outcome |
|---|
frappe traces | always exits 0 |
frappe trace REF with no match | exits 1 |
frappe errors --agent with any group | prints one MRDP block each and exits 1; otherwise it prints OK errors 0 |
frappe logs with a missing log | exits 1 |
| Stored events | the newest 500 traces, 200 standalone errors and 50 builds, in events.jsonl in the site’s log folder; at 8 MiB it rolls into one .previous file |
| Diagnostic | Meaning |
|---|
ERR RUNTIME:STATUS | One per error group since the newest good build, :500 for requests. MSG gives the class, message and count; FIX points at frappe trace FINGERPRINT --md. |
ERR REPEATED_QUERY | One per route and statement that ran five or more times in one request, with the most runs shown. FIX suggests preloading the association. |
Latte adds statistics, an access log and a collector
| Part | Adds |
|---|
| PostgreSQL | Latte’s PostgreSQL loads pg_stat_statements (top-level statements, no utility commands) and auto_explain (plans of statements over 250 ms, without timings or parameters, in postgres.log). The caramel_stats schema exists in the development database only, and frappe db dump leaves it out. |
| Access log | Caddy writes one JSON line for each request to the site’s access.log, mode 0600, rolling at 1 MB with one previous file. It is kept out of proxy.log. |
| Collector | Latte listens on 127.0.0.1:4318 for POST /v1/traces with JSON only. It rejects bodies over 4 MiB, keeps the newest 2,000 traces in memory, 200 spans per trace and 32 attributes per span, and forgets them when it stops. If the port is busy, frappe services reports the collector as unavailable and everything else works. |
| Control API 2 | GET /v2/status adds the collector’s state; GET /v2/traces?limit=N (default 50, at most 200) and /v2/traces/ID read the collector; GET /v2/sites adds each site’s error count and last error, which holds a class, a location and a fingerprint, never a message. |
| Menu bar | Latte.app refreshes every 10 seconds. It shows each site’s error count beside its state and notifies on a new error fingerprint or a build failure. A notification needs permission and opens the inspector. |
Prove it
Replay what the session kept
Serve the application, open a page in the browser, then read the events frappe dev kept. Caramel’s own scripts/check crema proves the production side: the JSON request line without a path, the X-Request-ID echo, the traceparent continuation, an error line without the message, the 0600 ops socket, the 421 on a bad Host, the metrics series and the debug token.
Terminal · inside the applicationfrappe dev --no-open
frappe traces --agent
frappe trace last --md
frappe errors --agent
Passing: frappe traces --agent prints one line per trace, starting request GET /, then OK traces N, and exits 0. frappe trace last --md prints the newest trace as Markdown. frappe errors --agent prints OK errors 0 and exits 0 when nothing failed since the newest successful build; a runtime error prints an ERR RUNTIME:500 block and exits 1, and frappe trace last-error --md then has a ## Backtrace section.