Skip to content
carameldocs · 0.16.0
Browse documentation
Reference / SugarORM

Give your data shape.

Declare each table as a typed schema, query it with checked names, guard writes with changesets, and derive migrations from the declaration.

ContractValue
Release0.15.0
NeedsA Caramel application and its development database
CoversSchemas, codecs, queries, preloads, changesets, migrations, branches
Proved byfrappe check and frappe migrate

Learning it for the first time? The recipe Create a resource generates a model, a changeset and a migration. This page lists every rule and limit.

Limits at a glance

LimitValue
primary: true fields per schemaexactly 1
version: true fields and timestamps per schemaat most 1 each
Check constraint name63 bytes
numeric(P,S) precision1,000
Extra queries per preload1
Migration namea lowercase letter, then up to 62 lowercase letters, digits or underscores
Branch namea lowercase letter, then up to 30 lowercase letters, digits or underscores
Statement, per connection30 s
Lock wait, per connection5 s

A model declares its table once

A model is a struct that extends SugarORM::Schema and declares its table once. The schema block defines the record type, its queries and its migrations. frappe make resource writes one for you.

Create a resource →
app/models/book.cr · assumes an App::Author schema with has_many books : Book
module App
  struct Book < SugarORM::Schema
    schema "books" do
      field id : Int64, primary: true
      field title : String
      field isbn : String?, renamed_from: :code
      field copies : Int32 = 1
      field archived : Bool = false
      timestamps
      belongs_to author : Author
      index :isbn, unique: true
      check copies: 0..
      drop_column :shelf
    end

    scope available { where(archived: false) }
    scope with_copies(count : Int32) { where("copies >= ?", count) }
  end
end

Fields take String, Int32, Int64, Bool, Float64 or Time, or any type through codec:; add ? for a nullable column.

ConditionOutcome
An unsupported field typefails compilation
A default that is not a literal (it becomes the column’s SQL default), or any default on a Time fieldfails compilation
A schema with no primary: true fieldfails compilation
A primary: true field that is nilable, not Int64, or has a defaultfails compilation
The one primary: true fieldis a non-nilable Int64 without a default, stored as an identity column
timestampsadds created_at and updated_at
timestamps declared twice, or a second version: true fieldfails compilation
An update that changes a fieldalso sets updated_at
A has_many or has_one whose foreign key the target lacksfails compilation
belongs_to with an on_delete: other than :cascade, :restrict or :set_null, or with any other optionfails compilation
on_delete: :set_null on a reference that is not nilable, or on a key from a tenanted schema to a tenanted schemafails compilation
A duplicate columnfails compilation
Two schemas that name one tablefails compilation
A reserved name such as query or updatefails compilation
A range check that is not an inclusive range of integer literals, an empty range such as 10..1, a check declared twice, or a check name over 63 bytesfails compilation
A recordis immutable: book.with(title: "Dune Messiah") returns a changed copy and never persists it
An insert or update that breaks a unique index or a CHECK the changeset did not mapraises SugarORM::UniqueViolation or SugarORM::CheckViolation
belongs_to author : AuthorAdds the author_id Int64 column, its foreign key and its index. Author? makes it nullable.
has_many books : BookReads the Book rows whose author_id points at this record.
has_one charter : CharterReads the first matching row by primary key, or nil.
index :isbn, unique: trueCreates index_books_on_isbn. Several columns join with _and_ in the name.
check copies: 0..Adds the CHECK constraint check_books_copies. Range checks take Int32 and Int64 fields and an inclusive range of integer literals: 0.., ..10 or 1..10.
check :dates, "starts_at < ends_at"A named SQL expression, compared by name: rename it to change it.
renamed_from: :codeRenames the old column instead of dropping it.
drop_column :shelfRecords the decision to drop a column and its data.

has_many and has_one expect a foreign key named after the owner, such as author_id; pass foreign_key: :column when it differs.

Codecs map other types to text

A field of any other type names a codec: field name : Type, codec: Codec. The codec maps the type to text. Rows read a codec column as text and writes bind the encoded text, so no value passes through a float.

Codec methodReturns
Codec.sql_typeThe column type, one of numeric, numeric(P,S), jsonb or text
Codec.encode(value)The text to store
Codec.decode(text)Builds the value back

SugarORM::JSONB(T) is a codec for any JSON::Serializable type, JSON::Any, or an Array or Hash of those. It stores the value in a jsonb column.

app/models/invoice.cr
module App
  struct Snapshot
    include JSON::Serializable
    getter lines : Array(Hash(String, String))

    def initialize(@lines)
    end
  end

  struct Invoice < SugarORM::Schema
    schema "invoices" do
      field id : Int64, primary: true
      field snapshot : Snapshot, codec: SugarORM::JSONB(Snapshot)
      field notes : Snapshot?, codec: SugarORM::JSONB(Snapshot)
    end
  end
end

A numeric column keeps exact decimals. This codec holds the digits as text, so the application picks its own decimal type and checks the text in decode.

app/models/rate.cr
module App
  record Rate, text : String

  module RateCodec
    def self.sql_type : String
      "numeric(20,8)"
    end

    def self.encode(rate : Rate) : String
      rate.text
    end

    def self.decode(text : String) : Rate
      raise ArgumentError.new("not a decimal: #{text}") unless text.matches?(/\A-?\d+(\.\d+)?\z/)
      Rate.new(text)
    end
  end

  struct Price < SugarORM::Schema
    schema "prices" do
      field id : Int64, primary: true
      field rate : Rate, codec: RateCodec
      field fee : Rate?, codec: RateCodec
    end
  end
end
ConditionOutcome
A codec field with a defaultfails compilation
A sql_type other than numeric, numeric(P,S) with S ≤ P ≤ 1000, jsonb or textraises ArgumentError
A changeset param on a codec fielduses the field’s type, such as Rate
The number validators on a codec fielddo not apply, because the stored value is text
validate_inclusion on a codec fieldtakes values of the field’s type
Raw SQL reading a codec columnselects it as text, as in SELECT rate::text AS rate FROM prices
frappe db diff on a codec fieldderives the column from sql_type
Adding a required codec field to an existing tablehalts the diff, because it has no default: add it nilable, backfill, then make it required

In where a codec field takes a value or nil, never an Array or a Range. Managed builds link GMP, so a codec may decode to BigDecimal after require "big".

Queries check names and types

Book.query starts an immutable query, and each clause returns a new one. Keyword where accepts only the schema’s columns, with their types.

where valueMatches
A valueCompares with =
An ArrayMatches any element
A RangeBounds a number or time column
The literal nilFinds NULL in a nullable column, as where_null(:isbn) does
A string fragmentBinds one value to each ?, in parentheses
where_condition(SugarORM::Condition.new(sql, values))Adds a fragment built elsewhere with its values, in parentheses
ConditionOutcome
Keyword where with a name that is not a column, or a value of the wrong typefails compilation
where with a value that may be nil, such as an optional contract field (it would match the NULL rows)fails compilation; narrow it first (if isbn = contract.isbn) or use where_null
where("copies >= ?", …) with a number of values other than its ? countraises ArgumentError
order_by with a name that is not a fieldfails compilation
order_by with a direction other than :asc or :descraises ArgumentError: order_by direction must be :asc or :desc, not :sideways
A negative limit or offsetraises ArgumentError
find and first when nothing matchesreturn nil
find! and first! when nothing matchesraise SugarORM::NotFound
find(db, id), first!(db) and create(db, …)run on the connection you pass, as the generated request spec does with Corretto’s
Inside an action · author is a loaded App::Author
books = App::Book.query
  .available
  .with_copies(2)
  .where(author_id: author.id, copies: 1..10)
  .order_by(:title, :desc)
  .limit(20)
books.to_a
App::Book.query.find(id)
App::Book.query.where(isbn: nil).first!
CallResult
order_by(:title, :desc)Orders by a field, :asc or :desc
offset, first, count, exists?, each, delete_allAlso available on a query
scope name { … }Becomes a chainable query method
to_sqlShows the statement

lock adds FOR UPDATE.

ConditionOutcome
lockadds FOR UPDATE: the rows the query returns stay locked until the transaction ends, and a competing transaction waits for them
A competing transaction that has waited 5 s (lock_timeout), or any statement that ran 30 s (statement_timeout)fails with PostgreSQL’s error; Crema reference lists CARAMEL_LOCK_TIMEOUT and CARAMEL_STATEMENT_TIMEOUT
A locking query outside a transactionraises SugarORM::Error, because the lock would end with the statement
count, exists? or delete_all on a locking queryraises ArgumentError
A tenanted model with lockkeeps its tenant scope
Preloaded associations under lockare not locked

Lock several rows in one order, as in order_by(:id).lock, so two transactions cannot wait on each other. A lock does not replace a permission check.

Inside an action
SugarORM::Repo.transaction do
  book = App::Book.query.lock.find!(id)
  book.update!(copies: book.copies - 1)
end

Associations load only when preloaded

An association stays unloaded until its query preloads it.

Inside an action · App::Author declares has_many books : Book
authors = App::Author.query.preload(:books).order_by(:name).to_a
authors.each { |loaded| puts "#{loaded.name}: #{loaded.books.size}" }
ConditionOutcome
A preloadadds one query that loads the association for every record in the result, so a list never runs one query per row
A preloaded resultis a wrapper around each record that answers its fields and the association
An association that was not preloadedfails compilation, not a hidden query: frappe check reports N_PLUS_ONE (Association 'books' of App::Author was not preloaded.) and its remediation names .preload(:books)

SugarORM.sql reads rows as named tuples

For CTEs, window functions and aggregates, SugarORM.sql runs a query with $1 binds and reads each row as a named tuple.

Inside an action
totals = SugarORM.sql(<<-SQL, as: {author_id: Int64, total: Int64})
  SELECT author_id, count(*) AS total FROM books GROUP BY author_id
  SQL
totals.each { |row| puts "#{row[:author_id]}: #{row[:total]}" }
SugarORM.sql_exec("UPDATE books SET archived = true WHERE copies < $1", 1)
ConditionOutcome
Result columns that differ from the as: keys, in orderraises SugarORM::ShapeError
SugarORM.sql_execreturns the number of rows affected
SugarORM.sql and SugarORM.sql_execuse the current connection, including an open transaction
Raw SQL on a tenanted modelis not tenant-scoped; add the tenant predicate yourself, or use App::Book.query.lock to lock a scoped row

Every write goes through a changeset

A changeset lists the fields a write may set with param, and its rules in validate(cs).

app/changesets/book.cr
module App
  class Book::Changeset < SugarORM::Changeset(App::Book)
    param title : String
    param isbn : String?
    param copies : Int32
    param author_id : Int64

    def validate(cs)
      cs.validate_presence(:title)
      cs.validate_length(:title, max: 200)
      cs.validate_greater_than(:copies, 0)
      cs.validate_format(:isbn, /\A[0-9-]+\z/, message: "needs only digits and dashes")
      cs.unique_constraint(:isbn)
    end
  end

  alias Book::CreateChangeset = Book::Changeset
  alias Book::UpdateChangeset = Book::Changeset
end

Before your rules run, SugarORM checks NOT NULL columns.

ConditionOutcome
A param that names no schema field, or has an incompatible typefails compilation
The primary key or a timestamp as a paramfails compilation
An insert that omits a NOT NULL column without a defaultadds the error is required
A write that sets a NOT NULL column to niladds the error is required
Building the changesetruns validate(cs) before any SQL
Every validator except validate_lengthtakes message: to replace its default
The number, length, format, inclusion and URL validatorsskip values that did not change and nil values
A number validator on a column that is not numeric, or a string validator on one that is not a Stringraises ArgumentError
check_constraint(:name) on an expression check without on: :fieldraises ArgumentError
cs.add_error(:title, "is checked out")adds your own error

These are the validators and their default messages:

ValidatorDefault message
validate_required(:title, :isbn)is required. Checks the current value: nil or a blank string fails.
validate_presence(:title)can't be blank. Checks a changed value only.
validate_length(:title, min: 2, max: 200)should be at least 2 character(s), or should be at most 200 character(s).
validate_greater_than(:copies, 0)must be greater than 0
validate_less_than(:copies, 100)must be less than 100
validate_format(:isbn, /\A[0-9-]+\z/)has invalid format
validate_inclusion(:copies, in: [1, 2, 3])is invalid
validate_url(:website)must be an absolute http or https URL. Caramel adds this validator.
unique_constraint(:isbn)has already been taken. Maps a unique violation of the index that starts with isbn.
check_constraint(:copies)must be at least 0, or must be at most N. Maps a violation of the field’s range check; an expression check takes on: :field and reads is invalid.

errors maps each field name to its messages, and "_base" holds errors about the whole record.

Translated applications replace these messages through the catalogs’ caramel.errors section.

Translate framework messages →

Saves return a changeset

Each schema generates create and update. Their keywords are the changeset’s params.

Inside an action · author is a loaded App::Author
changeset = App::Book.create(title: "Dune", author_id: author.id)
if changeset.saved?
  book = changeset.record
else
  errors = changeset.errors
end

book = App::Book.create!(title: "Emma", author_id: author.id)
book = book.update!(copies: 3)
book.delete

SugarORM::Repo.transaction do
  shelf = App::Book.create!(title: "Persuasion", author_id: author.id)
  shelf.update!(copies: 2)
end
ConditionOutcome
create and updateuse CreateChangeset and UpdateChangeset when the model defines them, as generated resources do; otherwise a default changeset that permits every field except the primary key and timestamps, and maps each unique index and range check
An unknown or mistyped keywordfails compilation at the call
create and updatereturn the changeset: saved? says whether the write happened, valid? whether validation passed, and record holds the stored row
create! and update!return the stored record or raise SugarORM::Invalid with the error messages
Updating a row that no longer existsadds Record no longer exists under _base
delete on a row already gonereturns false
SugarORM::Repo.transactioncommits when its block returns and rolls back when it raises
A nested transactionbecomes a savepoint; SugarORM::Repo.rollback undoes the innermost block
A unique violation mapped by unique_constraintleaves the transaction usable

A changeset can say what to do when a row with the same key exists.

ConditionOutcome
upsert on: :tea_id, update: [:sold]makes the insert INSERT … ON CONFLICT on that unique index
An index the schema does not declarefails compilation
A key or update: field that is not a param of the changesetfails compilation
A tenanted schemaincludes the tenant in the index
Defaults, validations and the tenant stampstill apply
The update: fields the changeset writesare set from the new row, along with updated_at and the version
No update: fieldsleave the existing row as it is
recordis the stored row either way
Concurrent inserts of one keyleave one row
app/changesets/sale.cr · App::Sale declares index :tea_id, unique: true
module App
  class Sale::Record < SugarORM::Changeset(App::Sale)
    param tea_id : Int64
    param sold : Int32
    upsert on: :tea_id, update: [:sold]
  end
end

A version field stops two people from overwriting each other.

ConditionOutcome
field lock_version : Int32, version: true that is nilable, not Int32 or Int64, or has a defaultfails compilation
The version columnstarts at 0; frappe db diff adds it as integer NOT NULL DEFAULT 0
Every changeset updatechecks the version it loaded and adds one
A row that changed since you loaded itwrites nothing and adds Record changed since you loaded it under _base; stale? is true
A changeset with param lock_version : Int32 given lock_version:checks that version instead, so a form carries the version it was rendered from
An insert that names a versionraises ArgumentError
upsert … update: that names the version fieldfails compilation
A deletedoes not check the version
The update action below, given a stale formanswers 409 with the errors, by its own render_errors call
The update action below, given an If-Match header that names another versionanswers 412, by its own if_match? check
The show action belowsets self.etag, so its JSON answer carries ETag; Actions and responses lists the header rules
app/models/book.cr · schema excerpt
schema "books" do
  field id : Int64, primary: true
  field title : String
  field lock_version : Int32, version: true
  timestamps
end
app/views/books/form.cr · beside the CSRF field
input type: "hidden", name: "_csrf", value: @csrf_token
input type: "hidden", name: "lock_version", value: @values["lock_version"]? || ""

The changeset takes the version as a param, and the update action declares field lock_version : Int32 in its contract. The 409 and 412 below are this action’s code, not a framework default.

app/changesets/book.cr · the version param
class Book::Changeset < SugarORM::Changeset(App::Book)
  param title : String
  param lock_version : Int32
end
app/actions/books/update.cr
module App::Books
  struct Update < App::ApplicationAction
    contract do
      field id : Int64
      field title : String
      field lock_version : Int32
    end

    def handle(contract : Contract)
      book = App::Book.query.find(contract.id) || return not_found
      unless if_match?(book.lock_version)
        return render_errors({"_base" => [SugarORM::Wording.record_stale]}, 412)
      end
      changes = book.update(title: contract.title, lock_version: contract.lock_version)
      return render_errors(changes.errors, changes.stale? ? 409 : 422) unless changes.saved?
      changes.record
    end
  end
end
app/actions/books/show.cr · inside handle
book = App::Book.query.find(contract.id) || return not_found
self.etag = book.lock_version
book

Migrations derive from the schema

Change the schema first, then derive the SQL.

Terminal · inside the application
frappe db diff --name add_copies

Expect: Prints Wrote PATH for each migration file it derives under db/migrations/, and a Note: line for each CHECK it leaves in place. If the declared schema already matches, it prints The development database already matches the declared schema; no migration was written. A --name that is not a lowercase letter followed by up to 62 lowercase letters, digits or underscores is refused.

Terminal · inside the application
frappe migrate

Expect: Lints every pending migration, then prints Applied N migrations. and reports schema drift without changing anything; The database matches the declared schema. when there is none.

Review each file before you apply it. Adding copies with a default and a unique isbn index to an existing books table writes two files; the first is:

db/migrations/20261002120000_add_copies.cr
# Derived from the declared schema. Once applied, a migration is immutable: change the schema and run frappe db diff again.
App::MIGRATIONS << SugarORM::Migration.new(20261002120000_i64, "add_copies", [
  <<-SQL,
    ALTER TABLE "books" ADD COLUMN "copies" integer NOT NULL DEFAULT 1
    SQL
])
ConditionOutcome
frappe db diffclones the development database into a scratch branch, applies pending migrations there, compares with the declared schema, writes the difference to db/migrations/, migrates the branch with the new files and compares again, removing them if anything still differs; the scratch branch is always dropped
An index on an existing tablegoes in a second file, …_add_copies_concurrently.cr, as CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS; it builds without blocking writes and runs outside a transaction
A check on an existing tablesplits in two: the transactional file adds it NOT VALID, which refuses new rows that break it, and the _concurrently file validates it
Existing rows that break the checkstop frappe migrate, which names the constraint; it stays NOT VALID until you fix or delete those rows and migrate again
A check_… constraint that no schema declaresis dropped
Every other CHECKstays in place with a note
frappe migratelints every pending migration before it runs any statement, applies them, then reports schema drift without changing anything
An applied migration that changedstops the next run

frappe make resource renders its create_ migration the same way a diff would. The linter checks these rules:

Lint ruleWhat it refuses
concurrent-indexCREATE INDEX on an existing table blocks its writes. Use CONCURRENTLY in a migration of its own.
not-null-defaultADD COLUMN … NOT NULL without a DEFAULT fails on a populated table.
destructive-columnDROP or RENAME COLUMN needs drop_column or renamed_from. Hand-written SQL needs a -- caramel:allow-drop or allow-rename table.column line.
mixed-concurrencyCONCURRENTLY statements cannot share a migration with statements that need a transaction.

Tables created in the same migration are exempt from the first two rules.

ConditionOutcome
A NOT NULL column without a default on an existing tablehalts the diff and prints its remediation
A column that no field declareshalts the diff
A type changehalts the diff
A new NOT NULL constrainthalts the diff
A primary key changeis never derived
frappe db diff --dev-overridederives those changes anyway
frappe migrate --dev-overrideturns lint violations into warnings
frappe migrate --dev-override outside CARAMEL_ENV=developmentrefuses (test and production always refuse)

A branch is a full copy of the database

A database branch is a full copy of the development database. Use one to try a migration or a data change without touching your development records.

Terminal · inside the application
frappe db branch create try_isbn

Expect: Prints the branch’s connection URL.

Terminal · inside the application
frappe dev --branch try_isbn

Expect: Serves the application against the branch; a branch that does not exist is refused.

Terminal · inside the application
frappe db branch list
frappe db branch delete try_isbn

Expect: list prints one branch name per line, or APP has no database branches. on stderr. delete prints Deleted database branch try_isbn.; a second run fails, because the branch is gone.

ConditionOutcome
A branch name that is not a lowercase letter followed by up to 30 lowercase letters, digits or underscoresrefuses: branch name NAME must be a lowercase letter followed by up to 30 lowercase letters, digits or underscores
While the copy is madeLatte closes other connections to the development database
frappe dev --branch with a branch that does not existrefuses: APP has no database branch NAME; create it with frappe db branch create NAME

Branches are for experiments. To keep a backup, use frappe db dump and frappe db restore.

Back up the development database →

Prove it

Check the schema against the database

Run these in the development branch, with the database frappe new created.

Terminal · inside the application
frappe check
frappe migrate

Passing: frappe check exits 0, with no compile error and no N_PLUS_ONE, and frappe migrate ends with The database matches the declared schema.