# API reference Source: https://django-search-query.git-pull.com/api/ (api)= # API reference The public surface is small: one call for the common case, {func}`~django_search_query.parse`/{func}`~django_search_query.build_q` when you want the AST in between, a field registry, and one exception. The optional admin integration adds a single mixin. See {doc}`query` for the syntax these compile, and the {doc}`tutorial` to run them end to end. ## Core `django-search-query`. {func}`~django_search_query.search_query_to_q` is the one call most sites need; it chains {func}`~django_search_query.parse` (string to AST) and {func}`~django_search_query.build_q` (AST to {class}`~django.db.models.Q`). Call the two directly only when you want to inspect or transform the AST before compiling. ```{eval-rst} .. autofunction:: django_search_query.search_query_to_q .. autofunction:: django_search_query.parse .. autofunction:: django_search_query.build_q .. autoclass:: django_search_query.registry.FieldRegistry :members: .. autoclass:: django_search_query.registry.FieldSpec .. autoclass:: django_search_query.errors.QueryParseError ``` ### Highlighting A second, presentation-only lexer colorizes a query for a live search box. Unlike the parser it never raises -- it tokenizes even a half-typed query so every keystroke can be highlighted -- which is why it is a separate function rather than a mode of {func}`~django_search_query.parse`. See {doc}`the pipeline ` for why the two lexers stay independent, and {doc}`colored-input ` for the JavaScript port that colors the admin search box. ```{eval-rst} .. autofunction:: django_search_query.highlight.highlight_query_spans .. autofunction:: django_search_query.highlight.apply_registry_errors .. autoclass:: django_search_query.highlight.Span .. autodata:: django_search_query.highlight.HIGHLIGHT_ROLES ``` ## Admin `django-admin-search-query` (optional). {class}`~django_admin_search_query.mixin.SearchQueryAdminMixin` is the one class this package ships: mount it ahead of {class}`~django.contrib.admin.ModelAdmin` (see the {doc}`admin how-to `) and it wires the query language into {meth}`~django.contrib.admin.ModelAdmin.get_search_results`, adds two JSON endpoints behind the colored input, and injects the URLs those endpoints need into the changelist template. Everything below is optional configuration on top of that one mixin -- most sites only ever set the three class attributes from the how-to guide. ```{eval-rst} .. autoclass:: django_admin_search_query.mixin.SearchQueryAdminMixin :members: ``` ### Hook contracts Each class attribute has a matching `get_*` method, so dynamic behavior -- a registry that depends on `request.user`, say -- only needs to override the method, not chase every place the attribute is read. Called with no overrides, the hooks read the class attributes straight through, and {meth}`~django_admin_search_query.mixin.SearchQueryAdminMixin.get_search_query_default_fields` falls back to {attr}`~django.contrib.admin.ModelAdmin.search_fields` (stripped of Django's lookup sigils -- `^`, `=`, `@`, `$`) whenever {attr}`~django_admin_search_query.mixin.SearchQueryAdminMixin.search_query_default_fields` is left empty: ```{doctest} >>> from django_admin_search_query.mixin import SearchQueryAdminMixin >>> class ExampleAdmin(SearchQueryAdminMixin): ... search_fields = ("^title", "=slug") >>> ExampleAdmin().get_search_query_default_fields() ('title', 'slug') ``` {meth}`~django_admin_search_query.mixin.SearchQueryAdminMixin.get_search_results` is the method Django's changelist actually calls. It strips the search term and returns `super().get_search_results(...)` unchanged -- Django's own `search_fields` behavior -- whenever the term is blank, no registry is configured, or {exc}`~django_search_query.errors.QueryParseError` is raised while compiling it; otherwise it returns a filtered {class}`~django.db.models.query.QuerySet`: `queryset.filter(q), False`. That second element is Django's `may_have_duplicates` flag; it is `False` for the local-column field maps this integration targets, where filtering by {class}`~django.db.models.Q` adds no joins. A field map that instead points at a to-many relation can introduce duplicate rows -- collapse them with `.distinct()` in that case. ### JSON endpoints {meth}`~django_admin_search_query.mixin.SearchQueryAdminMixin.get_urls` prepends two model-scoped endpoints to the admin's URLs, each wrapped in `admin_site.admin_view` so it inherits the admin's staff-login gate, and each additionally checking {meth}`~django.contrib.admin.ModelAdmin.has_view_permission` before responding: - `search-tokens/` (named `__search_tokens`) -- the registry's fields, kinds, operators, enum values, and field aliases, plus the default fields, as {class}`~django.http.JsonResponse`. - `search-highlight/` (named `__search_highlight`) -- the highlight spans for a `?q=` query string, as {class}`~django.http.JsonResponse`. Both endpoints are always registered once the mixin is mounted. The changelist template only *advertises* them -- as `data-*` attributes the colored input reads -- when {attr}`~django_admin_search_query.mixin.SearchQueryAdminMixin.search_query_registry` is configured; without a registry, the search box renders as stock admin with no extra markup. See {doc}`colored-input ` for the exact JSON shape of each endpoint and how the JavaScript input consumes it. --- # Changelog Source: https://django-search-query.git-pull.com/history/ (history)= ```{include} ../CHANGES ``` --- # django-search-query Source: https://django-search-query.git-pull.com/ (index)= # django-search-query A reusable, [Lucene]-inspired search query language for [Django] -- and an optional [Django admin] integration built on top of it. ::::{grid} 1 1 2 3 :gutter: 2 2 3 3 :::{grid-item-card} {octicon}`rocket` Tutorial :link: tutorial :link-type: doc Build a search registry and run a query against a Django queryset. ::: :::{grid-item-card} {octicon}`search` Query language :link: query :link-type: doc The search syntax, and the magic: a query string becomes a Q object and filters a queryset. ::: :::{grid-item-card} {octicon}`download` Install :link: install :link-type: doc Add the core package or the admin integration to a Django project. ::: :::{grid-item-card} {octicon}`book` API reference :link: api :link-type: doc search_query_to_q, parse, build_q, the field registry, and the admin mixin. ::: :::{grid-item-card} {octicon}`package` Packages :link: packages/index :link-type: doc The core query language and the optional admin integration. ::: :::{grid-item-card} {octicon}`tools` Project :link: project/index :link-type: doc Development setup, contributing, and the release process. ::: :::{grid-item-card} {octicon}`log` Changelog :link: history :link-type: doc Per-package release notes. ::: :::: ## Install ```console $ pip install django-search-query ``` ```console $ uv add django-search-query ``` The admin integration installs separately; see {doc}`install`. ## What this is {doc}`django-search-query ` accepts a structured search string and translates it into Django-compatible query behavior, so an application gets a consistent {doc}`search syntax ` without committing to a particular user interface, admin integration, or search backend. Under the hood that string becomes a {class}`~django.db.models.Q` you hand to any {class}`~django.db.models.query.QuerySet` -- see {doc}`query` for the syntax and the worked query-to-queryset examples. Its scope is intentionally loose: the syntax is Lucene-inspired without claiming full Lucene compatibility. {doc}`django-admin-search-query ` is an optional add-on that brings that same structured search to Django admin changelist pages while keeping the core language usable on its own -- the relationship stays loose so the core package never couples to admin behavior or presentation concerns. It also optionally ships a self-contained, vanilla-JavaScript search input with syntax highlighting, contextual suggestions, and semantic autocomplete that degrades to a plain text field when JavaScript is unavailable. [Lucene]: https://lucene.apache.org/ [Django]: https://docs.djangoproject.com/ [Django admin]: https://docs.djangoproject.com/en/stable/ref/contrib/admin/ ```{toctree} :hidden: tutorial query install api packages/index project/index history GitHub ``` --- # Install Source: https://django-search-query.git-pull.com/install/ (install)= # Install `django-search-query` is the core query language, independent of any particular UI, admin integration, or search backend; add it to any Django project's environment. `django-admin-search-query` is the optional {doc}`admin integration ` built on top of it -- reach for it only when you want structured search on an admin changelist page. ```{package-install} ``` Register whichever package you installed in `INSTALLED_APPS`: ```python INSTALLED_APPS = [ "django.contrib.contenttypes", "django.contrib.auth", "django.contrib.admin", "django.contrib.messages", "django.contrib.sessions", "django_search_query", "django_admin_search_query", # optional: admin integration ... ] ``` `django_admin_search_query` needs `django.contrib.admin` (and its own dependencies, `contenttypes`, `auth`, `messages`, `sessions`) already installed -- skip it if you only use the core query language. (developmental-releases)= ## Developmental releases New versions are published to PyPI as alpha, beta, or release candidates. In their versions you will see notation like `a1`, `b1`, and `rc1`, respectively. `1.0.0b4` would mean the 4th beta release of `1.0.0` before general availability. - [pip]\: ```console $ pip install --upgrade --pre django-search-query ``` - [pipx]\: ```console $ pipx install --suffix=@next 'django-search-query' --pip-args '\--pre' --force ``` - [uv]\: ```console $ uv add django-search-query --prerelease allow ``` - [uvx]\: ```console $ uvx --from 'django-search-query' --prerelease allow django-search-query ``` See {doc}`tutorial` for what to do once the package is installed. [pip]: https://pip.pypa.io/en/stable/ [pipx]: https://pypa.github.io/pipx/docs/ [uv]: https://docs.astral.sh/uv/ [uvx]: https://docs.astral.sh/uv/guides/tools/ --- # Colored search input Source: https://django-search-query.git-pull.com/packages/django-admin-search-query/colored-input/ (colored-input)= # Colored search input The colored input is a progressive enhancement of the Django admin changelist search box. Highlighting is **client-side**: `search-lexer.js` is a faithful JavaScript port of the Python highlighter, so the box recolors synchronously on every keystroke with no per-edit network round-trip. A parity test keeps the two copies of the grammar from drifting. ## How it works `SearchQueryAdminMixin` renders the search box through a per-admin `change_list.html` that carries a `data-*` config element, and pulls in `search-lexer.js`, `search-input.js`, and `search-input.css` through its `class Media` (the lexer loads first; the widget reads it as `window.DSQLexer`). On load, the script enhances the stock ``: it copies the themed box metrics off `#searchbar` (font, line-height, padding, border-width, box-sizing) onto a new native `` and an `aria-hidden` `
` behind it. The editor's text is transparent (the caret stays visible); the mirror paints the role-classed spans. Because an ``'s auto height equals a one-line `
`'s once those metrics match, height and vertical alignment fall out for free -- no row math, no descender clipping. ```text keystroke -> DSQLexer.highlightQuerySpans + applyRegistryErrors -> render spans ``` Coloring never leaves the browser, so there is no stagger and no intermediate un-colored frame: the mirror is rebuilt on the same tick as the keystroke. ## Python <-> JavaScript parity `search-lexer.js` mirrors `_TOKEN_RE`, `highlight_query_spans`, and `apply_registry_errors` alternative-for-alternative. Two Python/JS regex differences are handled explicitly: - **Whitespace.** Python's Unicode `\s` counts U+001C-U+001F and U+0085 as whitespace but not U+FEFF; JS `\s` is the reverse. The JS class is spelled out (`\t\n\v\f\r\x1c-\x1f\x85\p{Zs}\p{Zl}\p{Zp}`) to match Python exactly. - **Word char.** The keyword lookahead uses Python's Unicode `\w`; JS `\w` is ASCII-only. The port uses `[\p{L}\p{N}_]` so `ORé` and `NOTx` stay values in both engines. Offsets are re-accumulated as code points (not UTF-16 units) so `start` matches Python for emoji and astral input. `tests/test_lexer_parity.py` runs one adversarial corpus through both engines and asserts identical `(start, role, text)` spans. It executes the JS lexer with `node` when available; otherwise it marks the run `SPIKE` and falls back to a captured golden fixture. ## Endpoints `get_urls()` prepends two model-scoped, staff-gated JSON endpoints to the admin's URLs (prepended so the admin's trailing `` catch-all does not swallow them). ### `search-tokens/` Named `__search_tokens`. Returns the registry schema the client needs -- one entry per field plus the default search fields -- so the browser can both flag `error` roles and drive autocomplete without a second schema: ```json { "fields": [ { "name": "status", "kind": "enum", "operators": [], "enum_values": ["open", "draft", "closed"], "aliases": [] } ], "default_fields": ["title", "body"] } ``` The script fetches this once and caches it. ### `search-highlight/` Named `__search_highlight`. Given `?q=`, it returns the same spans the client computes. The enhanced input no longer calls it per keystroke; it is retained as a **no-JS fallback reference** and as the canonical Python output the parity test pins the JS port against. ```json { "query": "status:bogus", "spans": [ {"start": 0, "role": "field", "text": "status"}, {"start": 6, "role": "punct", "text": ":"}, {"start": 7, "role": "error", "text": "bogus"} ] } ``` ## Roles Each span carries one role; the CSS maps roles to colors (light and dark): `whitespace`, `field`, `punct`, `keyword`, `negation`, `operator`, `wildcard`, `phrase`, `value`, and `error`. Roles are color-only -- no `font-weight` change -- so glyph advance widths stay identical between the editor and its mirror. ## Autocomplete Typing offers a keyboard-navigable listbox driven by the token schema: field names on a bare prefix, and enum values after `field:` for enum fields. The combobox is hand-rolled to the @github/combobox-nav contract -- DOM focus never leaves the input, navigation is `aria-activedescendant`, and the active option alone carries `aria-selected`. `ArrowUp`/`ArrowDown`/`Home`/`End` move the selection, `Enter`/`Tab` commit it, and `Escape` closes the list. A commit splices the active token in place by offsets rather than rebuilding the string. The suggestion source is local today, but flows through a debounced, abortable controller keyed by `(field, fragment)`, so a server-backed field can be added without changing the call sites. ## Resilience - **Offline / network failure.** Coloring needs no network, so it works offline; only the one-time schema fetch (autocomplete + `error` roles) needs the server, and it degrades to plain coloring if it fails. - **No JavaScript.** The original input still submits `?q=`, so the box never regresses below stock admin. ## Development server `just dev` migrates a file-backed SQLite database, seeds a superuser (`admin` / `admin`) and a few articles, and runs `runserver` with static files and browser auto-reload so you can watch the input react to edits. --- # Explanation Source: https://django-search-query.git-pull.com/packages/django-admin-search-query/explanation/ (admin-search-query-explanation)= # Explanation ## Loose coupling to the core package `django-admin-search-query` depends on `django-search-query` through an ordinary version floor in its `pyproject.toml` (`django-search-query>=0.1.0a0`), the same way any other consumer of the query language would. There is no private coupling underneath that pin: the mixin only ever calls the same public surface documented in {doc}`../../api` -- {func}`~django_search_query.search_query_to_q`, {exc}`~django_search_query.errors.QueryParseError`, and the highlighter that backs {doc}`colored-input`. Nothing in the core package imports or knows about the admin integration at all; the dependency arrow points one way. That looseness is deliberate. `django-search-query` stays usable in any Django project -- a REST endpoint, a management command, a plain view -- whether or not the admin is even installed. The admin integration is the optional layer built on top, not a requirement the core package carries for everyone. ## No-JavaScript degradation The colored input is a progressive enhancement of an admin feature that already works: the stock `` and Django's own `search_fields` matching. {class}`~django_admin_search_query.mixin.SearchQueryAdminMixin` renders the search box through a per-admin `change_list.html` that calls `{{ block.super }}` to render Django's own search form untouched -- facets, hidden filter parameters, and help text stay whatever the installed Django version renders -- and only adds one thing after it: a hidden `data-*` config element carrying the URLs of the `search-tokens/` and `search-highlight/` endpoints (see {doc}`../../api`), emitted only when `search_query_registry` is set. `search-input.js` reads that config element on load, finds the stock `#searchbar` input next to it, and enhances it into the colored editor described in {doc}`colored-input`. If JavaScript never runs -- disabled, blocked, or the request never reaches the browser at all -- the config element is inert markup and `#searchbar` is still the plain Django input: typing and submitting the form sends `?q=` exactly as it always did, and the mixin still compiles it server-side. The search box never depends on JavaScript to function; JavaScript only makes it nicer to type into. The same fallback also covers a query that *is* JavaScript-typed but does not parse. A half-finished quote, an unknown field, a stray operator -- anything that raises {exc}`~django_search_query.errors.QueryParseError` (or an empty term, or an admin with no `search_query_registry` configured) makes {meth}`~django_admin_search_query.mixin.SearchQueryAdminMixin.get_search_results` fall back to `super().get_search_results(...)`, Django's own `search_fields` behavior. No-JS submissions and unparseable JS submissions land on the same code path, so the admin search box is never worse than stock `ModelAdmin` -- only ever better, when the query happens to parse. --- # How-to guides Source: https://django-search-query.git-pull.com/packages/django-admin-search-query/how-to/ (admin-search-query-how-to)= # How-to guides {class}`~django_admin_search_query.mixin.SearchQueryAdminMixin` is opt-in -- drop it in front of a {class}`~django.contrib.admin.ModelAdmin` on the changelists that need structured search, and leave every other admin untouched. ## Add structured search to a ModelAdmin Put the mixin *before* `admin.ModelAdmin` in the base list. Python resolves methods left to right, so the mixin's {meth}`~django.contrib.admin.ModelAdmin.get_search_results` and {meth}`~django.contrib.admin.ModelAdmin.get_urls` run first and call `super()` to reach Django's own implementation -- reversing the order would skip the mixin entirely. This is the real registration the project tests against, from `test_app/admin.py`: ```python from django.contrib import admin from django_admin_search_query import SearchQueryAdminMixin from django_search_query.registry import FieldRegistry, FieldSpec from .models import Article ARTICLE_REGISTRY = FieldRegistry( specs=( FieldSpec(name="title", kind="string"), FieldSpec(name="body", kind="string"), FieldSpec(name="author", kind="string"), FieldSpec( name="status", kind="enum", enum_values=("open", "draft", "closed"), ), FieldSpec( name="created", kind="date", supports_comparison=True, supports_range=True, ), ), ) ARTICLE_FIELD_MAP = {spec.name: spec.path for spec in ARTICLE_REGISTRY.specs} @admin.register(Article) class ArticleAdmin(SearchQueryAdminMixin, admin.ModelAdmin): list_display = ("title", "status", "author", "created") search_fields = ("title", "body") search_query_registry = ARTICLE_REGISTRY search_query_field_map = ARTICLE_FIELD_MAP search_query_default_fields = ("title", "body") ``` Three class attributes wire the language into this admin: - `search_query_registry` -- the same kind of {class}`~django_search_query.registry.FieldRegistry` you would build for any other site; it decides which fields {dsq}`status:open` and friends may touch on this changelist. - `search_query_field_map` -- field name to ORM lookup path, exactly like the core package's `field_map` (see {doc}`../django-search-query/how-to`). - `search_query_default_fields` -- ORM paths a bare term or phrase searches. `search_fields` stays in place and keeps doing double duty: it is still {attr}`~django.contrib.admin.ModelAdmin.search_fields`, so it is what a bare term falls back to when `search_query_default_fields` is left unset, and it is also where Django's own search takes over whenever the query does not parse -- a plain word, an unterminated quote, an unknown field. Set `search_query_registry` and the box upgrades to structured search; leave it unset and the admin behaves exactly like stock `ModelAdmin.search_fields`, so you can add the mixin ahead of writing a registry without changing behavior. See {doc}`explanation` for why that fallback never regresses the box below stock admin, and {doc}`colored-input` for the optional syntax-highlighting input that comes with the mixin. --- # django-admin-search-query Source: https://django-search-query.git-pull.com/packages/django-admin-search-query/ (django-admin-search-query)= # django-admin-search-query An optional Django admin integration for {doc}`django-search-query <../django-search-query/index>` -- reach for it when you want a changelist search box that understands the structured query language; the core package works without it. ::::{grid} 1 1 2 2 :gutter: 2 2 3 3 :::{grid-item-card} {octicon}`rocket` Tutorial :link: ../../tutorial :link-type: doc Build a search registry and run a query against a Django queryset. ::: :::{grid-item-card} {octicon}`checklist` How-to :link: how-to :link-type: doc Add `SearchQueryAdminMixin` to a `ModelAdmin`, using the worked `ArticleAdmin` example. ::: :::{grid-item-card} {octicon}`book` API reference :link: ../../api :link-type: doc The `SearchQueryAdminMixin` API, its hooks, and the JSON endpoints behind the colored input. ::: :::{grid-item-card} {octicon}`light-bulb` Explanation :link: explanation :link-type: doc Loose coupling to the core package, and how the search box degrades without JavaScript. ::: :::{grid-item-card} {octicon}`paintbrush` Colored input :link: colored-input :link-type: doc The optional vanilla-JavaScript search input: syntax highlighting and autocomplete. ::: :::: ```{toctree} :hidden: how-to explanation colored-input ``` --- # Explanation Source: https://django-search-query.git-pull.com/packages/django-search-query/explanation/ (dsq-explanation)= # Explanation ## The pipeline A search string becomes a {class}`~django.db.models.Q` through three stages, each owned by one module. The tokenizer classifies characters into a token stream and records their source offsets, deferring every grammar decision to the parser. The parser is a precedence-climbing (Pratt) parser: it consumes the token stream and builds an AST, validating each field it meets against a {class}`~django_search_query.registry.FieldRegistry` as it goes, so an unknown field or an unsupported operator raises {exc}`~django_search_query.errors.QueryParseError` before compilation ever starts. {func}`~django_search_query.build_q` lowers that AST into a Django `Q` with a single structural `match` -- adding a new ORM mapping never touches parsing, and adding new syntax never touches the ORM mapping. ```dsq status:open OR status:draft ``` ```text "status:open OR status:draft" -> tokenize() characters -> Token stream (ident, colon, term, or, ...) -> parse() Pratt parser -> AST: Term | Field | Cmp | Range | Exists | Not | And | Or -> build_q() structural match -> Django Q ``` {func}`~django_search_query.parse` and `build_q` are the two halves of that middle-and-last stage; {func}`~django_search_query.search_query_to_q` chains tokenizing, parsing, and building for the common case in one call. `AND` and `OR` are left-associative, `OR` binds loosest, and `NOT`/`-`/`+` bind tightest as prefixes, so `NOT a AND b` parses as `(NOT a) AND b` -- juxtaposition (`a b`) is an implicit `AND` sharing its binding power, so `a b` and `a AND b` parse identically. ## Why two lexers exist The tokenizer above is strict on purpose: it raises {exc}`~django_search_query.errors.QueryParseError` on the first malformed character, which is exactly what a query compiler should do -- fail loud rather than silently compile the wrong query. A live search box needs the opposite guarantee: it has to colorize every keystroke, including a query that is only half-typed -- an unterminated quote, a dangling `[`. For that, the package ships a second, presentation-only lexer, {func}`~django_search_query.highlight.highlight_query_spans`, that never raises. It runs one regex over the entire string, with a catch-all group absorbing any character it cannot classify, so the span list it returns always covers the source end to end. The two lexers share no code: keeping them independent means a change meant to loosen highlighting can never accidentally loosen what the parser accepts, and vice versa. It is also why `?` can color as a wildcard in the search box while the compiler still treats it as a literal character -- see {doc}`../../query` for that boundary case, and {doc}`../django-admin-search-query/colored-input` for the JavaScript port of this same lexer that colors the admin search box client-side. ## Lucene-inspired, not Lucene-compatible The syntax borrows familiar [Lucene] shapes -- field-scoped terms ({dsq}`status:open`), quoted phrases, `AND`/`OR`/`NOT`, grouping, comparisons, and ranges -- but the package does not claim full Lucene compatibility, and some gaps are permanent rather than pending. There is no fuzzy match (`~`), no boosting (`^`), and no proximity search. `?` is not a single-character wildcard, even though Lucene defines it as one: the compiler only ever implements `*`. Treat the syntax as a small, predictable subset of Lucene's rather than a drop-in replacement for it. [Lucene]: https://lucene.apache.org/ --- # How-to guides Source: https://django-search-query.git-pull.com/packages/django-search-query/how-to/ (dsq-how-to)= # How-to guides Task recipes for tuning {func}`~django_search_query.search_query_to_q` beyond the tutorial's defaults: which ORM path a field name resolves to, which fields a query may touch at all, how wildcards and ranges compile, and how to degrade gracefully when a query does not parse. ## Map field names to ORM paths `field_map` decouples the field name a user types from the ORM lookup path the compiler builds. A registered field defaults to its own name when `field_map` leaves it out, so most sites only add an entry for a field whose ORM path really differs -- an alias, or a lookup that crosses a relation with Django's `__` traversal. ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry(specs=(FieldSpec(name="author", kind="string"),)) >>> search_query_to_q( ... "author:tony", ... registry=registry, ... field_map={"author": "author"}, ... default_fields=("title", "body"), ... ) ``` `field_map={}` would compile identically here, since `author`'s ORM path already matches its name. ## Restrict which fields a query can touch A {class}`~django_search_query.registry.FieldRegistry` is a strict allow list: only names declared as a {class}`~django_search_query.registry.FieldSpec` are queryable. A term for anything else -- a stray internal field, a typo -- raises {exc}`~django_search_query.errors.QueryParseError` at parse time instead of silently matching nothing. ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry( ... specs=( ... FieldSpec(name="title", kind="string"), ... FieldSpec(name="status", kind="enum"), ... ), ... ) >>> search_query_to_q( ... "status:open", ... registry=registry, ... field_map={}, ... default_fields=("title",), ... ) ``` `status` is declared, so it compiles; a query for a field the registry never listed raises -- see {ref}`dsq-how-to-fallback` for how to catch that. ## Wildcards and ranges String fields support two wildcards: a trailing `*` anchors the match to the start of the value (`istartswith`), a leading `*` anchors it to the end (`iendswith`); a bare value, or one wrapped in `*` on both sides, stays `icontains`. A `date` or `number` field whose {class}`~django_search_query.registry.FieldSpec` opts in also accepts ordered comparisons (`>`, `>=`, `<`, `<=`) and inclusive or exclusive ranges (`[a TO b]` / `{a TO b}`). ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry(specs=(FieldSpec(name="title", kind="string"),)) >>> search_query_to_q( ... "title:report*", ... registry=registry, ... field_map={}, ... default_fields=(), ... ) ``` ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry( ... specs=(FieldSpec(name="created", kind="date", supports_range=True),), ... ) >>> search_query_to_q( ... "created:[2024-01-01 TO 2024-12-31]", ... registry=registry, ... field_map={}, ... default_fields=(), ... ) ``` `?` looks like it should be a single-character wildcard, Lucene-style -- it is not. The presentation lexer that drives syntax highlighting recognizes `?` as a `wildcard`-role character (see {doc}`explanation`), but the compiler only ever implements `*`. A literal `?` in a value passes straight through to `icontains`: ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry(specs=(FieldSpec(name="title", kind="string"),)) >>> search_query_to_q( ... "title:a?c", ... registry=registry, ... field_map={}, ... default_fields=(), ... ) ``` A bare value on a `date` or `number` field is not parsed into a date or number either -- without a comparison or range operator it takes the same `icontains` path as a string field: ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry(specs=(FieldSpec(name="created", kind="date"),)) >>> search_query_to_q( ... "created:2024", ... registry=registry, ... field_map={}, ... default_fields=(), ... ) ``` (dsq-how-to-fallback)= ## Catch parse errors and fall back Any malformed query -- an unknown field, an unsupported operator, an unterminated quote -- raises {exc}`~django_search_query.errors.QueryParseError` with the character offset of the failure. A search endpoint that always needs to return *something* can catch it and fall back to a plain substring search: ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.errors import QueryParseError >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry(specs=(FieldSpec(name="status", kind="enum"),)) >>> try: ... search_query_to_q( ... "bogus:1", registry=registry, field_map={}, default_fields=("title",), ... ) ... except QueryParseError as err: ... print(f"fell back: {err}") fell back: unknown field 'bogus'; known fields: status ``` `err.position` is the offset into the original string -- handy for pointing a caret at the exact character that broke. {doc}`../django-admin-search-query/colored-input` uses the same lexer output to color a live search box. --- # django-search-query Source: https://django-search-query.git-pull.com/packages/django-search-query/ (django-search-query)= # django-search-query A [Lucene]-inspired search query language that compiles to Django {class}`~django.db.models.Q` objects. ::::{grid} 1 1 2 2 :gutter: 2 2 3 3 :::{grid-item-card} {octicon}`rocket` Tutorial :link: ../../tutorial :link-type: doc Build a search registry and run a query against a Django queryset. ::: :::{grid-item-card} {octicon}`checklist` How-to :link: how-to :link-type: doc Task recipes: map field names, restrict searchable fields, use wildcards and ranges, and catch parse errors. ::: :::{grid-item-card} {octicon}`search` Query language :link: ../../query :link-type: doc The search syntax and the query-string-to-queryset showcase. ::: :::{grid-item-card} {octicon}`book` API reference :link: ../../api :link-type: doc The public API: parse, build_q, search_query_to_q, the field registry. ::: :::{grid-item-card} {octicon}`light-bulb` Explanation :link: explanation :link-type: doc How a search string becomes a `Q` object, and why the highlighter uses a second lexer. ::: :::: [Lucene]: https://lucene.apache.org/ ```{toctree} :hidden: how-to explanation ``` --- # Packages Source: https://django-search-query.git-pull.com/packages/ (packages)= # Packages The workspace ships two independently-installable, independently-versioned packages. The core query language stands on its own; the admin integration is opt-in and depends on it through a loose version floor. ::::{grid} 1 1 2 2 :gutter: 2 2 3 3 :::{grid-item-card} {octicon}`package` django-search-query :link: django-search-query/index :link-type: doc The core query language: compiles a search string to Django ORM `Q` objects. No UI, admin, or backend assumptions. ::: :::{grid-item-card} {octicon}`browser` django-admin-search-query :link: django-admin-search-query/index :link-type: doc Optional Django admin integration built on the core, with a vanilla-JavaScript search input. Depends on it through a loose version floor. ::: :::: ```{toctree} :caption: Packages :hidden: django-search-query/index django-admin-search-query/index ``` --- # Contributing Source: https://django-search-query.git-pull.com/project/contributing/ (contributing)= # Contributing ## Workspace layout ```text packages/ django-search-query/ # core query language src/django_search_query/ django-admin-search-query/ # optional admin integration src/django_admin_search_query/ tests/ # shared test suite (settings, smoke tests) docs/ # this documentation site ``` All shared tooling -- ruff, ty, pytest, coverage -- is configured once in the root `pyproject.toml` and applies to every package. ## Standards - Every Python file starts with `from __future__ import annotations`. - NumPy-style docstrings; ruff enforces `pydocstyle`. - Type hints are required and checked with [ty]. - Public functions and methods carry working doctests once they exist. See `AGENTS.md` at the repository root for the full contributor guide. [ty]: https://docs.astral.sh/ty/ --- # Project Source: https://django-search-query.git-pull.com/project/ (project)= # Project Development happens in a single [uv] workspace: one lockfile, one virtual environment, two publishable packages under `packages/`. ## Set up ```console $ uv sync --all-packages --group dev ``` This installs both workspace packages editable plus the docs, testing, and lint tooling. ## Development loop Run every change through this loop: ```console $ uv run ruff format . ``` ```console $ uv run pytest ``` ```console $ uv run ruff check . --fix --show-fixes ``` ```console $ uv run ty check ``` The [just] recipes wrap these: `just test`, `just ruff`, `just ruff-format`, `just ty`, and `just build-docs`. ```{toctree} :hidden: contributing releasing ``` [uv]: https://docs.astral.sh/uv/ [just]: https://just.systems/ --- # Releasing Source: https://django-search-query.git-pull.com/project/releasing/ (releasing)= # Releasing Each package is versioned and released **independently**. A release bumps one package's version and records its changes in the shared root changelog -- which consolidates both packages under one set of sections -- without touching the other package. The version literal lives in two places per package: - `packages//pyproject.toml` -- `[project].version` - `packages//src//__about__.py` -- `__version__` Publishing is driven by CI on tag push. Tags are created and pushed by a maintainer, never by tooling. ```{note} Because the admin integration depends on the core package through a loose floor (`django-search-query>=X`), the two can be released on separate cadences. ``` --- # Query language Source: https://django-search-query.git-pull.com/query/ (query)= # Query language You give your users a real search box -- field-scoped terms, quoted phrases, boolean logic -- and {func}`~django_search_query.search_query_to_q` turns what they type into a Django {class}`~django.db.models.Q`. Pass that to any {class}`~django.db.models.query.QuerySet` and the ORM does the rest: **string in, queryset out**. A field map plus that one call covers most sites, so if the defaults fit you can stop after the {doc}`tutorial`. The syntax is Lucene-*inspired*, not Lucene-compatible -- familiar forms mostly just work. A query the parser cannot read raises {exc}`~django_search_query.errors.QueryParseError`; catch it to fall back to a plain search, which is exactly what the {doc}`admin integration ` does. Underneath, a tokenizer lexes the string, {func}`~django_search_query.parse` builds an AST, and {func}`~django_search_query.build_q` compiles it to a `Q`; {doc}`the pipeline ` walks that chain. ## See it work Seed a handful of articles: | title | status | author | created | | --- | --- | --- | --- | | Closed ticket | closed | tony | 2023-11-20 | | Draft memo | draft | jane | 2024-03-15 | | Launch plan | open | tony | 2024-06-01 | | Report Q3 | open | mint | 2024-09-01 | Each search string compiles to a `Q`, and {class}`~django.db.models.query.QuerySet` `.filter(q)` runs it as SQL: | You type | It returns | The SQL Django runs | | --- | --- | --- | | {dsq}`status:open` | Launch plan, Report Q3 | `WHERE status LIKE 'open'` | | {dsq}`author:tony` | Closed ticket, Launch plan | `WHERE author LIKE '%tony%'` | | {dsq}`status:open OR status:draft` | Draft memo, Launch plan, Report Q3 | `WHERE status LIKE 'open' OR status LIKE 'draft'` | | {dsq}`-status:closed` | Draft memo, Launch plan, Report Q3 | `WHERE NOT (status LIKE 'closed')` | | {dsq}`title:report*` | Report Q3 | `WHERE title LIKE 'report%'` | | {dsq}`created:>2024-01-01` | Draft memo, Launch plan, Report Q3 | `WHERE created > '2024-01-01'` | | {dsq}`status:open author:tony` | Launch plan | `WHERE status LIKE 'open' AND author LIKE '%tony%'` | Rows are listed in `created` order. Every row is checked against a live `Article` table in `tests/test_query_showcase.py`, so the SQL and matches stay honest. (The SQL is cleaned for reading and rendered on SQLite; `str(qs.query)` shows Django's fuller debug form, and your own database renders the equivalent lookups -- PostgreSQL, for instance, wraps case-insensitive matches in `UPPER(...)`.) Here is the `Q` behind the first row -- the whole magic in one call: ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry(specs=(FieldSpec(name="status", kind="enum"),)) >>> q = search_query_to_q( ... "status:open", ... registry=registry, ... field_map={"status": "status"}, ... default_fields=("title", "body"), ... ) >>> q ``` `Article.objects.filter(q)` then returns the matching rows. The other rows declare `author`, `title`, and `created` in the registry the same way -- the {doc}`tutorial` builds a registry like that end to end. ## Syntax Every row compiles exactly as shown. `default_fields` is where bare terms and phrases search; a {class}`~django_search_query.registry.FieldRegistry` decides which `field:` names are allowed and how each compiles. | Syntax | Example | Compiles to | | --- | --- | --- | | Bare term | {dsq}`hello` | [`icontains`](https://docs.djangoproject.com/en/stable/ref/models/querysets/#std-fieldlookup-icontains), OR'd across `default_fields` | | Quoted phrase | {dsq}`"exact phrase"` | `icontains`, verbatim (no wildcard expansion) | | String field | {dsq}`author:tony` | `author__icontains='tony'` | | Enum field | {dsq}`status:open` | `status__iexact='open'` | | Trailing wildcard | {dsq}`title:report*` | `title__istartswith='report'` | | Leading wildcard | {dsq}`title:*report` | `title__iendswith='report'` | | Comparison | {dsq}`created:>2024-01-01` | `created__gt='2024-01-01'` | | Inclusive range | {dsq}`created:[2024-01-01 TO 2024-12-31]` | `created__gte='2024-01-01'` and `created__lte='2024-12-31'` | | Exclusive range | {dsq}`created:{2024-01-01 TO 2024-12-31}` | `created__gt='2024-01-01'` and `created__lt='2024-12-31'` | | Existence | {dsq}`status:*` | `status` is non-empty and non-null | | OR | {dsq}`status:open OR status:draft` | `Q(...) \| Q(...)` | | Implicit AND | {dsq}`status:open author:tony` | `Q(...) & Q(...)` | | Negation | {dsq}`-status:closed` | `~Q(status__iexact='closed')` | | `?` (reserved) | {dsq}`title:a?c` | literal `title__icontains='a?c'` -- `?` is not a wildcard the compiler implements | | Bare value on a `date`/`number` field | {dsq}`created:2024` | literal `created__icontains='2024'` -- not a date or number comparison | The last two rows are deliberate, not bugs: `?` is a wildcard character to the highlighter only, and a field's `kind` controls formatting, not parsing -- a bare value never becomes a Python `date` or `int` before it reaches the ORM. ## Next - {doc}`tutorial` -- build a {class}`~django_search_query.registry.FieldRegistry` and run a query end to end. - {doc}`api` -- {func}`~django_search_query.search_query_to_q`, {func}`~django_search_query.parse`, {func}`~django_search_query.build_q`, and the field registry. - {doc}`How-to guides ` -- field maps, restricting fields, wildcards and ranges, catching {exc}`~django_search_query.errors.QueryParseError`. - {doc}`Admin integration ` -- put this in the Django admin changelist. --- # Tutorial Source: https://django-search-query.git-pull.com/tutorial/ (tutorial)= # Tutorial This tutorial builds one search-enabled queryset from a raw string. You describe which fields a query is allowed to touch, compile a query string into a Django {class}`~django.db.models.Q`, and run it against a real {class}`~django.db.models.query.QuerySet`. That is the whole path -- most projects never need more than this. ## Describe your searchable fields {func}`~django_search_query.search_query_to_q` needs to know which fields a query string may touch, and what kind of value each one holds -- a plain string, an enum, a date. You declare that once with a {class}`~django_search_query.registry.FieldRegistry` built from {class}`~django_search_query.registry.FieldSpec` entries; the registry is what turns {dsq}`status:open` into a validated lookup instead of an arbitrary attribute access. A registry for an `Article` model with a title, a body, an author, a status, and a creation date looks like this: ```python from django_search_query.registry import FieldRegistry, FieldSpec ARTICLE_REGISTRY = FieldRegistry( specs=( FieldSpec(name="title", kind="string"), FieldSpec(name="body", kind="string"), FieldSpec(name="author", kind="string"), FieldSpec( name="status", kind="enum", enum_values=("open", "draft", "closed"), ), FieldSpec( name="created", kind="date", supports_comparison=True, supports_range=True, ), ), ) ``` This tutorial only queries `status`, so the rest of it works with a single-field registry. ## Compile a query `search_query_to_q` takes the query string, the registry, a field map (canonical field name to ORM lookup path), and which fields a bare term or phrase searches by default -- then returns a `Q` you pass straight to `.filter()`. ```{doctest} >>> from django_search_query import search_query_to_q >>> from django_search_query.registry import FieldRegistry, FieldSpec >>> registry = FieldRegistry(specs=(FieldSpec(name="status", kind="enum"),)) >>> search_query_to_q( ... "status:open", ... registry=registry, ... field_map={"status": "status"}, ... default_fields=("title", "body"), ... ) ``` {dsq}`status:open` matched the `status` field spec, so it compiled to an `__iexact` lookup on the `status` column. Terms without a `field:` prefix would instead search every path listed in `default_fields`. ## Run it against a queryset The `Q` above is unremarkable to Django -- pass it to `.filter()` like any other: ```python q = search_query_to_q( "status:open", registry=registry, field_map={"status": "status"}, default_fields=("title", "body"), ) Article.objects.filter(q) ``` This snippet is illustrative rather than an executed doctest: filtering a real queryset needs a migrated `Article` table, which the docs build does not provide. `tests/test_query_examples.py` pins the exact `Q` this {dsq}`status:open` call compiles to, so the claim above stays checked against real behavior. ## Where to go next - {doc}`install` -- add the packages to a project and read about developmental releases. - {doc}`packages/django-search-query/index` -- the field-map dict, allowed fields, validation hooks, and the syntax the parser accepts. - {doc}`packages/django-admin-search-query/index` -- the opt-in admin integration and its JavaScript search input. ---