Contributing to literalizer

Contributions to this repository must pass tests and linting.

CI is the canonical source of truth.

Install contribution dependencies

Install Python dependencies in a virtual environment.

$ pip install --editable . --group dev

Install prek hooks:

$ prek install

Linting

Run lint tools either by committing, or with:

$ prek run --all-files --hook-stage pre-commit --verbose
$ prek run --all-files --hook-stage pre-push --verbose
$ prek run --all-files --hook-stage manual --verbose

Running tests

$ uv run --group dev pytest

Golden verification preserves existing fixtures when rendering skips. A stale skipped fixture fails verification with its path. To remove it, run the corresponding golden test with --regen-all. Regeneration also rewrites supported golden files, so select only the affected test and review its diff.

Find a golden test

The manifest declares coverage; the following test modules collect the matching cases.

Golden test entry points

Coverage

Test module

literalize scenarios

tests/integration/test_literalize_golden.py

Default and variant calls

tests/integration/test_calls.py

Reference cases and variants

tests/integration/test_literalize_ref_golden.py

Default reference behavior

tests/integration/test_literalize_ref_default.py

Collect test names before selecting an affected case. These focused commands use -n 0 to run without parallel workers:

$ uv run --group dev pytest -n 0 tests/integration/test_calls.py --collect-only -q

Use -k to verify matching cases, for example the Python call cases:

$ uv run --group dev pytest -n 0 tests/integration/test_calls.py -k Python

After changing coverage, run the shared inventory check:

$ uv run --group dev pytest -n 0 tests/integration/harness/test_orphan_golden_files.py

The Python tests compare generated source with the stored fixture. Native language jobs in .github/workflows/lint.yml check those programs with a compiler or runtime.

Changelog entries

Describe each user-facing change in a news fragment at newsfragments/<issue>.change.md. Write fragments in Markdown using the .change.md suffix. Towncrier rejects invalid fragment names at release time. The release workflow assembles the fragments into CHANGELOG.md and the GitHub release notes, then deletes them. Do not re-add a fragment that a release has already consumed, for example when merging main into a long-running branch.

Golden case manifests

Every directory under tests/integration/cases/ contains a versioned case.toml. The manifest is the source of truth for the input’s suite, variant-axis and literalize_call coverage; language capability checks and expansion remain in the typed Python runner.

Portable scenarios discover every supported language and version. Narrow participation through declared capabilities, supported options or concrete syntax contracts. Root, reference and call manifests reject the legacy languages and languages_reason fields. Manifests and variant axes also reject language_id metadata gates; language IDs remain available for fixture naming and toolchain routing.

A YAML case that deliberately reuses an anchor declares its expected parser warning:

[expected_reused_anchor_warning]
anchor = "x"
count = 1

The anchor must be non-empty and the count must be a positive integer. Fixture discovery and public-API golden rendering both assert the declared warning type, anchor and count. Omitting the table leaves warnings visible, and unrelated warnings remain visible when the table is present. Other input formats cannot declare this YAML warning.

Ordinary format variants exercise every supported language version. An axis that explicitly selects language_version stays pinned to that version; a format requiring newer syntax uses its effective version once. To exercise only its default version, a typed axis can declare per_version = false. The orphan check uses those same effective specs for ordinary and call variants, so a compiling file still needs a rendering that compares its contents.

The literalize runner and orphan check share the rendering definitions in tests/integration/golden_scenarios.py. Adding a scenario to that list also adds its fixtures to the inventory; no separate orphan-check registration is needed. The shared records in tests/integration/golden_renderings.py keep spec and path construction separate from the scenario list. Each GoldenRendering supplies its effective specs and versioned golden_path.

Default and variant literalize_call cases share the call_golden_cases collection in tests/integration/call_variant_cases.py. Each case supplies CallRendering objects with its effective spec, golden path and expected exception. The call runner checks these renderings, and the orphan check accepts only paths whose rendering expects success. Expected rejections check the exception and apply the same regeneration-only cleanup policy in both call families.

An ordinary case participates in the base and combined suites. Its input.yaml is inferred because it is the directory’s sole input:

schema_version = 1
suites = ["base", "combined"]

The default module-name suite checks each constructor’s public default across its language versions:

schema_version = 1
suites = ["default-module-name"]
gates = [{ kind = "capability_flag", flag = "supports_module_name" }]

Its input lives beside the manifest, while its golden files remain under tests/integration/default_module_names/. Those files retain the default module names; regular compiler fixtures use unique names derived from their paths.

A variant-only case declares its specialized owner and the axes that consume it. A suffix distinguishes multiple logical uses of one input:

schema_version = 1
owner = "variant"

[[variants]]
axis = "nested_tuple_strategy"
suffix = "_mixed"

Native Python values that TOML cannot encode belong in base_context.record_null_value_sources. Use the shared scalar, sequence, and mapping recipes to substitute them through the public record_null_substitutions API; native_datetime_ruby_offsets demonstrates exact microsecond UTC offsets.

Simple render context also belongs beside the input. This base case replaces null record fields before inference:

schema_version = 1
suites = ["base", "combined"]

[base_context.record_null_substitutions]
replacement = -1

Some runners render one chosen input rather than the whole inventory: the non-default indent, the bare value at file scope, the pre-indent shapes. The input declares the part it plays in a roles list, and the runner looks it up by role, so the directory name stays a single source of truth on disk:

schema_version = 1
suites = ["base", "combined"]
roles = ["indent-input"]

A literalize_call case declares that owner and describes its call in a [call] table, so the whole case lives in one directory:

schema_version = 1
owner = "literalize-call"

[call]
target_function = "throttler.check"
parameter_names = ["user_id", "ts"]
per_element = true
call_transform = "emit({call})"
transform_stub_names = ["emit"]
requires_call_returns_expression = true

owner = "literalize-call" and a [call] table require each other. call_transform is a template, not code: it may substitute only {call} and {zipped}, and any other placeholder fails at load time. Enum- and type-valued fields are named by string and resolved by the loader: call_style (keyword, positional, object, or command), zip_input_format, and the variable_form pair (new or existing). variant_only = true keeps a case out of the default per-language call matrix, leaving it to the call-variant suite.

A $ref case declares one of the two ref owners and describes itself in a [ref] table:

schema_version = 1
owner = "literalize-ref"

[ref]
ref_case_override = "camel"

[ref.value_sources]
my_int = "42"

owner = "literalize-ref" (the case renders with an explicit ref_case) or owner = "literalize-ref-default" (the case renders without one) and a [ref] table require each other, so the table is spelled even when it is empty. ref_key defaults to "$ref". ref_case_override names an identifier case (snake, camel, pascal, upper_snake, or kebab) that replaces the language’s default and skips any language whose supported_ref_cases excludes it. Each [ref.value_sources] entry maps a ref name to a JSON source that seeds the bound value for that ref.

Keep regressions with the same input and API context together. A shared [call] or [ref] table can own named [[call_examples]] or [[ref_examples]] that exercise specific source syntax or representation options:

schema_version = 1
owner = "literalize-ref"

[ref]
value_sources = { shared = '[1,"x"]' }

[[ref_examples]]
example = "tagged_enum"
gates = [{ kind = "enum_member_present", option = "heterogeneous_strategy", member = "TAGGED_ENUM" }]
heterogeneous_strategy = "TAGGED_ENUM"
value_sources = { shared = '[1,"x"]' }

The primary table exercises the shared scenario across languages; each example selects the capabilities its additional contract needs. Examples share the owning input and use the same renderer, version expansion, compiler checks and orphan checks as the primary table. Each example declares its complete API configuration rather than inheriting options from the primary table. Its unique example name uses lowercase letters, digits and underscores and is appended to its golden filenames and test identifiers. Keep the original regression provenance with the example when consolidating cases.

A [ref] case with variable_form = "none" and a consumer splices the unwrapped LiteralizeResult.bare_code into a native function argument. Its compiling golden checks that record declarations remain outside the expression. Optional caller_preamble entries supply caller-owned headers after rendering, without teaching the renderer about the caller’s class ownership.

A case renders under every language unless it narrows. gates names the property the narrowing follows from, in the same vocabulary the variant axes and the rejection manifests use, so a language that later gains that property is covered without editing the manifest:

schema_version = 1
suites = ["base"]
gates = [{ kind = "capability_flag", flag = "default_allows_empty_dict_keys" }]

This gate belongs to an input containing an empty mapping key. Use the input’s actual requirements to select its languages, and keep source-syntax regressions as capability-gated examples of shared scenarios. The [call], [ref] and named example tables use the same gate vocabulary.

Non-default inputs can be explicit with input = "input.toml". Supported variant context fields are variable_form (new, existing, or both), collection_layout, pre_indent_level, and record_null_substitutions. The loader rejects unknown fields or axes, missing inputs, duplicate logical cases, and duplicate golden targets.

Rejection manifests

Each scenario under tests/errors/rejections/ declares an API call in rejection.toml and records its results in an expected.toml golden file. Scenarios run across every language by default; the input manifest has no language list. The golden counts languages accepting and rejecting, groups exact messages by exception type, and records successful calls in an Accepted table. For example, a scalar null scenario includes these entries:

[Accepted]
Python = ""

[UnrepresentableInputError]
Toml = "TOML cannot represent null values; remove the null or choose a language with a null literal"

A case is keyed by its language, then by the option member it ran under and the declared value it substituted, if the manifest varies either. Option members use their names ([CJSON]), and declared values are quoted (['9Entry']). A language may accept one source and reject another, so it can appear in both population counts. Changes between acceptance and rejection, exception types, or messages fail golden comparison.

The manifest declares what provokes the rejection:

schema_version = 1
summary = "A JSON value type keys its objects with strings."
exceptions = ["UnrepresentableInputError"]
option = "json_type"
gates = [{ kind = "spec_field_present", field = "json_type" }]

[call]
api = "literalize"
sources = ["1: one"]
input_format = "yaml"

Optional gates narrow the scenario by capability using the same vocabulary as the golden suite’s variant axes. A new language joining those gates is covered automatically. option runs each applicable language once per member of that option, and values runs a case per declared value, which any {value} in a constructor argument substitutes. Languages lacking an enum member or declaration modifier needed to express the call do not contribute that combination. This selection reads option declarations, without executing the call to decide whether it belongs.

Declaration modifiers normally resolve against the target’s enum. modifier_source = "all" repeats the call using every enum that declares the requested modifier names, covering both local and foreign modifier ownership without naming an implementation in the manifest. Those cases add a ['modifiers=Owner'] part to the golden identifier, so distinct enum sources cannot overwrite each other’s results.

api is constructor, literalize, literalize_call, or literalize_call_with_declarations, and the loader rejects an argument the named API does not take. Composition scenarios supply ordered call.declarations entries with source, variable_form (new or existing), and variable_name. The fragments and call use the same target and reference inference hints. Direct option operations use sequence_type_annotation with option = "sequence_format" or declaration_formatter with option = "declaration_style". These run every enum member declaring the corresponding callable interface or standard declaration configuration, without constructing a target with that member.

For literalize scenarios, call.source_codepoints maps named {placeholders} to Unicode code points, including lone surrogates that TOML cannot store. call.invalid_values declares cyclic sequences or a fixed depth of nested sequences for ref_values, bound_refs, or record_null_substitutions; the runner builds fresh native values for each call.

An optional top-level integer_digit_limit scopes the interpreter’s conversion limit to the scenario and restores it even if the call fails. Use zero to disable the limit, or at least sys.int_info.str_digits_check_threshold to enable it. With record_output = true, successful rendering or formatter calls record their exact code in [Accepted]; other scenarios record acceptance as an empty string. Each type in exceptions must occur in at least one case, keeping a scenario from silently losing the rejection it covers. Other public Literalizer errors are recorded with their exact types and messages too; unexpected exceptions escape the runner. For parser diagnostics, an optional top-level position = [line, column] asserts the public error’s one-based line and column attributes when the declared exception occurs. The same rule applies to optional attributes assertions; language_name = "{language}" checks the current language name. Python-version-specific message overrides remain in the golden, and regeneration preserves inactive versions.

Documentation

Documentation is built on GitHub Pages.

Run the following commands to build and view documentation locally:

$ uv run --group=dev sphinx-build -M html docs/source docs/build -W
$ python -c 'import os, webbrowser; webbrowser.open("file://" + os.path.abspath("docs/build/html/index.html"))'

Continuous integration

Tests are run on GitHub Actions. The configuration for this is in .github/workflows/.

Code generation benchmarks

The benchmarks in tests/benchmarks/test_bench_literalize.py measure code generation with CodSpeed’s CPU simulation instrument. The six benchmark shards in .github/workflows/ci.yml publish results to the same CodSpeed report. They use Ubuntu 24.04, managed CPython 3.14.8 and a fixed PYTHONHASHSEED of 0. Changing the benchmark runtime requires a new baseline for meaningful performance comparisons. Compiler correctness is checked by the integration golden tests and language round-trip checks. Individual compiler commands have deadlines to bound pathological workloads.

Performing a release

See Release Process.