CLI

Bowtie is a versatile tool which you can use to investigate any or all of the implementations it supports. Below are a few sample command lines you might be interested in.

Running Commands Against All Implementations

Many of Bowtie’s subcommands take a -i / --implementation option which specifies which implementations you wish to run against. In general, these same subcommands allow repeating this argument multiple times to run across multiple implementations. In many or even most cases, you may be interested in running against all implementations Bowtie supports. In the future, Bowtie’s CLI will default to running against all implementations in a number of additional cases where it makes sense to do so.

For now, if you wish to run against all implementations you can make use of the filter-implementations command to simply output the full list, along with some simple shell vomit to insert the needed -i options. Specifically, running $(bowtie filter-implementations | sed 's/^/-i /') will expand out to something you can use to run against all implementations. See below for a concrete example.

Examples

Validating a Specific Instance Against One or More Implementations

The bowtie validate subcommand can be used to test arbitrary schemas and instances against any implementation Bowtie supports.

Given some collection of implementations to check – here perhaps two Javascript implementations – it takes a single schema and one or more instances to check against it:

$ bowtie validate -i js-ajv -i js-hyperjump <(printf '{"type": "integer"}') <(printf 37) <(printf '"foo"')

Note that the schema and instance arguments are expected to be files, and that therefore the above makes use of normal shell process substitution to pass some examples on the command line.

Piping this output to bowtie summary is often the intended outcome (though not always, as you also may upload the output it gives to https://bowtie.report/ as a local report). For summarizing the results in the terminal however, the above command when summarized produces:

$ bowtie validate -i js-ajv -i js-hyperjump <(printf '{"type": "integer"}') <(printf 37) <(printf '"foo"') | bowtie summary

                                            Bowtie
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Schema                                                                                    ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│                                                                                           │
│ {                      Instance   hyperjump-json-schema (javascript)   ajv (javascript)   │
│   "type": "integer"   ──────────────────────────────────────────────────────────────────  │
│ }                      37         valid                                valid              │
│                        "foo"      invalid                              invalid            │
│                                                                                           │
└─────────────────────┴──────────────────────────────────────────────────────────────────────┘
                                        2 tests ran

Running the Official Suite Across Implementations

The following will run all Draft 7 tests from the official test suite (which it will automatically retrieve) across all implementations supporting Draft 7, showing a summary of any test failures.

$ bowtie suite $(bowtie filter-implementations | sed 's/^/-i /') draft7 | bowtie summary --show failures

Running a Single Test Suite File

To run the draft 7 type-keyword tests on the Lua jsonschema implementation, run:

$ bowtie suite -i lua-jsonschema https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft7/type.json | bowtie summary --show failures

Running Test Suite Tests From Local Checkouts

Providing a local path to the test suite can be used as well, which is useful if you have additional tests in development locally:

$ bowtie suite $(bowtie filter-implementations | sed 's/^/-i /') ~/path/to/json-schema-org/suite/tests/draft2020-12/ | bowtie summary --show failures

Checking An Implementation Functions On Basic Input

If you wish to verify that a particular implementation works on your machine (e.g. if you suspect a problem with the container image, or otherwise aren’t seeing results), you can run bowtie smoke. E.g., to verify the Golang jsonschema implementation is functioning, you can run:

$ bowtie smoke -i go-jsonschema

Connectables

In all of the examples presented above, we passed our desired implementations to Bowtie’s -i / --implementation option.

In truth, this option is more flexible than indicated above, though generally this extra flexibility is useful for more advanced use cases, which this section elaborates on.

The full syntax of the -i option is known as a connectable.

Connectables implement a mini-language for connecting to supported harnesses.

They allow connecting to implementations supported by Bowtie without making assumptions about the specific mechanism used to run or execute them.

In most simple use cases, users likely will not interact directly with the connectable syntax. Passing -i some-supported-implementation will generally do The Right Thing™. But for more advanced usage, the full collection of supported connectables will now be described.

The general syntax for connectables looks like:

[<kind>:]<id>[:<arguments>*]

The kind of connectable is optional, and when unspecified defaults to the happy connectable, making an example of the simplest connectable look like lua-jsonschema (which expands to happy:lua-jsonschema). A slightly longer description of what the happy connectable means is below, but in brief, the happy connectable will speak directly to any implementation which can be imported from Python. For any other implementation, an OCI container will be created (and later destroyed) using the image connectable.

More generally, the id is an identifier whose exact form depends on each kind of connectable. Arguments customize the actual connection to the implementation, and which arguments are supported again depends on the kind of connectable.

Connectables are loosely inspired by Twisted's strports.

image

A container image which Bowtie will start, stop and delete when finished.

The image id should be the image name. Providing a repository is optional, and if unprovided, will default to pulling images from Bowtie’s own public repository of images.

The image must be an image whose entrypoint speaks Bowtie’s harness protocol (which of course all of Bowtie’s own published harnesses images will do).

Examples:

  • image:example: an image named example, retrieved from Bowtie’s repository

  • example: with no explicit image, referring to the same image as previous

  • image:foo/bar:latest: an image with fully specified OCI container repository which will be pulled if not already present

container

An externally running container which Bowtie will connect to.

The container must be listening on standard input for input valid under Bowtie’s harness protocol.

Bowtie will not attempt to manage the container, so this connectable is suitable for cases where you wish to spin up a container externally, leave it running and potentially have Bowtie connect to it multiple times.

The id is a connector-specific identifier and should indicate the specific intended implementation. For example, for container images, it must be the name of a container image which will be pulled if needed. It need not be fully qualified (i.e. include the repository), and will default to pulling from Bowtie’s own image repository.

Examples:

  • container:deadbeef: an OCI container with ID deadbeef which is assumed to be running (and will be attached to)

direct

An implementation which is directly importable from Python

This connectable is useful for implementations Bowtie can speak directly to without inter-process communication. The list of such implementations is small relative to those reachable using containers.

Examples:

  • direct:python-jsonschema: a direct connection to the Python implementation known as jsonschema

happy

A connectable which intelligently picks the best possible way to connect to an implementation

The happy connectable is a sort of “higher-level” connectable. Given the name of an implementation, the happy connectable will pick the best possible way to connect to the implementation. Best here is defined to be “most reliable” and/or “fastest”.

Generally this means “connect directly when possible, falling back to spinning up a container”.

This connectable is used when no connector is specified, effectively giving the best behavior when no specific connectable is requested.

Examples:

  • happy:python-jsonschema: a direct connection to the Python implementation known as jsonschema, equivalent to direct:python-jsonschema

  • happy:some-not-directly-connectable-implementation: for any non-directly connectable implementation, equivalent to image:some-not-directly-connectable-implementation

Enabling Shell Tab Completion

The Bowtie CLI supports tab completion using the click module's built-in support. Below are short instructions for your shell using the default configuration paths.

Add this to ~/.bashrc:

$ eval "$(_BOWTIE_COMPLETE=bash_source bowtie)"

Using eval means that the command is invoked and evaluated every time a shell is started, which can delay shell responsiveness. To speed it up, write the generated script to a file, then source that.

Save the script somewhere.

$ _BOWTIE_COMPLETE=bash_source bowtie > ~/.bowtie-complete.bash

Source the file in ~/.bashrc.

$ . ~/.bowtie-complete.bash

After modifying your shell configuration, you may need to start a new shell in order for the changes to be loaded.

Reference

bowtie

A meta-validator for the JSON Schema specifications.

Bowtie gives you access to the JSON Schema ecosystem across every programming language and implementation.

It lets you compare implementations either to each other or to known correct results from the official JSON Schema test suite.

bowtie [OPTIONS] COMMAND [ARGS]...

Options

--version

Show the version and exit.

-L, --log-level <log_level>

How verbose should Bowtie be?

Default:

'warning'

Options:

debug | info | warning | error | critical

If you don’t know where to begin, bowtie validate --help or bowtie suite --help are likely good places to start.

Full documentation can also be found at https://docs.bowtie.report/

badges

Generate Bowtie badges for implementations using a previous Bowtie run.

Will generate badges for any existing dialects, and ignore any for which a report was not generated.

bowtie badges [OPTIONS]

Options

--site <site>

The path to a previously generated collection of reports, used to generate the badges.

Default:

PosixPath('site')

filter-dialects

Output dialect URIs matching a given criteria.

If any implementations are provided, filter dialects supported by all the given implementations.

bowtie filter-dialects [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>

A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-T, --read-timeout <SECONDS>

An explicit timeout to wait for each implementation to respond to each instance being validated. Set this to 0 if you wish to wait forever, though note that this means you may end up waiting … forever!

Default:

2.0

-d, --dialect <URI_OR_NAME>

Filter from the given list of dialects only.

-l, --latest

Show only the latest dialect.

-b, --boolean-schemas, -B, --no-boolean-schemas

If provided, show only dialects which do (or do not) support boolean schemas. Otherwise show either kind.

filter-implementations

Output implementations which match the given criteria.

Useful for piping or otherwise using the resulting output for further Bowtie commands.

bowtie filter-implementations [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>

A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-T, --read-timeout <SECONDS>

An explicit timeout to wait for each implementation to respond to each instance being validated. Set this to 0 if you wish to wait forever, though note that this means you may end up waiting … forever!

Default:

2.0

-f, --format <format>

What format to use for the output

Default:

'plain'

Options:

plain | json

--schema

Show the JSON Schema for this command’s JSON output.

-d, --supports-dialect <URI_OR_NAME>

Only include implementations supporting the given dialect URI or dialect shortname. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, draft7.

-l, --language <LANGUAGE>

Only include implementations in the given programming language.

Options:

c++ | clojure | cpp | dotnet | go | java | javascript | js | kotlin | lua | php | python | ruby | rust | scala | ts | typescript

--direct

Only include implementations with direct connectable functionality (i.e. which can run without the presence of a container runtime).

info

Show information about a supported implementation.

bowtie info [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>

A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-T, --read-timeout <SECONDS>

An explicit timeout to wait for each implementation to respond to each instance being validated. Set this to 0 if you wish to wait forever, though note that this means you may end up waiting … forever!

Default:

2.0

-f, --format <format>

What format to use for the output

Default:

'pretty if stdout is a tty, otherwise JSON'

Options:

json | pretty | markdown

--schema

Show the JSON Schema for this command’s JSON output.

latest-report

Output the latest published report from Bowtie’s website.

bowtie latest-report [OPTIONS]

Options

-D, --dialect <URI_OR_NAME>

A URI or shortname identifying the dialect of each test. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, draft7.

Default:

'Draft 2020-12'

run

Run test cases written directly in Bowtie’s testing format.

This is generally useful if you wish to hand-author which schemas to include in the schema registry, or otherwise exactly control the contents of a test case.

bowtie run [OPTIONS] [INPUT]

Options

-D, --dialect <URI_OR_NAME>

A URI or shortname identifying the dialect of each test. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, draft7.

Default:

'Draft 2020-12'

-i, --implementation <IMPLEMENTATION>

Required A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-k, --filter <GLOB>

Only run cases whose description match the given glob pattern.

-x, --fail-fast

Stop running immediately after the first failure or error.

--max-fail <COUNT>

Stop running once COUNT tests fail in total across implementations.

--max-error <COUNT>

Stop running once COUNT tests error in total across implementations.

-S, --set-schema

Explicitly set $schema in all (non-boolean) case schemas sent to implementations. Note this of course means what is passed to implementations will differ from what is provided in the input.

Default:

False

-T, --read-timeout <SECONDS>

An explicit timeout to wait for each implementation to respond to each instance being validated. Set this to 0 if you wish to wait forever, though note that this means you may end up waiting … forever!

Default:

2.0

-V, --validate-implementations

When speaking to implementations (provided via -i), validate the requests and responses sent to them under Bowtie’s JSON Schema specification. Generally, this option protects against broken Bowtie implementations and can be left at its default (of off) unless you are developing a new implementation container.

Arguments

INPUT

Optional argument

smoke

Smoke test implementations for basic correctness against Bowtie’s protocol.

bowtie smoke [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>

A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-T, --read-timeout <SECONDS>

An explicit timeout to wait for each implementation to respond to each instance being validated. Set this to 0 if you wish to wait forever, though note that this means you may end up waiting … forever!

Default:

2.0

-q, --quiet

Don’t print any output, just exit with nonzero status on failure.

-f, --format <format>

What format to use for the output

Default:

'pretty if stdout is a tty, otherwise JSON'

Options:

json | pretty | markdown

--schema

Show the JSON Schema for this command’s JSON output.

statistics

Show summary statistics for a Bowtie generated report.

If stdin is a TTY, the most recent public report for the latest JSON Schema dialect is downloaded. Otherwise, if it is not a TTY (e.g. if it is a pipe) then it should contain report data.

Piping input via:

$ bowtie latest-report –dialect <some-dialect> | bowtie statistics

can be useful to retrieve the latest report for a specific dialect.

bowtie statistics [OPTIONS] [REPORT]

Options

-f, --format <format>

What format to use for the output

Default:

'pretty if stdout is a tty, otherwise JSON'

Options:

json | pretty | markdown

--schema

Show the JSON Schema for this command’s JSON output.

--quantiles <n>

How many quantiles should be emitted for the compliance numbers? Computing quantiles only is sensical if this number is more than the number of implementations reported on. By default, we compute quartiles.

Arguments

REPORT

Optional argument

suite

Run the official JSON Schema test suite against any implementation.

Supports a number of possible inputs:

  • file paths found on the local file system containing tests, e.g.:

    • {PATH}/tests/draft7 to run the draft 7 version’s tests out of a local checkout of the test suite

    • {PATH}/tests/draft7/foo.json to run just one file from a checkout

  • URLs to the test suite repository hosted on GitHub, e.g.:

    • https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft7/ to run a version directly from any branch which exists in GitHub

    • https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft7/foo.json to run a single file directly from a branch which exists in GitHub

  • short name versions of the previous URLs (similar to those providable to bowtie validate --dialect, e.g.:

    • 7, to run the draft 7 tests directly from GitHub (as in the URL example above)

bowtie suite [OPTIONS] INPUT

Options

-i, --implementation <IMPLEMENTATION>

Required A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-k, --filter <GLOB>

Only run cases whose description match the given glob pattern.

-x, --fail-fast

Stop running immediately after the first failure or error.

--max-fail <COUNT>

Stop running once COUNT tests fail in total across implementations.

--max-error <COUNT>

Stop running once COUNT tests error in total across implementations.

-S, --set-schema

Explicitly set $schema in all (non-boolean) case schemas sent to implementations. Note this of course means what is passed to implementations will differ from what is provided in the input.

Default:

False

-T, --read-timeout <SECONDS>

An explicit timeout to wait for each implementation to respond to each instance being validated. Set this to 0 if you wish to wait forever, though note that this means you may end up waiting … forever!

Default:

2.0

-V, --validate-implementations

When speaking to implementations (provided via -i), validate the requests and responses sent to them under Bowtie’s JSON Schema specification. Generally, this option protects against broken Bowtie implementations and can be left at its default (of off) unless you are developing a new implementation container.

Arguments

INPUT

Required argument

summary

Generate an (in-terminal) summary of a Bowtie run.

bowtie summary [OPTIONS] [REPORT]

Options

-f, --format <format>

What format to use for the output

Default:

'pretty if stdout is a tty, otherwise JSON'

Options:

json | pretty | markdown

--schema

Show the JSON Schema for this command’s JSON output.

-s, --show <show>

Configure whether to display validation results (whether instances are valid or not) or test failure results (whether the validation results match expected validation results)

Default:

'validation'

Options:

failures | validation

Arguments

REPORT

Optional argument

validate

Validate instances under a schema across any supported implementation.

bowtie validate [OPTIONS] SCHEMA [INSTANCES]...

Options

-D, --dialect <URI_OR_NAME>

A URI or shortname identifying the dialect of each test. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, draft7.

-i, --implementation <IMPLEMENTATION>

Required A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-S, --set-schema

Explicitly set $schema in all (non-boolean) case schemas sent to implementations. Note this of course means what is passed to implementations will differ from what is provided in the input.

Default:

False

-T, --read-timeout <SECONDS>

An explicit timeout to wait for each implementation to respond to each instance being validated. Set this to 0 if you wish to wait forever, though note that this means you may end up waiting … forever!

Default:

2.0

-V, --validate-implementations

When speaking to implementations (provided via -i), validate the requests and responses sent to them under Bowtie’s JSON Schema specification. Generally, this option protects against broken Bowtie implementations and can be left at its default (of off) unless you are developing a new implementation container.

-d, --description <description>

A (human-readable) description for this test case.

--expect <expect>

Expect the given input to be considered valid or invalid, or else (with ‘any’) to allow either result.

Default:

'any'

Options:

valid | invalid | any

Arguments

SCHEMA

Required argument

INSTANCES

Optional argument(s)