Plan to replace the hard-coded list of 'may be skipped' framework tests in
skippable() with annotations in test.json which record 'will be skipped
in these specific CI jobs'.
If the value of the MESON_CI_JOBNAME env var (an arbitrary string
expected to be unique for each CI configuration) contains any of the
strings in the `skip_on_jobname` key in test.json, the test is expected
to output MESON_SKIP_TEST.
Unexpected skips or runs are treated as an error.
Future work: Maybe we should add additional count categories 'unexpected
skip' and 'unexpected not skipped', rather than counting those as 'skipped'
and 'failed', respectively.
I'm not sure what the current schema is describing, but it's not doing
anything useful to validate 'matrix' entries, as I discover when I come
to add something to that.
'matrix' is a dict with possible keys 'options' and 'exclude'.
'options' is a dict with arbitrary keys, whose's values are arrays
Once we describe those dicts correctly, also fix that 'val' keys may be
string, boolean or null, and the corresponding 'exclude' keys may be
string or boolean.
v2:
Also allow 'val' and 'exclude' to be an array of strings.
Improve test.json schema to disallow arbitrary keys in all dicts which
have a defined set of keys (the 'installed', matrix 'options' and
'stdout' dicts).
Add 'count' and 'comment' keys to 'stdout' dict.
Add myself as a maintainer of these files since I'll be keeping them
in sync with the Vim repository.
Signed-off-by: Liam Beguin <liambeguin@gmail.com>
This is still missing completions for promote, but I can't figure out
how to find the wraps in subprojects that are not in the parent project
when those projects haven't been fetched yet.
This is useful for automatically generated docs (doxygen, hotdoc)
with a lot of generated files that may differ with different
versions of the generator.
Update the test.json schema, adding the 'stdout' property.
Also amend the test.json schema so the presence of an unexpected
property on the root object causes a validation error.
v2:
Also add 'tools' property to json schema.
Amend the documentation not to use the word 'list' to describe a dict.
JUnit is pretty ubiquitous, lots of services and results viewers
understand it, in particular gitlab and jenkins know how to consume
JUnit xml. This means projects using CI services can have their test
results consumed automatically.
Fixes: #6972
This does a couple of nice things, one is that editors like vscode can
be configured to use this schema to provide auto completion and error
highlighting if invalid values are added or required values are missing.
It also allows us test that the format of the test matrix work in a unit
test, which I've added. It does require that the python jsonschema
package is installed.
Vim can automatically comment and format comments. Set the necessary
variable to enable that feature.
See `:help format-comments` for more information.
Previously, this called out to the %ninja_test macro to run the
tests, but that limits us to only the arguments that ninja can
understand. In particular, it is not possible to add a test
timeout multiplier (such as is sometimes needed when building for
slow architectures such as armv7hl). With this patch, it will be
possible to specify `%meson_test -t 5` in the RPM spec file
without needing to patch the sources to extend the timeouts,
making life easier for packagers.
Related: https://github.com/mesonbuild/meson/issues/2037
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Eli Schwartz