Commit Graph

344 Commits

Author SHA1 Message Date
Eli Schwartz e8a85fa8a2
various python neatness cleanups
All changes were created by running

"pyupgrade --py3-only"

and committing the results. Although this has been performed in the
past, newer versions of pyupgrade can automatically catch more
opportunities, notably list comprehensions can use generators instead,
in the following cases:
- unpacking into function arguments as function(*generator)
- unpacking into assignments of the form x, y = generator
- as the argument to some builtin functions such as min/max/sorted

Also catch a few creeping cases of new code added using older styles.
2021-10-04 16:29:30 -04:00
Christian Clauss d5b0673079 Fix typos 2021-09-30 14:11:54 -07:00
Mat Cross 174bc1c786 Presence of nagfor signifies that Fortran tests can be run. 2021-09-10 16:08:57 -04:00
Xavier Claessens 0063eb251e python: Workaround broken install path 2021-08-22 22:14:59 -04:00
Eli Schwartz 44e123dd90
test runner: add the ability to configure test.json for python paths
Adds new installed file types with @VAR@ substitution.
2021-08-18 17:58:30 -04:00
Nirbheek Chauhan 1799658614 Revert "Disable Rust + clangcl."
This reverts commit 6c55fc5cb0
2021-08-17 19:58:23 -07:00
Nirbheek Chauhan 339c768862 Revert "Disable vs2017 + 32bit Rust combo as it is broken."
This reverts commit 75688240cf

Should've been fixed by c95bffb295
2021-08-18 05:43:39 +05:30
Eli Schwartz 00bfbcbfa2 tests: make detecting broken compilers always run without tracebacks
Fixes regression in commit 75688240cf.
Even though this function is *currently* only invoked on Windows, these
environment variables may not actually exist -- and apparently don't in
at least the "UnusedMissingReturn / windows" test run, which... did not
get triggered by that commit, since it only edited the testsuite runner,
not any test cases. \o/
2021-08-15 11:35:44 +03:00
Jussi Pakkanen 6c55fc5cb0 Disable Rust + clangcl. 2021-08-15 03:06:01 +03:00
Jussi Pakkanen 75688240cf Disable vs2017 + 32bit Rust combo as it is broken. 2021-08-15 00:31:18 +03:00
Jussi Pakkanen 8c0c92e1a6 Allow gettext test to skip on macOS. 2021-08-12 23:25:33 +03:00
Jussi Pakkanen 61ec215860 Remove workaround now that we have a new Ninja. 2021-07-25 21:33:18 +03:00
Jon Turney 3c81ff8f00
Make it an error if MESON_CI_JOBNAME isn't set under CI
Also check skip is as expected for CI all runs, not just those that set
MESON_CI_JOBNAME.
2021-07-13 15:26:08 +01:00
Jon Turney b20cfec4bc Use None as Environment object build_dir in detect_system_compiler()
The Environment object constructor accepts None as build_dir (for quite
a while now), so don't bother with creating a temporary directory for
use as the build_dir, if we're not going to need it.

Future work: Environment.__init__() sets scratch_dir to '' if build_dir
is None, which seems a little wonky, as it isn't a path.
2021-07-07 20:08:12 +02:00
Jon Turney 609864a66d Annotate framework tests with where they are expected to skip
Remove hard-coded framework test skip logic in skippable(), instead
annotate test.json with environments in which skip is expected.

(Mainly this is done with by testing the value of MESON_CI_JOBNAME now
set for linux jobs)
2021-07-07 20:08:12 +02:00
Jon Turney c78e6cac80 Allow `skip_*` test.json keys at top-level or in `matrix:` 2021-07-07 20:08:12 +02:00
Jon Turney a2e44296c9 Add `skip_on_os` to test.json 2021-07-07 20:08:12 +02:00
Jon Turney 7126fbaf81 Add `skip_on_jobname` to test.json
Plan to replace the hard-coded list of 'may be skipped' framework tests in
skippable() with annotations in test.json which record 'will be skipped
in these specific CI jobs'.

If the value of the MESON_CI_JOBNAME env var (an arbitrary string
expected to be unique for each CI configuration) contains any of the
strings in the `skip_on_jobname` key in test.json, the test is expected
to output MESON_SKIP_TEST.

Unexpected skips or runs are treated as an error.

Future work: Maybe we should add additional count categories 'unexpected
skip' and 'unexpected not skipped', rather than counting those as 'skipped'
and 'failed', respectively.
2021-07-07 20:08:12 +02:00
Jon Turney 6bf86f4783 Drop non-framework test names from skippable()
In all these cases, the test has already been determined as skippable
because it's not part of the 'frameworks' suite.
2021-07-07 20:08:12 +02:00
Eli Schwartz c4f5f6364e pyupgrade 2021-07-05 17:55:04 +03:00
Daniel Mensinger 3e396b3782
fix: Always explicitly set encoding for text files (fixes #8263) 2021-06-29 11:28:08 +02:00
Daniel Mensinger 28175bbee2
pathlib: Patch pathlib to work around some bugs (fixes #7295) 2021-06-29 11:28:08 +02:00
Daniel Mensinger 3f889606c7 Split compiler detection from Environment
This moves all the compiler detection logic into the new
compilers.detect module. This dramatically reduces the size
and complexity of Environment.
2021-06-25 19:34:48 +02:00
Jon Turney bee4dc9f78 Fix project tests category name used in skippable() and XML output
Since 25df6e7d split the iteration over tests to start them from the
iteration to collect their results, the variable 'name' is only being
set in the first iteratiorn, so all tests are treated as being in the
last test category read (probably 'wasm') for skipppable() and in the
XML output.

Store the category name in the TestDef object
Use it in skippable()
Use it in classname attribute of XML test results
2021-06-22 16:23:09 +02:00
Daniel Mensinger b2112bc4f6 tests: Always enable the traceback in run_project_tests.py 2021-06-18 23:48:33 +02:00
Daniel Mensinger 10afec575b tests: Add visual seperator to the test logs in run_project_tests.py 2021-06-18 23:48:33 +02:00
Jon Turney 636bf91586 CI: Split out file format test as a separate GitHub workflow
Split out checking of file format as a separate GitHub workflow, rather
than running it as part of the project tests for every platform and
toolchain combination in CI, so that this test is not effected by the
changed paths constraints which are applied to the project tests.
2021-06-11 15:28:07 +02:00
Daniel Mensinger e987a88c19 tests: Force colorize CI output 2021-06-09 13:25:36 +02:00
Daniel Mensinger 7fac515acf tests: Don't even start running skipped tests 2021-06-09 13:25:36 +02:00
Daniel Mensinger 30ff36dc91 tests: Fix global state issue 2021-06-09 13:25:36 +02:00
Daniel Mensinger 0aa8843fb2 tests: Add option to limit the number of workers
Also, increase the minimum ammount of workers from 1 to
2 when the core count can not be determined.
2021-06-09 13:25:36 +02:00
Daniel Mensinger 098e3f9994 tests: Optional progress bar with tqdm 2021-06-09 13:25:36 +02:00
Daniel Mensinger 25df6e7d16 tests: Refactored test runner 2021-06-09 13:25:36 +02:00
Daniel Mensinger e1708d3de9 tests: Reduce complexity of the run_tests signature 2021-06-09 13:25:36 +02:00
Daniel Mensinger 99d3897caf tests: reduce global variables 2021-06-09 13:25:36 +02:00
Daniel Mensinger 7fc755b334 typing: Fully annotate run_project_tests.py 2021-06-09 13:25:36 +02:00
Dylan Baker 74ed27f776 cython: add 1 basic test 2021-06-07 09:17:40 -07:00
Jussi Pakkanen 0b2a7300c0 Set up VS environment automatically when run. 2021-05-13 11:03:10 +03:00
Ralf Gommers 9bbf2dd07c Add validation for --only in `run_project_tests.py`
This now gives a clear error rather than silently passes for
unrecognized categories, like:

    python run_project_tests.py --only nonexisting

or

    python run_project_tests.py --only objc  # should be 'objective c'
2021-05-02 22:27:40 +03:00
Jussi Pakkanen 54767ab482
Merge pull request #8568 from dcbaker/submit/qt-dependency-factory
QT: use a proper dependency factory
2021-03-24 22:52:15 +02:00
Dylan Baker 1767c57492 run_project_tests: add annotations 2021-03-23 15:28:08 -07:00
Dylan Baker 54c55f77a9 run_project_tests.py: Allow qt tests to skip on !macOS
We don't always have qmake installed (and it's good to test failure
paths too!) so we can't expect this to succeed in all cases. With the
following commit we'll use a test.json to test both pkg-config and
qmake, so we need to be able to skip.
2021-03-23 15:28:08 -07:00
Xavier Claessens 66d62a224e test.json: Add support for not matching stdout lines
By default expected line must be matched in order. When an expected line
is matched it does not matter if it's matched again later or not.

When defining "count", it means that line must be matched exactly that
many times before matching the next expected line. Once all occurences
have been matched for an expected line, it not must appear any more in
all next lines.
2021-03-23 18:07:18 -04:00
Xavier Claessens d0d81ab84e run_project_tests.py: Allow "--only common/240" syntax 2021-03-23 18:07:18 -04:00
Jussi Pakkanen ecd8c7e4af Update VS module version check. 2021-03-16 21:06:51 +02:00
Jussi Pakkanen 3ac3e125e4 Delete leftover files from some tests. 2021-03-16 16:37:02 +02:00
Eli Schwartz 6a0fabc647
mass rewrite of string formatting to use f-strings everywhere
performed by running "pyupgrade --py36-plus" and committing the results
2021-03-04 17:16:11 -05:00
Eli Schwartz 4340bf34fa
various python neatness cleanups
All changes were created by running

"pyupgrade --py3-only --keep-percent-format"

and committing the results. I have not touched string formatting for
now.

- use set literals
- simplify .format() parameter naming
- remove __future__
- remove default "r" mode for open()
- use OSError rather than compatibility aliases
- remove stray parentheses in function(generator) scopes
2021-03-04 17:11:26 -05:00
Dylan Baker 3e11da6db5 run_*_tests: Add some type annotations 2021-02-23 08:38:19 -08:00
Dylan Baker 8a12115741 run_project_tests: Delete a build directory if it already exists
This can happen if you abort a run before the cleanup finishes
2021-02-23 08:38:19 -08:00
Dylan Baker 44d5ea2a1f run_project_tests: split json parsing code from test gathering code 2021-02-23 08:38:19 -08:00
Dylan Baker 80ec157e5e ci: Add bindgen to CI images 2021-02-06 10:27:04 -08:00
Paolo Bonzini 9ba7b0ebd0 mlog: add __str__ method to AnsiDecorator
Automatically colorize the text when printing the AnsiDecorator, based
on the result of mlog.colorize_console().  This is how AnsiDecorator
is used most of the time anyway.
2021-01-21 08:54:35 -05:00
Daniel Mensinger 8f1d9bb7b0 cmake: add PATH logic to preliminary dep check (fixes #8133) 2021-01-10 21:48:23 +00:00
Christoph Reiter 58640bdff9 Remove AutoDeletedDir
Replace the only usage with a simple try/finally and remove the class.
2020-11-23 09:26:41 -08:00
Christoph Reiter edbcb2bfc8 Replace various calls to AutoDeletedDir with TemporaryDirectoryWinProof
AutoDeletedDir was a workaround to TemporaryDirectory not using
windows_proof_rmtree(). TemporaryDirectoryWinProof does now, so replace
it.
2020-11-23 09:26:41 -08:00
Jussi Pakkanen d27bad7d6d
Merge pull request #8011 from dcbaker/submit/post-python36-cleanups
Python 3.6 cleanups
2020-11-22 04:28:19 +02:00
Jason Ekstrand 9dab27f814 run_project_tests: Use the test environment for install and clean 2020-11-21 07:55:10 -08:00
Dylan Baker f6672c7a19 use real pathlib module
We added the _pathlib module to work around defeciencies in python 3.5's
implementation, since we now rely on 3.6 lets drop this
2020-11-20 15:08:40 -08:00
Sam Thursfield 21897a10ca gnome: Handle libraries that are not in the current build dir
The generate_gir() function previously assumed all library
inputs were in the current build dir. This would fail if they
weren't.
2020-11-13 17:39:47 +02:00
Jussi Pakkanen bca90722ec Bump minimum supported Python version to 3.6. Closes #6297. 2020-10-30 11:18:11 +02:00
Dylan Baker e8399c8c6e
depenencies/llvm: Handle llvm-config --shared-mode failing (#7379)
* depenencies/llvm: Handle llvm-config --shared-mode failing

Fixes: #7371
Fixes: #7878

* test cases/llvm: Refactor to use test.json

Instead of trying to cover everything internally
2020-10-22 17:21:14 +03:00
Daniel Mensinger b27af7e465
cmake: Add cross tests 2020-10-13 17:04:20 +02:00
Daniel Mensinger 1dfaccfd91 pathlib: Fix resolve() by overriding it in Python 3.5 2020-10-04 10:45:48 +02:00
Nirbheek Chauhan 946aeb6947 project tests: Also check whether introspect works 2020-09-17 22:25:37 +00:00
Nirbheek Chauhan 4c93da6531 project tests: Rename a variable for clarity 2020-09-17 22:25:37 +00:00
Jon Turney 6053da7a00 Add build machine compilers to project tests compiler report
When cross-compiling, also report build machine compilers in the
compiler report produced by run_project_tests.py
2020-09-11 20:45:09 +00:00
Jon Turney 88e13c5f7c Split tests out from 'common' which require a native compiler
Split out tests (and parts of tests) which require a native compiler
from the 'common' suite to a new suite called 'native', so we can
selectively avoid running those tests when only a cross-compiler is
available.

Also move test '211 cmake module' to 'cmake' suite, since it appears
that the way we use cmake requires a native compiler.
2020-09-10 07:20:41 +00:00
Daniel Mensinger e681235e5f
typing: fix code review 2020-09-08 20:15:58 +02:00
Daniel Mensinger 17439fa3e8 test: Add 'dir' support for installed files in test.json
This is useful for automatically generated docs (doxygen, hotdoc)
with a lot of generated files that may differ with different
versions of the generator.
2020-08-27 00:08:57 +00:00
Jussi Pakkanen 587e159d6c
Merge pull request #7460 from mensinda/fixDeepcpy
deps: Do not deepcopy internal libraries (fixes #7457)
2020-07-23 12:45:49 +03:00
cmcneish a7a6a4833f
Add boost_root support to properties files (#7210)
* Add boost_root support to properties files

This commit implements `boost_root`, `boost_includedir`, and
`boost_librarydir` variable support to native and cross properties
files. The search order is currently environment variables, then
these variables, and finally a platform-dependent search.

* Add preliminary boost_root / boost_includedir tests

Each test contains a fake "version.hpp", as that's how boost detection is
currently being done. We look for this file relative to the root directory,
which probably shouldn't be allowed (it previously was for BOOST_LIBRARYDIR
but not for BOOST_ROOT). It also cannot help with breakage detection in
libraries, however it looks like this wasn't getting tested beforehand.

I've given the two unique version numbers that shouldn't be present in any
stock version of boost (001 and 002).

* Add return type to detect_split_root

* Return empty list when nothing found in BOOST_ROOT, rather than None

* Update boost_root tests

* Create nativefile.ini based on location of run_project_tests.py
* Add fake libraries to ensure boost_librarydir is being used

* Require all search paths for boost to be absolute

* Redo boost search ordering

To better match things like pkg-config, we now look through native/cross files,
then environment variables, then system locations for boost installations.

Path detection does not fall back from one method to the next for properties or
environment variables--if boost_root, boost_librarydir, or boost_includedir is
specified, they must be sufficient to find boost. Likewise for BOOST_ROOT and
friends. pkg-config detection is still optional falling back to system-wide
detection, for Conan.

(Also, fix a typo in test 33's nativefile)

* Correct return type for detect_roots

* Correct boost dependency search order in documentation

* Print debug information for boost library finding, to resolve CI issues

* Handle native/cross file templates in a more consistent way

All tests can now create a `nativefile.ini.in` if they need to use some
parameter that the testing framework knows about but they can't.

* Pass str--rather than PosixPath--to os.path.exists, for Python35

* Look for boost minor versions, rather than boost patch versions in test cases

* Drop fake dylib versions of boost_regex

* Prefer get_env_var to use of os.environ

* Correct error reporting for relative BOOST_ROOT paths

* Bump version this appears in. Also, change "properties file" to "machine file" as that appears to be the more common language.
2020-07-21 16:40:29 +02:00
Daniel Mensinger 62f3aa1dad
Make virtualenv inside the project work 2020-07-16 14:30:07 +02:00
Michael Hirsch 7851495064
remove redundant syntax 2020-07-12 09:13:00 -04:00
Michael Hirsch 895de87b90
some python test cases don't care about backend, so run them in any case 2020-07-12 09:12:16 -04:00
Michael Hirsch 1ec84c570f run_*tests*.py: print Meson version at start of test suite
on some systems, tests may take over an hour to run--only to find
you might have used an unintended Meson version (e.g. release instead
of dev). This change prints the Meson version at the start of the
run_*tests*.py scripts.

Also, raise SystemExit(main()) is preferred in general over
sys.exit(main())
2020-06-15 23:33:40 +03:00
Daniel Mensinger a44dc67cb7 ci: Use test.json to skip VS2017 azure tests
See also #7307 #7314 #7316

cc @nirbheek
2020-06-15 05:46:25 +00:00
Daniel Mensinger 96379e51ed tests: reset CMakeDependency.class_cmakeinfo 2020-06-13 11:05:28 +00:00
Jussi Pakkanen 1e7f1ad4c9
Merge pull request #7197 from jon-turney/test-output-check-mandatory
Make the expected output check mandatory for failing-meson and warning-meson tests
2020-05-26 02:52:12 +03:00
Jussi Pakkanen 527536dd4a Clear internal caches before running each test. 2020-05-23 22:45:30 +03:00
Jon Turney 0ac4376990
Make expected stdout mandatory for warning-meson and failing-meson tests
Unify present or absent test.json file cases in gather_tests

Make expected stdout mandatory in test.json for some test categories

Use a trivial TestCategory class rather than a tuple, to make it easier
to default category attributes
2020-05-23 19:55:50 +01:00
Jussi Pakkanen f2e2e910d9 Remove stray print call. 2020-05-23 17:50:21 +03:00
Jussi Pakkanen 22bc0d46a5 Revert "Merge pull request #7172 from jon-turney/test-output-check-mandatory"
This reverts commit 0871b1032c, reversing
changes made to 9dc3ca2c1c.
2020-05-23 16:02:30 +03:00
Jussi Pakkanen 0871b1032c
Merge pull request #7172 from jon-turney/test-output-check-mandatory
Make the expected output check mandatory for failing-meson and warning-meson tests
2020-05-22 17:06:23 +03:00
Jon Turney 19b44575d1
Make expected stdout mandatory for warning-meson and failing-meson tests
Unify present or absent test.json file cases in gather_tests

Make expected stdout mandatory in test.json for some test categories

Use a trivial TestCategory class rather than a tuple, to make it easier
to default category attributes
2020-05-15 15:25:00 +01:00
Dylan Baker 4e9e35f3bd interpreterbase: Allow passing an extra message in feature/deprecation warnings
The intended use it to tell people the new thing to do.
2020-05-14 11:04:51 -07:00
Dylan Baker 85708facae
Merge pull request #6620 from jon-turney/test-output-check
Add a mechanism for validating meson output in tests
2020-05-13 11:32:28 -07:00
Paolo Bonzini 7e15295018 rename unstable-kconfig to unstable-keyval
Discussions in #6524 have shown that there are various possible uses of the
kconfig module and even disagreements in the exact file format between
Python-based kconfiglib and the tools in Linux.  Instead of trying to
reconcile them, just rename the module to something less suggestive and
leave any policy to meson.build files.

In the future it may be possible to add some kind of parsing through
keyword arguments such as bool_true, quoted_strings, etc. and possibly
creation of key-value lists too.  For now, configuration_data objects
provide an easy way to access quoted strings.  Note that Kconfig stores
false as "absent" so it was already necessary to write "x.has_key('abc')"
rather than the more compact "x['abc']".  Therefore, having to use
configuration_data does not make things much more verbose.
2020-05-08 20:56:14 +03:00
Jon Turney f867bfbce0
Add a mechanism for validating meson output in tests
Expected stdout lines must match lines from the actual stdout, in the
same order. Lines with match type 're' are regex matched.

v2:
Ignore comment lines in expected_stdout

v3:
Automatically adjust path separators for location in expected output

v4:
Put expected stdout in test.json, rather than a separate file
2020-04-30 20:41:29 +01:00
Jon Turney f107f9b396
Make colourize_console() a function
Currently, colourize_console is a constant, set at process
initialization.

To allow the actual stdout to be easily compared with the expected when
running tests, we want to allow colourization to be on for the test
driver, but not for the in-process configure done by run_configure,
which has stdout redirected from a tty to a pipe.

v2:
Cache _colorize_console per file object

v3:
Reset cache on setup_console()
2020-04-30 20:11:31 +01:00
Daniel Mensinger 2a586f00b7
tests: Skip hotdoc test if hotdoc is not installed 2020-04-28 17:54:02 +02:00
Daniel Mensinger 4a1f197743
tests: Add support for specifying tool requirements
Adds the `tools` section to `tests.json` to specify requirements
for the tools in the environment. All tests that fail at least
one tool requirements check are skipped.
2020-04-28 17:54:02 +02:00
Dylan Baker a3f39fde1b run_project_tests: Allow matrix tests to skip
Since they generate some tests that are only valid on specific operating
systems.
2020-04-17 10:49:08 -07:00
Dylan Baker d887a8faff run_project_tests: Add an option to put temprorary files in /tmp
I have an NVME drive, I really don't want to be thrashing it with
temporary files. It's also annoying to watch vscode thrash about with
files that are quickly created and deleted. I'd rather put them in /tmp,
so I've added a non-default option to do so.
2020-04-17 10:49:08 -07:00
Jussi Pakkanen 84e216fd64
Merge pull request #6636 from jon-turney/machine-detection-problems
Redetect machines when languages change
2020-03-20 20:35:59 +02:00
Dylan Baker 7470617953 project_tests: Add the option to the test format to mark the language
This is needed when mixing D and C code, as it's possible to end up
witha  combination of linkers and compilres such that C produces pdb
files but D does not.
2020-03-09 16:55:08 -07:00
Dylan Baker fc8fffaf47 run_project_tests: Sort imports
It was impossible to figure out what was coming from where before.
2020-03-09 16:55:08 -07:00
Dylan Baker 6d4df978e2 run_project_tests: fix argument type 2020-03-09 16:55:08 -07:00
Dylan Baker c9ecfc84ff tests: Add pdb files for d tests 2020-03-09 16:55:08 -07:00
Dylan Baker 5ccda6878d project_tests: Add "version" to "shared_lib" and "pdb" types
This allows the harness to apply the version correctly, putting it in the right
place, dropping the right amount of numbers, etc.

pdb taking a version allows it to be more easily copied from the
shared_lib type.
2020-03-09 11:57:26 -07:00