Commit Graph

67 Commits

Author SHA1 Message Date
Eli Schwartz c5e766af75
be more explicit about asyncio usage
`get_event_loop()` would always implicitly create one on demand, but
Python upstream has decided it's a bad/confusing API. Explicitly
starting our own, then setting it, is exactly equivalent other than not
being scheduled for deprecation.
2023-01-04 16:46:52 -05:00
Xavier Claessens 7737398cd3 msubprojects: Fix issues when updating all wraps
When updating all wraps, it is not an error if some have not been
downloaded or some does not have a wrap file.
2022-10-21 17:35:26 +03:00
Xavier Claessens aaabd6224f Make `meson wrap update` command update all wraps in parallel
This moves the implementation into msubprojects because it has all the
infrastructure to update wraps in parallel while keeping "meson wrap"
UX.
2022-10-11 15:54:59 -04:00
Tristan Partin 838367ca60 Fix mypy lint
Unused "type:ignore"
2022-09-27 11:14:19 -04:00
Daniel Carson 004575874f Warn if wrap file changes
Save off the hash of the wrap file when first configuring a subproject.
When reconfiguring a subproject, check the hash of the wrap file
against the stored hash. If they don't match then warn the user.
2022-09-18 22:48:50 -04:00
Eli Schwartz a49cd00d64 treewide: various cleanups to move imports for mypy into typechecking blocks
Along the way, add __future__ annotations where lacking.
2022-06-10 09:15:48 -04:00
Paweł Marczewski 9061c3a52d wrap: Add support for applying a list of patch files
Co-authored-by: Xavier Claessens <xavier.claessens@collabora.com>
2022-06-07 06:43:39 -04:00
Eli Schwartz bb171c2dff
pyupgrade --py37-plus
Some more old style code crept in again.

Additionally, pyupgrade learned to catch more if/elsed code based on the
python version, and delete it.
2022-04-14 18:37:03 -04:00
Eli Schwartz 479a84455f wrap: implement allow-insecure for 'meson wrap' 2022-03-27 18:57:07 -04:00
Eli Schwartz b28e6aead4 wrap: add functionality to specify whether insecure downloads should be used
We have a fallback route in `meson subprojects download` and friends,
which tries to retrieve wrapdb urls via http, if Python was not built
with SSL support.

Stop doing this. Replace it with a command line option to specify that
insecure downloads are wanted, and reference it in the error message if
downloading fails due to SSL issues.
2022-03-27 18:57:07 -04:00
Eli Schwartz 60c893f1c2
flake8: remove some redundant separators 2022-02-16 18:19:13 -05:00
Eli Schwartz 4b351aef26
first pass at migrating to dataclasses
In some cases, init variables that accept None as a sentinel and
immediately overwrite with [], are migrated to dataclass field
factories. \o/

Note: dataclasses by default cannot provide eq methods, as they then
become unhashable. In the future we may wish to opt into declaring them
frozen, instead/additionally.
2022-01-10 18:36:57 -05:00
Tristan Partin b361fc52dd Add typing to msubprojects.py 2021-11-28 07:10:12 -05:00
Eli Schwartz e72da5ff5a msubprojects: implement "packagefiles --save"
Copies packagefiles from the subproject back to the patch_directory for
safe storage and to check into git etc.
2021-10-10 10:17:55 -04:00
Eli Schwartz 8e50ba732e msubprojects: add new subcommand "packagefiles"
This will re-apply the meson.build patch overlay, ensuring it is up to
date. Also take the opportunity offered by this infrastructure to
repatch when performing `update --reset` since internally this will run
`git stash` and thus cause the (possibly locally modified) meson.build
files to disappear.
2021-10-10 10:17:55 -04:00
Eli Schwartz 352380f659 Revert "msubprojects: if fetching the remote fails, gracefully fallback to local copy"
This reverts commit 4568482316.

As it turns out, the rationale for this was completely bogus. This
command doesn't re-apply the patch_directory etc. and in fact there is
no command whatsoever that does this. So, this command does not have two
purposes, and we are not making one of them more robust -- instead it
has one purpose, and we are making it lie about whether it failed.

Instead of trying to freeload off of this command, we will just add
another command to properly apply patch overlays.
2021-10-10 10:17:55 -04:00
Eli Schwartz 99112d6c79
msubprojects: when revision is an available commit, do not fetch updates
It's not even worth bothering to contact the network because it can
never be anything other than what we have already.
2021-09-19 23:34:33 -04:00
Eli Schwartz 4568482316
msubprojects: if fetching the remote fails, gracefully fallback to local copy
This command is useful to e.g. update a cloned subproject which does not
have its packagefiles merged, or which has updated packagefiles. This
does not strictly require internet, so if we can satisfy the ref, simply
log a warning and continue.

This enables a workflow where for network-isolated builds, the
subproject is cloned and moved into place and later on the build
orchestrator runs `meson subprojects update` without a network interface
to initialize the subproject.
2021-09-19 23:34:33 -04:00
Eli Schwartz 82e68e98ad
indent code in preparation for refactoring
Because figuring out what changed on a single line in the middle of a
reindent is pain.
2021-09-19 23:29:26 -04:00
Eli Schwartz a387e8d743
fix traceback while trying to print error message 2021-09-14 15:50:29 -04:00
Andrei Alexeyev 254969b128 msubprojects: make purge print full paths to comply with the tests 2021-07-27 13:11:46 +03:00
Andrei Alexeyev 5d88c45861 msubprojects: do not follow symlinks when purging
Fixes #8891
2021-07-27 13:11:46 +03:00
Xavier Claessens 2379fc72bc msubprojects: catch exceptions and continue with others
If the command fails for one subproject we should still continue with
others. Failed subprojects are reported at the end.

This issue became more problematic when doing parallel tasks because the
reason the command fails was completely hidden by other parallel tasks.
2021-07-21 19:45:37 +02:00
Tristan Partin f21685a833 Delete redirected wrap files in subprojects purge
We need to store the original filename as well as whether the wrap was
redirected in order to properly purge the redirected wrap.
2021-07-03 13:14:20 +03:00
Xavier Claessens 4aaccdb6ba msubprojects: Display progress while tasks are running
Logs are printed only when the task is done to not interleave logs while
running multiple tasks in parallel. That means that nothing is printed
until the task is done and that could take time.
2021-06-25 15:16:55 -04:00
Xavier Claessens 27970b1d85 msubprojects: Add message at the end of purge 2021-06-25 15:16:55 -04:00
Xavier Claessens 2b6fe00e3d msubprojects: Run action on all subprojects in parallel
This moves all the code into a class and call its run() method in a
thread. The class queues all logs to print them at the end to avoid
mixing output of multiple actions.
2021-06-25 15:16:55 -04:00
Xavier Claessens 62c53b834d wrap: Port to v2 protocol
Fixes: #8754.
2021-06-04 15:35:29 -04:00
Tristan Partin bb12587e0b Add subprojects purge wrap-git coverage 2021-04-26 12:06:12 -04:00
Tristan Partin b6d277c140 Add 'subprojects purge' command
This will help facilitate cache busting in certain situations, and
replaces hand-rolled solutions of writing a length command to remove
various files/folders within the subprojects directory.
2021-04-26 16:42:29 +03:00
Xavier Claessens 1ad469342b msubprojects: wrap-file should be re-extracted with --reset
When using --reset we should guarantee that next reconfigure will pick
the latest code. For wrap-file we have no way to know if the revision
changed, so we have to delete the source tree and extract again.

It is unlikely that user has local changes in non-git subprojects, and
--reset is known to be dangerous.
2021-03-28 23:24:01 +03:00
Eli Schwartz 6a0fabc647
mass rewrite of string formatting to use f-strings everywhere
performed by running "pyupgrade --py36-plus" and committing the results
2021-03-04 17:16:11 -05:00
Nirbheek Chauhan 1c631ec8ab msubprojects: Ignore worktrees when checking out
Otherwise it's not possible to share git subprojects via worktrees
when creating a worktree of a git repository that uses meson
subprojects.

The downside is that the user needs to be careful while adding commits
to each tree's index.
2020-12-07 09:02:44 -05:00
Dylan Baker f6672c7a19 use real pathlib module
We added the _pathlib module to work around defeciencies in python 3.5's
implementation, since we now rely on 3.6 lets drop this
2020-11-20 15:08:40 -08:00
Xavier Claessens 6333ee88c1 Merge wraps from subprojects into wraps from main project
wraps from subprojects are now merged into the list of wraps from main
project, so they can be used to download dependencies of dependencies
instead of having to promote wraps manually. If multiple projects
provides the same wrap file, the first one to be configured wins.

This also fix usage of sub-subproject that don't have wrap files. We can
now configure B when its source tree is at
`subprojects/A/subprojects/B/`. This has the implication that we cannot
assume that subproject "foo" is at `self.subproject_dir / 'foo'` any
more.
2020-10-13 17:55:16 -04:00
Xavier Claessens dccff1f2bc msubprojects: Handle wrap-file to wrap-git case 2020-10-13 11:12:49 +00:00
Xavier Claessens 3ade5bbd92 msubprojects: Handle change of URL in wrap-git 2020-10-13 11:12:49 +00:00
Daniel Mensinger 1dfaccfd91 pathlib: Fix resolve() by overriding it in Python 3.5 2020-10-04 10:45:48 +02:00
Xavier Claessens 17f07a53d0 msubprojects: Fix unit tests on Windows 2020-09-18 03:01:15 +00:00
Xavier Claessens da72e6284e msubprojects: Revert back to rev-parse instead of --show-current
It is undocumented in manpage, but --show-current is new in git 2.22
which is too recent for distros we want to support like Ubuntu 18.04.
2020-09-18 03:01:15 +00:00
Xavier Claessens e0cd54a322 msubprojects: Stop trying to guess subproject type
It was done to include them in `meson subprojects foreach` without
--types argument, but it's better to special case missing --types and
include wraps that have type=None too. It was a bad idea because that
was messing them in `meson subprojects update`, now they are ignored by
that command.
2020-09-18 03:01:15 +00:00
Xavier Claessens 6fbf368fde msubprojects: Avoid useless "no local changes to save" message 2020-09-18 03:01:15 +00:00
Xavier Claessens ce347f2f8c msubprojects: Fix checkout of new tag/branch
If revision is a tag that does not exist locally, `git fetch origin
<revision>` won't create it and checkout will fail. Using --refmap
ensures that references exists locally.
2020-09-18 03:01:15 +00:00
Xavier Claessens 49b61e476f msubprojects: Fix --types default value 2020-09-18 03:01:15 +00:00
Xavier Claessens 14c1a6983d msubprojects: Allow comma separated list of types 2020-09-10 11:39:30 -04:00
Xavier Claessens a3ac25b0c3 msubprojects: Fix review comments 2020-09-10 11:39:30 -04:00
Xavier Claessens 552432e507 git: Use Popen_safe to avoid encoding issues 2020-09-10 11:39:30 -04:00
Xavier Claessens d3c06bb023 msubprojects: Rework update command
Besides refactoring code into smaller functions:
- Makes the --rebase behaviour the default for consistency: it was
  already rebasing when current branch and revision are the same, it is
  less confusing to rebase when they are different too.
- Add --reset mode that checkout the new branch and hard reset that
  branch to remote commit. This new mode guarantees that every
  subproject are exactly at the wrap's revision.
- Local changes are always stashed first to avoid any data loss. In the
  worst case scenario the user can always check reflog and stash list to
  rollback.

Fixes: #7526
2020-09-10 11:39:30 -04:00
Xavier Claessens 8cb5e3d7d2 msubprojects: Return failure if some commands failed
If the command fails on some subprojects continue with the rest but
return non-0 code. This is useful for CI scripts to ensure it tests
latest code instead of old cached code in case of network error or
something.
2020-09-10 11:39:30 -04:00
Xavier Claessens 064f7241d8 msubprojects: Stash changes before checkout 2020-09-10 11:39:30 -04:00