Files
llvm/lldb/source/Core/DynamicLoader.cpp
Tom Yang 66d5f6a605 [lldb] fix parallel module loading deadlock for Linux DYLD (#166480)
Another attempt at resolving the deadlock issue @GeorgeHuyubo discovered
(his previous
[attempt](https://github.com/llvm/llvm-project/pull/160225)).

This change can be summarized as the following:
* Plumb through a boolean flag to force no preload in
`GetOrCreateModules` all the way through to `LoadModuleAtAddress`.
* Parallelize `Module::PreloadSymbols` separately from
`Target::GetOrCreateModule` and its caller `LoadModuleAtAddress` (this
is what avoids the deadlock).

These changes roughly maintain the performance characteristics of the
previous implementation of parallel module loading. Testing on targets
with between 5000 and 14000 modules, I saw similar numbers as before,
often more than 10% faster in the new implementation across multiple
trials for these massive targets. I think it's because we have less lock
contention with this approach.

# The deadlock

See [bt.txt](https://github.com/user-attachments/files/22524471/bt.txt)
for a sample backtrace of LLDB when the deadlock occurs.

As @GeorgeHuyubo explains in his
[PR](https://github.com/llvm/llvm-project/pull/160225), the deadlock
occurs from an ABBA deadlock that happens when a thread context-switches
out of `Module::PreloadSymbols`, goes into `Target::GetOrCreateModule`
for another module, possibly entering this block:
```
      if (!module_sp) {
        // The platform is responsible for finding and caching an appropriate
        // module in the shared module cache.
        if (m_platform_sp) {
          error = m_platform_sp->GetSharedModule(
              module_spec, m_process_sp.get(), module_sp, &search_paths,
              &old_modules, &did_create_module);
        } else {
          error = Status::FromErrorString("no platform is currently set");
        }
      }
```
`Module::PreloadSymbols` holds a module-level mutex, and then
`GetSharedModule` *attempts* to hold the mutex of the global shared
`ModuleList`. So, this thread holds the module mutex, and waits on the
global shared `ModuleList` mutex.

A competing thread may execute `Target::GetOrCreateModule`, enter the
same block as above, grabbing the global shared `ModuleList` mutex.
Then, in `ModuleList::GetSharedModule`, we eventually call
`ModuleList::FindModules` which eventually waits for the `Module` mutex
held by the first thread (via `Module::GetUUID`). Thus, we deadlock.

## Reproducing the deadlock

It might be worth noting that I've never been able to observe this
deadlock issue during live debugging (e.g. launching or attaching to
processes), however we were able to consistently reproduce this issue
with coredumps when using the following settings:
```
(lldb) settings set target.parallel-module-load true
(lldb) settings set target.preload-symbols true
(lldb) settings set symbols.load-on-demand false
(lldb) target create --core /some/core/file/here
# deadlock happens
```

## How this change avoids this deadlock

This change avoids concurrent executions of `Module::PreloadSymbols`
with `Target::GetOrCreateModule` by waiting until after the
`Target::GetOrCreateModule` executions to run `Module::PreloadSymbols`
in parallel. This avoids the ordering of holding a Module lock *then*
the ModuleList lock, as `Target::GetOrCreateModule` executions maintain
the ordering of the shared ModuleList lock first (from what I've read
and tested).

## Why not read-write lock?

Some feedback in https://github.com/llvm/llvm-project/pull/160225 was to
modify mutexes used in these components with read-write locks. This
might be a good idea overall, but I don't think it would *easily*
resolve this specific deadlock. `Module::PreloadSymbols` would probably
need a write lock to Module, so even if we had a read lock in
`Module::GetUUID` we would still contend. Maybe the `ModuleList` lock
could be a read lock that converts to a write lock if it chooses to
update the module, but it seems likely that some thread would try to
update the shared module list and then the write lock would contend
again.

Perhaps with deeper architectural changes, we could fix this issue?

# Other attempts

One downside of this approach (and the former approach of parallel
module loading) is that each DYLD would need to implement this pattern
themselves. With @clayborg's help, I looked at a few other approaches:
* In `Target::GetOrCreateModule`, backgrounding the
`Module::PreloadSymbols` call by adding it directly to the thread pool
via `Debugger::GetThreadPool().async()`. This required adding a lock to
`Module::SetLoadAddress` (probably should be one there already) since
`ObjectFileELF::SetLoadAddress` is not thread-safe (updates sections).
Unfortunately, during execution, this causes the preload symbols to run
synchronously with `Target::GetOrCreateModule`, preventing us from truly
parallelizing the execution.
* In `Module::PreloadSymbols`, backgrounding the `symtab` and `sym_file`
`PreloadSymbols` calls individually, but similar issues as the above.
* Passing a callback function like
https://github.com/swiftlang/llvm-project/pull/10746 instead of the
boolean I use in this change. It's functionally the same change IMO,
with some design tradeoffs:
* Pro: the caller doesn't need to explicitly call
`Module::PreloadSymbols` itself, and can instead call whatever function
is passed into the callback.
* Con: the caller needs to delay the execution of the callback such that
it occurs after the `GetOrCreateModule` logic, otherwise we run into the
same issue. I thought this would be trickier for the caller, requiring
some kinda condition variable or otherwise storing the calls to execute
afterwards.

# Test Plan:
```
ninja check-lldb
```

---------

Co-authored-by: Tom Yang <toyang@fb.com>
2025-11-14 15:58:43 -08:00

391 lines
14 KiB
C++

//===-- DynamicLoader.cpp -------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include "lldb/Target/DynamicLoader.h"
#include "lldb/Core/Debugger.h"
#include "lldb/Core/Module.h"
#include "lldb/Core/ModuleList.h"
#include "lldb/Core/ModuleSpec.h"
#include "lldb/Core/PluginManager.h"
#include "lldb/Core/Progress.h"
#include "lldb/Core/Section.h"
#include "lldb/Symbol/ObjectFile.h"
#include "lldb/Target/MemoryRegionInfo.h"
#include "lldb/Target/Platform.h"
#include "lldb/Target/Process.h"
#include "lldb/Target/Target.h"
#include "lldb/Utility/ConstString.h"
#include "lldb/Utility/LLDBLog.h"
#include "lldb/Utility/Log.h"
#include "lldb/lldb-private-interfaces.h"
#include "llvm/ADT/StringRef.h"
#include <memory>
#include <cassert>
using namespace lldb;
using namespace lldb_private;
DynamicLoader *DynamicLoader::FindPlugin(Process *process,
llvm::StringRef plugin_name) {
DynamicLoaderCreateInstance create_callback = nullptr;
if (!plugin_name.empty()) {
create_callback =
PluginManager::GetDynamicLoaderCreateCallbackForPluginName(plugin_name);
if (create_callback) {
std::unique_ptr<DynamicLoader> instance_up(
create_callback(process, true));
if (instance_up)
return instance_up.release();
}
} else {
for (uint32_t idx = 0;
(create_callback =
PluginManager::GetDynamicLoaderCreateCallbackAtIndex(idx)) !=
nullptr;
++idx) {
std::unique_ptr<DynamicLoader> instance_up(
create_callback(process, false));
if (instance_up)
return instance_up.release();
}
}
return nullptr;
}
DynamicLoader::DynamicLoader(Process *process) : m_process(process) {}
// Accessors to the global setting as to whether to stop at image (shared
// library) loading/unloading.
bool DynamicLoader::GetStopWhenImagesChange() const {
return m_process->GetStopOnSharedLibraryEvents();
}
void DynamicLoader::SetStopWhenImagesChange(bool stop) {
m_process->SetStopOnSharedLibraryEvents(stop);
}
ModuleSP DynamicLoader::GetTargetExecutable() {
Target &target = m_process->GetTarget();
ModuleSP executable = target.GetExecutableModule();
if (executable) {
if (FileSystem::Instance().Exists(executable->GetFileSpec())) {
ModuleSpec module_spec(executable->GetFileSpec(),
executable->GetArchitecture());
auto module_sp = std::make_shared<Module>(module_spec);
// If we're a coredump and we already have a main executable, we don't
// need to reload the module list that target already has
if (!m_process->IsLiveDebugSession()) {
return executable;
}
// Check if the executable has changed and set it to the target
// executable if they differ.
if (module_sp && module_sp->GetUUID().IsValid() &&
executable->GetUUID().IsValid()) {
if (module_sp->GetUUID() != executable->GetUUID())
executable.reset();
} else if (executable->FileHasChanged()) {
executable.reset();
}
if (!executable) {
executable = target.GetOrCreateModule(module_spec, true /* notify */);
if (executable.get() != target.GetExecutableModulePointer()) {
// Don't load dependent images since we are in dyld where we will
// know and find out about all images that are loaded
target.SetExecutableModule(executable, eLoadDependentsNo);
}
}
}
}
return executable;
}
void DynamicLoader::UpdateLoadedSections(ModuleSP module, addr_t link_map_addr,
addr_t base_addr,
bool base_addr_is_offset) {
UpdateLoadedSectionsCommon(module, base_addr, base_addr_is_offset);
}
void DynamicLoader::UpdateLoadedSectionsCommon(ModuleSP module,
addr_t base_addr,
bool base_addr_is_offset) {
bool changed;
module->SetLoadAddress(m_process->GetTarget(), base_addr, base_addr_is_offset,
changed);
}
void DynamicLoader::UnloadSections(const ModuleSP module) {
UnloadSectionsCommon(module);
}
void DynamicLoader::UnloadSectionsCommon(const ModuleSP module) {
Target &target = m_process->GetTarget();
const SectionList *sections = GetSectionListFromModule(module);
assert(sections && "SectionList missing from unloaded module.");
const size_t num_sections = sections->GetSize();
for (size_t i = 0; i < num_sections; ++i) {
SectionSP section_sp(sections->GetSectionAtIndex(i));
target.SetSectionUnloaded(section_sp);
}
}
const SectionList *
DynamicLoader::GetSectionListFromModule(const ModuleSP module) const {
SectionList *sections = nullptr;
if (module) {
ObjectFile *obj_file = module->GetObjectFile();
if (obj_file != nullptr) {
sections = obj_file->GetSectionList();
}
}
return sections;
}
ModuleSP DynamicLoader::FindModuleViaTarget(const FileSpec &file) {
Target &target = m_process->GetTarget();
ModuleSpec module_spec(file, target.GetArchitecture());
if (UUID uuid = m_process->FindModuleUUID(file.GetPath())) {
// Process may be able to augment the module_spec with UUID, e.g. ELF core.
module_spec.GetUUID() = uuid;
}
if (ModuleSP module_sp = target.GetImages().FindFirstModule(module_spec))
return module_sp;
if (ModuleSP module_sp =
target.GetOrCreateModule(module_spec, /*notify=*/false))
return module_sp;
return nullptr;
}
ModuleSP DynamicLoader::LoadModuleAtAddress(const FileSpec &file,
addr_t link_map_addr,
addr_t base_addr,
bool base_addr_is_offset) {
if (ModuleSP module_sp = FindModuleViaTarget(file)) {
UpdateLoadedSections(module_sp, link_map_addr, base_addr,
base_addr_is_offset);
return module_sp;
}
return nullptr;
}
static ModuleSP ReadUnnamedMemoryModule(Process *process, addr_t addr,
llvm::StringRef name) {
char namebuf[80];
if (name.empty()) {
snprintf(namebuf, sizeof(namebuf), "memory-image-0x%" PRIx64, addr);
name = namebuf;
}
return process->ReadModuleFromMemory(FileSpec(name), addr);
}
ModuleSP DynamicLoader::LoadBinaryWithUUIDAndAddress(
Process *process, llvm::StringRef name, UUID uuid, addr_t value,
bool value_is_offset, bool force_symbol_search, bool notify,
bool set_address_in_target, bool allow_memory_image_last_resort) {
ModuleSP memory_module_sp;
ModuleSP module_sp;
PlatformSP platform_sp = process->GetTarget().GetPlatform();
Target &target = process->GetTarget();
Status error;
StreamString prog_str;
if (!name.empty()) {
prog_str << name.str() << " ";
}
if (uuid.IsValid())
prog_str << uuid.GetAsString();
if (value_is_offset == 0 && value != LLDB_INVALID_ADDRESS) {
prog_str << " at 0x";
prog_str.PutHex64(value);
}
if (!uuid.IsValid() && !value_is_offset) {
memory_module_sp = ReadUnnamedMemoryModule(process, value, name);
if (memory_module_sp) {
uuid = memory_module_sp->GetUUID();
if (uuid.IsValid()) {
prog_str << " ";
prog_str << uuid.GetAsString();
}
}
}
ModuleSpec module_spec;
module_spec.SetTarget(target.shared_from_this());
module_spec.GetUUID() = uuid;
FileSpec name_filespec(name);
if (FileSystem::Instance().Exists(name_filespec))
module_spec.GetFileSpec() = name_filespec;
if (uuid.IsValid()) {
Progress progress("Locating binary", prog_str.GetString().str());
// Has lldb already seen a module with this UUID?
// Or have external lookup enabled in DebugSymbols on macOS.
if (!module_sp)
error =
ModuleList::GetSharedModule(module_spec, module_sp, nullptr, nullptr);
// Can lldb's symbol/executable location schemes
// find an executable and symbol file.
if (!module_sp) {
FileSpecList search_paths = Target::GetDefaultDebugFileSearchPaths();
StatisticsMap symbol_locator_map;
module_spec.GetSymbolFileSpec() =
PluginManager::LocateExecutableSymbolFile(module_spec, search_paths,
symbol_locator_map);
ModuleSpec objfile_module_spec =
PluginManager::LocateExecutableObjectFile(module_spec,
symbol_locator_map);
module_spec.GetFileSpec() = objfile_module_spec.GetFileSpec();
if (FileSystem::Instance().Exists(module_spec.GetFileSpec()) &&
FileSystem::Instance().Exists(module_spec.GetSymbolFileSpec())) {
module_sp = std::make_shared<Module>(module_spec);
}
if (module_sp) {
module_sp->GetSymbolLocatorStatistics().merge(symbol_locator_map);
}
}
// If we haven't found a binary, or we don't have a SymbolFile, see
// if there is an external search tool that can find it.
if (!module_sp || !module_sp->GetSymbolFileFileSpec()) {
PluginManager::DownloadObjectAndSymbolFile(module_spec, error,
force_symbol_search);
if (FileSystem::Instance().Exists(module_spec.GetFileSpec())) {
module_sp = std::make_shared<Module>(module_spec);
} else if (force_symbol_search && error.AsCString("") &&
error.AsCString("")[0] != '\0') {
*target.GetDebugger().GetAsyncErrorStream() << error.AsCString();
}
}
// If we only found the executable, create a Module based on that.
if (!module_sp && FileSystem::Instance().Exists(module_spec.GetFileSpec()))
module_sp = std::make_shared<Module>(module_spec);
}
// If we couldn't find the binary anywhere else, as a last resort,
// read it out of memory.
if (allow_memory_image_last_resort && !module_sp.get() &&
value != LLDB_INVALID_ADDRESS && !value_is_offset) {
if (!memory_module_sp)
memory_module_sp = ReadUnnamedMemoryModule(process, value, name);
if (memory_module_sp)
module_sp = memory_module_sp;
}
Log *log = GetLog(LLDBLog::DynamicLoader);
if (module_sp.get()) {
// Ensure the Target has an architecture set in case
// we need it while processing this binary/eh_frame/debug info.
if (!target.GetArchitecture().IsValid())
target.SetArchitecture(module_sp->GetArchitecture());
target.GetImages().AppendIfNeeded(module_sp, false);
bool changed = false;
if (set_address_in_target) {
if (module_sp->GetObjectFile()) {
if (value != LLDB_INVALID_ADDRESS) {
LLDB_LOGF(log,
"DynamicLoader::LoadBinaryWithUUIDAndAddress Loading "
"binary %s UUID %s at %s 0x%" PRIx64,
name.str().c_str(), uuid.GetAsString().c_str(),
value_is_offset ? "offset" : "address", value);
module_sp->SetLoadAddress(target, value, value_is_offset, changed);
} else {
// No address/offset/slide, load the binary at file address,
// offset 0.
LLDB_LOGF(log,
"DynamicLoader::LoadBinaryWithUUIDAndAddress Loading "
"binary %s UUID %s at file address",
name.str().c_str(), uuid.GetAsString().c_str());
module_sp->SetLoadAddress(target, 0, true /* value_is_slide */,
changed);
}
} else {
// In-memory image, load at its true address, offset 0.
LLDB_LOGF(log,
"DynamicLoader::LoadBinaryWithUUIDAndAddress Loading binary "
"%s UUID %s from memory at address 0x%" PRIx64,
name.str().c_str(), uuid.GetAsString().c_str(), value);
module_sp->SetLoadAddress(target, 0, true /* value_is_slide */,
changed);
}
}
if (notify) {
ModuleList added_module;
added_module.Append(module_sp, false);
target.ModulesDidLoad(added_module);
}
} else {
if (force_symbol_search) {
lldb::StreamUP s = target.GetDebugger().GetAsyncErrorStream();
s->Printf("Unable to find file");
if (!name.empty())
s->Printf(" %s", name.str().c_str());
if (uuid.IsValid())
s->Printf(" with UUID %s", uuid.GetAsString().c_str());
if (value != LLDB_INVALID_ADDRESS) {
if (value_is_offset)
s->Printf(" with slide 0x%" PRIx64, value);
else
s->Printf(" at address 0x%" PRIx64, value);
}
s->Printf("\n");
}
LLDB_LOGF(log,
"Unable to find binary %s with UUID %s and load it at "
"%s 0x%" PRIx64,
name.str().c_str(), uuid.GetAsString().c_str(),
value_is_offset ? "offset" : "address", value);
}
return module_sp;
}
int64_t DynamicLoader::ReadUnsignedIntWithSizeInBytes(addr_t addr,
int size_in_bytes) {
Status error;
uint64_t value =
m_process->ReadUnsignedIntegerFromMemory(addr, size_in_bytes, 0, error);
if (error.Fail())
return -1;
else
return (int64_t)value;
}
addr_t DynamicLoader::ReadPointer(addr_t addr) {
Status error;
addr_t value = m_process->ReadPointerFromMemory(addr, error);
if (error.Fail())
return LLDB_INVALID_ADDRESS;
else
return value;
}
void DynamicLoader::LoadOperatingSystemPlugin(bool flush)
{
if (m_process)
m_process->LoadOperatingSystemPlugin(flush);
}