mirror of
https://github.com/intel/llvm.git
synced 2026-01-13 11:02:04 +08:00
Another attempt at resolving the deadlock issue @GeorgeHuyubo discovered (his previous [attempt](https://github.com/llvm/llvm-project/pull/160225)). This change can be summarized as the following: * Plumb through a boolean flag to force no preload in `GetOrCreateModules` all the way through to `LoadModuleAtAddress`. * Parallelize `Module::PreloadSymbols` separately from `Target::GetOrCreateModule` and its caller `LoadModuleAtAddress` (this is what avoids the deadlock). These changes roughly maintain the performance characteristics of the previous implementation of parallel module loading. Testing on targets with between 5000 and 14000 modules, I saw similar numbers as before, often more than 10% faster in the new implementation across multiple trials for these massive targets. I think it's because we have less lock contention with this approach. # The deadlock See [bt.txt](https://github.com/user-attachments/files/22524471/bt.txt) for a sample backtrace of LLDB when the deadlock occurs. As @GeorgeHuyubo explains in his [PR](https://github.com/llvm/llvm-project/pull/160225), the deadlock occurs from an ABBA deadlock that happens when a thread context-switches out of `Module::PreloadSymbols`, goes into `Target::GetOrCreateModule` for another module, possibly entering this block: ``` if (!module_sp) { // The platform is responsible for finding and caching an appropriate // module in the shared module cache. if (m_platform_sp) { error = m_platform_sp->GetSharedModule( module_spec, m_process_sp.get(), module_sp, &search_paths, &old_modules, &did_create_module); } else { error = Status::FromErrorString("no platform is currently set"); } } ``` `Module::PreloadSymbols` holds a module-level mutex, and then `GetSharedModule` *attempts* to hold the mutex of the global shared `ModuleList`. So, this thread holds the module mutex, and waits on the global shared `ModuleList` mutex. A competing thread may execute `Target::GetOrCreateModule`, enter the same block as above, grabbing the global shared `ModuleList` mutex. Then, in `ModuleList::GetSharedModule`, we eventually call `ModuleList::FindModules` which eventually waits for the `Module` mutex held by the first thread (via `Module::GetUUID`). Thus, we deadlock. ## Reproducing the deadlock It might be worth noting that I've never been able to observe this deadlock issue during live debugging (e.g. launching or attaching to processes), however we were able to consistently reproduce this issue with coredumps when using the following settings: ``` (lldb) settings set target.parallel-module-load true (lldb) settings set target.preload-symbols true (lldb) settings set symbols.load-on-demand false (lldb) target create --core /some/core/file/here # deadlock happens ``` ## How this change avoids this deadlock This change avoids concurrent executions of `Module::PreloadSymbols` with `Target::GetOrCreateModule` by waiting until after the `Target::GetOrCreateModule` executions to run `Module::PreloadSymbols` in parallel. This avoids the ordering of holding a Module lock *then* the ModuleList lock, as `Target::GetOrCreateModule` executions maintain the ordering of the shared ModuleList lock first (from what I've read and tested). ## Why not read-write lock? Some feedback in https://github.com/llvm/llvm-project/pull/160225 was to modify mutexes used in these components with read-write locks. This might be a good idea overall, but I don't think it would *easily* resolve this specific deadlock. `Module::PreloadSymbols` would probably need a write lock to Module, so even if we had a read lock in `Module::GetUUID` we would still contend. Maybe the `ModuleList` lock could be a read lock that converts to a write lock if it chooses to update the module, but it seems likely that some thread would try to update the shared module list and then the write lock would contend again. Perhaps with deeper architectural changes, we could fix this issue? # Other attempts One downside of this approach (and the former approach of parallel module loading) is that each DYLD would need to implement this pattern themselves. With @clayborg's help, I looked at a few other approaches: * In `Target::GetOrCreateModule`, backgrounding the `Module::PreloadSymbols` call by adding it directly to the thread pool via `Debugger::GetThreadPool().async()`. This required adding a lock to `Module::SetLoadAddress` (probably should be one there already) since `ObjectFileELF::SetLoadAddress` is not thread-safe (updates sections). Unfortunately, during execution, this causes the preload symbols to run synchronously with `Target::GetOrCreateModule`, preventing us from truly parallelizing the execution. * In `Module::PreloadSymbols`, backgrounding the `symtab` and `sym_file` `PreloadSymbols` calls individually, but similar issues as the above. * Passing a callback function like https://github.com/swiftlang/llvm-project/pull/10746 instead of the boolean I use in this change. It's functionally the same change IMO, with some design tradeoffs: * Pro: the caller doesn't need to explicitly call `Module::PreloadSymbols` itself, and can instead call whatever function is passed into the callback. * Con: the caller needs to delay the execution of the callback such that it occurs after the `GetOrCreateModule` logic, otherwise we run into the same issue. I thought this would be trickier for the caller, requiring some kinda condition variable or otherwise storing the calls to execute afterwards. # Test Plan: ``` ninja check-lldb ``` --------- Co-authored-by: Tom Yang <toyang@fb.com>
391 lines
14 KiB
C++
391 lines
14 KiB
C++
//===-- DynamicLoader.cpp -------------------------------------------------===//
|
|
//
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
//
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
#include "lldb/Target/DynamicLoader.h"
|
|
|
|
#include "lldb/Core/Debugger.h"
|
|
#include "lldb/Core/Module.h"
|
|
#include "lldb/Core/ModuleList.h"
|
|
#include "lldb/Core/ModuleSpec.h"
|
|
#include "lldb/Core/PluginManager.h"
|
|
#include "lldb/Core/Progress.h"
|
|
#include "lldb/Core/Section.h"
|
|
#include "lldb/Symbol/ObjectFile.h"
|
|
#include "lldb/Target/MemoryRegionInfo.h"
|
|
#include "lldb/Target/Platform.h"
|
|
#include "lldb/Target/Process.h"
|
|
#include "lldb/Target/Target.h"
|
|
#include "lldb/Utility/ConstString.h"
|
|
#include "lldb/Utility/LLDBLog.h"
|
|
#include "lldb/Utility/Log.h"
|
|
#include "lldb/lldb-private-interfaces.h"
|
|
|
|
#include "llvm/ADT/StringRef.h"
|
|
|
|
#include <memory>
|
|
|
|
#include <cassert>
|
|
|
|
using namespace lldb;
|
|
using namespace lldb_private;
|
|
|
|
DynamicLoader *DynamicLoader::FindPlugin(Process *process,
|
|
llvm::StringRef plugin_name) {
|
|
DynamicLoaderCreateInstance create_callback = nullptr;
|
|
if (!plugin_name.empty()) {
|
|
create_callback =
|
|
PluginManager::GetDynamicLoaderCreateCallbackForPluginName(plugin_name);
|
|
if (create_callback) {
|
|
std::unique_ptr<DynamicLoader> instance_up(
|
|
create_callback(process, true));
|
|
if (instance_up)
|
|
return instance_up.release();
|
|
}
|
|
} else {
|
|
for (uint32_t idx = 0;
|
|
(create_callback =
|
|
PluginManager::GetDynamicLoaderCreateCallbackAtIndex(idx)) !=
|
|
nullptr;
|
|
++idx) {
|
|
std::unique_ptr<DynamicLoader> instance_up(
|
|
create_callback(process, false));
|
|
if (instance_up)
|
|
return instance_up.release();
|
|
}
|
|
}
|
|
return nullptr;
|
|
}
|
|
|
|
DynamicLoader::DynamicLoader(Process *process) : m_process(process) {}
|
|
|
|
// Accessors to the global setting as to whether to stop at image (shared
|
|
// library) loading/unloading.
|
|
|
|
bool DynamicLoader::GetStopWhenImagesChange() const {
|
|
return m_process->GetStopOnSharedLibraryEvents();
|
|
}
|
|
|
|
void DynamicLoader::SetStopWhenImagesChange(bool stop) {
|
|
m_process->SetStopOnSharedLibraryEvents(stop);
|
|
}
|
|
|
|
ModuleSP DynamicLoader::GetTargetExecutable() {
|
|
Target &target = m_process->GetTarget();
|
|
ModuleSP executable = target.GetExecutableModule();
|
|
|
|
if (executable) {
|
|
if (FileSystem::Instance().Exists(executable->GetFileSpec())) {
|
|
ModuleSpec module_spec(executable->GetFileSpec(),
|
|
executable->GetArchitecture());
|
|
auto module_sp = std::make_shared<Module>(module_spec);
|
|
// If we're a coredump and we already have a main executable, we don't
|
|
// need to reload the module list that target already has
|
|
if (!m_process->IsLiveDebugSession()) {
|
|
return executable;
|
|
}
|
|
// Check if the executable has changed and set it to the target
|
|
// executable if they differ.
|
|
if (module_sp && module_sp->GetUUID().IsValid() &&
|
|
executable->GetUUID().IsValid()) {
|
|
if (module_sp->GetUUID() != executable->GetUUID())
|
|
executable.reset();
|
|
} else if (executable->FileHasChanged()) {
|
|
executable.reset();
|
|
}
|
|
|
|
if (!executable) {
|
|
executable = target.GetOrCreateModule(module_spec, true /* notify */);
|
|
if (executable.get() != target.GetExecutableModulePointer()) {
|
|
// Don't load dependent images since we are in dyld where we will
|
|
// know and find out about all images that are loaded
|
|
target.SetExecutableModule(executable, eLoadDependentsNo);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
return executable;
|
|
}
|
|
|
|
void DynamicLoader::UpdateLoadedSections(ModuleSP module, addr_t link_map_addr,
|
|
addr_t base_addr,
|
|
bool base_addr_is_offset) {
|
|
UpdateLoadedSectionsCommon(module, base_addr, base_addr_is_offset);
|
|
}
|
|
|
|
void DynamicLoader::UpdateLoadedSectionsCommon(ModuleSP module,
|
|
addr_t base_addr,
|
|
bool base_addr_is_offset) {
|
|
bool changed;
|
|
module->SetLoadAddress(m_process->GetTarget(), base_addr, base_addr_is_offset,
|
|
changed);
|
|
}
|
|
|
|
void DynamicLoader::UnloadSections(const ModuleSP module) {
|
|
UnloadSectionsCommon(module);
|
|
}
|
|
|
|
void DynamicLoader::UnloadSectionsCommon(const ModuleSP module) {
|
|
Target &target = m_process->GetTarget();
|
|
const SectionList *sections = GetSectionListFromModule(module);
|
|
|
|
assert(sections && "SectionList missing from unloaded module.");
|
|
|
|
const size_t num_sections = sections->GetSize();
|
|
for (size_t i = 0; i < num_sections; ++i) {
|
|
SectionSP section_sp(sections->GetSectionAtIndex(i));
|
|
target.SetSectionUnloaded(section_sp);
|
|
}
|
|
}
|
|
|
|
const SectionList *
|
|
DynamicLoader::GetSectionListFromModule(const ModuleSP module) const {
|
|
SectionList *sections = nullptr;
|
|
if (module) {
|
|
ObjectFile *obj_file = module->GetObjectFile();
|
|
if (obj_file != nullptr) {
|
|
sections = obj_file->GetSectionList();
|
|
}
|
|
}
|
|
return sections;
|
|
}
|
|
|
|
ModuleSP DynamicLoader::FindModuleViaTarget(const FileSpec &file) {
|
|
Target &target = m_process->GetTarget();
|
|
ModuleSpec module_spec(file, target.GetArchitecture());
|
|
if (UUID uuid = m_process->FindModuleUUID(file.GetPath())) {
|
|
// Process may be able to augment the module_spec with UUID, e.g. ELF core.
|
|
module_spec.GetUUID() = uuid;
|
|
}
|
|
|
|
if (ModuleSP module_sp = target.GetImages().FindFirstModule(module_spec))
|
|
return module_sp;
|
|
|
|
if (ModuleSP module_sp =
|
|
target.GetOrCreateModule(module_spec, /*notify=*/false))
|
|
return module_sp;
|
|
|
|
return nullptr;
|
|
}
|
|
|
|
ModuleSP DynamicLoader::LoadModuleAtAddress(const FileSpec &file,
|
|
addr_t link_map_addr,
|
|
addr_t base_addr,
|
|
bool base_addr_is_offset) {
|
|
if (ModuleSP module_sp = FindModuleViaTarget(file)) {
|
|
UpdateLoadedSections(module_sp, link_map_addr, base_addr,
|
|
base_addr_is_offset);
|
|
return module_sp;
|
|
}
|
|
|
|
return nullptr;
|
|
}
|
|
|
|
static ModuleSP ReadUnnamedMemoryModule(Process *process, addr_t addr,
|
|
llvm::StringRef name) {
|
|
char namebuf[80];
|
|
if (name.empty()) {
|
|
snprintf(namebuf, sizeof(namebuf), "memory-image-0x%" PRIx64, addr);
|
|
name = namebuf;
|
|
}
|
|
return process->ReadModuleFromMemory(FileSpec(name), addr);
|
|
}
|
|
|
|
ModuleSP DynamicLoader::LoadBinaryWithUUIDAndAddress(
|
|
Process *process, llvm::StringRef name, UUID uuid, addr_t value,
|
|
bool value_is_offset, bool force_symbol_search, bool notify,
|
|
bool set_address_in_target, bool allow_memory_image_last_resort) {
|
|
ModuleSP memory_module_sp;
|
|
ModuleSP module_sp;
|
|
PlatformSP platform_sp = process->GetTarget().GetPlatform();
|
|
Target &target = process->GetTarget();
|
|
Status error;
|
|
|
|
StreamString prog_str;
|
|
if (!name.empty()) {
|
|
prog_str << name.str() << " ";
|
|
}
|
|
if (uuid.IsValid())
|
|
prog_str << uuid.GetAsString();
|
|
if (value_is_offset == 0 && value != LLDB_INVALID_ADDRESS) {
|
|
prog_str << " at 0x";
|
|
prog_str.PutHex64(value);
|
|
}
|
|
|
|
if (!uuid.IsValid() && !value_is_offset) {
|
|
memory_module_sp = ReadUnnamedMemoryModule(process, value, name);
|
|
|
|
if (memory_module_sp) {
|
|
uuid = memory_module_sp->GetUUID();
|
|
if (uuid.IsValid()) {
|
|
prog_str << " ";
|
|
prog_str << uuid.GetAsString();
|
|
}
|
|
}
|
|
}
|
|
ModuleSpec module_spec;
|
|
module_spec.SetTarget(target.shared_from_this());
|
|
module_spec.GetUUID() = uuid;
|
|
FileSpec name_filespec(name);
|
|
if (FileSystem::Instance().Exists(name_filespec))
|
|
module_spec.GetFileSpec() = name_filespec;
|
|
|
|
if (uuid.IsValid()) {
|
|
Progress progress("Locating binary", prog_str.GetString().str());
|
|
|
|
// Has lldb already seen a module with this UUID?
|
|
// Or have external lookup enabled in DebugSymbols on macOS.
|
|
if (!module_sp)
|
|
error =
|
|
ModuleList::GetSharedModule(module_spec, module_sp, nullptr, nullptr);
|
|
|
|
// Can lldb's symbol/executable location schemes
|
|
// find an executable and symbol file.
|
|
if (!module_sp) {
|
|
FileSpecList search_paths = Target::GetDefaultDebugFileSearchPaths();
|
|
StatisticsMap symbol_locator_map;
|
|
module_spec.GetSymbolFileSpec() =
|
|
PluginManager::LocateExecutableSymbolFile(module_spec, search_paths,
|
|
symbol_locator_map);
|
|
ModuleSpec objfile_module_spec =
|
|
PluginManager::LocateExecutableObjectFile(module_spec,
|
|
symbol_locator_map);
|
|
module_spec.GetFileSpec() = objfile_module_spec.GetFileSpec();
|
|
if (FileSystem::Instance().Exists(module_spec.GetFileSpec()) &&
|
|
FileSystem::Instance().Exists(module_spec.GetSymbolFileSpec())) {
|
|
module_sp = std::make_shared<Module>(module_spec);
|
|
}
|
|
|
|
if (module_sp) {
|
|
module_sp->GetSymbolLocatorStatistics().merge(symbol_locator_map);
|
|
}
|
|
}
|
|
|
|
// If we haven't found a binary, or we don't have a SymbolFile, see
|
|
// if there is an external search tool that can find it.
|
|
if (!module_sp || !module_sp->GetSymbolFileFileSpec()) {
|
|
PluginManager::DownloadObjectAndSymbolFile(module_spec, error,
|
|
force_symbol_search);
|
|
if (FileSystem::Instance().Exists(module_spec.GetFileSpec())) {
|
|
module_sp = std::make_shared<Module>(module_spec);
|
|
} else if (force_symbol_search && error.AsCString("") &&
|
|
error.AsCString("")[0] != '\0') {
|
|
*target.GetDebugger().GetAsyncErrorStream() << error.AsCString();
|
|
}
|
|
}
|
|
|
|
// If we only found the executable, create a Module based on that.
|
|
if (!module_sp && FileSystem::Instance().Exists(module_spec.GetFileSpec()))
|
|
module_sp = std::make_shared<Module>(module_spec);
|
|
}
|
|
|
|
// If we couldn't find the binary anywhere else, as a last resort,
|
|
// read it out of memory.
|
|
if (allow_memory_image_last_resort && !module_sp.get() &&
|
|
value != LLDB_INVALID_ADDRESS && !value_is_offset) {
|
|
if (!memory_module_sp)
|
|
memory_module_sp = ReadUnnamedMemoryModule(process, value, name);
|
|
if (memory_module_sp)
|
|
module_sp = memory_module_sp;
|
|
}
|
|
|
|
Log *log = GetLog(LLDBLog::DynamicLoader);
|
|
if (module_sp.get()) {
|
|
// Ensure the Target has an architecture set in case
|
|
// we need it while processing this binary/eh_frame/debug info.
|
|
if (!target.GetArchitecture().IsValid())
|
|
target.SetArchitecture(module_sp->GetArchitecture());
|
|
target.GetImages().AppendIfNeeded(module_sp, false);
|
|
|
|
bool changed = false;
|
|
if (set_address_in_target) {
|
|
if (module_sp->GetObjectFile()) {
|
|
if (value != LLDB_INVALID_ADDRESS) {
|
|
LLDB_LOGF(log,
|
|
"DynamicLoader::LoadBinaryWithUUIDAndAddress Loading "
|
|
"binary %s UUID %s at %s 0x%" PRIx64,
|
|
name.str().c_str(), uuid.GetAsString().c_str(),
|
|
value_is_offset ? "offset" : "address", value);
|
|
module_sp->SetLoadAddress(target, value, value_is_offset, changed);
|
|
} else {
|
|
// No address/offset/slide, load the binary at file address,
|
|
// offset 0.
|
|
LLDB_LOGF(log,
|
|
"DynamicLoader::LoadBinaryWithUUIDAndAddress Loading "
|
|
"binary %s UUID %s at file address",
|
|
name.str().c_str(), uuid.GetAsString().c_str());
|
|
module_sp->SetLoadAddress(target, 0, true /* value_is_slide */,
|
|
changed);
|
|
}
|
|
} else {
|
|
// In-memory image, load at its true address, offset 0.
|
|
LLDB_LOGF(log,
|
|
"DynamicLoader::LoadBinaryWithUUIDAndAddress Loading binary "
|
|
"%s UUID %s from memory at address 0x%" PRIx64,
|
|
name.str().c_str(), uuid.GetAsString().c_str(), value);
|
|
module_sp->SetLoadAddress(target, 0, true /* value_is_slide */,
|
|
changed);
|
|
}
|
|
}
|
|
|
|
if (notify) {
|
|
ModuleList added_module;
|
|
added_module.Append(module_sp, false);
|
|
target.ModulesDidLoad(added_module);
|
|
}
|
|
} else {
|
|
if (force_symbol_search) {
|
|
lldb::StreamUP s = target.GetDebugger().GetAsyncErrorStream();
|
|
s->Printf("Unable to find file");
|
|
if (!name.empty())
|
|
s->Printf(" %s", name.str().c_str());
|
|
if (uuid.IsValid())
|
|
s->Printf(" with UUID %s", uuid.GetAsString().c_str());
|
|
if (value != LLDB_INVALID_ADDRESS) {
|
|
if (value_is_offset)
|
|
s->Printf(" with slide 0x%" PRIx64, value);
|
|
else
|
|
s->Printf(" at address 0x%" PRIx64, value);
|
|
}
|
|
s->Printf("\n");
|
|
}
|
|
LLDB_LOGF(log,
|
|
"Unable to find binary %s with UUID %s and load it at "
|
|
"%s 0x%" PRIx64,
|
|
name.str().c_str(), uuid.GetAsString().c_str(),
|
|
value_is_offset ? "offset" : "address", value);
|
|
}
|
|
|
|
return module_sp;
|
|
}
|
|
|
|
int64_t DynamicLoader::ReadUnsignedIntWithSizeInBytes(addr_t addr,
|
|
int size_in_bytes) {
|
|
Status error;
|
|
uint64_t value =
|
|
m_process->ReadUnsignedIntegerFromMemory(addr, size_in_bytes, 0, error);
|
|
if (error.Fail())
|
|
return -1;
|
|
else
|
|
return (int64_t)value;
|
|
}
|
|
|
|
addr_t DynamicLoader::ReadPointer(addr_t addr) {
|
|
Status error;
|
|
addr_t value = m_process->ReadPointerFromMemory(addr, error);
|
|
if (error.Fail())
|
|
return LLDB_INVALID_ADDRESS;
|
|
else
|
|
return value;
|
|
}
|
|
|
|
void DynamicLoader::LoadOperatingSystemPlugin(bool flush)
|
|
{
|
|
if (m_process)
|
|
m_process->LoadOperatingSystemPlugin(flush);
|
|
}
|