feature: add new environment variables for compiler cache on Windows

Changes:
- replaced registry keys with environment variables
for cl_cache in OCL
- added compiler cache helpers
- implemented support for new env vars on Windows
- added tests

New env vars mechanism works as follows:
If `PERSISTENT_CACHE` is set,
driver checks if `NEO_CACHE_DIR` is set.
If `NEO_CACHE_DIR` is not set,
driver uses `%LocalAppData%\NEO\neo_compiler_cache`
as `cl_cache` destination folder.
If `NEO_CACHE_DIR` is not set and `%LocalAppData%`
path could not be obtained,
compiler cache is disabled.

In the current Windows implementation,
special characters in the folder path are not supported.

Related-To: NEO-8092
Signed-off-by: Fabian Zwolinski <fabian.zwolinski@intel.com>
This commit is contained in:
Fabian Zwolinski
2023-09-08 11:22:54 +00:00
committed by Compute-Runtime-Automation
parent 5569dac6d0
commit e96dd344c3
12 changed files with 468 additions and 67 deletions

View File

@@ -54,7 +54,7 @@ which improves performance.
#### Official instructions
##### Environment flags (Linux only)
##### Environment flags
NEO_CACHE_PERSISTENT - integer value to enable (1)/disable (0) on-disk binary cache. When enabled
Neo will try to cache and reuse compiled binaries. Default is on.
@@ -68,7 +68,7 @@ NEO_CACHE_MAX_SIZE - Cache eviction is triggered once total size of cached bin
##### How cl_cache works (Linux implementation)
When persistent cache is enabled at first occurance driver create config.file which contains amount directory
When persistent cache is enabled at first occurance driver create config.file which contains the directory
size and is also entry point to caching mechanism.
Each write to disk has following steps:
@@ -116,31 +116,51 @@ reuse previously cached kernel binaries instead of compiling kernels from source
#### Official instructions (implementation pending)
##### Environment flags
NEO_CACHE_PERSISTENT - integer value to enable (1)/disable (0) on-disk binary cache. When enabled
Neo will try to cache and reuse compiled binaries. Default is off.
NEO_CACHE_DIR - path to persistent cache directory. Default values are %LocalAppData%\NEO\neo_compiler_cache
if %LocalAppData% is found. If none of environment
variables are set then on-disk cache is disabled.
NEO_CACHE_MAX_SIZE - Cache eviction is triggered once total size of cached binaries exceeds the value in
bytes (default is 1GB). Set to 0 to disable size-based cache eviction.
##### How cl_cache works (Windows implementation)
When persistent cache is enabled at first occurance driver create config.file which contains the directory
size and is also entry point to caching mechanism.
Each write to disk has following steps:
1. lock config.file (advisor lock)
2. create temporary file
3. write content to file
4. rename temporary file to proper hash name
Reads are unblocked
Eviction mechanism is working as follow:
1. lock config.file (advisor lock)
2. windows system calls will gather all entries created by the driver
3. check last usage time
4. sort files
5. remove least recently used files with 1/3 amount size
#### Legacy approach
##### Windows configuration
To set the new location of cl_cache directory - in the registry `HKEY_LOCAL_MACHINE\SOFTWARE\Intel\IGFX\OCL`:
1. add key `cl_cache_dir`
1. add string value named <path_to_app> to `cl_cache_dir` key
1. set data of added value to desired location of cl_cache
To set the new location of cl_cache directory - add new environment variable:
1. variable name: `cl_cache_dir`
1. variable value: <destination_directory_for_cl_cache>
##### Example:
If application is located in `C:\Program Files\application\app.exe`,
by default cl_cache will be stored in `C:\Program Files\application\cl_cache`.
If the new path should be `C:\Users\USER\Documents\application\cl_cache`,
to subkey `HKEY_LOCAL_MACHINE\SOFTWARE\Intel\IGFX\OCL\cl_cache_dir`
add string value named `C:\Program Files\application\app.exe`
with data `C:\Users\USER\Documents\application\cl_cache`.
e.g.
string value : `HKEY_LOCAL_MACHINE\SOFTWARE\Intel\IGFX\OCL\cl_cache_dir\C:\Program Files\application\app.exe`
data : `C:\Users\USER\Documents\application\cl_cache`
Neo will look for string value (REG_SZ) `C:\Program Files\application\app.exe`
in key `HKEY_LOCAL_MACHINE\SOFTWARE\Intel\IGFX\OCL\cl_cache_dir`.
Data of this string value will be used as new cl_cache dump directory for this specific application.
If the new path should be `C:\Users\USER\Documents\application\cl_cache`, create a new environment variable named `cl_cache_dir` with the value `C:\Users\USER\Documents\application\cl_cache`.
##### What are the known limitations of cl_cache for Windows?

View File

@@ -9,11 +9,10 @@
#include "shared/source/compiler_interface/os_compiler_cache_helper.h"
#include "shared/source/helpers/constants.h"
#include "shared/source/os_interface/debug_env_reader.h"
#include "shared/source/os_interface/sys_calls_common.h"
#include "shared/source/utilities/debug_settings_reader.h"
#include "opencl/source/os_interface/ocl_reg_path.h"
#include "config.h"
#include "os_inc.h"
@@ -27,18 +26,15 @@ std::string ClCacheDir = "cl_cache_dir";
CompilerCacheConfig getDefaultCompilerCacheConfig() {
CompilerCacheConfig ret;
NEO::EnvironmentVariableReader envReader;
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, oclRegPath));
auto cachePersistentKey = oclRegPath + NeoCachePersistent;
if (settingsReader->getSetting(settingsReader->appSpecificLocation(cachePersistentKey), defaultCacheEnabled()) != 0) {
if (envReader.getSetting(NeoCachePersistent.c_str(), defaultCacheEnabled()) != 0) {
ret.enabled = true;
std::string emptyString = "";
ret.cacheDir = settingsReader->getSetting(settingsReader->appSpecificLocation(NeoCacheDir), emptyString);
ret.cacheDir = envReader.getSetting(NeoCacheDir.c_str(), emptyString);
if (ret.cacheDir.empty()) {
if (!checkDefaultCacheDirSettings(ret.cacheDir, settingsReader.get())) {
if (!checkDefaultCacheDirSettings(ret.cacheDir, envReader)) {
ret.enabled = false;
return ret;
}
@@ -51,7 +47,7 @@ CompilerCacheConfig getDefaultCompilerCacheConfig() {
}
ret.cacheFileExtension = ".cl_cache";
ret.cacheSize = static_cast<size_t>(settingsReader->getSetting(settingsReader->appSpecificLocation(NeoCacheMaxSize), static_cast<int64_t>(MemoryConstants::gigaByte)));
ret.cacheSize = static_cast<size_t>(envReader.getSetting(NeoCacheMaxSize.c_str(), static_cast<int64_t>(MemoryConstants::gigaByte)));
if (ret.cacheSize == 0u) {
ret.cacheSize = std::numeric_limits<size_t>::max();
@@ -60,7 +56,7 @@ CompilerCacheConfig getDefaultCompilerCacheConfig() {
return ret;
}
ret.cacheDir = settingsReader->getSetting(settingsReader->appSpecificLocation(ClCacheDir), static_cast<std::string>(CL_CACHE_LOCATION));
ret.cacheDir = envReader.getSetting(ClCacheDir.c_str(), static_cast<std::string>(CL_CACHE_LOCATION));
if (NEO::SysCalls::pathExists(ret.cacheDir)) {
ret.enabled = true;

View File

@@ -7,30 +7,264 @@
#include "shared/source/compiler_interface/default_cache_config.h"
#include "shared/source/os_interface/sys_calls_common.h"
#include "shared/source/os_interface/windows/sys_calls.h"
#include "shared/source/utilities/debug_settings_reader.h"
#include "shared/test/common/helpers/variable_backup.h"
#include "shared/test/common/mocks/mock_io_functions.h"
#include "shared/test/common/test_macros/test.h"
namespace NEO {
namespace SysCalls {
extern DWORD getLastErrorResult;
extern bool pathExistsMock;
extern const size_t pathExistsPathsSize;
extern std::string pathExistsPaths[];
extern size_t createDirectoryACalled;
extern BOOL createDirectoryAResult;
extern HRESULT shGetKnownFolderPathResult;
extern const size_t shGetKnownFolderSetPathSize;
extern wchar_t shGetKnownFolderSetPath[];
} // namespace SysCalls
struct CompilerCacheTest : public ::testing::Test {
CompilerCacheTest()
: mockableEnvValuesBackup(&IoFunctions::mockableEnvValues, &mockableEnvs),
getLastErrorResultBackup(&SysCalls::getLastErrorResult),
shGetKnownFolderPathResultBackup(&SysCalls::shGetKnownFolderPathResult),
createDirectoryACalledBackup(&SysCalls::createDirectoryACalled),
createDirectoryAResultBackup(&SysCalls::createDirectoryAResult) {}
void SetUp() override {
SysCalls::createDirectoryACalled = 0u;
}
TEST(CompilerCache, GivenDefaultClCacheConfigWithPathExistsThenValuesAreProperlyPopulated) {
void TearDown() override {
std::wmemset(SysCalls::shGetKnownFolderSetPath, 0, SysCalls::shGetKnownFolderSetPathSize);
for (size_t i = 0; i < SysCalls::pathExistsPathsSize; i++) {
std::string().swap(SysCalls::pathExistsPaths[i]);
}
}
protected:
VariableBackup<std::unordered_map<std::string, std::string> *> mockableEnvValuesBackup;
std::unordered_map<std::string, std::string> mockableEnvs;
VariableBackup<DWORD> getLastErrorResultBackup;
VariableBackup<HRESULT> shGetKnownFolderPathResultBackup;
VariableBackup<size_t> createDirectoryACalledBackup;
VariableBackup<BOOL> createDirectoryAResultBackup;
};
TEST_F(CompilerCacheTest, GivenDefaultClCacheConfigWithPathExistsThenValuesAreProperlyPopulated) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "0";
bool pathExistsMock = true;
VariableBackup<bool> pathExistsMockBackup(&NEO::SysCalls::pathExistsMock, pathExistsMock);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
EXPECT_STREQ("cl_cache", cacheConfig.cacheDir.c_str());
EXPECT_STREQ(".cl_cache", cacheConfig.cacheFileExtension.c_str());
EXPECT_TRUE(cacheConfig.enabled);
}
TEST(CompilerCache, GivenDefaultClCacheConfigWithNonExistingPathThenValuesAreProperlyPopulated) {
TEST_F(CompilerCacheTest, GivenDefaultClCacheConfigWithNonExistingPathThenValuesAreProperlyPopulated) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "0";
bool pathExistsMock = false;
VariableBackup<bool> pathExistsMockBackup(&NEO::SysCalls::pathExistsMock, pathExistsMock);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
EXPECT_STREQ("cl_cache", cacheConfig.cacheDir.c_str());
EXPECT_STREQ(".cl_cache", cacheConfig.cacheFileExtension.c_str());
EXPECT_FALSE(cacheConfig.enabled);
}
TEST_F(CompilerCacheTest, GivenAllEnvVarWhenProperlySetThenCorrectConfigIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
mockableEnvs["NEO_CACHE_DIR"] = "ult\\directory\\";
bool pathExistsMock = true;
VariableBackup<bool> pathExistsMockBackup(&NEO::SysCalls::pathExistsMock, pathExistsMock);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
EXPECT_TRUE(cacheConfig.enabled);
EXPECT_EQ(cacheConfig.cacheFileExtension, ".cl_cache");
EXPECT_EQ(cacheConfig.cacheSize, 22u);
EXPECT_EQ(cacheConfig.cacheDir, "ult\\directory\\");
}
TEST_F(CompilerCacheTest, GivenNonExistingPathWhenGetCompilerCacheConfigThenConfigWithDisabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
mockableEnvs["NEO_CACHE_DIR"] = "ult\\directory\\";
bool pathExistsMock = false;
VariableBackup<bool> pathExistsMockBackup(&NEO::SysCalls::pathExistsMock, pathExistsMock);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
EXPECT_FALSE(cacheConfig.enabled);
EXPECT_TRUE(cacheConfig.cacheDir.empty());
}
TEST_F(CompilerCacheTest, GivenLocalAppDataCachePathSetWhenGetCompilerCacheConfigThenConfigWithEnabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
SysCalls::pathExistsPaths[0] = "C:\\Users\\user1\\AppData\\Local\\NEO";
SysCalls::pathExistsPaths[1] = "C:\\Users\\user1\\AppData\\Local\\NEO\\neo_compiler_cache";
SysCalls::shGetKnownFolderPathResult = S_OK;
const wchar_t *localAppDataPath = L"C:\\Users\\user1\\AppData\\Local";
wcscpy(SysCalls::shGetKnownFolderSetPath, localAppDataPath);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
const std::string expectedCacheDirPath = "C:\\Users\\user1\\AppData\\Local\\NEO\\neo_compiler_cache";
EXPECT_TRUE(cacheConfig.enabled);
EXPECT_EQ(cacheConfig.cacheFileExtension, ".cl_cache");
EXPECT_EQ(cacheConfig.cacheSize, 22u);
EXPECT_EQ(cacheConfig.cacheDir, expectedCacheDirPath);
}
TEST_F(CompilerCacheTest, GivenNeoCacheDirNotSetAndLocalAppDataCachePathNotSetWhenGetCompilerCacheConfigThenConfigWithDisabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
SysCalls::shGetKnownFolderPathResult = S_FALSE;
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
EXPECT_FALSE(cacheConfig.enabled);
EXPECT_TRUE(cacheConfig.cacheDir.empty());
}
TEST_F(CompilerCacheTest, GivenLocalAppDataSetAndNonExistingNeoDirectoryWhenGetCompilerCacheConfigThenNeoDirectoryIsCreatedAndConfigWithEnabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
SysCalls::pathExistsPaths[0] = "C:\\Users\\user1\\AppData\\Local\\NEO\\neo_compiler_cache";
SysCalls::shGetKnownFolderPathResult = S_OK;
const wchar_t *localAppDataPath = L"C:\\Users\\user1\\AppData\\Local";
wcscpy(SysCalls::shGetKnownFolderSetPath, localAppDataPath);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
const std::string expectedCacheDirPath = "C:\\Users\\user1\\AppData\\Local\\NEO\\neo_compiler_cache";
EXPECT_TRUE(cacheConfig.enabled);
EXPECT_EQ(cacheConfig.cacheFileExtension, ".cl_cache");
EXPECT_EQ(cacheConfig.cacheSize, 22u);
EXPECT_EQ(cacheConfig.cacheDir, expectedCacheDirPath);
EXPECT_EQ(1u, SysCalls::createDirectoryACalled);
}
TEST_F(CompilerCacheTest, GivenLocalAppDataSetAndNonExistingNeoDirectoryWhenGetCompilerCacheConfigAndNeoDirCreationFailsThenConfigWithDisabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
SysCalls::pathExistsPaths[0] = "C:\\Users\\user1\\AppData\\Local\\NEO\\neo_compiler_cache";
SysCalls::shGetKnownFolderPathResult = S_OK;
SysCalls::createDirectoryAResult = FALSE;
const wchar_t *localAppDataPath = L"C:\\Users\\user1\\AppData\\Local";
wcscpy(SysCalls::shGetKnownFolderSetPath, localAppDataPath);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
EXPECT_FALSE(cacheConfig.enabled);
EXPECT_TRUE(cacheConfig.cacheDir.empty());
EXPECT_EQ(1u, SysCalls::createDirectoryACalled);
}
TEST_F(CompilerCacheTest, GivenLocalAppDataSetAndNonExistingNeoCompilerCacheDirectoryWhenGetCompilerCacheConfigThenNeoCompilerCacheDirectoryIsCreatedAndConfigWithEnabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
SysCalls::pathExistsPaths[0] = "C:\\Users\\user1\\AppData\\Local\\NEO";
SysCalls::shGetKnownFolderPathResult = S_OK;
const wchar_t *localAppDataPath = L"C:\\Users\\user1\\AppData\\Local";
wcscpy(SysCalls::shGetKnownFolderSetPath, localAppDataPath);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
const std::string expectedCacheDirPath = "C:\\Users\\user1\\AppData\\Local\\NEO\\neo_compiler_cache";
EXPECT_TRUE(cacheConfig.enabled);
EXPECT_EQ(cacheConfig.cacheFileExtension, ".cl_cache");
EXPECT_EQ(cacheConfig.cacheSize, 22u);
EXPECT_EQ(cacheConfig.cacheDir, expectedCacheDirPath);
EXPECT_EQ(1u, SysCalls::createDirectoryACalled);
}
TEST_F(CompilerCacheTest, GivenLocalAppDataSetAndNonExistingNeoCompilerCacheDirectoryWhenGetCompilerCacheConfigAndDirectoryCreationFailsThenConfigWithDisabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
SysCalls::pathExistsPaths[0] = "C:\\Users\\user1\\AppData\\Local\\NEO";
SysCalls::shGetKnownFolderPathResult = S_OK;
SysCalls::createDirectoryAResult = FALSE;
SysCalls::getLastErrorResult = ERROR_ALREADY_EXISTS + 1;
const wchar_t *localAppDataPath = L"C:\\Users\\user1\\AppData\\Local";
wcscpy(SysCalls::shGetKnownFolderSetPath, localAppDataPath);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
EXPECT_FALSE(cacheConfig.enabled);
EXPECT_TRUE(cacheConfig.cacheDir.empty());
EXPECT_EQ(1u, SysCalls::createDirectoryACalled);
}
TEST_F(CompilerCacheTest, GivenLocalAppDataSetWhenGetCompilerCacheConfigAndNeoCompilerCacheDirectoryAlreadyExistsThenConfigWithEnabledCacheIsReturned) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "22";
SysCalls::pathExistsPaths[0] = "C:\\Users\\user1\\AppData\\Local\\NEO";
SysCalls::shGetKnownFolderPathResult = S_OK;
SysCalls::createDirectoryAResult = FALSE;
SysCalls::getLastErrorResult = ERROR_ALREADY_EXISTS;
const wchar_t *localAppDataPath = L"C:\\Users\\user1\\AppData\\Local";
wcscpy(SysCalls::shGetKnownFolderSetPath, localAppDataPath);
auto cacheConfig = NEO::getDefaultCompilerCacheConfig();
const std::string expectedCacheDirPath = "C:\\Users\\user1\\AppData\\Local\\NEO\\neo_compiler_cache";
EXPECT_TRUE(cacheConfig.enabled);
EXPECT_EQ(cacheConfig.cacheFileExtension, ".cl_cache");
EXPECT_EQ(cacheConfig.cacheSize, 22u);
EXPECT_EQ(cacheConfig.cacheDir, expectedCacheDirPath);
EXPECT_EQ(1u, SysCalls::createDirectoryACalled);
}
TEST_F(CompilerCacheTest, GivenCacheMaxSizeSetTo0WhenGetDefaultConfigThenCacheSizeIsSetToMaxSize) {
mockableEnvs["NEO_CACHE_PERSISTENT"] = "1";
mockableEnvs["NEO_CACHE_MAX_SIZE"] = "0";
mockableEnvs["NEO_CACHE_DIR"] = "ult\\directory\\";
SysCalls::pathExistsPaths[0] = "ult\\directory\\";
auto cacheConfig = getDefaultCompilerCacheConfig();
EXPECT_TRUE(cacheConfig.enabled);
EXPECT_EQ(cacheConfig.cacheFileExtension, ".cl_cache");
EXPECT_EQ(cacheConfig.cacheSize, std::numeric_limits<size_t>::max());
EXPECT_EQ(cacheConfig.cacheDir, "ult\\directory\\");
}
} // namespace NEO

View File

@@ -8,8 +8,8 @@
#include "shared/source/compiler_interface/os_compiler_cache_helper.h"
#include "shared/source/helpers/path.h"
#include "shared/source/os_interface/debug_env_reader.h"
#include "shared/source/os_interface/linux/sys_calls.h"
#include "shared/source/utilities/debug_settings_reader.h"
#include "shared/source/utilities/io_functions.h"
#include "os_inc.h"
@@ -44,12 +44,12 @@ bool createCompilerCachePath(std::string &cacheDir) {
return false;
}
bool checkDefaultCacheDirSettings(std::string &cacheDir, SettingsReader *reader) {
bool checkDefaultCacheDirSettings(std::string &cacheDir, NEO::EnvironmentVariableReader &reader) {
std::string emptyString = "";
cacheDir = reader->getSetting(reader->appSpecificLocation("XDG_CACHE_HOME"), emptyString);
cacheDir = reader.getSetting("XDG_CACHE_HOME", emptyString);
if (cacheDir.empty()) {
cacheDir = reader->getSetting(reader->appSpecificLocation("HOME"), emptyString);
cacheDir = reader.getSetting("HOME", emptyString);
if (cacheDir.empty()) {
return false;
}
@@ -73,7 +73,7 @@ bool checkDefaultCacheDirSettings(std::string &cacheDir, SettingsReader *reader)
time_t getFileModificationTime(const std::string &path) {
struct stat st;
if (NEO::SysCalls::stat(path, &st) == 0) {
return st.st_mtim.tv_sec;
return st.st_mtime;
}
return 0;
}

View File

@@ -9,9 +9,9 @@
#include <string>
namespace NEO {
class SettingsReader;
class EnvironmentVariableReader;
int64_t defaultCacheEnabled();
bool checkDefaultCacheDirSettings(std::string &cacheDir, SettingsReader *reader);
bool checkDefaultCacheDirSettings(std::string &cacheDir, NEO::EnvironmentVariableReader &reader);
time_t getFileModificationTime(const std::string &path);
size_t getFileSize(const std::string &path);
} // namespace NEO

View File

@@ -7,17 +7,96 @@
#include "shared/source/compiler_interface/os_compiler_cache_helper.h"
#include "shared/source/helpers/path.h"
#include "shared/source/os_interface/windows/sys_calls.h"
#include "shared/source/utilities/debug_settings_reader.h"
#include <ShlObj.h>
#include <algorithm>
namespace NEO {
int64_t defaultCacheEnabled() {
return 0l;
}
bool checkDefaultCacheDirSettings(std::string &cacheDir, SettingsReader *reader) {
std::string getKnownFolderPath(REFKNOWNFOLDERID rfid) {
PWSTR path = nullptr;
auto result = SysCalls::shGetKnownFolderPath(rfid, 0, nullptr, &path);
if (result != S_OK) {
SysCalls::coTaskMemFree(path);
return std::string();
}
std::wstring temp(path);
SysCalls::coTaskMemFree(path);
std::string ret(temp.length(), 0);
std::transform(temp.begin(), temp.end(), ret.begin(), [](wchar_t c) {
return static_cast<char>(c);
});
return ret;
}
bool createCompilerCachePath(std::string &cacheDir) {
if (NEO::SysCalls::pathExists(cacheDir)) {
cacheDir = joinPath(cacheDir, "neo_compiler_cache");
if (NEO::SysCalls::pathExists(cacheDir)) {
return true;
}
auto result = SysCalls::createDirectoryA(cacheDir.c_str(), NULL);
if (result) {
return true;
}
if (SysCalls::getLastError() == ERROR_ALREADY_EXISTS) {
return true;
}
}
cacheDir = "";
return false;
}
bool checkDefaultCacheDirSettings(std::string &cacheDir, NEO::EnvironmentVariableReader &reader) {
cacheDir = getKnownFolderPath(FOLDERID_LocalAppData);
if (cacheDir.empty()) {
return false;
}
cacheDir = joinPath(cacheDir, "NEO\\");
if (!SysCalls::pathExists(cacheDir)) {
SysCalls::createDirectoryA(cacheDir.c_str(), NULL);
}
if (NEO::SysCalls::pathExists(cacheDir)) {
return createCompilerCachePath(cacheDir);
}
cacheDir = "";
return false;
}
time_t getFileModificationTime(const std::string &path) {
WIN32_FIND_DATAA ffd;
HANDLE hFind = INVALID_HANDLE_VALUE;
hFind = SysCalls::findFirstFileA(path.c_str(), &ffd);
if (hFind == INVALID_HANDLE_VALUE) {
return 0;
}
ULARGE_INTEGER uli;
uli.LowPart = ffd.ftLastWriteTime.dwLowDateTime;
uli.HighPart = ffd.ftLastWriteTime.dwHighDateTime;
return uli.QuadPart;
}
size_t getFileSize(const std::string &path) {
return 0;
WIN32_FIND_DATAA ffd;
HANDLE hFind = INVALID_HANDLE_VALUE;
hFind = SysCalls::findFirstFileA(path.c_str(), &ffd);
if (hFind == INVALID_HANDLE_VALUE) {
return 0u;
}
return static_cast<size_t>((ffd.nFileSizeHigh * (MAXDWORD + 1)) + ffd.nFileSizeLow);
}
} // namespace NEO

View File

@@ -88,6 +88,10 @@ BOOL getOverlappedResult(HANDLE hFile, LPOVERLAPPED lpOverlapped, LPDWORD lpNumb
return GetOverlappedResult(hFile, lpOverlapped, lpNumberOfBytesTransferred, bWait);
}
BOOL createDirectoryA(LPCSTR lpPathName, LPSECURITY_ATTRIBUTES lpSecurityAttributes) {
return CreateDirectoryA(lpPathName, lpSecurityAttributes);
}
HANDLE createFileA(LPCSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode, LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile) {
return CreateFileA(lpFileName, dwDesiredAccess, dwShareMode, lpSecurityAttributes, dwCreationDisposition, dwFlagsAndAttributes, hTemplateFile);
}
@@ -96,6 +100,10 @@ BOOL deleteFileA(LPCSTR lpFileName) {
return DeleteFileA(lpFileName);
}
HRESULT shGetKnownFolderPath(REFKNOWNFOLDERID rfid, DWORD dwFlags, HANDLE hToken, PWSTR *ppszPat) {
return SHGetKnownFolderPath(rfid, dwFlags, hToken, ppszPat);
}
BOOL readFileEx(HANDLE hFile, LPVOID lpBuffer, DWORD nNumberOfBytesToRead, LPOVERLAPPED lpOverlapped, LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine) {
return ReadFileEx(hFile, lpBuffer, nNumberOfBytesToRead, lpOverlapped, lpCompletionRoutine);
}
@@ -120,6 +128,10 @@ DWORD getFileAttributesA(LPCSTR lpFileName) {
return GetFileAttributesA(lpFileName);
}
void coTaskMemFree(LPVOID pv) {
CoTaskMemFree(pv);
}
LSTATUS regOpenKeyExA(HKEY hKey, LPCSTR lpSubKey, DWORD ulOptions, REGSAM samDesired, PHKEY phkResult) {
return RegOpenKeyExA(hKey, lpSubKey, ulOptions, samDesired, phkResult);
}

View File

@@ -24,8 +24,10 @@ BOOL moveFileExA(LPCSTR lpExistingFileName, LPCSTR lpNewFileName, DWORD dwFlags)
BOOL lockFileEx(HANDLE hFile, DWORD dwFlags, DWORD dwReserved, DWORD nNumberOfBytesToLockLow, DWORD nNumberOfBytesToLockHigh, LPOVERLAPPED lpOverlapped);
BOOL unlockFileEx(HANDLE hFile, DWORD dwReserved, DWORD nNumberOfBytesToLockLow, DWORD nNumberOfBytesToLockHigh, LPOVERLAPPED lpOverlapped);
BOOL getOverlappedResult(HANDLE hFile, LPOVERLAPPED lpOverlapped, LPDWORD lpNumberOfBytesTransferred, BOOL bWait);
BOOL createDirectoryA(LPCSTR lpPathName, LPSECURITY_ATTRIBUTES lpSecurityAttributes);
HANDLE createFileA(LPCSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode, LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile);
BOOL deleteFileA(LPCSTR lpFileName);
HRESULT shGetKnownFolderPath(REFKNOWNFOLDERID rfid, DWORD dwFlags, HANDLE hToken, PWSTR *ppszPat);
BOOL readFileEx(HANDLE hFile, LPVOID lpBuffer, DWORD nNumberOfBytesToRead, LPOVERLAPPED lpOverlapped, LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine);
BOOL writeFile(HANDLE hFile, LPCVOID lpBuffer, DWORD nNumberOfBytesToWrite, LPDWORD lpNumberOfBytesWritten, LPOVERLAPPED lpOverlapped);
HANDLE findFirstFileA(LPCSTR lpFileName, LPWIN32_FIND_DATAA lpFindFileData);
@@ -33,6 +35,8 @@ BOOL findNextFileA(HANDLE hFindFile, LPWIN32_FIND_DATAA lpFindFileData);
BOOL findClose(HANDLE hFindFile);
DWORD getFileAttributesA(LPCSTR lpFileName);
void coTaskMemFree(LPVOID pv);
LSTATUS regOpenKeyExA(HKEY hKey, LPCSTR lpSubKey, DWORD ulOptions, REGSAM samDesired, PHKEY phkResult);
LSTATUS regQueryValueExA(HKEY hKey, LPCSTR lpValueName, LPDWORD lpReserved, LPDWORD lpType, LPBYTE lpData, LPDWORD lpcbData);

View File

@@ -1,5 +1,5 @@
/*
* Copyright (C) 2018-2021 Intel Corporation
* Copyright (C) 2018-2023 Intel Corporation
*
* SPDX-License-Identifier: MIT
*
@@ -9,6 +9,7 @@
#if _WIN32
#include <windows.h>
#include <ShlObj.h>
#include <winternl.h>
#pragma warning(push)

View File

@@ -9,6 +9,8 @@
#include "shared/test/common/os_interface/windows/mock_sys_calls.h"
#include "os_inc.h"
#include <cstdint>
#include <vector>
@@ -36,6 +38,8 @@ const HKEY validHkey = reinterpret_cast<HKEY>(0);
bool getNumThreadsCalled = false;
bool mmapAllowExtendedPointers = false;
bool pathExistsMock = false;
extern const size_t pathExistsPathsSize = 5;
std::string pathExistsPaths[pathExistsPathsSize];
const char *driverStorePath = nullptr;
size_t closeHandleCalled = 0u;
@@ -49,6 +53,9 @@ BOOL lockFileExResult = TRUE;
size_t unlockFileExCalled = 0u;
BOOL unlockFileExResult = TRUE;
size_t createDirectoryACalled = 0u;
BOOL createDirectoryAResult = TRUE;
size_t createFileACalled = 0u;
extern const size_t createFileAResultsCount = 4;
HANDLE createFileAResults[createFileAResultsCount] = {nullptr, nullptr, nullptr, nullptr};
@@ -57,6 +64,10 @@ size_t deleteFileACalled = 0u;
const size_t deleteFilesCount = 4;
std::string deleteFiles[deleteFilesCount];
HRESULT shGetKnownFolderPathResult = 0;
extern const size_t shGetKnownFolderSetPathSize = 50;
wchar_t shGetKnownFolderSetPath[shGetKnownFolderSetPathSize];
bool callBaseReadFileEx = true;
BOOL readFileExResult = TRUE;
size_t readFileExCalled = 0u;
@@ -78,6 +89,25 @@ size_t getFileAttributesCalled = 0u;
DWORD getFileAttributesResult = TRUE;
bool pathExists(const std::string &path) {
std::string tempP1 = path;
if (!path.empty() && path.back() == PATH_SEPARATOR) {
tempP1.pop_back();
}
for (const auto &p : pathExistsPaths) {
if (p.empty())
continue;
std::string tempP2 = p;
if (tempP2.back() == PATH_SEPARATOR) {
tempP2.pop_back();
}
if (tempP1 == tempP2) {
return true;
}
}
return pathExistsMock;
}
@@ -145,6 +175,19 @@ BOOL getOverlappedResult(HANDLE hFile, LPOVERLAPPED lpOverlapped, LPDWORD lpNumb
return TRUE;
}
BOOL createDirectoryA(LPCSTR lpPathName, LPSECURITY_ATTRIBUTES lpSecurityAttributes) {
createDirectoryACalled++;
if (createDirectoryAResult) {
for (size_t i = 0; i < pathExistsPathsSize; i++) {
if (pathExistsPaths[i].empty()) {
pathExistsPaths[i] = lpPathName;
break;
}
}
}
return createDirectoryAResult;
}
HANDLE createFileA(LPCSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode, LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile) {
HANDLE retVal = nullptr;
@@ -164,6 +207,11 @@ BOOL deleteFileA(LPCSTR lpFileName) {
return TRUE;
}
HRESULT shGetKnownFolderPath(REFKNOWNFOLDERID rfid, DWORD dwFlags, HANDLE hToken, PWSTR *ppszPat) {
*ppszPat = shGetKnownFolderSetPath;
return shGetKnownFolderPathResult;
}
BOOL readFileEx(HANDLE hFile, LPVOID lpBuffer, DWORD nNumberOfBytesToRead, LPOVERLAPPED lpOverlapped, LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine) {
readFileExCalled++;
if (callBaseReadFileEx) {
@@ -210,6 +258,10 @@ DWORD getFileAttributesA(LPCSTR lpFileName) {
return getFileAttributesResult;
}
void coTaskMemFree(LPVOID pv) {
return;
}
LSTATUS regOpenKeyExA(HKEY hKey, LPCSTR lpSubKey, DWORD ulOptions, REGSAM samDesired, PHKEY phkResult) {
if (regOpenKeySuccessCount > 0) {
regOpenKeySuccessCount--;

View File

@@ -13,7 +13,7 @@
#include "shared/source/helpers/hash.h"
#include "shared/source/helpers/hw_info.h"
#include "shared/source/helpers/string.h"
#include "shared/source/utilities/debug_settings_reader.h"
#include "shared/source/os_interface/debug_env_reader.h"
#include "shared/source/utilities/io_functions.h"
#include "shared/test/common/helpers/debug_manager_state_restore.h"
#include "shared/test/common/helpers/default_hw_info.h"
@@ -603,7 +603,7 @@ bool pathExistsMock(const std::string &path) {
} // namespace NonExistingPathIsSet
TEST(CompilerCacheHelper, GivenNonExistingPathWhenCheckDefaultCacheDirSettingsThenFalseIsReturned) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["NEO_CACHE_DIR"] = "ult/directory/";
@@ -612,7 +612,7 @@ TEST(CompilerCacheHelper, GivenNonExistingPathWhenCheckDefaultCacheDirSettingsTh
VariableBackup<decltype(NEO::SysCalls::sysCallsPathExists)> pathExistsBackup(&NEO::SysCalls::sysCallsPathExists, NonExistingPathIsSet::pathExistsMock);
std::string cacheDir = "";
EXPECT_FALSE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_FALSE(checkDefaultCacheDirSettings(cacheDir, envReader));
}
namespace XDGEnvPathIsSet {
@@ -628,7 +628,7 @@ bool pathExistsMock(const std::string &path) {
} // namespace XDGEnvPathIsSet
TEST(CompilerCacheHelper, GivenXdgCachePathSetWhenCheckDefaultCacheDirSettingsThenProperPathIsReturned) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["XDG_CACHE_HOME"] = "xdg/directory/";
@@ -638,7 +638,7 @@ TEST(CompilerCacheHelper, GivenXdgCachePathSetWhenCheckDefaultCacheDirSettingsTh
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "xdg/directory/neo_compiler_cache");
}
@@ -655,7 +655,7 @@ bool pathExistsMock(const std::string &path) {
} // namespace HomeEnvPathIsSet
TEST(CompilerCacheHelper, GivenHomeCachePathSetWhenCheckDefaultCacheDirSettingsThenProperDirectoryIsSet) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["HOME"] = "home/directory/";
@@ -664,7 +664,7 @@ TEST(CompilerCacheHelper, GivenHomeCachePathSetWhenCheckDefaultCacheDirSettingsT
VariableBackup<decltype(NEO::SysCalls::sysCallsPathExists)> pathExistsBackup(&NEO::SysCalls::sysCallsPathExists, HomeEnvPathIsSet::pathExistsMock);
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "home/directory/.cache/neo_compiler_cache");
}
@@ -691,7 +691,8 @@ int mkdirMock(const std::string &dir) {
} // namespace XdgPathIsSetAndOtherProcessCreatesPath
TEST(CompilerCacheHelper, GivenXdgEnvWhenOtherProcessCreatesNeoCompilerCacheFolderThenProperDirectoryIsReturned) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["XDG_CACHE_HOME"] = "xdg/directory/";
bool mkdirCalledTemp = false;
@@ -702,7 +703,7 @@ TEST(CompilerCacheHelper, GivenXdgEnvWhenOtherProcessCreatesNeoCompilerCacheFold
VariableBackup<bool> mkdirCalledBackup(&XdgPathIsSetAndOtherProcessCreatesPath::mkdirCalled, mkdirCalledTemp);
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "xdg/directory/neo_compiler_cache");
EXPECT_TRUE(XdgPathIsSetAndOtherProcessCreatesPath::mkdirCalled);
}
@@ -723,7 +724,8 @@ int mkdirMock(const std::string &dir) {
} // namespace XdgPathIsSetAndNeedToCreate
TEST(CompilerCacheHelper, GivenXdgEnvWhenNeoCompilerCacheNotExistsThenCreateNeoCompilerCacheFolder) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["XDG_CACHE_HOME"] = "xdg/directory/";
@@ -732,12 +734,13 @@ TEST(CompilerCacheHelper, GivenXdgEnvWhenNeoCompilerCacheNotExistsThenCreateNeoC
VariableBackup<decltype(NEO::SysCalls::sysCallsMkdir)> mkdirBackup(&NEO::SysCalls::sysCallsMkdir, XdgPathIsSetAndNeedToCreate::mkdirMock);
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "xdg/directory/neo_compiler_cache");
}
TEST(CompilerCacheHelper, GivenXdgEnvWithoutTrailingSlashWhenNeoCompilerCacheNotExistsThenCreateNeoCompilerCacheFolder) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["XDG_CACHE_HOME"] = "xdg/directory";
@@ -746,7 +749,7 @@ TEST(CompilerCacheHelper, GivenXdgEnvWithoutTrailingSlashWhenNeoCompilerCacheNot
VariableBackup<decltype(NEO::SysCalls::sysCallsMkdir)> mkdirBackup(&NEO::SysCalls::sysCallsMkdir, XdgPathIsSetAndNeedToCreate::mkdirMock);
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "xdg/directory/neo_compiler_cache");
}
@@ -766,7 +769,7 @@ int mkdirMock(const std::string &dir) {
} // namespace HomePathIsSetAndNeedToCreate
TEST(CompilerCacheHelper, GivenHomeCachePathSetWithoutTrailingSlashWhenCheckDefaultCacheDirSettingsThenProperDirectoryIsCreated) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["HOME"] = "home/directory";
@@ -776,12 +779,12 @@ TEST(CompilerCacheHelper, GivenHomeCachePathSetWithoutTrailingSlashWhenCheckDefa
VariableBackup<decltype(NEO::SysCalls::sysCallsMkdir)> mkdirBackup(&NEO::SysCalls::sysCallsMkdir, HomePathIsSetAndNeedToCreate::mkdirMock);
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "home/directory/.cache/neo_compiler_cache");
}
TEST(CompilerCacheHelper, GivenHomeCachePathSetWhenCheckDefaultCacheDirSettingsThenProperDirectoryIsCreated) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["HOME"] = "home/directory/";
@@ -791,7 +794,7 @@ TEST(CompilerCacheHelper, GivenHomeCachePathSetWhenCheckDefaultCacheDirSettingsT
VariableBackup<decltype(NEO::SysCalls::sysCallsMkdir)> mkdirBackup(&NEO::SysCalls::sysCallsMkdir, HomePathIsSetAndNeedToCreate::mkdirMock);
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "home/directory/.cache/neo_compiler_cache");
}
@@ -818,7 +821,8 @@ int mkdirMock(const std::string &dir) {
} // namespace HomePathIsSetAndOtherProcessCreatesPath
TEST(CompilerCacheHelper, GivenHomeEnvWhenOtherProcessCreatesNeoCompilerCacheFolderThenProperDirectoryIsReturned) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::unordered_map<std::string, std::string> mockableEnvs;
mockableEnvs["HOME"] = "home/directory/";
bool mkdirCalledTemp = false;
@@ -829,7 +833,7 @@ TEST(CompilerCacheHelper, GivenHomeEnvWhenOtherProcessCreatesNeoCompilerCacheFol
VariableBackup<bool> mkdirCalledBackup(&HomePathIsSetAndOtherProcessCreatesPath::mkdirCalled, mkdirCalledTemp);
std::string cacheDir = "";
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_TRUE(checkDefaultCacheDirSettings(cacheDir, envReader));
EXPECT_EQ(cacheDir, "home/directory/.cache/neo_compiler_cache");
EXPECT_TRUE(HomePathIsSetAndOtherProcessCreatesPath::mkdirCalled);
}

View File

@@ -9,8 +9,8 @@
#include "shared/source/compiler_interface/os_compiler_cache_helper.h"
#include "shared/source/helpers/constants.h"
#include "shared/source/helpers/string.h"
#include "shared/source/os_interface/debug_env_reader.h"
#include "shared/source/os_interface/windows/sys_calls.h"
#include "shared/source/utilities/debug_settings_reader.h"
#include "shared/test/common/helpers/debug_manager_state_restore.h"
#include "shared/test/common/helpers/gtest_helpers.h"
#include "shared/test/common/helpers/variable_backup.h"
@@ -83,10 +83,9 @@ class CompilerCacheMockWindows : public CompilerCache {
};
TEST(CompilerCacheHelper, GivenHomeEnvWhenOtherProcessCreatesNeoCompilerCacheFolderThenProperDirectoryIsReturned) {
std::unique_ptr<SettingsReader> settingsReader(SettingsReader::createOsReader(false, ""));
NEO::EnvironmentVariableReader envReader;
std::string cacheDir = "";
EXPECT_FALSE(checkDefaultCacheDirSettings(cacheDir, settingsReader.get()));
EXPECT_FALSE(checkDefaultCacheDirSettings(cacheDir, envReader));
}
namespace SysCalls {