Updated 7.-Jami-plugins.md (markdown)
This commit is contained in:
agsantos
parent
307888a0a7
commit
c901ae4be9
|
|
@ -1,7 +1,7 @@
|
|||
**NOTE: this page describes what is a Jami Plugin and how to install and use them.**
|
||||
|
||||
# Jami Plugins
|
||||
As from September of 2020, Jami team has added plugins as a call feature for Linux and Android users.
|
||||
As from September of 2020, Jami team has added plugins as a call feature for Linux, Windows, and Android users.
|
||||
This meaning that now you can personalize your call experience by using one of our available plugins.
|
||||
But that is not all, you can also transform your awesome ideas into a brand new plugin!
|
||||
|
||||
|
|
@ -16,20 +16,18 @@ command. Therefore, Daemon has a `JamiPluginManager` class that among other acti
|
|||
Despite Daemon importance, what a plugin effectivelly does to a call video is unknown to it the same way Daemon does not know what is effectivelly done by LRC or the clients interfaces.
|
||||
Plugins then can be seen as a forth interacting component in Jami.
|
||||
|
||||
Plugins are composed by two main components: the media handler and the process. While the latter actually implements plugin's functionality and waits data to perform, the former is the interface between Daemon and the plugin.
|
||||
It is, the media handler is attached or detached to a media stream from Daemon and passes data to plugin's process.
|
||||
The plugin system inside Jami exposes different APIs that can be used by the plugins. For instance, the Media Handler API enables the plugins to modify audio and video streams from Jami calls. This API is used by our GreenScreen plugin but could also be used to build a YouTube streaming system, various instagram-style filters, a real time translation service, etc.
|
||||
|
||||
To use a plugin it is necessary that the `JamiPluginManager` knows plugin's media handlers and the available media streams.
|
||||
Plugin's media handlers are created once a plugin is loaded and they are shared with Daemon's Plugin Manager.
|
||||
And in the event of a new SIP call, the `JamiPluginManager` creates and stores the corresponding media stream subjects.
|
||||
Finally, once a user puts a plugin in action the `JamiPluginManager` attaches that plugin's media handler and the available call media streams. When deactivated, our Plugin Manager dettaches these two instances.
|
||||
Plugins can be composed by one or multiple media handlers that are responsible for attaching/detaching a media stream from Jami and a data process. Each media handler represent a functionality that can be totally different between them or can be a modified versions of the same core process. In our example, we have only one functionality, it being, the GreenScreen plugin has one media handler wich data process is responsible for segmenting the foreground from a video frame and applying another image to the background, just like it is done with the green screens in movies!
|
||||
|
||||
To use one custom functionality, it is necessary that Jami knows all plugins' media handlers, wich one is going to be used and the data that should be processed. Plugin's media handlers are created once a plugin is loaded and are shared with Daemon's Plugin Manager. The data is inside Jami flow (for a call plugin, in the event of a new call, Jami creates and stores the corresponding media stream subjects). Finally, once a user puts a plugin functionality in action Jami tells this media handler to attach the available data. When deactivated, Jami tells the media handler to dettach.
|
||||
|
||||
## How to use it?
|
||||
|
||||
### Setup
|
||||
A Jami plugin is a `pluginname.jpl` file, and it must be installed to your Jami.
|
||||
Once installed, Jami will add your new plugin to the available plugins list but they will not be ready for use yet. Plugins are libraries and must be loaded if you want to expose them.
|
||||
Moreover, plugin may have preferences and besides install/uninstall and load/unload actions, it is possible to modify those preferences. For example, our Foreground Segmentation plugin allows the user to change the background image displayed.
|
||||
Moreover, plugin may have preferences and besides install/uninstall and load/unload actions, it is possible to modify those preferences. For example, our GreenScreen plugin allows the user to change the background image displayed.
|
||||
|
||||
#### Android
|
||||
To setup a plugin for Android you must go under Setting, enable plugins if they are disabled, and select a plugin file from your phone.
|
||||
|
|
@ -45,17 +43,20 @@ In order to a preference modification can take effect the plugin has to be reloa
|
|||
|
||||
|
||||
|
||||
#### Linux
|
||||
Similarly, for Linux, you must go to Jami Prefences, enable plugins if it is disabled, and select a plugins file from your computer.
|
||||
For Linux, each plugins in plugins list shows a uninstall button and a checkbox button corresponding to the load/unload action.
|
||||
For client-gnome it is not yet possible to change plugin's preferences.
|
||||
#### Linux/Windows
|
||||
Similarly, for the client-qt available on Linux and Windows and for the client-gnome available only on Linux, you must go to Prefences, enable plugins if it is disabled, and select a plugins file from your computer.
|
||||
Each plugin in the shown list is linked to two buttons beeing:
|
||||
|
||||
- Client-qt: a load/unload button and a preferences button;
|
||||
- Client-gnome: a load/unload button and a uninstall button;
|
||||
For client-gnome it is not possible to change plugin's preferences.
|
||||
|
||||
|
||||
|
||||
|
||||
### Use!
|
||||
A plugins only takes place if you turn them on during a call.
|
||||
For either Android or Linux you can do so by clicking on the plugins icon on your call screen.
|
||||
For either Android, Linux or Windows you can do so by clicking on the plugins icon on your call screen.
|
||||
|
||||
|
||||
|
||||
|
|
@ -70,17 +71,58 @@ Other ones should work as long they and the plugin are correctly built!
|
|||
|
||||
#### OpenCV 4.1.1
|
||||
We kindly added OpenCV 4.1.1 as a contrib in [daemon](https://git.jami.net/savoirfairelinux/ring-daemon/tree/master/contrib).
|
||||
This way you can easily build OpenCV for Android or Linux. You only have to set `export DAEMON=<path/to/daemon>` and follow the corresponding instructions.
|
||||
This way you can easily build OpenCV for Android, Linux, and Windows. You only have to follow the corresponding instructions.
|
||||
|
||||
##### Windows
|
||||
```bash
|
||||
set DAEMON=<path/to/daemon>
|
||||
cd ${DAEMON}/compat/msvc
|
||||
python3 winmake.py -fb opencv
|
||||
```
|
||||
|
||||
##### Linux
|
||||
With Docker (recommended):
|
||||
```bash
|
||||
export DAEMON=<path/to/daemon>
|
||||
cd ${DAEMON}/../
|
||||
docker build -f plugins/docker/Dockerfile_ubuntu_20.04 -t plugins-validation .
|
||||
docker run --rm -it -v ${DAEMON}/../:/home/plugins/jami:rw plugins-validation:latest /bin/bash
|
||||
cd jami/plugins/contrib
|
||||
tar -xf libs.tar.gz
|
||||
cd ../../daemon/contrib
|
||||
mkdir native
|
||||
cd native
|
||||
../bootstrap --disable-argon2 --disable-asio --disable-fmt --disable-gcrypt --disable-gmp --disable-gnutls --disable-gpg-error --disable-gsm --disable-http_parser --disable-iconv --disable-jack --disable-jsoncpp --disable-libarchive --disable-libressl --disable-msgpack --disable-natpmp --disable-nettle --enable-opencv --disable-opendht --disable-pjproject --disable-portaudio --disable-restinio --disable-secp256k1 --disable-speexdsp --disable-upnp --disable-uuid --disable-yaml-cpp --disable-zlib
|
||||
make list
|
||||
make fetch opencv opencv_contrib
|
||||
make
|
||||
```
|
||||
|
||||
Using your own system:
|
||||
```bash
|
||||
export DAEMON=<path/to/daemon>
|
||||
cd ${DAEMON}/contrib/native
|
||||
./../bootstrap --enable-opencv --enable-opencv_contrib
|
||||
../bootstrap --disable-argon2 --disable-asio --disable-fmt --disable-gcrypt --disable-gmp --disable-gnutls --disable-gpg-error --disable-gsm --disable-http_parser --disable-iconv --disable-jack --disable-jsoncpp --disable-libarchive --disable-libressl --disable-msgpack --disable-natpmp --disable-nettle --enable-opencv --disable-opendht --disable-pjproject --disable-portaudio --disable-restinio --disable-secp256k1 --disable-speexdsp --disable-upnp --disable-uuid --disable-yaml-cpp --disable-zlib
|
||||
make list
|
||||
make fetch opencv opencv_contrib
|
||||
make
|
||||
```
|
||||
|
||||
##### Android
|
||||
Using Docker (recommended):
|
||||
```bash
|
||||
export DAEMON=<path/to/daemon>
|
||||
cd ${DAEMON}/../
|
||||
docker build -f plugins/docker/Dockerfile_android -t plugins-validation .
|
||||
docker run --rm -it ${DAEMON}/:/home/gradle/src:rw plugins-validation:latest /bin/bash
|
||||
cd plugins/contrib
|
||||
tar -xf libs.tar.gz
|
||||
ANDROID_ABI="arm64-v8a" sh build-dependencies.sh
|
||||
```
|
||||
|
||||
Using your own system:
|
||||
```bash
|
||||
export DAEMON=<path/to/daemon>
|
||||
cd ${DAEMON}
|
||||
export ANDROID_NDK=<NDK>
|
||||
export ANDROID_ABI=arm64-v8a
|
||||
|
|
@ -102,7 +144,9 @@ make
|
|||
```
|
||||
|
||||
#### Tensorflow 2.1.0
|
||||
A difficulty for a lot of people working with tensorflow is how to properly build it. Here we carefully guide you through the proper build of tensorflow LITE Native and Tensorflow C++ API.
|
||||
A difficulty for a lot of people working with tensorflow is how to properly build it.
|
||||
With that in mind we created the docker files available for [here]("https://hub.docker.com/repository/docker/sflagsantos/tensorflow-cuda") and [here]("https://hub.docker.com/repository/docker/sflagsantos/tensorflowlite"). These docker can be used to build plugins for Linux and Android, however they cannot handle Windows.
|
||||
Here we carefully guide you through the proper build of tensorflow LITE Native and Tensorflow C++ API for our three supported platforms.
|
||||
|
||||
You will need:
|
||||
|
||||
|
|
@ -115,8 +159,7 @@ You will need:
|
|||
cd tensorflow
|
||||
git checkout -b v2.1.0
|
||||
```
|
||||
|
||||
Assemble instructions for Tensorflow LITE Native and C++ API are available under [jami-plugins](https://git.jami.net/savoirfairelinux/jami-plugins) README_ASSEMBLE file
|
||||
We assembled Tensorflow headers needed to build plugins, to access them you only have to extract `libs.tar.gz` file found under `jami-project/plugins/contrib`. However, if you are using another tensorflow version or you want to do it by yourself, you can follow the assemble instructions for Tensorflow LITE Native and C++ API are available under [jami-plugins](https://git.jami.net/savoirfairelinux/jami-plugins) README_ASSEMBLE file.
|
||||
|
||||
#### Linux
|
||||
Tensorflow LITE does not support desktops GPU. If you want to use them, please consider using C++ API
|
||||
|
|
@ -142,9 +185,37 @@ Setup your build options with `./configure`.
|
|||
* Tensorflow C++ API
|
||||
|
||||
```bash
|
||||
bazel build --config=v1 --define framework_shared_object=false //tensorflow:libtensorflow_cc.so
|
||||
bazel build --config=v1 --define framework_shared_object=false --define=no_tensorflow_py_deps=true //tensorflow:libtensorflow_cc.so
|
||||
```
|
||||
|
||||
#### Windows
|
||||
Tensorflow LITE does not support desktops GPU. If you want to use them, please consider using C++ API
|
||||
|
||||
If you want to build Tensorflow C++ API with GPU suport, be sure to have a CUDA capable GPU and that you have
|
||||
followed all installation steps for the Nvidia drivers, CUDA Toolkit, CUDNN, Tensor RT, that their versions
|
||||
matches and that they are correct for the Tensorflow version you want to build.
|
||||
|
||||
The following links may be very helpfull:
|
||||
|
||||
* https://www.tensorflow.org/install/source
|
||||
* https://developer.nvidia.com/cuda-gpus
|
||||
* https://developer.nvidia.com/cuda-toolkit-archive
|
||||
* https://developer.nvidia.com/cudnn
|
||||
|
||||
Setup your build options with `python3 configure.py`.
|
||||
|
||||
* Tensorflow LITE Native
|
||||
|
||||
```bash
|
||||
bazel build //tensorflow/lite:tensorflowlite.dll
|
||||
```
|
||||
* Tensorflow C++ API
|
||||
|
||||
```bash
|
||||
bazel build --config=v1 --define framework_shared_object=false --config=cuda --define=no_tensorflow_py_deps=true //tensorflow:tensorflow_cc.dll
|
||||
```
|
||||
There may be some missign references while compilling a plugin with Tensorflow C++ API. If that happens you have to rebuild you tensorflow and explicitly export the missing symbols. Fortunatelly Tensorflow now has a easy workaround to do so, you only have to feed [this]("https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/tools/def_file_filter/def_file_filter.py.tpl") file with the desired symbols.
|
||||
|
||||
#### Android - Tensorflow LITE Native
|
||||
For mobile applications Tensorflow LITE is the only option you want to consider and to succesfully build it you will also need:
|
||||
|
||||
|
|
@ -173,27 +244,35 @@ And build as desired:
|
|||
```
|
||||
|
||||
### Plugin
|
||||
To exemplify a plugin build, we will use the Foreground Segmentation plugin available [here](https://git.jami.net/savoirfairelinux/jami-plugins).
|
||||
To exemplify a plugin build, we will use the GreenScreen plugin available [here](https://git.jami.net/savoirfairelinux/jami-plugins).
|
||||
|
||||
First you need to clone the repository to your ring-project path:
|
||||
#### Windows
|
||||
Windows build of plugins are linked with the daemon repository and its build scripts. So to build our example plugins you have to:
|
||||
|
||||
```bash
|
||||
cd ring-project
|
||||
git clone https://review.jami.net/jami-plugins
|
||||
cd <ring-project>/daemon/compat/msvc
|
||||
python3 winmake.py -fb GreenScreen
|
||||
```
|
||||
|
||||
Inside ForegroundSegmentation folder there is three possible build files:
|
||||
The GreenScreen.jpl file will be available under `<ring-project/plugins/build/>`.
|
||||
|
||||
* build.sh
|
||||
* builtfcc.sh
|
||||
* buildandroid.sh
|
||||
#### Linux/Android
|
||||
First you need to go to the plugins repository in your cloned ring-project:
|
||||
|
||||
Each of these files corresponds to a platform and tensorflow build to be used.
|
||||
```bash
|
||||
cd <ring-project>/plugins
|
||||
```
|
||||
|
||||
* Android: for this platform we do `sh buildandroid.sh`;
|
||||
* Linux: for this platform we can do `sh build.sh` for Tensorflow LITE Native and `sh buildtfcc.sh` for Tensoflorw C++ API;
|
||||
- Linux - TF C++ API + GPU
|
||||
`python3 buildPlugin.py --projects=GreenScreen`
|
||||
- Linux - TF C++ API + CPU
|
||||
`python3 buildPlugin.py --projects=GreenScreen --processor=CPU`
|
||||
- Linux - TF LITE
|
||||
`TF="_tensorflowLite" python3 buildPlugin.py --projects=GreenScreen`
|
||||
- Android
|
||||
`ANDROID_ABI="arm64-v8a armeabi-v7a x86_64" python3 buildPlugin.py --projects=GreenScreen --distribution=android`
|
||||
|
||||
The foregroundsegmentation.jpl file will be available under `<jami-plugins/build/>`.
|
||||
The GreenScreen.jpl file will be available under `<ring-project/plugins/build/>`.
|
||||
|
||||
|
||||
Related articles:
|
||||
|
|
|
|||
Loading…
Reference in New Issue