Manticore Buddy: pluggable design

Hey folks. Exciting news about Manticore Buddy: we have finished the migration to a pluggable design! That means you can build your own Manticore Search SQL/JSON query as a plugin, publish it on packagist.org, and install it using the CREATE PLUGIN SQL command. Let's dive into the architecture of the pluggable system and a simple tutorial that can help you get started.

Architecture

Intro

If you are not familiar with Buddy, we recommend reading blogposts:

that we published previously.

Originally, Buddy was developed in PHP, with all extensions and extra command handlers in the same code base. After Manticore Search 6 had been released, we received questions about adding custom functionality and realized the current system lacked flexibility. That's why we decided to migrate to a pluggable architecture.

We believe that the most important principle in software development is not to reinvent the wheel and to keep things simple. As a result, we didn't spend too much time deliberating and chose the most well-known and popular package manager in the PHP world to manage the plugin system - Packagist.

Plugin Types

There are three types of plugins:

core - included by default with Buddy and are part of the package you receive when you install Manticore.
local - used for development and debugging before publishing.
external - installed using the CREATE PLUGIN command.

This tutorial explains how to first create a local plugin and then make it an external one, enabling any Manticore user to enhance their setup by using your plugin.

Request -> response flow

To make it work, we extracted commonly used components in Buddy and all plugins into a new package - Buddy Core. It should be included when developing your plugin to make things easier in your IDE and support hints and autocompletes.

The code that enables Buddy to function as a daemon and perform internal tasks, including connecting the Core and the Plugin source code, is called Buddy Base. This is a ReactPHP application that also uses Buddy Core and has basic logic to make everything work and find the correct plugin to launch on command.

For a clearer understanding of the process, here's a diagram illustrating the Request → Response flow:

Tutorial

To make things easier for you and to dive into developing your own plugin, let's examine how we can build a simple plugin for Buddy that supports the SHOW HOSTNAME command and returns the current host machine's hostname. This will help you understand the process you should follow when creating your own plugin, and it's also an incredibly straightforward example to start with. If we examine how SHOW HOSTNAME functions in Manticore Search before implementing its logic, we can expect a result like this:

mysql> SHOW HOSTNAME;
ERROR 1064 (42000): sphinxql: syntax error, unexpected identifier, 
expecting VARIABLES near 'HOSTNAME'

Preparation

The first thing we should do is prepare our development environment. You only need to do this once and can later update Buddy from the GIT repository to receive the latest updates. Let's open the terminal and clone Buddy to our machine into a folder called manticoresearch-buddy and navigate to the cloned folders.

git clone https://github.com/manticoresoftware/manticoresearch-buddy.git 
cd manticoresearch-buddy

Now you have the entire source code of Buddy, but none of the packages are installed. Buddy uses Composer to manage its dependencies, and we need to install them first to run the application. However, before that, we'll run a special Docker container that we've prepared specifically for development. The Manticore Executor Kit image includes all the necessary tools for a smooth Manticore Buddy development experience, whether you're creating new plugins or contributing to Buddy itself.

docker pull manticoresearch/manticore-executor-kit:latest
docker create --privileged --entrypoint bash \
  -v $(pwd):/workdir -w /workdir --name manticore-buddy \
  --network host -it manticoresearch/manticore-executor-kit:latest
docker start manticore-buddy

Congratulations! You have now started a Docker container with Manticore Search and Manticore Executor to develop Buddy and plugins. The working directory is set to /workdir, and your source folder is bound to the container. Your next step is to install Composer dependencies. Remember, this and all further commands must be executed inside manticore-buddy container.

docker exec -it manticore-buddy bash
composer install

Your Buddy is now ready for use, development, debugging, and testing. Next, you need to complete two things:

First, edit /etc/manticore.conf and add buddy_path with our sources there. This will let Manticore Search know that we need to use the custom path of running Buddy from sources, not from the installed modules. Add this to the searchd section:

buddy_path = manticore-executor /workdir/src/main.php --debug

Second, try running searchd (the Manticore Search server) and ensure that it works. You should see a few lines starting with[BUDDY]. When you want to stop the process, simply use Ctrl + C.

# Launch daemon inside the container and keep it foreground
searchd --nodetach

At this point, everything is set for preparing your development environment for Buddy and beginning the implementation of a new plugin. You can still experiment with it and attempt to execute some queries to ensure it's functioning properly. For instance, open another terminal window and run the following:

$ docker exec -it manticore-buddy mysql -h0 -P9306
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 6.0.5 3bcbd00fa@230320 dev (columnar 2.0.5 8171c1a@230320) 
(secondary 2.0.5 8171c1a@230320) git branch HEAD (no branch)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> show queries;
+------+--------------+----------+-----------------+
| id   | query        | protocol | host            |
+------+--------------+----------+-----------------+
|   10 | select       | http     | 127.0.0.1:19148 |
|    9 | show queries | mysql    | 127.0.0.1:54484 |
+------+--------------+----------+-----------------+
2 rows in set (0.008 sec)

MySQL [(none)]>

You've just sent the command SHOW QUERIES to Manticore Search, which was then routed to Buddy and executed successfully. You can see what Buddy received and its response in the searchd log in the first terminal tab. So, we can confirm that Buddy is functioning correctly and everything has been set up right.

Plugin Template

We've created a special GitHub template repository that we highly recommend using to create any Manticore Buddy plugin. All plugins should be prefixed with buddy-plugin-[your-name], where [your-name] is the name of your plugin. For example, since we're developing a hostname selection plugin, we'll name it buddy-plugin-show-hostname. So:

Open https://github.com/manticoresoftware/buddy-plugin-template.
Click Use this template and then Create a new repository.
Fill out the form with your repository's name and create it. Make sure your repo name starts with buddy-plugin- as it's mandatory.
git clone your new repo to the plugins directory of the Manticore Buddy repo you previously deployed. Note that it's better to do this outside the container, especially if you're cloning from a private repository.

In https://github.com/manticoresoftware/buddy-plugin-show-hostname.git, you can find the entire task implemented. If you don't want to delve into the details, you can simply git clone that instead of your forked repo. However, if you do want to master creating a plugin, let's continue.

Add real code

Now we need to follow the next steps and update our template with the related data to our new plugin.

First, we open composer.json and update the name of the plugin, description, and namespaces. So finally, your changes will look like this (just make sure you use your plugin repo name):

diff --git a/composer.json b/composer.json
index 23c252b..f36cb6e 100644
--- a/composer.json
+++ b/composer.json
@@ -1,11 +1,11 @@
 {
- "name": "manticoresoftware/buddy-plugin-template",
- "description": "The Buddy template handler plugin",
+ "name": "manticoresoftware/buddy-plugin-show-hostname",
+ "description": "The Buddy SHOW hostname handler plugin",
  "type": "library",
  "license": "GPL-2.0-or-later",
  "autoload": {
    "psr-4": {
-     "Manticoresearch\\Buddy\\Plugin\\Template\\": "src/"
+     "Manticoresearch\\Buddy\\Plugin\\ShowHostname\\": "src/"
    }
  },
  "authors": [

We also need to update the namespaces of two classes - Payload and Handler. The diff should look like this:

diff --git a/src/Handler.php b/src/Handler.php
index 3756dc3..07fcfd0 100644
--- a/src/Handler.php
+++ b/src/Handler.php
@@ -8,7 +8,7 @@
   version. You should have received a copy of the GPL license along with this
   program; if you did not, you can find it at http://www.gnu.org/
 */
-namespace Manticoresearch\Buddy\Plugin\Template;
+namespace Manticoresearch\Buddy\Plugin\ShowHostname;

 use Manticoresearch\Buddy\Core\Plugin\BaseHandler;
 use Manticoresearch\Buddy\Core\Task\Task;
diff --git a/src/Payload.php b/src/Payload.php
index b170340..a35f201 100644
--- a/src/Payload.php
+++ b/src/Payload.php
@@ -8,7 +8,7 @@
   version. You should have received a copy of the GPL license along with this
   program; if you did not, you can find it at http://www.gnu.org/
 */
-namespace Manticoresearch\Buddy\Plugin\Template;
+namespace Manticoresearch\Buddy\Plugin\ShowHostname;

 use Manticoresearch\Buddy\Core\Network\Request;
 use Manticoresearch\Buddy\Core\Plugin\BasePayload;

As a final step, do not forget to install the new dependency by running composer install inside the container.

The request flow from Manticore Search to your plugin and back

We are ready to implement our logic now. But first, let's learn a little bit more about how it's handled internally.

Let's assume we're sending a SHOW HOSTNAME query to the launched searchd process using a MySQL client. Manticore Search can't handle it and sends it to Buddy to await a response. Buddy can return the exact response, which will simply be proxied back to the MySQL client.

Once Buddy receives the query from Manticore Search, it parses it and runs validation across all plugins: first core plugins, and then external plugins. To run this validation, it executes the Payload::hasMatch($request) method, passing the request to it, which contains the query and some metadata, such as the requested endpoint, type (JSON or SQL), and so on.

Payload represents the structure containing all required data for handling the command. If a plugin's Payload returns true in the hasMatch method, it means that we will use this plugin to handle the request. It's created from the Request, and at this stage, we need to parse the query, extract valuable information, and set it to our payload data, which will be passed to the Handler.

Handler is the class that actually executes the implemented command. It does so in a threaded environment using the parallel extension without blocking the main server loop. There's a simple method run that creates a closure, which must perform a task and return the response back to Manticore Search, which is then proxied to the user.

In our case, we just need to add a check for supporting the SHOW HOSTNAME query syntax and then write the Handler logic to call the gethostname PHP function and return it as a response. That's it! Let's start with the Payload.

Implementing Payload

When you open src/Payload.php, you will see there are just two methods: fromRequest and hasMatch. We have marked the lines we need to edit with TODO comments, so it should be easy to find what we need to do. In our case, the query is very simple and not flexible at all, so we don't have to change anything in the fromRequest method. We can just remove the TODO comments. What we need to do is update the static function hasMatch(). There we need to implement a check that the query strictly matches the string SHOW HOSTNAME in a case-insensitive way. Once it has a match, we should return true. Otherwise, we should return false. That's how the base system understands that it needs to use this plugin in case we receive a query which shows a hostname.

The final diff we made to the Payload should look like this:

diff --git a/src/Payload.php b/src/Payload.php
index a35f201..bd0795e 100644
--- a/src/Payload.php
+++ b/src/Payload.php
@@ -26,7 +26,6 @@ final class Payload extends BasePayload {
   */
  public static function fromRequest(Request $request): static {
    $self = new static();
-   // TODO: add logic of parsing request into payload here
    // We just need to do something, but actually its' just for PHPstan
    $self->path = $request->path;
    return $self;
@@ -37,7 +36,6 @@ final class Payload extends BasePayload {
   * @return bool
   */
  public static function hasMatch(Request $request): bool {
-   // TODO: validate $request->payload and return true, if your plugin should handle it
-   return $request->payload === 'template';
+   return stripos($request->payload, 'show hostname') !== false;
  }
 }

We are done with the Payload changes and can move on to the Handler to implement the real logic.

Implementing Handler

Now, let's open src/Handler.php and examine the TODO marks, along with a brief description of what's happening. We need to navigate to the run method and update it.

We can see a closure that returns a TaskResult, which is a special result type that wraps a standard JSON response.

We will include a gethostname() call and prepare the response by wrapping it within the TaskResult that will be returned to the client. The final diff will appear as follows:

diff --git a/src/Handler.php b/src/Handler.php
index 07fcfd0..944899b 100644
--- a/src/Handler.php
+++ b/src/Handler.php
@@ -11,6 +11,7 @@
 namespace Manticoresearch\Buddy\Plugin\ShowHostname;

 use Manticoresearch\Buddy\Core\Plugin\BaseHandler;
+use Manticoresearch\Buddy\Core\Task\Column;
 use Manticoresearch\Buddy\Core\Task\Task;
 use Manticoresearch\Buddy\Core\Task\TaskResult;
 use RuntimeException;
@@ -33,9 +34,11 @@ final class Handler extends BaseHandler {
   * @throws RuntimeException
   */
  public function run(Runtime $runtime): Task {
-   // TODO: your logic goes into closure and should return TaskResult as response
    $taskFn = static function (): TaskResult {
-     return TaskResult::none();
+     $hostname = gethostname();
+     return TaskResult::withRow([
+       'hostname' => $hostname,
+     ])->column('hostname', Column::String);
    };

    return Task::createInRuntime(

Debugging and development

Now that you've completed the implementation, it's time to test and debug (if necessary) your plugin. There is a special type of plugin called local designed specifically for this purpose. Since your plugin is already in the plugins directory, it automatically becomes a local plugin. To use it, simply:

In the root Buddy directory, run the command composer require [your-plugin-name]:dev-main to include the plugin. your-plugin-name is the name from you plugin's composer.jsonyou edited previously.
Restart searchd to ensure that the code is updated.

This method allows you to develop, debug, edit, and test your plugin live without having to push it to the Git repository and publish it on packagist.org.

Publishing and installing

When you're done with debugging and ready to publish your plugin, simply commit your changes to the Git repository and publish your package on packagist.org.

Afterward, check that it's working as expected by running the CREATE PLUGIN [your-plugin-name] TYPE 'buddy' VERSION 'dev-main' command, which will attempt to download the plugin using Composer and install it to the plugin_dir.

Like this:

$ docker exec -it manticore-buddy mysql -h0 -P9306
MySQL [(none)]> CREATE PLUGIN manticoresoftware/buddy-plugin-show-hostname type 'buddy' VERSION 'dev-main';

Query OK, 0 rows affected (1 min 12.213 sec)

MySQL [(none)]> show hostname;
+----------+
| hostname |
+----------+
| dev      |
+----------+
1 row in set (0.011 sec)

MySQL [(none)]>

Congrats! Your plugin works fine and it allows you to fetch hostname by using SHOW hostname query.

Conclusion

We hope you enjoyed this article and gained a deeper understanding of the new Buddy pluggable architecture, already available in the Linux dev packages and coming soon to the next release (following Manticore 6.0.4, so if there's a newer version when you read this, it's likely already included). You can start developing your plugin right away by following the instructions we provided. While this is a basic introduction to the pluggable system, we plan to create and publish a more advanced article on developing a complex plugin. Don't hesitate to ask us questions in our Slack or Telegram chats, and feel free to suggest topics for the complex plugin tutorial.
Remember that it’s also a good idea to take a look at our core plugin’s code to delve deeper and gain a better understanding of how it works in more complex cases.

Sincerely yours,
Manticore Team.