Setting up an automatically updated custom Composer repository

13 Nov 2020

#automation #composer

We had a bunch of public Drupal modules/PHP libraries that needed to be distributed to multiple Drupal sites using composer. There’s a few different ways to solve that issue:

1. Publish packages on drupal.org and packagist.org
2. Include repositories in site’s composer.json, like:

"repositories": [
  {
    "type": "vcs",
    "url": "https://github.com/your-organization/submodule.git"
  }
]

Not being able to delete packages on drupal.org made it non-option for us. We also sometimes had to fork third party libraries and distribute them as dependencies as well.

The second option is probably the best choice for most installations, but our custom modules also had dependencies to other custom modules, and since Composer doesn’t load repositories recursively, meant we couldn’t define dependencies to other custom packages in our modules’ composer.json file.

Setting up a webhook server to update the repository

After playing around using GitHub to build and host the repository, we eventually ended up using adnanh/webhook library instead.

Requirements

• A web server to serve static files
• PHP to run Satis and Composer
• GO to run adnanh/webhook library

Set up the Satis repository

Install Satis using composer:

$ composer require composer/satis

Create satis.json:

{
    "name": "your-organization/composer-repository",
    "description": "Repository for your custom libraries",
    "homepage": "https://your-custom-repository.url",
    "include-filename": "all.json",
    "repositories": [
        {
            "packagist.org": false
        },
        {
            "name": "your-organization/custom_module",
            "type": "vcs",
            "url": "git@github.com:your-organization/custom_module.git"
        }
    ]
}

Make sure your custom module has a composer.json file, something like:

{
    "name": "your-organization/custom_module",
    "type": "drupal-module",
    "license": "GPL-2.0-or-later"
}

Run php vendor/bin/satis build satis.json dist/ to verify it works as expected.

Set up the webhook server

We are using a custom Docker image based on almir/docker-webhook and official nginx Docker image to serve the static build file.

You can check roughly what we use from this GitHub Gist. It doesn’t include Traefik configuration, but hopefully it’s enough to get you started.

Create hooks.json:

[
  {
    "id": "update-repository",
    "execute-command": "php",
    "command-working-directory": ".",
    "response-message": "Success",
    "response-headers":
    [
      {
        "name": "Access-Control-Allow-Origin",
        "value": "*"
      }
    ],
    "pass-arguments-to-command":
    [
      {
        "source": "string",
        "name": "vendor/bin/satis"
      },
      {
        "source": "string",
        "name": "build"
      },
      {
        "source": "string",
        "name": "satis.json"
      },
      {
        "source": "string",
        "name": "dist"
      }
    ],
    "trigger-rule":
    {
      "and": [
        {
          "match":
          {
            "type": "payload-hash-sha1",
            "secret": "{{ getenv "WEBHOOK_SECRET" }}",
            "parameter":
            {
              "source": "header",
              "name": "X-Hub-Signature"
            }
          }
        }
      ]
    }
  }
]

The hooks.json above assumes that webhook requests are coming from GitHub (check out the documentation for other VCS providers). The payload is validated against X-Hub-Signature value using the secret defined in trigger-rule.

The secret is read from an environment variable called WEBHOOK_SECRET. This requires webhook server be to started using -template argument. You can also hard-code your secret there instead.

Start the webhook server: /path/to/webhook -hooks /path/to/hooks.json -verbose -template.

Add a webhook in your project:

• Payload URL should be http://webhookserver.example.com:9000/hooks/update-repository (it’s running on port 9000 by default).
• Secret is same as the one defined in trigger-rule.

Then point your webserver to serve the dist/ folder and add the repository to your project’s composer.json file:

"repositories": [
    {
        "type": "composer",
        "url": "https://yourrepository.example.com"
    },
]

and you should be ready to go.

Partial updates

You might eventually run into same issue as we did, the more releases your packages have, the longer it takes to rebuild your index.

There are a few ways to speed things up, like removing older releases and unused Git branches.

You can also tell Satis to selectively update only particular packages by passing the package name as an argument to your satis build command: php vendor/bin/satis build satis.json build/ your-organization/custom_module.

The package name can be parsed automatically from an incoming payload and then be passed to Satis with something like this:

{
    "pass-arguments-to-command":
    [
      ...
      {
        "source": "payload",
        "name": "repository.full_name"
      }
    ]
}