Drupal Gitlab CI – Running contrib tests locally

Gitlab CI is now the preferred default for running tests in contrib in Drupal. This is great! Gitlab gives us a lot of new control over how tests are run. While moving project update bot to gitlab I wanted local developement really badly.

Drupal chose to create a gitlab templates repository to get control on how Gitlab is run for contrib testing. Great! There is pretty good guide available on drupal.org to get you up and running easily. The basics are really simple though.

Getting started with Drupal’s Gitlab

To get started you just include the a .gitlab-ci.yml based on the template. See below for the important part.

include: 
  ################
  # DrupalCI includes: 
  # As long as you include this, any future includes added by the Drupal Association will be accessible to your pipelines automatically.
  # View these include files at https://git.drupalcode.org/project/gitlab_templates/
  ################
  - project: $_GITLAB_TEMPLATES_REPO
    ref: $_GITLAB_TEMPLATES_REF
    file:
      - '/includes/include.drupalci.main.yml'
      - '/includes/include.drupalci.variables.yml'
      - '/includes/include.drupalci.workflows.yml'

So in short is just tells gitlab where to look for some templates and include those into your project. Easy stuff!

The (not so fun) testing loop

But what if you want to customize, or test your gitlab workflow? This almost always ends up the the following loop:

  1. Commit .gitlab-ci.yml
  2. Wait for run (to fail)
  3. Adjust .gitlab-ci.ml
  4. Repeat
  5. Profit.

This is not really optimal, you keep waiting for gitlab to trigger and rub your stuff. But fear not, there is a way to get Gitlab CI running locally.

The deprecated Gitlab runner

When searching for a way to run locally I started at Gitlab ofcourse. I did find the Gitlab runner documentation. Which seemed fine, until i noticed it didn’t support includes and apparently a lot more.

It seems like Gitlab doesn’t want to support local runners, perhaps in the future, it’s a well requested feature.

Our saviour? firecow/gitlab-ci-local

So while scouring the internet I found a little gem: firecow/gitlab-ci-local.

This repository aims to bring gitlab-ci to your local machine. After interacting with the maintainer a few times after running into some problems I feel pretty confident this will be well supported.

So how to use is? Basically the easiest way is to use NPM to install the package globally.

npm install -g gitlab-ci-local

You do need to make sure you have docker installed. Its a fully docker based solution.

Playing nicely with templates

As a test I’ve used the ‘keycdn’ module, which is also used by the Gitlab team to test the templates. You’d recon you can just clone the repository and go right? Not entirely.

bjorn@1WNLK63:~/projects/keycdn$ gitlab-ci-local
Project include could not be fetched { project: , ref: HEAD, file: includes/include.drupalci.main.yml }
Error: Command failed with exit code 2: git archive --remote=ssh://[email protected]:22/.git HEAD includes/include.drupalci.main.yml | tar -f - -xC .gitlab-ci-local/includes/git.drupal.org//HEAD/
remote:
remote: ========================================================================
remote:
remote: The namespace you were looking for could not be found.
remote:
remote: ========================================================================
remote:
fatal: the remote end hung up unexpectedly
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

The error here is pretty logical. In a normal environment the variables are set globally by gitlab and point to the right gitlab repository. We need to set those also for ourselves. We can do that in multiple ways, but the easiest is probably making your own little shell script.

I’ve been using this one:

~/.local/bin/drupal-ci-local

#!/bin/bash

gitlab-ci-local \
  --remote-variables [email protected]:project/gitlab_templates=includes/include.drupalci.variables.yml=1.0.x \
  --variable="_GITLAB_TEMPLATES_REPO=project/gitlab_templates" \
  "$@"

This script does nothing more than making sure we use the correct variables to run our setup. If you put this is an directory that can be executed, like ~/.local/bin or /usr/local/bin you can use it anywhere in your shell.

So for example:

$ git clone [email protected]:project/keycdn
$ cd keycdn
$ drupal-ci-local composer # runs the composer step
$ drupal-ci-local # runs all steps!

What’s really cool about this is the fact that all the changes that the gitlab makes to your project. So should you run into something really weird, you can actually debug it! Yay!

Running chromedrivertests

There is are still some quirks, when using chromedriver will need some changes to know where to go. There are multiple containers:

  1. The main container is called: “build”
  2. The database is called: “database”
  3. The chromedriver is called: “chrome”

The current implementation for Drupal CI kinda cheats. It makes all services available on localhost on the runner. This is not default behaviour and uses a featureflag. The proper implementation would be to use the container names to get things up and running. This should be possible, but will need some work to make sure testing url’s and chromedriver urls are correct.

RTFM – there is more!

Really really really read the readme on the repository: firecow/gitlab-ci-local. There is loads of other things you can do like global variables and such. Lovely!

Conclusion

You can run Gitlab locally pretty easy. There will be quircks, but lets see if we can fix those together 🙂

Issue #3365438 on drupal.org.

DrupalCon Pittsburgh 2023: Project update bot – Road to Drupal 11

This year I ended up at DrupalCon again. Was loads of fun my US friends again. And finally meet so many that weren’t here last year. This year I gave a session on Project update bot, and how we will make the road to Drupal 11 even easier.

Drupal.org’s Project Update Bot was originally built for the Drupal 8 to 9 upgrade and is running again helping maintainers to make drupal.org projects compatible with Drupal 10 as well. It automatically opens issues in drupal.org projects’ issue queues when there is a Drupal 10 incompatibility identified, and provides a best effort fix as well. Then project maintainers and users can further refine the fix and include it in the project at their own timeframe.

With Drupal 10 launched it’s time to look forward at Drupal 11. The spike in work around the Drupal 10 launch put a lot of stress on core maintainers. the people working on PHPStan, the update bot and even project maintainers.

We will use this session to evaluate what went well, what should improve and how we can make Drupal.org’s Project Update Bot a continuous process that will help module developers during the whole Drupal 10 lifecycle, helping everyone’s transition into Drupal 11 be an even smoother experience.

Download the slides here: https://www.swis.nl/project-update-bot-road-to-drupal-11

The session is also available on YouTube if you’ve missed it 🙂

Decoupled and integration survey: help shape Drupal’s JSON:API’s future

I joined the JSON:API team last summer to help JSON:API move forward faster. Since then quite a few bugs have been fixed, new PHP API’s through hooks and events have been added and a lot of inconsistencies have been fixed.

Today we ask for your help. Are you using JSON:API to build decoupled applications or integrate Drupal in your organization’s IT landscape? Consider filling out our survey. The results of the survey will be used as the basis of the JSON:API roadmap for next year. Developing what the community and Drupal needs to stay relevant.

Have you been using Drupal’s JSON:API for building headless applications or system integration? Or have you explicitly chosen not to? Please consider helping us by filling out our ‘decoupled and integration survey’, it won’t take more than 5 minuten of your time.

You can find the blogpost with survey here: https://www.drupal.org/about/core/blog/decoupled-and-integration-survey-help-shape-drupals-jsonapis-future

Or open the survey directly

Building the trusted web with blockchain timestamps – DrupalCon Europe 2021

The internet is broken. We are slowly loosing trust in what we see and read on the internet. How can we bring back accountability and transparency on the net. Trust must become part of the internet’s DNA.

The solution is timestamping content on blockchain technology. From news outlets, to legal documents, ecommerce and social media. Timestamping is an open source solution to restore trust in the internet.

Each piece of content has an unique fingerprint, the hash. This hash is unique to that state of the content. Change any detail in the content and the hash will change. By storing the hash on a blockchain transaction you can forever prove that state of the content at a certain time.

The session was recorded and published on YouTube.

Drupal core maintainer for JSON:API

After helping out with jsonapi in Drupal core for a while now and taking on the maintenance on https://drupal.org/project/jsonapi_extras. I’ve been asked to join the subsystem maintainer team for the Drupal core JSON:API module. I gracefully accepted with support of SWIS.

I feel JSON:API is an essential part of keeping Drupal relevant in the future. I’m super excited to support Drupal in this role and keep the positive momentum of jsonapi moving forward. Thanks to Win Leers, Gabe Sullice and Mateu Aguiló Bosch for asking me to join the team.

Drupal Tech Talk 28: Where would we be without API’s?

Thursday the 12th of september SWIS is hosting a Drupal Tech Talk. There will be pizza, drinks, snacks and two awesome talks about oAuth and JSON:API.

Care to come? Sign up on meetup.

Drupal Tech Talk 28: Where would we be without API’s?

Thursday, Sep 12, 2019, 5:30 PM

SWIS Full Service Internet Bureau
3e Binnenvestgracht 23 T1 Leiden, ZH

13 Drupalistas Attending

Zonder API’s was het een stuk moeilijker om verschillende stukjes software te laten samenwerken en zou onze code voortdurend stuk gaan. Dus bij de 28e Drupal Tech Talk heffen we het glas op de API’s en duiken we in de werelden van OAuth en JSON:API. Voor deze Drupal Tech Talk zijn we te gast bij SWIS in Leiden. Zoals altijd zijn we de gastheer zeer…

Check out this Meetup →

Stop the noise! Introduction to the JSON:API specification

If you’ve ever argued about the way your JSON responses should be formatted, JSON:API can be your anti-bikeshedding tool. JSON:API is a great way to expose a consistent API in your application.

In this session, we will talk about how JSON:API got to where it is today and how it can help you make Drupal the core of all your online transactions. We will check out the specifications and look at the main benefits of JSON:API.

Expect to learn the structure and features of the JSON:API specifications and why it should be your smart default. You should be able to get started right away with some examples we will provide in this session.

https://www.slideshare.net/slideshow/embed_code/key/49nwSHRxPDRZbB
The presentation given on DrupalJam 2019

Further reading

Want to help out?

Presenting: Stop the noise! Introduction to JSON:API @ DrupalJam 2019

Friday the 28th of june i’ll be giving a talk on DrupalJam. Quite a lot of the guys from SWIS will be there too.

Stop the noise! Introduction to the JSON:API specification

If you’ve ever argued about the way your JSON responses should be formatted, JSON:API can be your anti-bikeshedding tool. JSON:API is a great way to expose a consistent API in your application.

In this session, we will talk about how JSON:API got to where it is today and how it can help you make Drupal the core of all your online transactions. We will check out the specifications and look at the main benefits of JSON:API.

Expect to learn the structure and features of the JSON:API specifications and why it should be your smart default. You should be able to get started right away with some examples we will provide in this session.

28th june – 11:00-12:00 in Kalvermelk 1