| layout | title |
|---|---|
doc |
12-ParallelExecution - Codeception - Documentation |
When execution time of your tests is longer than a coffee break, it is a good reason to think about making your tests faster. If you have already tried to run them on SSD drive, and the execution time still upsets you, it might be a good idea to run your tests in parallel.
Codeception does not provide a command like run-parallel. There is no common solution that can play well for everyone. Here are the questions you will need to answer:
- How parallel processes will be executed?
- How parallel processes won't affect each other?
- Will they use different databases?
- Will they use different hosts?
- How should I split my tests across parallel processes?
There are two approaches to achieve parallelization. We can use Docker and run each process inside isolated containers, and have those containers executed simultaneously.
Docker works really well for isolating testing environments. By the time of writing this chapter, we didn't have an awesome tool like it. This chapter demonstrates how to manage parallel execution manually. As you will see we spend too much effort trying to isolate tests which Docker does for free. Today we recommend using Docker for parallel testing.
Please make sure you have docker installed. Docker experience is required as well.
Run official Codeception image from DockerHub:
docker run codeception/codeception
Running tests from a project, by mounting the current path as a host-volume into the container.
The default working directory in the container is /project.
docker run -v ${PWD}:/project codeception/codeception run
To prepare application and tests to be executed inside containers you will need to use Docker Compose to run multiple containers and connect them together.
Define all required services in docker-compose.yml file. Make sure to follow Docker philisophy: 1 service = 1 container. So each process should be defined as its own service. Those services can use official Docker images pulled from DockerHub. Directories with code and tests should be mounted using volume directive. And exposed ports should be explicitly set using ports directive.
We prepared a sample config with codeception, web server, database, and selenium with Chrome to be executed together.
{% highlight yaml %}
version: '3' services: codecept: image: codeception/codeception depends_on: - chrome - web volumes: - .:/project web: image: php:7-apache depends_on: - db volumes: - .:/var/www/html db: image: percona:5.6 chrome: image: selenium/standalone-chrome
{% endhighlight %}
Codeception service will execute command codecept run but only after all services are started. This is defined using depends_on parameter.
It is easy to add more custom services. For instance to use Redis you just simple add this lines:
{% highlight yaml %}
redis: image: redis:3
{% endhighlight %}
By default the image has codecept as its entrypoint, to run the tests simply supply the run command
{% highlight yaml %} docker-compose run --rm codecept help
{% endhighlight %}
Run suite
{% highlight yaml %} docker-compose run --rm codecept run acceptance
{% endhighlight %}
{% highlight yaml %} docker-compose run --rm codecept run acceptance LoginCest
{% endhighlight %}
Development bash
{% highlight yaml %} docker-compose run --rm --entrypoint bash codecept
{% endhighlight %}
And finally to execute testing in parallel you should define how you split your tests and run parallel processes for docker-compose. Here we split tests by suites, but you can use different groups to split your tests. In section below you will learn how to do that with Robo.
{% highlight yaml %}
docker-compose --project-name test-web run -d --rm codecept run --html report-web.html web &
docker-compose --project-name test-unit run -d --rm codecept run --html report-unit.html unit &
docker-compose --project-name test-functional run -d --rm codecept run --html report-functional.html functional
{% endhighlight %}
At the end, it is worth specifying that Docker setup can be complicated and please make sure you understand Docker and Docker Compose before proceed. We prepared some links that might help you:
- Acceptance Tests Demo Repository
- Dockerized Codeception Internal Tests
- Phundament App with Codeception
If you want to automate splitting tests by parallel processes, and executing them using PHP script you should use Robo task runner to do that.
Parallel Test Execution consists of 3 steps:
- splitting tests
- running tests in parallel
- merging results
We propose to perform those steps using a task runner. In this guide we will use Robo task runner. It is a modern PHP task runner that is very easy to use. It uses Symfony Process to spawn background and parallel processes. Just what we need for the step 2! What about steps 1 and 3? We have created robo tasks for splitting tests into groups and merging resulting JUnit XML reports.
To conclude, we need:
- Robo, a task runner.
- robo-paracept - Codeception tasks for parallel execution.
Execute this command in an empty folder to install Robo and Robo-paracept : {% highlight bash %}
$ composer require codeception/robo-paracept --dev
{% endhighlight %}
You need to install Codeception after, if codeception is already installed it will not work. {% highlight bash %}
$ composer require codeception/codeception
{% endhighlight %}
Initializes basic RoboFile in the root of your project
{% highlight bash %}
$ robo init
{% endhighlight %}
Open RoboFile.php to edit it
{% highlight php %}
taskSplitTestFilesByGroups(5) ->projectRoot('.') ->testsFrom('tests/acceptance') ->groupsTo('tests/_data/paracept_') ->run(); /* // Split your tests by single tests (alternatively) $this->taskSplitTestsByGroups(5) ->projectRoot('.') ->testsFrom('tests/acceptance') ->groupsTo('tests/_data/paracept_') ->run(); */ } {% endhighlight %} Let's prepare group files: {% highlight bash %} $ robo parallel:split-tests [Codeception\Task\SplitTestFilesByGroupsTask] Processing 33 files [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_1 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_2 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_3 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_4 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_5 {% endhighlight %} Now we have group files. We should update `codeception.yml` to load generated group files. In our case we have groups: *paracept_1*, *paracept_2*, *paracept_3*, *paracept_4*, *paracept_5*. {% highlight yaml %} groups: paracept_*: tests/_data/paracept_* {% endhighlight %} Let's try to execute tests from the second group: {% highlight bash %} $ codecept run acceptance -g paracept_2 {% endhighlight %} #### Step 2: Running Tests Robo has `ParallelExec` task to spawn background processes. ##### Inside Container If you are using [Docker](#docker) containers you can launch multiple Codeception containers for different groups: {% highlight php %} public function parallelRun() { $parallel = $this->taskParallelExec(); for ($i = 1; $i <= 5; $i++) { $parallel->process( $this->taskExec('docker-compose run --rm codecept run') ->option('group', "paracept_$i") // run for groups paracept_* ->option('xml', "tests/_log/result_$i.xml") // provide xml report ); } return $parallel->run(); } {% endhighlight %} ##### Locally If you want to run tests locally just use preinstalled `taskCodecept` task of Robo to define Codeception commands and put them inside `parallelExec`. {% highlight php %} taskParallelExec(); for ($i = 1; $i <= 5; $i++) { $parallel->process( $this->taskCodecept() // use built-in Codecept task ->suite('acceptance') // run acceptance tests ->group("paracept_$i") // for all paracept_* groups ->xml("tests/_log/result_$i.xml") // save XML results ); } return $parallel->run(); } {% endhighlight %} In case you don't use containers you can isolate processes by starting different web servers and databases per each test process. We can define different databases for different processes. This can be done using [Environments](https://codeception.com/docs/07-AdvancedUsage#Environments). Let's define 5 new environments in `acceptance.suite.yml`: {% highlight yaml %} actor: AcceptanceTester modules: enabled: - Db: dsn: 'mysql:dbname=testdb;host=127.0.0.1' user: 'root' dump: 'tests/_data/dump.sql' populate: true cleanup: true - WebDriver: url: 'http://localhost/' env: env1: modules: config: Db: dsn: 'mysql:dbname=testdb_1;host=127.0.0.1' WebDriver: url: 'http://test1.localhost/' env2: modules: config: Db: dsn: 'mysql:dbname=testdb_2;host=127.0.0.1' WebDriver: url: 'http://test2.localhost/' env3: modules: config: Db: dsn: 'mysql:dbname=testdb_3;host=127.0.0.1' WebDriver: url: 'http://test3.localhost/' env4: modules: config: Db: dsn: 'mysql:dbname=testdb_4;host=127.0.0.1' WebDriver: url: 'http://test4.localhost/' env5: modules: config: Db: dsn: 'mysql:dbname=testdb_5;host=127.0.0.1' WebDriver: url: 'http://test5.localhost/' {% endhighlight %} ---- After the `parallelRun` method is defined you can execute tests with {% highlight bash %} $ robo parallel:run {% endhighlight %} #### Step 3: Merge Results In case of `parallelExec` task we recommend to save results as JUnit XML, which can be merged and plugged into Continuous Integration server. {% highlight php %} taskMergeXmlReports(); for ($i=1; $i<=5; $i++) { $merge->from("tests/_output/result_paracept_$i.xml"); } $merge->into("tests/_output/result_paracept.xml")->run(); } {% endhighlight %} Now, we can execute : {% highlight bash %} $ robo parallel:merge-results {% endhighlight %} `result_paracept.xml` file will be generated. It can be processed and analyzed. #### All Together To create one command to rule them all we can define new public method `parallelAll` and execute all commands. We will save the result of `parallelRun` and use it for our final exit code: {% highlight php %} parallelSplitTests(); $result = $this->parallelRun(); $this->parallelMergeResults(); return $result; } {% endhighlight %} ## Conclusion Codeception does not provide tools for parallel test execution. This is a complex task and solutions may vary depending on a project. We use [Robo](https://robo.li) task runner as an external tool to perform all required steps. To prepare our tests to be executed in parallel we use Codeception features of dynamic groups and environments. To do even more we can create Extensions and Group classes to perform dynamic configuration depending on a test process. * **Previous Chapter: [< ContinuousIntegration](/docs/12-ContinuousIntegration)**