Last updated 11th May 2021

Force a redeploy

There are times where you might want to trigger a redeployment of your application.

You can redeploy each environment in the management console by clicking the "Redeploy" button on the top right corner.

Redeploy button in Console

Redeploys are also available using the following Web PaaS CLI command:

webpaas redeploy

Please note that the redeploy will happen after any scheduled builds in either "Running" or "Pending" state.

Clear the build cache

In rare circumstances the build cache, used to speed up the build process, may become corrupted. That may happen if, for example, code is being downloaded from a 3rd party language service like Packagist or NPM while that service is experiencing issues. To flush the build cache entirely run the following command:

webpaas project:clear-build-cache

That will wipe the build cache for the current project entirely. Naturally the next build for each environment will likely be longer as the cache rebuilds.

HTTP responses 502 Bad Gateway or 503 Service Unavailable

These errors indicate your application (or application runner, like PHP-FPM) is crashing or unavailable. Typical causes include:

  • Your configuration has an error and the process is not starting or requests are not able to be forwarded to it correctly. Check your web.commands.start entry or that your passthru configuration is correct.
  • The amount of traffic coming to your site exceeds the processing power of your application.
  • Certain code path(s) in your application are too slow and timing out.
  • A PHP process is crashing because of a segmentation fault (see below).
  • A PHP process is killed by the kernel out-of-memory killer (see below).

Large file upload failing (10MB limit)

When trying to upload a large JSON file to your API you might see a 400 response code (Malformed request).

Web PaaS enforces a 10MB limit on sending files with the application/json Content-Type header. If you want to send large files through, you will have to send them with multipart/form-data instead:

$ curl -XPOST '' --header 'Content-Type: multipart/form-data' -F file=large_file.json

Claimed domains

The error

This domain is already claimed by another project. If this is incorrect or you are trying to add a subdomain, please open a ticket with support.

is related to Web PaaS's subdomain highjacking prevention assumptions, and likely occurred during an attempt to assign subdomains across multiple projects. Consult the documentation linked above for instructions for how to modify your DNS records to bypass some those assumptions regarding project domain ownership.

Error provisioning the new certificate

One reason Let's Encrypt certificates may fail to provision on your environments has to do with the 64 character limit Let's Encrypt places on URLs. If the names of your branches are too long, the Web PaaS generated environment URL will go over this limit, and the certificate will be rejected.

See Let's Encrypt limits and branch names for a more detailed breakdown of this issue.

Total disk usage exceeds project maximum

One of the billable parameters in your project's settings is Storage. This global storage pool is allocated among the various services and application containers in your project via the disk parameter. The sum of all disk parameters in your project's YAML config files must be less than or equal to the global project storage number.

Error: Resources exceeding plan limit; disk: 8192.00MB > 5120.00MB; try removing a service, or add more storage to your plan

This means that you have allocated, for example, disk: 4096 in a MySQL service in services.yaml and also disk: 4096 in the for your application, while only having the minimum default of 5GB storage for your project as a whole. The solution is either to lower the disk parameters to within the limits of 5GB of storage, or raise the global storage parameter on your project's settings to at least 10GB.

Because storage is a billable component of your project, only the project's owner can make this change.

Check your application's disk space

Run webpaas ssh within your project folder to login to the container's shell. Then use the df command to check the available writable space for your application.

df -h -x tmpfs -x squashfs | grep -v /run/shared

This command will show the writable mounts on the system, similar to:

Filesystem                                                       Size  Used Avail Use% Mounted on
/dev/mapper/platform-syd7waxqy4n5q--master--7rqtwti----app       2.0G   37M  1.9G   2% /mnt
/dev/mapper/platform-tmp--syd7waxqy4n5q--master--7rqtwti----app  3.9G   42M  3.8G   2% /tmp

The first line shows the storage device that is shared by all of your persistent disk mounts. All defined mounts use a common storage pool. In this example, the application container has allocated 2 GB of the total disk space. Of those 2GB, 2% (37 MB) is used by all defined mounts.

The second line is the operating system temporary directory, which is always the same size. While you can write to the /tmp directory files there are not guaranteed to persist and may be deleted on deploy.

Increase the disk space available

The sum of all disk keys defined in your project's and .platform/services.yaml files must be equal or less than the available storage in your plan.

1. Buy extra storage for your project

Each project comes with 5GB of Disk Storage available to each environment. To increase the disk space available for your project, click on "Edit Plan" to increase your storage in bulks of 5GB.

2. Increase your application and services disk space

Once you have enough storage available, you can increase the disk space allocated for your application and services using disk keys in your and .platform/services.yaml.

Check the following resources for more details:

No space left on device

During the build hook, you may run into the following error depending on the size of your application:

W: [Errno 28] No space left on device: ...

The cause of this issue has to do with the amount of disk provided to the build container before it is deployed. Application images are restricted to 4 GB during build, no matter how much writable disk has been set aside for the deployed application.

Some build tools (yarn/npm) store cache for different versions of their modules. This can cause the build cache to grow over time beyond the maximum of 4GB. Try clearing the build cache and redeploying. In most cases, this will resolve the issue.

If for some reason your application requires more than 4 GB during build, you can open a support ticket to have this limit increased. The most disk space available during build still caps off at 8 GB in these cases.

MySQL lock wait timeout

If you receive MySQL error messages like this:

SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded;

This means a process running in your application acquired a lock from MySQL for a long period of time. That is typically caused by one of the following:

  • There are multiple places acquiring locks in different order. For example, code path 1 first locks record A and then locks record B. Code path 2, in contrast, first locks record B and then locks record A.
  • There is a long running background process executed by your application that holds the lock until it ends.

If you're using MariaDB 10+, you can use the SQL query SHOW FULL PROCESSLIST \G to list DB queries waiting for locks. Find output like the following, and start debugging.

< skipped >
Command: Query
Time: ...
State: Waiting for table metadata lock
Info: SELECT ...
< skipped >

To find active background processes, run ps aufx on your application container.

Also, please make sure that locks are acquired in a pre-defined order and released as soon as possible.

MySQL: definer/invoker of view lack rights to use them

There is a single MySQL user, so you can not use "DEFINER" Access Control mechanism for Stored Programs and Views.

When creating a VIEW, you may need to explicitly set the SECURITY parameter to INVOKER:

VIEW `view_name` AS

MySQL server has gone away

Disk space issues

Errors such as "PDO Exception 'MySQL server has gone away'" are usually simply the result of exhausting your existing diskspace. Be sure you have sufficient space allocated to the service in .platform/services.yaml.

The current disk usage can be checked using the CLI command webpaas db:size. Because of issues with the way InnoDB reports its size, this can out by up to 20%. As table space can grow rapidly, it is usually advisable to make your database mount size twice the size reported by the db:size command.

Worker timeout

Another possible cause of "MySQL server has gone away" errors is a server timeout. MySQL has a built-in timeout for idle connections, which defaults to 10 minutes. Most typical web connections end long before that is ever approached, but it's possible that a long-running worker may idle and not need the database for longer than the timeout. In that case the same "server has gone away" message may appear.

If that's the case, the best way to handle it is to wrap your connection logic in code that detects a "server has gone away" exception and tries to re-establish the connection.

Alternatively, if your worker is idle for too long it can self-terminate. Web PaaS will automatically restart the worker process, and the new process can establish its own new database connection.

Packet size limitations

Another cause of the "MySQL server has gone away" errors can be the size of the database packets. If that is the case, the logs may show warnings like "Error while sending QUERY packet" before the error. One way to resolve the issue is to use the max_allowed_packet parameter described above.

ERROR: permission denied to create database

The provided user does not have permission to create databases.
The database is created for you and can be found in the path field of the $PLATFORM_RELATIONSHIPS environment variable.

"Read-only file system" error

Everything will be read-only, except the writable mounts you declare. Writable mounts are there for your data: for file uploads, logs and temporary files. Not for your code. In order to change code on Web PaaS you have to go through Git.

This is what gives you all of the benefits of having repeatable deployments, consistent backups, traceability, and the magically fast creation of new staging/dev environments.

In Web PaaS, you cannot just "hack production". It is a constraint, but it is a good constraint.

During the build phase of your application, the main filesystem is writable. So you can do whatever you want (e.g. compile code or generate anything you need). But during and after the deploy phase, the main filesystem will be read-only.

Failed to connect to the Git repository

If you have been granted access to a Web PaaS project, but are unable to clone the repository via the management console or with the CLI command webpaas get <projectID>, this is likely due to the fact that 1) the project has an external repository integration configured to GitHub, GitLab, or Bitbucket, and 2) you have not been granted access to that integrated repository.

It will be necessary to update your access settings on the integrated repository before you can clone and commit to the project.

RootNotFoundException from the CLI

If you check out a project via Git directly and not using the webpaas get command, you may end up with the CLI unable to determine what project it's in. If you run a CLI command from within the project directory you've checked out but get an error like this:

[RootNotFoundException] Project root not found. This can only be run from inside a project directory.

Then the CLI hasn't been able to determine the project to use. To fix that, run:

webpaas project:set-remote <project_id>

where <project_id> is the random-character ID of the project. That can be found by running webpaas projects from the command line to list all accessible projects. Alternatively, it can be found in the management console after the webpaas get command shown or in the URL of the management console or project domain.

"File not found" in Drupal

If you see a bare "File not found" error when accessing your Drupal site with a browser, this means that you've pushed your code as a vanilla project but no index.php has been found.

Make sure your repository contains an index.php file in the web location root, or that your Drush make files are properly named.

PHP-specific error messages

server reached max_children

You may see a line like the following in the /var/log/app.log file:

WARNING: [pool web] server reached max_children setting (2), consider raising it

That indicates that the server is receiving more concurrent requests than it has PHP processes allocated, which means some requests will have to wait until another finishes. In this example there are 2 PHP processes that can run concurrently.

Web PaaS sets the number of workers based on the available memory of your container and the estimated average memory size of each process. There are two ways to increase the number of workers:

  • Adjust the worker sizing hints for your project.
  • Upgrade your subscription on Web PaaS to get more computing resources. To do so, log into your account and edit the project.

Execution timeout

If your PHP application is not able to handle the amount of traffic or it is slow, you should see log lines from /var/log/app.log like any of the below:

WARNING: [pool web] child 120, script '/app/public/index.php' (request: "GET /index.php") execution timed out (358.009855 sec), terminating

That means your PHP process is running longer than allowed. You can adjust the max_execution_time value in php.ini, but there is still a 5 minute hard cap on any web request that cannot be adjusted.

The most common cause of a timeout is either an infinite loop (which is a bug that you should fix) or the work itself requires a long time to complete. For the latter case, you should consider putting the task into a background job.

The following command will identify the 20 slowest requests in the last hour, which can provide an indication of what code paths to investigate.

grep $(date +%Y-%m-%dT%H --date='-1 hours') /var/log/php.access.log | sort -k 4 -r -n | head -20

Otherwise, you may check if the following options are applicable:

  • Find the most visited pages and see if they can be cached and/or put behind a CDN. You may refer to how caching works.
  • Upgrade your subscription on Web PaaS to get more computing resources. To do so, log into your account and edit the project subscription.

PHP process crashed

If your PHP process crashed with a segmentation fault, you should see log lines in /var/log/app.log like below:

WARNING: [pool web] child 112 exited on signal 11 (SIGSEGV) after 7.405936 seconds from start

This is complicated, either a PHP extension is hitting a segmentation fault or your PHP application code is crashing. You should review recent changes in your application and try to find the cause of it, probably with the help of XDebug.

PHP process is killed

If your PHP process is killed by the kernel, you should see log lines in /var/log/app.log like this:

WARNING: [pool web] child 429 exited on signal 9 (SIGKILL) after 50.938617 seconds from start

That means the memory usage of your container exceeds the limit allowed on your plan so the kernel kills the offending process. You should try the following:

  • Check if the memory usage of your application is expected and try to optimize it.
  • Use sizing hints to reduce the amount of PHP workers which reduces the memory footprint.
  • Upgrade your subscription on Web PaaS to get more computing resources. To do so, log into your account and edit the project.

Stuck build or deployment

If you see a build or deployment running longer than expected, that may be one of the following cases:

1. The build is blocked by a process in your build hook.

2. The deployment is blocked by a long running process in your deploy hook.

3. The deployment is blocked by a long running cron job in the environment.

4. The deployment is blocked by a long running cron job in the parent environment.

To determine if your environment is being stuck in the build or the deployment, you can look at the build log available in the management console. If you see a line similar to the following:

Re-deploying environment w6ikvtghgyuty-drupal8-b3dsina.

It means the build has completed successfully and the system is trying to deploy. If that line never appears then it means the build is stuck.

When a deployment is blocked, you should try the following:

1. Use SSH to connect to your environment. Find any long-running cron jobs or deploy hooks on the environment by running ps afx. Once you have identified the long running process on the environment, kill it with kill <PID>. PID stands for the process id shown by ps afx.

2. If you're performing "Sync" or "Activate" on an environment and the process is stuck, use SSH to connect to the parent environment and identify any long running cron jobs with ps afx. Kill the job(s) if you see any.

Slow or failing build or deployment

Builds that take long time or fail is a common problem. Most of the time it's related to an application issue and they can be hard to troubleshoot without guidance.

Here are a few tips that can help you solve the issues you are experiencing.

Check for errors in the logs

Invisible errors during the build and deploy phase can cause increased wait times, failed builds and other problems. Investigating each log and fixing errors is essential.

Related documentation: Accessing logs

Build and deploy hooks

Hooks are frequently the cause of long build time. If they run into problem they can cause the build to fail or hang indefinitely.

The build hook can be tested in your local environment. Because the deployed environment on Web PaaS is read-only the build hooks cannot be rerun there.

Deploy hooks can be tested either locally or by logging into the application over SSH and running them there. They should execute safely but be aware that depending on what your scripts are doing they may have an adverse impact on the running application (e.g., flushing all caches).

Furthermore, you can test your hooks with these Linux commands to help figure out any problems:

time $cmd # Print execution time
strace -T $cmd # Print a system call report

Related documentation: Build and deploy hooks

Cron jobs

Containers cannot be shutdown while long-running tasks are active. That means long-running cron jobs will block a container from being shut down to make way for a new deploy.

For that reason, make sure your custom cron jobs execution times are low and that they are running properly. Be aware that cron jobs may invoke other services in unexpected ways, which can increase execution time.

note Drupal's drush core-cron run installed module's cron task. Those can be, for example; evicting invalid cache, updating database records, regenerating assets. Be sure to frequently benchmark the drush core-cron command in all your environments, as it is a common source of performance issues.

Related documentation: Cron and scheduled tasks

Did you find this guide useful?

Please feel free to give any suggestions in order to improve this documentation.

Whether your feedback is about images, content, or structure, please share it, so that we can improve it together.

Your support requests will not be processed via this form. To do this, please use the "Create a ticket" form.

Thank you. Your feedback has been received.

These guides might also interest you...

OVHcloud Community

Access your community space. Ask questions, search for information, post content, and interact with other OVHcloud Community members.

Discuss with the OVHcloud community

In accordance with the 2006/112/CE Directive, modified on 01/01/2015, prices incl. VAT may vary according to the customer's country of residence
(by default, the prices displayed are inclusive of the UK VAT in force).