Running shopware 5 on a Debian Jessie machine with nginx and php5-fpm, we get very often a 502 Bad Gateway. This happens mostly in backend when longer operations are working like thumbnail creation, even if this is done within small chunks of single ajax requests.
The used server with 64 GB RAM and 16 Cores is sleeping at all, because there is no real traffic on it. We use it like a staging system currently unless we have fixed all errors like this one.
Error log:
In the nginx-error log the following lines can be found then:
[error] 20524#0: *175 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "POST /backend/MediaManager/createThumbnails HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/" [error] 20524#0: *175 no live upstreams while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "POST /backend/Log/createLog HTTP/1.1", upstream: "fastcgi://php-fpm", host: "www.domain.com", referrer: "http://www.domain.com/backend/" [error] 20524#0: *175 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "GET /backend/login/getLoginStatus?_dc=1457014588680 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/" [error] 20522#0: *209 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "GET /backend/login/getLoginStatus?_dc=1457014618682 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/"
Maybe it is notable, that at first lot of “*175 connect” errors occure and then finally a “*209 connect”.
Config files:
I’ll try to post only significant lines related to this topic and will leave out all those lines which are commented out.
php-fpm:
/etc/php5-fpm/pool.d/www.conf:
[www] user = www-data group = www-data listen = /var/run/php5-fpm.sock listen.owner = www-data listen.group = www-data pm.max_children = 5 pm.start_servers = 2 pm.min_spare_servers = 1 pm.max_spare_servers = 3
nginx:
/etc/nginx/nginx.conf:
user www-data; worker_processes auto; pid /run/nginx.pid; events { worker_connections 768; multi_accept on; } http { ## MIME types. include /etc/nginx/mime.types; default_type application/octet-stream; ## Default log and error files. access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## Use sendfile() syscall to speed up I/O operations and speed up ## static file serving. sendfile on; ## Handling of IPs in proxied and load balancing situations. # set_real_ip_from 192.168.1.0/24; # set to your proxies ip or range # real_ip_header X-Forwarded-For; ## Timeouts. client_body_timeout 60; client_header_timeout 60; keepalive_timeout 10 10; send_timeout 60; ## Reset lingering timed out connections. Deflect DDoS. reset_timedout_connection on; ## Body size. client_max_body_size 10m; ## TCP options. tcp_nodelay on; ## Optimization of socket handling when using sendfile. tcp_nopush on; ## Compression. gzip on; gzip_buffers 16 8k; gzip_comp_level 1; gzip_http_version 1.1; gzip_min_length 10; gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon application/vnd.ms-fontobject font/opentype application/x-font-ttf; gzip_vary on; gzip_proxied any; # Compression for all requests. gzip_disable "msie6"; ## Hide the Nginx version number. server_tokens off; ## Upstream to abstract backend connection(s) for PHP. upstream php-fpm { server unix:/var/run/php5-fpm.sock; # server 127.0.0.1:9000; ## Create a backend connection cache. keepalive 32; } ## Include additional configs include /etc/nginx/conf.d/*.conf; ## Include all vhosts. include /etc/nginx/sites-enabled/*; }
/etc/nginx/sites-available/site.conf:
server { listen 80; listen 443 ssl; server_name xxxxxxxx.com; root /var/www/shopware; ## Access and error logs. access_log /var/log/nginx/xxxxxxxx.com.access.log; error_log /var/log/nginx/xxxxxxxx.com.error.log; ## leaving out lots of shopware/mediafiles-related settings ## .... ## continue: location ~ .php$ { try_files $uri $uri/ =404; ## NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini fastcgi_split_path_info ^(.+.php)(/.+)$; ## required for upstream keepalive # disabled due to failed connections #fastcgi_keep_conn on; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param SHOPWARE_ENV $shopware_env if_not_empty; fastcgi_param ENV $shopware_env if_not_empty; # BC for older SW versions fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; client_max_body_size 24M; client_body_buffer_size 128k; ## upstream "php-fpm" must be configured in http context fastcgi_pass php-fpm; } }
What to do now? Please let me now if i should provide further information to this question.
Update
After applying nginx- and fpm-settings from @peixotorms, the errors in nginx-logs changed to:
30 upstream timed out (110: Connection timed out) while reading response header from upstream
But the issue itself isn’t solved. It has just another face…
Advertisement
Answer
It might sound strange to you, but your problem is most probably due to the fact that you’re running PHP on a socket instead of a tcp port. You will start seeing 502 errors (and others) when you have around 300 concurrent requests (sometimes less) to php on a socket configuration.
Also your pm.max_children
is way too low, unless you want to limit your server to around 5 simultaneous php requests maximum: http://php.net/manual/en/install.fpm.configuration.php
Configure it this way, and those errors should go away:
For your nginx.conf change the following values:
worker_processes 4; worker_rlimit_nofile 750000; # handles connection stuff events { worker_connections 50000; multi_accept on; use epoll; } upstream php-fpm { keepalive 30; server 127.0.0.1:9001; }
Your /etc/php5-fpm/pool.d/www.conf
(Use these settings because you have plenty or RAM and CPU)
[www] user = www-data group = www-data listen.owner = www-data listen.group = www-data listen.mode = 0660 listen = 127.0.0.1:9001 listen.allowed_clients = 127.0.0.1 listen.backlog = 65000 pm = dynamic pm.max_children = 1024 pm.start_servers = 8 pm.min_spare_servers = 4 pm.max_spare_servers = 16 pm.max_requests = 10000
Also add this on your location ~ .php$ {
block:
location ~ .php$ { try_files $uri $uri/ =404; ## NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini fastcgi_split_path_info ^(.+.php)(/.+)$; ## required for upstream keepalive # disabled due to failed connections #fastcgi_keep_conn on; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param SHOPWARE_ENV $shopware_env if_not_empty; fastcgi_param ENV $shopware_env if_not_empty; # BC for older SW versions fastcgi_keep_conn on; fastcgi_connect_timeout 20s; fastcgi_send_timeout 60s; fastcgi_read_timeout 60s; fastcgi_pass php-fpm; }
EDIT:
Change the values below on your /etc/php5/fpm/php.ini
file to this and restart:
safe_mode = Off output_buffering = Off zlib.output_compression = Off max_execution_time = 900 max_input_time = 900 memory_limit = 2048M post_max_size = 120M file_uploads = On upload_max_filesize = 120M