W3 Total Cache

easyengine (ee) note: If you are using easyengine, you can accomplish everything in this article using following commands:

ee site create example.com --w3tc

Assumption:

  1. You already installed PHP, MySQL, Nginx & Postfix
  2. You already installed a fresh WordPress or moved an existing WordPress to current server

Based on these assumptions, we will jump to directly a WordPress-Nginx configuration part.

Standard WordPress-Nginx configuration with W3 Total Cache:

W3 Total Cache plugin provides many options. I will recommend using the given below:

  • Page Cache – Disk Enhanced
  • Minify – Disable this unless you are 100% sure your theme/plugin will support it nicely. Automatic minification rarely works for us.
  • Database Cache – Memcache
  • Object Cache – Memcache
  • Browser Cache – Disabled
  • CDN – You can use any CDN. Origin Pull is recommended. Check CDN Setup Guides for details.

If you have noticed, I have highlighted two options only. These are the only two options we are  going to handle in our configuration.

server {
    server_name example.com www.example.com;

    access_log   /var/log/nginx/example.com.access.log;
    error_log    /var/log/nginx/example.com.error.log debug;

    root /var/www/example.com/htdocs;
    index index.php;

    set $cache_uri $request_uri;

    # POST requests and urls with a query string should always go to PHP
    if ($request_method = POST) {
        set $cache_uri 'null cache';
    }   
    if ($query_string != "") {
        set $cache_uri 'null cache';
    }   

    # Don't cache uris containing the following segments
    if ($request_uri ~* "(/wp-admin/|/xmlrpc.php|/wp-(app|cron|login|register|mail).php|wp-.*.php|/feed/|index.php|wp-comments-popup.php|wp-links-opml.php|wp-locations.php|sitemap(_index)?.xml|[a-z0-9_-]+-sitemap([0-9]+)?.xml)") {
        set $cache_uri 'null cache';
    }   

    # Don't use the cache for logged in users or recent commenters
    if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in") {
        set $cache_uri 'null cache';
    }

    # Use cached or actual file if they exists, otherwise pass request to WordPress
    location / {
        try_files /wp-content/cache/page_enhanced/${host}${cache_uri}_index.html $uri $uri/ /index.php?$args ;
    }

        location ~ ^/wp-content/cache/minify/[^/]+/(.*)$ {
                try_files $uri /wp-content/plugins/w3-total-cache/pub/minify.php?file=$1;
        }    

    location = /favicon.ico { log_not_found off; access_log off; }
    location = /robots.txt  { log_not_found off; access_log off; }

    location ~ \.php$ {
        try_files $uri =404; 
        include fastcgi_params;
                fastcgi_pass 127.0.0.1:9000;
    }

    # Cache static files for as long as possible
    location ~* .(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
               expires max; log_not_found off; access_log off;
    }
}

Notes:

  1. To simplify configuration, I haven’t added Mobile user agent checks. Its better to use responsive design than maintaining a separate site for mobile devices.
  2. You may not see special rules for minify feature. Above rules can handle W3 total minification without any change.

Don’t Forget:

Always test your Nginx configuration and then reload it. All changes to Nginx config must be follow with these commands:

nginx -t && service nginx reload

Must Read:

64 responses to “W3 Total Cache”

    • Line expires max; in Nginx configuration takes care of browser caching at nginx-level ver well. It sets cache duration for 25-years.

      I am using w3 total cache from long time. In fact, I wrote about Nginx+W3 total cache 2 years ago. 🙂

      There is nothing wrong in “Browse Cache” option provided by your plugin. It’s just that nginx can handle it faster and itself. IMHO its always better to save PHP from some work. For this reason, I also recommend disabling gzip/compression related options.

      By the way, I already scheduled a W3 Total Cache + Multisite article around weekend. That will also have few special configuration options.

      • Sorry, I think there’s some confusion. If you check the Install tab for example, you can see that W3TC generates the directives for your .conf file dynamically when you change policy (settings). So PHP isn’t actually doing anything in that regard, that would be non-performant. I hope that clears things up.

        • I have checked generated .conf files sometime back and I tried to cover everything in above configuration.
          Does “Browser Caching” option is responsible for adding some rules into .conf file only?
          I will check .conf file on my end again.

          By the way, I just verified browser caching for this site by using following command:
          curl -I http://rtcamp.com/files/2012/09/wordpress-nginx.jpeg
          As expected, Nginx returned correct Expires header in its output.
          Expires: Thu, 31 Dec 2037 23:55:55 GMT

          Please correct me if I am missing anything here.

          • The intent is for W3TC to generate Apache / Nginx rules that you can include into your main web server conf. So in this case the difference between using W3TC’s settings and doing it yourself is only that you can manage everything in one place by using W3TC. The W3TC is trying to take all of the policies you’re setting in the plugin and create the appropriate rules.

  2. Rahul,

    have you tried benchmarking WP + Nginx + fastcgi_cache versus WP + Nginx + W3TC? What are your thoughts and preferred setup between the two?

    I have been using WP + Nginx + fastcgi_cache flawlessly after reading your tutorial, but wonder if I would have any performance gain with W3TC instead.

    Keep up the great work!

    • @Resende

      Glad to know that you found this useful. 🙂

      I haven’t done any benchmarking yet, but once a page is cached, fastcgi_cache and W3 Total Cache may not differ much. Once a page is cached Nginx alone can serve it, assuming cache is properly configured.

      Technically, fastcgi_cache will have upper age as:

      1. It is using tmpfs i.e. RAM-based storage. (Note: with some extra work we can move W3 Total Cache’s storage folder on tmpfs)
      2. It will need less CPU-cycles as fastcgi_cache rules are relatively simpler
      3. Nginx uses hashing for efficient storage & retrieval so on large site time it take to locate correct page for a requested URL will be less for fastcgi_cache.

      For a cache-miss, definitely fastcgi_cache method will be faster as PHP-process won’t be executing any codes for cache-generation.

      These days, I am running some benchmarks on a client server. I will ask them if I can run fastcgi_cache v/s W3 Total Cache benchmark on their server. 🙂

  3. Thanks for these great tutorials!

    Have you done any of the benchmarks yet? I would be really interested in seeing Nginx + APC with W3 Total Cache versus the same with Wp Super Cache and if I should be disabling fastcgi_cache when using either of these?

    • Various caching types should be compared unless your goal is to learn how they are different. Memory caching is not always “faster” than disk caching, the problem you’re solving and how you’re measuring / testing determine what you will learn even with all the same testing environment.

  5. I want to know is there any way to use minify on CSS and JS while using fastcgi_cache alone..Minify is one of the feature in w3 total cache..(I used w3 total cache with fastcgi_cache and just removed the plugin).

    I know there is a pagespeed module for nginx but that is not production ready?? Looking forward for your suggestion on this issue.

      • Thanks Rahul for speedy response.In the meantime I checked you are using W3 total cache on this site (no fastcgi_cache if I am right)..is there are specific reason for it..I saw in another post you recommended fastcgi_cache + nginx over nginx + w3 and nginx + wp super cache.

        • We are using Nginx’s fastcgi_cache for page-caching here. W3 Total Cache is used for only MySQL & Object cache.

          I recommend Nginx’s fastcgi_cache because I think its faster. I do not have any benchmark data to support it though.

          You can use any page-cache. As long as cache can serve pages with PHP/MYSQL crashed, there is no issue with any cache.

    • ISPConfig generates nginx config itself.

      You can copy-paste part of config in ISPConfig’s web-based panel. There is a section to paste extra nginx rules per site in ISPConfig.

      You can paste following there:

      
	set $cache_uri $request_uri;

	# POST requests and urls with a query string should always go to PHP
	if ($request_method = POST) {
		set $cache_uri 'null cache';
	}   
	if ($query_string != "") {
		set $cache_uri 'null cache';
	}   

	# Don't cache uris containing the following segments
	if ($request_uri ~* "(/wp-admin/|/xmlrpc.php|/wp-(app|cron|login|register|mail).php|wp-.*.php|/feed/|index.php|wp-comments-popup.php|wp-links-opml.php|wp-locations.php|sitemap(_index)?.xml|[a-z0-9_-]+-sitemap([0-9]+)?.xml)") {
		set $cache_uri 'null cache';
	}   

	# Don't use the cache for logged in users or recent commenters
	if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in") {
		set $cache_uri 'null cache';
	}

	# Use cached or actual file if they exists, otherwise pass request to WordPress
	location / {
		try_files /wp-content/cache/page_enhanced/${host}${cache_uri}_index.html $uri $uri/ /index.php?$args ;
	}

  7. I use a $5 VPS with 512MB RAM, 1 Core and 20 GB SSD. Despite the fact that it’s SSD, and my Nginx configuration is as optimized as yours (and thank you for all these guides :), I figured that I’d give memcached a try for Page Cache option of W3 Total Cache.

    Here’s the Blitz report when I use the same W3 configuration as you’ve mentioned in this post:
    https://www.blitz.io/report/2cc5c638f7c471adbe42f5c099572c84

    And here’s the report after I switched Page Caching using Memcached:
    https://www.blitz.io/report/2cc5c638f7c471adbe42f5c0995723f6

    I monitored CPU usage (poor 1 core VPS) during the Blitz Rush – using Memcached allowed a constant <5% CPU occupation (combination of php-fpm worker and nginx worker mostly), while using Disk page cache resulted a constant 100% after concurrency reaches about 150.

    Just my two cents 🙂

    • Thanks for your time and efforts.

      However, there are many things which are not clear from your reports.

          For first report, did you turn on pagecaching using Disk Enhanced method as I suggested in article?
          Have you ran tests in reverse order? Memcached first and disk-cache second. Did you see any difference?
          Have you verified that in disk-based config, cache was configured correctly? Have a look at this checklist. 100% CPU utilisation might be because of for every pageview, cached version of page getting regenerated.

      Off-topic, if its possible for you, can you use config presented here – http://rtcamp.com/tutorials/nginx-wordpress-fastcgi_cache-with-conditional-purging/ ?. In other config, we use disk-based caching only but rather than HDD, its a RAMDISK (tmpfs) in action.

  8. Aren’t you missing the host in the rewrite for cached pages? Request_uri doesn’t include the hostname but the cache folder is in page_enhanced then host the uri subfolder.

  9. I’d also recommend adding a rewrite for ?replytocom=* and ?utm_source*. These circumvent caching but should be rewritten to the URI (no args) or served from cache. Other wise bots our hackers blow up the load since the query string doesn’t hit the cach but php/mysql dierectly. Also a good idea to avoid “dup” content.

  10. Instead of Ubuntu, I used arch linux and followed the articles instructions. It apparently works without anything in the directory. If I go to my domain name (which the wordpress.conf file point to the html directory of that domain) I get a 403 forbidden error. Which is fine because the directory is empty and it means that pointing to the directory is working. However, after I download wordpress to that directory and extract it and then point my browser to the domain again, the screen stays blank. I can’t seem to figure out why.

  11. Hi Rahul ,

    Nginx helper seems to work fine for page level caching , but due to high traffic we require object level caching too , Can nginx caching and w3tc run together ? if yes is their any tutorial you can provide.

    Regards,
    Abhishek

  12. Hi, I installed w3 total cache on wordpress, when I enable Database & Object Cache (opcode xcache), some of my web pages and wp-logins pages give an error “The page you’re looking for are tem… unavailable… Please try again.”

    I’ve set “xcache_admin_enable_auth=Off” at /etc/php5/conf.d/xcache.ini, /wordpressroot/wp-content/plugins/w3tc…/ini/xcache.ini. Still not work!

    Please help!

    Much thanks!

  13. I think WP super cache is better because its much more easier.The speed of cached pages are good and it doesn’t use PHP commands for caching pages. W3 total cache does the same but its a little complicated for most users to configure it properly and if you can’t configure it right, it even slows down your website.

      • Is fastcgi_cache plugin better than WP super cache or W3TC ? I have not seen this plugin in official wordpress plugins yet. I installed WP super cache. Is it better that i switch to fastcgi_cache plugin ? Please guide me. I want to use lowest hardware resources.

        • Mike,
          Fastcgi_cache is not a plugin, it is a setting of nginx. First user that hits the page, nginx sends request to php5-fpm, and cache the result (rendered html). It then serves cached page from tmpfs or from disk for as long as you set its time to live. You can purge the cache using nginx helper plugin. It works great, its fast and does not depend on php to generate cache. I am running some benchmarks on our wordpress websites with various different approaches, maybe I can come up with some results, but I believe its generation and retriaval time is faster (under heavy load) than any wp cache plugin.

          • @Dominique

            Thanks for your detailed reply. 🙂

            I would love to see your benchmarks.

            I also believe that fastcgi-cache’s generation and retrieval time will be better. Apart from heavy-load, another factor is number of cached files.

            For sites, which uses /post-name/ as permalinks, w3-total-cache will be storing 1000’s of files under single-dir (assuming site has 1000’s of posts) whereas fastcgi-cache will be hashing them across multiple directories.

  14. Raul,
    I would like to hear your opinion about barrier that I fond on my way to implementing a good wordpress setup for our websites. I am stuck at minification and combining assets.
    ngx_pagespeed eats alot of cpu. It is flawless indeed, but it is hungry.
    All plugin minification tools that I came across have one problem – they do not make it possible to take advantage of gzip_static. gzipping on-the-fly is also cpu hungry.
    If you do not have to gzip on the fly you save alot of cpu.

    I need a minification solution that does it once, and creates a .gz file in order make it possible for nginx to find it. For instance, W3tc creates a file with .gzip extension. NGINX do not find it. It gzip again eating cpu.

    Since I run alot of wordpress websites and plan to run more, I need a solution that can take care of minification and combining on the fly and cache everything. It seems simple enough for me, but I cannot find anything that does it.

    fastcgi_cache min hits: 2.
    First visit: CSS and JS’s get combined and minified. .js and .gz files are created. Files are touched with same timestamp.
    Second visit: nginx creates cache version of the page (fastcgi_cache) with assets pointing to minified ones.
    All visits from now on, until expire of cache: All hits the cache. Only thing that gets gzipped on the fly is the page itself, fetched from tmpfs.

    What do you think? Can you share your opinion on this?

    Best regards.

  16. Hmm… Has anyone here tested by adding Redis (apt-get install redis-server, google redis + wp) in the equation of LEMP + W3?
    My site runs LEMP + W3 Total Cache + Redis and the combination is simply amazing.

    • I am aware of redis and its on my list from long-time.

      How are you using redis in your setup? For page-cache or object-cache or both.

      There is also one wrong method of replacing wordpress’s index.php to get/set complete page from redis. I hope you are not using it as that method doesn’t improve concurrency much.

      • I am using ngx_pagespeed with Redis and it works pretty great.I will not recommend using W3 Total cache or any other caching plugin if you are on VPS or dedicated server as ngx_pagespeed takes care of all.

        You can read my ngx_pagespeed configuration here

        • I don’t see redis in your article. Can you elaborate more?

          By the way W3TC is still recommended for object-cache which nginx_pagespeed doesn’t handle.

          Speeding up backend is equally important.

          • The article covers ngx_pagespeed only.

            I have not included Redis in that as it is separate topic.I am using Predis for connectivity with Redis.Since Redis is persistent is serves well.

            Also for backend caching (database) if you are on VPS or dedicated server you can tune mysql for better performance.I am also using PHP with APC.

            Note I have used fastcgi_cache but it did not worked better than above configuration (just tested with webpagetest and other sites)..But if you wish you can dive more into that

          • Predis is redis library for PHP only.

            If you are using index.php method to talk to redis server then it won’t scale well.

            Ideally, you need a nginx-redis module to read from cache and something on wordpress side which will write into redis cache. Still this may not be faster than nginx’s fastcgi_cache using ramdisk.

            Using php+apc and mysql-tweaking won’t fulfill gap left by object-cache. As you are using APC, you can use w3tc or any other object-cache plugin to use APC as object-store.

            Finally, webpagetest/gtmatrix gives you feedback about how well a single page is optimized. You need to run apachebench, siege or blitz.io tests to measure concurrancy which is directly affected by caching mechanism.

            For example, if “php” is used to retrieve values from redis-cache, then total-system’s concurrency will be limited by how much load php-fpm can handle alone! This is a classic case of “a chain is as strong as its weakest link…”

  17. Thanks for detailed response but I have confusion

    Performance is how fast page is generated and served (measured by TTFB or speed index of webpagetest or other similar tests)

    number of concurrent users per second is how your server scales under load (tested by blitz.io and others as you mentioned above)

    • Your both points are correct.

      My only concern is getting php involved which affects scalability for sure. It can also affect TTFB.

      Just imagine: nginx -> RAM v/s nginx -> php -> redis -> RAM.

      What is on my todo-list is testing: nginx -> redis -> RAM sequence. But in extreme and rare case also, IMHO, it won’t beat nginx -> RAM by much difference.

      That is why redis experiment is low on priority list.

      By the way, w3-total-cache config in this article also gives you nginx -> RAM sequence. As long as RAM is free, OS will do hard-disk to RAM caching for static files read from disk.

  19. After I added this code to my nginx vhost, frontend working without any issue. But when I try to navigate to “wp-admin” pages (new posts, plugin, etc), nginx give me “502 Bad Gateway”. However WP-admin dashboard loading without any issue. I have Apache+Nginx+CentOS WordPress installation.

    • Nginx vhost error, I can see this one.

      connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or
      directory)

      So, if I removed

      location ~ .php$ {
      try_files $uri /index.php;
      include fastcgi_params;
      fastcgi_pass unix:/var/run/php5-fpm.sock;
      }

      OK?

  20. The Page cache disk:enhanced is not working on my pages.

    <!– W3 Total Cache: Page cache debug info: Engine: disk: enhanced
    Cache key:
    domain.com//2014/02/sample-post/_index.html
    Caching: disabled Reject reason: Requested URI is rejected Creation
    Time: 0.379s

    Seems that the Cache key didn’t get the right url. My url is domain.com/index.php/2014/2/sample-post

    Please help. I have asked this in w3 total cache support forum, but no one answered me.

    Thanks!

  21. Hi, thanks for the tutorial. I am not really sure about nginx. So hope you can help me with my questions!

    I have this in my nginx default file.

    location / {
    try_files $uri $uri/ /index.html;
    }
    error_page 404 /404.html;

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    root /usr/share/nginx/www;
    }

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    location ~ \.php$ {
    try_files $uri =404;
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_nam$
    include fastcgi_params;

    error_page 404 /404.html;

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    root /usr/share/nginx/www;
    }

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    location ~ \.php$ {

    Should I replace the above with yours:

    set $cache_uri $request_uri;

    # POST requests and urls with a query string should always go to PHP
    if ($request_method = POST) {
    set $cache_uri 'null cache';
    }
    if ($query_string != "") {
    set $cache_uri 'null cache';
    }

    # Don't cache uris containing the following segments
    if ($request_uri ~* "(/wp-admin/|/xmlrpc.php|/wp-(app|cron|login|register|mail).php|wp-.*.php|/feed/|index.php|wp-comments-popup.php|wp-links-opml.php|wp-locations.php|sitemap(_index)?.xml|[a-z0-9_-]+-sitemap([0-9]+)?.xml)") {
    set $cache_uri 'null cache';
    }

    # Don't use the cache for logged in users or recent commenters
    if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in") {
    set $cache_uri 'null cache';
    }

    # Use cached or actual file if they exists, otherwise pass request to WordPress
    location / {
    try_files /wp-content/cache/page_enhanced/${host}${cache_uri}_index.html $uri $uri/ /index.php?$args ;
    }

    location ~ ^/wp-content/cache/minify/[^/]+/(.*)$ {
    try_files $uri /wp-content/plugins/w3-total-cache/pub/minify.php?file=$1;
    }

    location = /favicon.ico { log_not_found off; access_log off; }
    location = /robots.txt { log_not_found off; access_log off; }

    location ~ .php$ {
    try_files $uri /index.php;
    include fastcgi_params;
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    }

    # Cache static files for as long as possible
    location ~* .(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
    expires max; log_not_found off; access_log off;
    }
    }

    Sorry for pasting so much code. Because I don’t know how else to explain what I am asking. If the above is wrong, can you kindly highlight the portion of my code I am suppose to replace with yours? thanks very much for your time and patience.

  22. There is another problem with your code.

    When SSL is enabled.This rule will cause the page broken when the visitor access your website via SSL.
    try_files /wp-content/cache/page_enhanced/${host}${cache_uri}_index.html $uri $uri/ /index.php?$args ;

    here is a simple fix:
    …
    set $w3tc_ssl “”;
    if ($scheme = https) {
    set $w3tc_ssl _ssl;
    }
    …
    try_files /wp-content/cache/page_enhanced/${host}${cache_uri}_index${w3tc_ssl}.html $uri $uri/ /index.php?$args ;
    …

  24. For those having problems with minify. There are two options:

    1) Disable the “Rewrite URL structure” in the minify tab … but services like cloudflare might no cache it

    2) Update nginx conf

    where you have

    location ~ ^/wp-content/cache/minify/(.+\.(css|js))$ {
    try_files $uri /wp-content/plugins/w3-total-cache/pub/minify.php?file=$1;
    }

    replace by

    location ~ ^/wp-content/cache/minify/(.+\.(css|js))$ {
    try_files $uri /index.php?w3tc_minify=$1;
    }