{"id":2175,"date":"2011-06-24T13:32:28","date_gmt":"2011-06-24T20:32:28","guid":{"rendered":"http:\/\/queenofsubtle.com\/404\/?page_id=2175"},"modified":"2011-06-24T13:32:28","modified_gmt":"2011-06-24T20:32:28","slug":"use-robots-txt","status":"publish","type":"page","link":"https:\/\/queenofsubtle.com\/404\/?page_id=2175","title":{"rendered":"Use robots.txt"},"content":{"rendered":"<p>If you have a section of your site with pages that move around or get added and deleted a lot, consider blocking search engines from indexing them in their databases. You definitely shouldn&#8217;t use META tags on pages that move a lot, either.<\/p>\n<p>Create a file called robots.txt in the root directory (the one you refer to as &#8216;\/&#8217;). Follow this format:<\/p>\n<pre style=\"padding-left: 30px;\"># robots.txt for http:\/\/www.yoursite.net\/\r\nUser-agent: *\r\nDisallow: \/disappearing\/ # These pages move a lot.\r\nDisallow: \/soontobe404.htm<\/pre>\n<p>&nbsp;<\/p>\n<p>&#8216;User-agent: *&#8217; means that it applies to all search engines. Anything preceded by a &#8216;#&#8217; is simply a note to yourself. You can block entire directories or individual files using robots.txt. Use it!<\/p>\n\r\n\t\t\t<div style=\"clear: both; float: none;\" class=\"ratepost-clear\"><\/div>\r\n\t\t\t<div id=\"ratepost-2175\" class=\"ratepost\" rel=\"3\">\r\n\t\t\t\t<div class=\"rating-info\">rating: 3.00\/5<\/div>\r\n\t\t\t\t\t<span class=\"rating3\"><div class=\"star1\" rel=\"1\"><\/div><div class=\"star2\" rel=\"2\"><\/div><div class=\"star3\" rel=\"3\"><\/div><div class=\"star4\" rel=\"4\"><\/div><div class=\"star5\" rel=\"5\"><\/div><\/span>\r\n\t\t\t\t<div class=\"status-info\"><\/div>\r\n\t\t<\/div><div style=\"clear: both; float: none;\" class=\"ratepost-clear\"><\/div>","protected":false},"excerpt":{"rendered":"<p>If you have a section of your site with pages that move around or get added and deleted a lot, consider blocking search engines from indexing them in their databases. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":2128,"menu_order":0,"comment_status":"open","ping_status":"open","template":"","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"_links":{"self":[{"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=\/wp\/v2\/pages\/2175"}],"collection":[{"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2175"}],"version-history":[{"count":3,"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=\/wp\/v2\/pages\/2175\/revisions"}],"predecessor-version":[{"id":2178,"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=\/wp\/v2\/pages\/2175\/revisions\/2178"}],"up":[{"embeddable":true,"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=\/wp\/v2\/pages\/2128"}],"wp:attachment":[{"href":"https:\/\/queenofsubtle.com\/404\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2175"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}