<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[My Botsy]]></title><description><![CDATA[My Botsy]]></description><link>https://mybotsy.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 18:58:05 GMT</lastBuildDate><atom:link href="https://mybotsy.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Why Your Page Isn't Indexed: 5 Real Noindex Cases (Meta Tag, X-Robots-Tag and robots.txt)]]></title><description><![CDATA[Where noindex hides, how robots.txt can make Google ignore it, and why tools sometimes give the wrong answer]]></description><link>https://mybotsy.hashnode.dev/why-your-page-isn-t-indexed-5-real-noindex-cases-meta-tag-x-robots-tag-and-robots-txt</link><guid isPermaLink="true">https://mybotsy.hashnode.dev/why-your-page-isn-t-indexed-5-real-noindex-cases-meta-tag-x-robots-tag-and-robots-txt</guid><category><![CDATA[Web Development]]></category><category><![CDATA[http]]></category><category><![CDATA[SEO]]></category><category><![CDATA[webdev]]></category><dc:creator><![CDATA[My Botsy]]></dc:creator><pubDate>Mon, 02 Nov 2026 16:14:59 GMT</pubDate><content:encoded><![CDATA[<p>A page is noindex if either of two things says so: a <code>&lt;meta name="robots" content="noindex"&gt;</code> tag in the HTML <code>&lt;head&gt;</code>, or an <code>X-Robots-Tag: noindex</code> HTTP response header. Google's docs say the two "have the same effect". The header one never shows up in View Source, which is why it causes so much confusion.</p>
<p>Here are five real cases I checked on November 2, 2026, with curl and two free web checkers, and what each one teaches.</p>
<h2>1. A login page with a meta noindex</h2>
<p><code>https://mybotsy.com/login</code> returns:</p>
<pre><code>robots   noindex, follow, noarchive
</code></pre>
<p>Deliberate and correct: a login form has no business in search results. <code>noarchive</code> does nothing on Google today; Google lists it among rules it no longer uses.</p>
<h2>2. An XML sitemap with a header noindex</h2>
<pre><code>$ curl -sI https://yoast.com/sitemap_index.xml
HTTP/2 200
content-type: text/xml; charset=UTF-8
x-robots-tag: noindex, follow
</code></pre>
<p>XML has no <code>&lt;head&gt;</code>, so the header is the only possible signal. Same for PDFs, images and JSON. A meta-tag-only check would report this file as indexable.</p>
<h2>3. A noindex Google can't see</h2>
<p>yoast.com's robots.txt has <code>Disallow: /wp-json/</code>, and <code>/wp-json/</code> also sends <code>x-robots-tag: noindex</code>. Those rules fight each other: Google says that if robots.txt blocks a URL, its indexing rules "will not be found and will therefore be ignored", and a blocked URL can still appear in results when other pages link to it. For an API endpoint, no big deal. For a page you really want gone, <strong>allow crawling and serve noindex</strong>.</p>
<h2>4. Two robots tags on one page</h2>
<p>A theme and an SEO plugin each printing a robots tag is a classic CMS bug. I served a test page with <code>noindex</code> followed by <code>index, follow</code>. One checker I tested (the one I help build) showed only the last value, <code>index, follow</code>. Google applies the <strong>more restrictive</strong> rule, so the page is noindex. When a page "looks indexable" but won't index, search the raw source for <code>robots</code> and count the hits.</p>
<h2>5. An error page in disguise</h2>
<p>A Wikipedia page-history URL contains <code>noindex,nofollow</code> in a browser, but a server-side checker without a browser user agent got a 403 error page and showed no robots tag at all. <strong>A missing robots tag only counts if the response was a 200.</strong> Check the status line first.</p>
<h2>Other traps</h2>
<ul>
<li><strong>Redirects:</strong> some tools follow 301s silently and merge headers from both responses. Check the final URL directly.</li>
<li><strong>JavaScript:</strong> Google may skip rendering once it sees a noindex, so removing noindex with JS may not work. The original HTML must be clean.</li>
<li><strong><code>none</code></strong> means <code>noindex, nofollow</code>, even though the word noindex never appears.</li>
<li><strong><code>googlebot</code> vs <code>robots</code>:</strong> a googlebot-only noindex still lets Bing index the page.</li>
<li><strong>WordPress "Discourage search engines"</strong> prints <code>noindex, nofollow</code> everywhere. First thing to check after a launch or migration.</li>
<li><strong>robots.txt can't noindex.</strong> Google says <code>noindex</code> in robots.txt "is not supported".</li>
</ul>
<h2>A quick checklist</h2>
<ol>
<li>Meta tags: look for <code>robots</code>/<code>googlebot</code> with <code>noindex</code> or <code>none</code>.</li>
<li>Headers: <code>curl -sI &lt;url&gt;</code> and look for <code>x-robots-tag</code>, and confirm the status is 200.</li>
<li>robots.txt: make sure the path isn't disallowed.</li>
<li>Search Console URL Inspection: "Indexing allowed?" plus Test live URL.</li>
</ol>
<hr />
<p>The full write-up, with sitemap-wide checks and an FAQ, is on the My Botsy blog: <a href="https://mybotsy.com/blog/how-to-check-if-a-page-is-noindex">How to Check If a Page Is Noindex</a>. If you'd rather not use curl, the free <a href="https://mybotsy.com/meta-tags-checker">meta tags checker</a> and <a href="https://mybotsy.com/http-headers-lookup">HTTP headers lookup</a> there do steps 1 and 2 (with the limitations noted in cases 4 and 5).</p>
]]></content:encoded></item><item><title><![CDATA[How Much Does Minifying CSS Really Save? Real Numbers on normalize.css and Bootstrap]]></title><description><![CDATA[Raw, gzip and Brotli sizes before and after minifying, plus the minifier bugs that change meaning]]></description><link>https://mybotsy.hashnode.dev/how-much-does-minifying-css-really-save-real-numbers-on-normalize-css-and-bootstrap</link><guid isPermaLink="true">https://mybotsy.hashnode.dev/how-much-does-minifying-css-really-save-real-numbers-on-normalize-css-and-bootstrap</guid><category><![CDATA[web performance]]></category><category><![CDATA[frontend]]></category><category><![CDATA[webdev]]></category><category><![CDATA[CSS]]></category><dc:creator><![CDATA[My Botsy]]></dc:creator><pubDate>Sun, 01 Nov 2026 16:24:10 GMT</pubDate><content:encoded><![CDATA[<p>Minifying CSS removes what the browser doesn't need: comments, line breaks, indentation and the last semicolon in each rule. Everyone says to do it. But browsers almost never download raw CSS; servers compress it with gzip or Brotli first, and compression already squeezes out repeated whitespace. So how much does minifying actually save?</p>
<p>I measured two well-known stylesheets on November 1, 2026, before and after minifying, raw and compressed (gzip level 9, Brotli quality 11):</p>
<table>
<thead>
<tr>
<th>File</th>
<th>Raw</th>
<th>Gzip</th>
<th>Brotli</th>
</tr>
</thead>
<tbody><tr>
<td>normalize.css 8.0.1, original</td>
<td>6,138 B</td>
<td>1,737</td>
<td>1,398</td>
</tr>
<tr>
<td>normalize.css, minified</td>
<td>1,852 (-69.8%)</td>
<td>780 (-55.1%)</td>
<td>606 (-56.7%)</td>
</tr>
<tr>
<td>bootstrap.css 5.3.3, original</td>
<td>281,046 B</td>
<td>32,865</td>
<td>24,501</td>
</tr>
<tr>
<td>bootstrap.css, minified</td>
<td>233,341 (-17.0%)</td>
<td>30,743 (-6.5%)</td>
<td>22,853 (-6.7%)</td>
</tr>
</tbody></table>
<h2>Why the results are so different</h2>
<p>normalize.css is mostly explanatory comments, and comments are text compression still has to send, so stripping them saves a lot even after gzip. Bootstrap's source is generated, tidy code with few comments, so after Brotli the saving is about 1.6 KB. For reference, Bootstrap's own bootstrap.min.css is 232,803 bytes raw and 22,709 with Brotli, within about 150 compressed bytes of the output above. I compared both outputs rule by rule: all 57 declarations in normalize.css and all 5,541 in Bootstrap kept the same meaning.</p>
<p><strong>Practical rule:</strong> minify <em>and</em> compress. Minifying pays off most on commented, hand-written stylesheets: WordPress child themes, email templates, one-off landing pages.</p>
<h2>Safe vs. "clever" minification</h2>
<p>Safe changes can't alter rendering: removing comments (keep <code>/*! license */</code> ones), whitespace between tokens, the final <code>;</code> before <code>}</code>, and extra zero decimals (<code>0.0px</code> to <code>0</code>). Rewriting values (<code>#FFFFFF</code> to <code>#fff</code>, <code>bold</code> to <code>700</code>) saves a few more bytes but is where minifier bugs live. Things a minifier must leave alone:</p>
<ul>
<li>spaces around <code>-</code> and <code>+</code> inside <code>calc(100% - 20px)</code> (without them the rule is invalid)</li>
<li>spaces in media queries (<code>@media (min-width: 600px) and (max-width: 900px)</code>)</li>
<li>quoted strings, including <code>content: " a /* not a comment */ "</code> and font names like <code>"Open Sans"</code></li>
</ul>
<p>Two typing habits also break some minifiers: <code>! important</code> with a space, and a missing <code>}</code> that silently swallows the next rule. Check your source for both before minifying.</p>
<h2>What minifying won't do</h2>
<p>It won't remove unused CSS. If most of a framework's rules never match your markup, you need a purge step (PurgeCSS, Tailwind's content scanning, or Chrome DevTools' Coverage tab to find them). That usually saves far more than minifying.</p>
<h2>Doing it</h2>
<p>In a build, Vite, webpack and most frameworks already minify CSS. For a one-off file I use a <a href="https://mybotsy.com/css-minifier">free online CSS minifier</a> on a small tools site I help run; it shows the before/after size so you can see whether it's worth it. The full guide, with the step-by-step and more test cases, is <a href="https://mybotsy.com/blog/how-to-minify-css">here</a>.</p>
]]></content:encoded></item></channel></rss>