Added blog article on search exclusion features

CHANGELOG

@@ -1,3 +1,12 @@
+mkdocs-material-7.3.0+insiders-3.1.1 (2021-09-26)
+
+  * Fixed ordering bug in search exclusion logic
+
+mkdocs-material-7.3.0+insiders-3.1.0 (2021-09-26)
+
+  * Added support for excluding pages, sections, and elements from search
+  * Fixed #2803: Code block annotations not visible when printing
+
 mkdocs-material-7.3.0 (2021-09-23)
 
   * Added support for sticky navigation tabs

docs/blog/2021/excluding-content-from-search.md

@@ -0,0 +1,216 @@
+---
+template: overrides/main.html
+description: >
+  How we rebuilt client-side search, delivering a better user experience while
+  making it faster and smaller at the same time
+disqus: mkdocs-material
+search:
+  exclude: true
+---
+
+# Excluding content from search
+
+__The latest Insiders release brings three new simple ways to exclude
+dedicated parts of a document from the search index, allowing for more
+fine-grained control.__
+
+<aside class="mdx-author" markdown="1">
+![@squidfunk][1]
+
+<span>__Martin Donath__ · @squidfunk</span>
+<span>
+:octicons-calendar-24: September 26, 2021 ·
+:octicons-clock-24: 5 min read ·
+[:octicons-tag-24: 7.3.0+insiders-3.1.1](../../insiders/changelog.md#311-_-september-26-2021)
+</span>
+</aside>
+
+  [1]: https://avatars.githubusercontent.com/u/932156
+
+---
+
+Two weeks ago, Material for MkDocs Insiders shipped a [brand new search
+plugin][2], yielding [massive improvements in usability][3], but also in [speed
+and size][4] of the search index. Interestingly, as discussed in the previous
+blog article, we only scratched the surface of what's now possible. This
+release brings some useful features that enhance the writing experience,
+allowing for more fine-grained control of what pages, sections and blocks of a
+Markdown file should be indexed by the built-in search functionality.
+
+_The following section discusses existing solutions for excluding pages and
+sections from the search index. If you immediately want to learn what's new,
+skip to the [section just after that][5]._
+
+  [2]: search-better-faster-smaller.md
+  [3]: search-better-faster-smaller.md#whats-new
+  [4]: search-better-faster-smaller.md#benchmarks
+  [5]: #whats-new
+
+## Prior art
+
+MkDocs has a rich and thriving ecosystem of [plugins][6], and it comes as no
+surprise that there's already a fantastic plugin by @chrieke to exclude specific
+sections of a Markdown file – the [mkdocs-exclude-search][7] plugin. It can be
+installed with:
+
+```
+pip install mkdocs-exclude-search
+```
+
+__How it works__: the plugin post-processes the `search_index.json` file that
+is generated by the built-in search plugin, giving the author the ability to
+exclude certain pages and sections by adding a few lines of configuration to
+`mkdocs.yml`. An example:
+
+``` yaml
+plugins:
+  - search
+  - exclude-search:
+      exclude:
+        - page.md
+        - page.md#section
+        - directory/*
+        - /*/page.md
+```
+
+It's easy to see that the plugin follows a configuration-centric approach, which
+adds support for advanced filtering techniques like infix- and suffix-filtering
+using wildcards. While this is a very powerful idea, it comes with some
+downsides:
+
+1. __Exclusion patterns and content are not co-located__: exclusion patterns
+   need to be defined in `mkdocs.yml`, and not as part of the respective
+   document or section to be excluded. This might result in stale exclusion
+   patterns, leading to unintended behavior:
+
+    - When a headline is changed, its slug (permalink) also changes, which might
+      suddenly match (or unmatch) a pattern, e.g., when an author fixes a typo
+      in a headline.
+
+    - As exclusion patterns support the use of wildcards, different authors
+      might overwrite each other's patterns without any immediate feedback since
+      the plugin does only report the number of excluded documents – not _what_
+      has been excluded.[^1]
+
+  [^1]:
+    When the log level is set to `DEBUG`, the plugin will report exactly which
+    pages and sections have been excluded from the search index, but MkDocs will
+    now flood the terminal with debug output from its core and other plugins.
+
+2. __Exclusion control might be too coarse__: The [mkdocs-exclude-search][7]
+   plugin only allows for the exclusion of pages and sections. It's not possible
+   to exclude parts of a section, e.g., content that is irrelevant to search but
+   must be included as part of the documentation.
+
+  [6]: https://github.com/mkdocs/mkdocs/wiki/MkDocs-Plugins
+  [7]: https://github.com/chrieke/mkdocs-exclude-search
+
+## What's new?
+
+The latest Insiders release brings fine-grained control for [__excluding pages,
+sections, and blocks__][8] from the search index, implemented through front
+matter, as well as the [Attribute List][9] extension. Note that it doesn't
+replace the [mkdocs-exclude-search][7] plugin but _complements_ it.
+
+  [8]: ../../setup/setting-up-site-search.md#search-exclusion
+  [9]: https://python-markdown.github.io/extensions/attr_list/
+
+### Excluding pages
+
+An entire page can be excluded from the search index by adding a simple
+directive to the front matter of the respective Markdown file. The good thing
+is that the author now only has to check the top of the document to learn
+whether it is excluded or not:
+
+``` bash
+---
+search:
+  exclude: true
+---
+
+# Document title
+...
+```
+
+### Excluding sections
+
+If a section should be excluded, the author can use the [Attribute List][9]
+extension to add a __pragma__ called `{ data-search-exclude }` at the end of a
+heading. The pragma is not included in the final HTML, as search pragmas are
+filtered by the search plugin before the page is rendered:
+
+=== "`docs/page.md`"
+
+    ``` markdown
+    # Document title
+
+    ## Section 1
+
+    The content of this section is included
+
+    ## Section 2 { data-search-exclude }
+
+    The content of this section is excluded
+    ```
+
+=== "`search_index.json`"
+
+    ``` json
+    {
+      ...
+      "docs": [
+        {
+          "location":"page/",
+          "text":"",
+          "title":"Document title"
+        },
+        {
+          "location":"page/#section-1",
+          "text":"<p>The content of this section is included</p>",
+          "title":"Section 1"
+        }
+      ]
+    }
+    ```
+
+### Excluding blocks
+
+If even more fine-grained control is desired, the __pragma__ can be added to
+any [block-level element][10] or [inline-level element][11] that is officially
+supported by the [Attribute List][9] extension:
+
+=== "`docs/page.md`"
+
+    ``` markdown
+    # Document title
+
+    The content of this block is included
+
+    The content of this block is excluded
+    { data-search-exclude }
+    ```
+
+=== "`search_index.json`"
+
+    ``` json
+    {
+      ...
+      "docs": [
+        {
+          "location":"page/",
+          "text":"<p>The content of this block is included</p>",
+          "title":"Document title"
+        },
+      ]
+    }
+    ```
+
+  [10]: https://python-markdown.github.io/extensions/attr_list/#block-level
+  [11]: https://python-markdown.github.io/extensions/attr_list/#inline-level
+
+## Conclusion
+
+The latest release brings three simple ways to control more precisely what goes
+into the search index and what doesn't. It complements the already very powerful
+[mkdocs-exclude-search][7] plugin, allowing for new methods of shaping the
+structure, size and content of the search index.

docs/blog/2021/search-better-faster-smaller.md

@@ -20,7 +20,8 @@ smaller at the same time.__
 <span>__Martin Donath__ · @squidfunk</span>
 <span>
 :octicons-calendar-24: September 13, 2021 ·
-:octicons-clock-24: 15 min read
+:octicons-clock-24: 15 min read ·
+[:octicons-tag-24: 7.2.6+insiders-3.0.0](../../insiders/changelog.md#300-_-september-13-2021)
 </span>
 </aside>
 

docs/blog/index.md

@@ -6,6 +6,25 @@ search:
 
 # Blog
 
+<h2>Excluding content from search</h2>
+
+__The latest Insiders release brings three new simple ways to exclude dedicated
+parts of a document from the search index, allowing for more fine-grained
+control.__
+
+Two weeks ago, Material for MkDocs Insiders shipped a brand new search plugin,
+yielding massive improvements in usability, but also in speed and size of the
+search index. Interestingly, as discussed in the previous blog article, we only
+scratched the surface of what's now possible. This release brings some useful
+features that enhance the writing experience, allowing for more fine-grained
+control of what pages, sections and blocks of a Markdown file should be indexed
+by the built-in search functionality.
+
+[Continue reading :octicons-arrow-right-24:][2]{ .md-button }
+
+  [2]: 2021/excluding-content-from-search.md
+
+
 <h2>Search: better, faster, smaller</h2>
 
 __This is the story of how we managed to completely rebuild client-side search,

docs/insiders/changelog.md

@@ -6,6 +6,15 @@ template: overrides/main.html
 
 ## Material for MkDocs Insiders
 
+### 3.1.1 <small>_ September 26, 2021</small>
+
+- Fixed ordering bug in search exclusion logic
+
+### 3.1.0 <small>_ September 26, 2021</small>
+
+- Added support for excluding pages, sections, and elements from search
+- Fixed #2803: Code block annotations not visible when printing
+
 ### 3.0.1 <small>_ September 19, 2021</small>
 
 - Added support for using literal `h1-6` tags for search plugin

docs/insiders/index.md

@@ -141,6 +141,7 @@ The following features are currently exclusively available to sponsors:
 - [x] [Rich search previews :material-new-box:][36]
 - [x] [Tokenizer with lookahead :material-new-box:][37]
 - [x] [Advanced search highlighting :material-new-box:][38]
+- [x] [Excluding content from search :material-new-box:][39]
 - [x] [Social cards][34]
 - [x] [Cookie consent][33]
 - [x] [Linking content tabs][32]
@@ -195,14 +196,14 @@ the public for general availability.
 - [x] [Custom admonition icons][31]
 - [x] [Linking content tabs][32]
 
-  [30]: ../setup/setting-up-site-search.md#boosting-a-page
+  [30]: ../setup/setting-up-site-search.md#search-boosting
   [31]: ../reference/admonitions.md#changing-the-icons
   [32]: ../reference/content-tabs.md#linking-content-tabs
 
 #### $ 7,000 – Royal Gold
 
 - [x] [Cookie consent][33]
-- [ ] Exclude pages from search
+- [ ] Was this page helpful?
 - [ ] Link cards
 
   [33]: ../setup/setting-up-site-analytics.md#cookie-consent
@@ -221,11 +222,13 @@ the public for general availability.
 - [x] [Rich search previews][36]
 - [x] [Tokenizer with lookahead][37]
 - [x] [Advanced search highlighting][38]
+- [x] [Excluding content from search][39]
 
   [35]: ../blog/2021/search-better-faster-smaller.md
   [36]: ../blog/2021/search-better-faster-smaller.md#rich-search-previews
   [37]: ../blog/2021/search-better-faster-smaller.md#tokenizer-lookahead
   [38]: ../blog/2021/search-better-faster-smaller.md#accurate-highlighting
+  [39]: ../setup/setting-up-site-search.md#search-exclusion
 
 ### Goals completed
 
@@ -300,17 +303,17 @@ implemented behind feature flags; all configuration changes are
 backward-compatible. This means that your users will be able to build the
 documentation locally with Material for MkDocs and when they push their changes,
 it can be built with Insiders (e.g. as part of GitHub Actions). Thus, it's
-recommended to [install Insiders][39] only in CI, as you don't want to expose
+recommended to [install Insiders][40] only in CI, as you don't want to expose
 your `GH_TOKEN` to users.
 
-  [39]: ../publishing-your-site.md#github-pages
+  [40]: ../publishing-your-site.md#github-pages
 
 ### Payment
 
 _We don't want to pay for sponsorship every month. Are there any other options?_
 
 Yes. You can sponsor on a yearly basis by [switching your GitHub account to a
-yearly billing cycle][40]. If for some reason you cannot do that, you could
+yearly billing cycle][41]. If for some reason you cannot do that, you could
 also create a dedicated GitHub account with a yearly billing cycle, which you
 only use for sponsoring (some sponsors already do that).
 
@@ -318,7 +321,7 @@ If you have any problems or further questions, don't hesitate to contact me at
 sponsors@squidfunk.com. Note that one-time payments are not eligible for
 Insiders, but of course, very appreciated.
 
-  [40]: https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/changing-the-duration-of-your-billing-cycle
+  [41]: https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/changing-the-duration-of-your-billing-cycle
 
 ### Terms
 
@@ -327,7 +330,7 @@ commercial project. Can we use Insiders under the same terms and conditions?_
 
 Yes. Whether you're an individual or a company, you may use _Material for MkDocs
 Insiders_ precisely under the same terms as Material for MkDocs, which are given
-by the [MIT license][41]. However, we kindly ask you to respect the following
+by the [MIT license][42]. However, we kindly ask you to respect the following
 guidelines:
 
 - Please __don't distribute the source code__ of Insiders. You may freely use
@@ -338,7 +341,7 @@ guidelines:
 - If you cancel your subscription, you're removed as a collaborator and will
   miss out on future updates of Insiders. However, you may __use the latest
   version__ that's available to you __as long as you like__. Just remember that
-  [GitHub deletes private forks][42].
+  [GitHub deletes private forks][43].
 
-  [41]: ../license.md
+  [42]: ../license.md
-  [42]: https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/removing-a-collaborator-from-a-personal-repository
+  [43]: https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/removing-a-collaborator-from-a-personal-repository

docs/setup/setting-up-site-search.md

@@ -238,7 +238,7 @@ For setup instructions, refer to the [official documentation][19].
 
 ## Usage
 
-### Boosting a page
+### Search boosting
 
 [:octicons-file-code-24: Source][20] ·
 :octicons-note-24: Metadata ·
@@ -259,7 +259,103 @@ search:
 ```
 
   [20]: ../insiders/index.md
-  [21]: ../../reference/meta-tags/#metadata
+  [21]: ../reference/meta-tags.md#metadata
+
+### Search exclusion
+
+[:octicons-file-code-24: Source][20] ·
+:octicons-beaker-24: Experimental ·
+[:octicons-heart-fill-24:{ .mdx-heart } Insiders only][20]{ .mdx-insiders }
+
+#### Excluding pages
+
+Sometimes, it's necessary to exclude a page from the search index, which can be
+achieved by adding the following front matter using the [Metadata][21]
+extension:
+
+``` bash
+---
+search:
+  exclude: true
+---
+
+# Document title
+...
+```
+
+#### Excluding sections
+
+With the help of the [Attribute List][22] extension, it's possible to exclude a
+specific section of a page from search by adding the `{ data-search-exclude }`
+pragma after the Markdown heading:
+
+=== "`docs/page.md`"
+
+    ``` markdown
+    # Document title
+
+    ## Section 1
+
+    The content of this section is included
+
+    ## Section 2 { data-search-exclude }
+
+    The content of this section is excluded
+    ```
+
+=== "`search_index.json`"
+
+    ``` json
+    {
+      ...
+      "docs": [
+        {
+          "location":"page/",
+          "text":"",
+          "title":"Document title"
+        },
+        {
+          "location":"page/#section-1",
+          "text":"<p>The content of this section is included</p>",
+          "title":"Section 1"
+        }
+      ]
+    }
+    ```
+
+  [22]: ../reference/images.md#attribute-list
+
+#### Excluding blocks
+
+If even more fine-grained control is needed, content blocks can be excluded
+from search sections with a `{ data-search-exclude }` pragma, so they will not
+be included in the search index:
+
+=== "`docs/page.md`"
+
+    ``` markdown
+    # Document title
+
+    The content of this block is included
+
+    The content of this block is excluded
+    { data-search-exclude }
+    ```
+
+=== "`search_index.json`"
+
+    ``` json
+    {
+      ...
+      "docs": [
+        {
+          "location":"page/",
+          "text":"<p>The content of this block is included</p>",
+          "title":"Document title"
+        }
+      ]
+    }
+    ```
 
 ## Customization
 
@@ -273,12 +369,12 @@ your needs.
 
 ### Query transformation
 
-[:octicons-file-code-24: Source][22] ·
+[:octicons-file-code-24: Source][23] ·
 :octicons-mortar-board-24: Difficulty: _easy_
 
 When a user enters a query into the search box, the query is pre-processed
 before it is submitted to the search index. Material for MkDocs will apply the
-following transformations, which can be customized by [extending the theme][23]:
+following transformations, which can be customized by [extending the theme][24]:
 
 ``` ts
 export function defaultTransform(query: string): string {
@@ -311,7 +407,7 @@ export function defaultTransform(query: string): string {
 If you want to switch to the default behavior of the `mkdocs` and `readthedocs`
 themes, both of which don't transform the query prior to submission, or
 customize the `transform` function, you can do this by [overriding the
-`config` block][24]:
+`config` block][25]:
 
 ``` html
 {% extends "base.html" %}
@@ -331,19 +427,19 @@ customize the `transform` function, you can do this by [overriding the
 The `transform` function will receive the query string as entered by the user
 and must return the processed query string to be submitted to the search index.
 
-  [22]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/transform/index.ts
+  [23]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/transform/index.ts
-  [23]: ../customization.md#extending-the-theme
+  [24]: ../customization.md#extending-the-theme
-  [24]: ../customization.md#overriding-blocks-recommended
+  [25]: ../customization.md#overriding-blocks-recommended
 
 ### Custom search
 
-[:octicons-file-code-24: Source][25] ·
+[:octicons-file-code-24: Source][26] ·
 :octicons-mortar-board-24: Difficulty: _challenging_
 
-Material for MkDocs implements search as part of a [web worker][26]. If you
+Material for MkDocs implements search as part of a [web worker][27]. If you
 want to switch the web worker with your own implementation, e.g. to submit
 search to an external service, you can add a custom JavaScript file to the
-`docs` directory and [override the `config` block][23]:
+`docs` directory and [override the `config` block][24]:
 
 ``` html
 {% block config %}
@@ -361,8 +457,8 @@ format using _discriminated unions_, i.e. through the `type` property of the
 message. See the following interface definitions to learn about the message
 formats:
 
-- [:octicons-file-code-24: `SearchMessage`][27]
+- [:octicons-file-code-24: `SearchMessage`][28]
-- [:octicons-file-code-24: `SearchIndex` and `SearchResult`][28]
+- [:octicons-file-code-24: `SearchIndex` and `SearchResult`][29]
 
 The sequence and direction of messages is rather intuitive:
 
@@ -371,7 +467,7 @@ The sequence and direction of messages is rather intuitive:
 - :octicons-arrow-right-24: `SearchQueryMessage`
 - :octicons-arrow-left-24: `SearchResultMessage`
 
-  [25]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker
+  [26]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker
-  [26]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
+  [27]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
-  [27]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker/message/index.ts
+  [28]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker/message/index.ts
-  [28]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/_/index.ts
+  [29]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/_/index.ts

mkdocs.yml

@@ -219,4 +219,5 @@ nav:
   - Blog:
     - blog/index.md
     - 2021:
+      - blog/2021/excluding-content-from-search.md
       - blog/2021/search-better-faster-smaller.md