Full text search in hugo website

2021-05-17 743 words 4 minutes

Contents

I wanted to embed full text search in this web site as content has increased a bit, and sometime I need to dig into the key word.

I tried two different ways:

Algolia
- Saas (Search as a service)
- Commercial, but satisfactory free plan
- high performance and low bandwidth, but need to synchronize index.json and limit for contentLength
Lunr
- JavaScript library
- simple, no need to synchronize index.json, no limit for contentLength, but high bandwidth and low performance (Especially for Chinese which needs a large segmentit library)

1 Algolia

1.1 Setup Algolia in LoveIt theme

General way to setup Algolia in Hugo is:

Create Index in Aloglia website

Setup Account in Algolia
Setup location (in my case, I setup HongKong as it is faster at this point)
Create Index
Check “Settings” > “API Keys”
- Application ID
- Search Only API Key

Generate Search index

For generating search index (e.g. public/index.json), there are instructions to do that setting: - [outputFormats.Algolia],[params.algolia] and [outputs] in config.toml - layouts/_default/list.algolia.json,

Ref. FORESTRY - 2) Generating Your Search Index

Comparing with LoveIt theme directory and its example configuration,

The template has:

themes/LovewIt/lib/algoliasearch/algoliasearch-lite.umd.min.js
themes/LovewIt/layouts/assets.html has below and it the template generate.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


{{- /* Search */ -}}
{{- if .Site.Params.search | and .Site.Params.search.enable -}}
    {{- $search := .Site.Params.search -}}
    {{- $source := $cdn.autocompleteJS | default "lib/autocomplete/autocomplete.min.js" -}}
    {{- dict "Source" $source "Fingerprint" $fingerprint | dict "Scratch" .Scratch "Data" | partial "scratch/script.html" -}}
    {{- $config = dict "maxResultLength" $search.maxResultLength "snippetLength" $search.snippetLength "highlightTag" $search.highlightTag "noResultsFound" (T "noResultsFound") | dict "search" | merge $config -}}
    {{- if eq $search.type "lunr" -}}
        {{- with .Site.Home.OutputFormats.Get "json" -}}
            {{- $config = dict "type" "lunr" "lunrIndexURL" .RelPermalink | dict "search" | merge $config -}}
        {{- end -}}
        {{- $source := $cdn.lunrJS | default "lib/lunr/lunr.min.js" -}}
        {{- dict "Source" $source "Fingerprint" $fingerprint | dict "Scratch" .Scratch "Data" | partial "scratch/script.html" -}}
        {{- if T "lunrLanguageLib" -}}
            {{- $config = T "lunrLanguageCode" | dict "lunrLanguageCode" | dict "search" | merge $config -}}
            {{- with T "lunrSegmentitLib" -}}
                {{- $config = dict "lunrSegmentitURL" (resources.Get .).RelPermalink | dict "search" | merge $config -}}
            {{- end -}}
            {{- dict "Source" "lib/lunr/lunr.stemmer.support.js" "Minify" true "Fingerprint" $fingerprint | dict "Scratch" .Scratch "Data" | partial "scratch/script.html" -}}
            {{- dict "Source" (T "lunrLanguageLib") "Minify" true "Fingerprint" $fingerprint | dict "Scratch" .Scratch "Data" | partial "scratch/script.html" -}}
        {{- end -}}
    {{- else if eq $search.type "algolia" -}}
        {{- $source := $cdn.algoliasearchJS | default "lib/algoliasearch/algoliasearch-lite.umd.min.js" -}}
        {{- dict "Source" $source "Fingerprint" $fingerprint | dict "Scratch" .Scratch "Data" | partial "scratch/script.html" -}}
        {{- $config = dict "type" "algolia" "algoliaIndex" $search.algolia.index "algoliaAppID" $search.algolia.appID "algoliaSearchKey" $search.algolia.searchKey | dict "search" | merge $config -}}
    {{- end -}}
{{- end -}}

For generating index.json file, I just setup config.toml and run hugo for building site to ./public directory.

config.toml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


  [params.search]
    enable = true
    type = "algolia"
    contentLength = 4000
    placeholder = ""
    maxResultLength = 10
    snippetLength = 30
    highlightTag = "em"
    absoluteURL = false
    [params.search.algolia]
      index = <Index name>
      appID = <Application ID>
      searchKey = <Search Only API Key>

Send search index in Algolia

Algolia > Indices > Add Records and upload index.json.

Then search function works 😄

Updeate search index in Algolia

Upload re-generated index.json manually or use Algolia Atomic. I did not try this as I moved to Lunr.

2 Lunr

Lunr works without syncing index files to external service. It’s good for simplifying site updating script and operation.

2.1 Setup Lunr

The template has:

themes/LovewIt/lib/lunr/lunr.min.js

Checking the lunr documentation, I just configured:

config.toml

1
2
3
4
5
6
7
8
9


  [params.search]
    enable = true
    type = "lunr"
    contentLength = 4000
    placeholder = ""
    maxResultLength = 10
    snippetLength = 30
    highlightTag = "em"
    absoluteURL = false

3 Findings / TODO

Note

Activated full-text search in this website, but there are some findings through above work

As Lunr does not support without patch, but I should should check there are themes/LovewIt/lib/lunr/lunr.ja.js in template.
Or, I should check atomic-algolia package to automate syncing index file. Check tiG - Algolia and Gitlab CI/CD
Algolia looks quick and support strict reference.