p-search is a local search-engine I developed for Emacs. I haven’t publicized it too much, mainly due to being extremely busy with a newborn, but I’m slowly working on resuming development on it. In this post, I want to dive into a particular use case that I feel many might find helpful: searching a local denote repository.

Background

The typical way that searching is done locally is via a tool like grep. You think of the term you want to search for, run grep, and get back a list of all entries using that term. This obviously works very well and has been quite effective for decades. Problems start to arise when you have too many matches.

Some notes may be centered around a particular topic while others may casually mention it. In this case, when searching for said topic, you would want the more relevant documents to have priority. This is what p-search does.

There are a few tricks that p-search utilizes to do this. The first is that p-search extracts various pieces of data from your document and lets you search on each one. Any search performed is not considered 100% definite, i.e. if the search fails that doesn’t mean the document is excluded from the search results. p-search also employs the BM25 algorithm for ranking search relevance by counting occurrences of terms and taking into account the document length (e.g. a very short document with lots of occurrences of the term “Emacs” is going to be more about Emacs than a very long document with only one occurrence).

Now lastly some terminology, p-search refers to the search criteria as a prior, taken from Bayesian statistics, meaning a priori belief as to what constitutes a relevant document. p-search searches on documents which are provided by a candidate generator. For example, you can have a candidate generator set up to provide all files within a certain directory as a search candidate.

Basic Setup

With p-search installed you’re now ready to start. I’m using a repository of denote notes I’ve found online for my example as I’ve just started to use denote myself and have few notes.

With any file open in your note repository, run the command p-search. You should now see the session buffer open, which should appear as follows:

Note that there are three sections:

• The “Candidate Generators” section is showing you that the search documents are coming from one place: files in the filesystem in directory /Users/zacharyromero/Documents/notes/denote/
• The “Priors” section. Empty since we haven’t searched for anything yet
• The “Search Results” section. This is showing all the documents with an equal probability (number on the right)

Now let’s search our notes related to Emacs!

To add a new search prior, press P. You’ll then see a transient menu with options of search priors to add. Let’s add a text query by pressing q. Now let’s type “Emacs” then press enter. We now get to another transient menu where we set up the prior. Let’s just press c now to create it.

After giving some time for this command to run, you’ll see the results updated with new re-ranked search results.

The topics more related to Emacs should be on top. You can browse through the search results with > (p-search-next-results-page) and see how the results should become less and less related to Emacs.

Extending p-search for Denote

(Special thanks to Samuel W. Flint for implementing this integration)

p-search was designed to be extendable. Candidate generators and priors can be defined and added to the system. Another class of things in p-search is called a mapping. A mapping is a function that takes a document and supplements it with fields or changes it in some way. A mapping may add supplemental searchable data to a document (e.g. extract denote keywords). A mapping may also split a larg document into smaller documents (e.g. splitting an org file into headings) or filter out documents of a certain type.

To load the extension, run (require 'p-search-x-denote). Refresh the p-search buffer with g (p-search-refresh-buffer) and you should now see a new section called “Mappings”.

Press M to add a mapping, select “denote” and then c to create it. You should now see the Denote mapping under the “Mappings” section.

Various metadata about the documents have now been added which can now be searched on. Let’s create another prior by pressing P. This time, let’s create a categorical prior, c.

This will now prompt you for the type of category you want to search on. Since our session now knows about denote documents, you will see that you can search on “category” or “denote-type”. Let’s choose category. Now select a category you want to search on and create this prior by pressing c. Your search should now be updated to prioritize this keyword.

Conclusion

Hopefully this guide was helpful to get you started using p-search. There are a lot more features in p-search that I didn’t cover. I plan on writing more of these guides in the future.