Searching in Rails with Solr, Sunspot and Docker

Jan 20, 2018 • Gaurav Koley

Cross posted from Blog.Koley.in

Over the past few months, I have been working on a project titled Goodwill Currency for the Web Science Lab @ IIIT-B. It is a Ruby on Rails based portal with a VueJS frontend.

One of the interesting challenges that I came across was search. Since most of the content on this portal will be hidden behind a login wall, we need a custom search solution. Enter Solr and Sunspot.

Solr is a OpenSource Search platform based on Apache Lucene and has an excellent community around it. The documentation is excellent although running it in production can be a bit challenging.

Sunspot is a ruby integration for solr and has excellent support for Rails. To integrate sunspot:

Add to the Gemfile

gem 'sunspot_rails'

Install using bundle install

Generate configuration file:

rails g sunspot_rails:install

For the app, let’s assume we’ve got a Resource model (as is the case in my app), with title, description and tag_list fields. Also, we have an association creators which is a has_many which in turn has a "name" field.

To set it up for keyword search, we use the searchable method:

class Resource < ApplicationRecord

  has_many :creators

  searchable do
    text :title, :default_boost => 2
    text :description
    text :creators do     # for associations
      creators.map { |creator| creator.name }
    end
    text :tag_list, :default_boost => 2
    time :created_at
  end

end

The text here means that we’re creating a field that’s fulltext-searchable. It’ll be broken apart into individual keywords, and then those keywords will be matched against the words in keyword search queries.

There are quite a few other field types, but text is the only one that is searched by keywords.

That default_boost parameter means that, unless otherwise specified, words matching the title field should be considered twice as relevant as words matching the description field.

Now that Sunspot knows how to index the Post model, we need to get the existing data into Solr.

rake sunspot:reindex

Apart from any change in the searchable definition, any time a Post is created, updated, or destroyed, Sunspot will automatically make the change to the index.

Now lets add a new action to the controller which will handle our search queries.

class ResourcesController < ApplicationController

  # GET /resources/search?q={query}
  # GET /resources/search.json?q={query}
  def search
    @search = Resource.search(:include => [:creators]) do
      keywords(params[:q])
    end
    @title = "Search results for "+params[:q]
    @resources = @search.results
    respond_to do |format|
      format.html { render 'resources/index' }
      format.json { render 'resources/search' }
    end
    end

end

The above code utilises search form where a user types in some keywords, which submits a :q param to the ResourcesController#search action which then renders the results using the view for the index action.

Thats it! We are done configuring Rails for Solr and Sunspot.

Now, lets start our Solr server.

Use sunspot_solr gem if you want to run Solr in development.

Sunspot embeds Solr inside the gem so there's no need to install it separately. This means that everything works straight out of the box which makes it far more convenient to use in development.

Add to Gemfile:

gem 'sunspot_rails'
gem 'sunspot_solr'

Run Solr server:

bundle exec rake sunspot:solr:start

You can now access the solr server from browser:

http://localhost:8983/solr/#/

Deploying Solr Search to production

I prefer to deploy all of my Rails apps on docker using docker-compose. So we will be using docker-compose to deploy Solr as well.

# docker-compose.yml

version: '3'
services:
  web:
    build: .
    ports:
      - "3000:3000"
    dns: "8.8.8.8"
    volumes:
      - ".:/app"
    env_file: .env
    links:
      - db:db
      - solr:solr
      # In production instead add external_links for db
    command: bash -c "bin/rake assets:precompile && bin/rake db:create && bin/rake db:migrate && bin/rails s" 

  # In production remove this and add an external link in web
  db:
    image: postgres:latest
    environment:
      - POSTGRES_PASSWORD=somePassword
    volumes:
      - ./database:/var/lib/postgresql
  solr:
    image: solr:7.0.1
    ports:
      - "8983:8983"
    volumes:
      - data:/opt/solr/server/solr/mycores
    entrypoint:
      - docker-entrypoint.sh
      - solr-precreate
      - mycore
    links:
      - db:db
volumes:
  data: {}

Go edit your config/sunspot.yml for the right settings for production mode.

production:
  solr:
    hostname: solr # since our solr instance is linked as solr
    port: 8983
    log_level: WARNING
    solr_home: solr
    path: /solr/mycore 
    # this path comes from the last command of our entrypoint as
    # specified in the last parameter for our solr container

And that is all.

Common Initial Troubleshooting.

If you see:

Errno::ECONNREFUSED (Connection refused - connect(2))

Then perhaps:

You have not started the solr server:

rake sunspot:solr:start

If you see

Solr::Error::Http (RSolr::Error::Http - 404 Not Found
Error:     Not Found

URI: http://localhost:8983/solr/development/select?wt=json

Create a new core using the admin interface at:

http://localhost:8983/solr/#/~cores

or by running the following command:

docker-compose exec solr solr create_core -c development

Resources: