Deploying a Rails 4 app on CentOS 7 production server with Apache and Passenger - Trouble shooting 2020

This article is the log of my trouble shooting while I was developing and deploying the Rails 4 app from:

1. Deploying a Rails 4 app on CentOS 7 production server with Apache and Passenger I
2. RubyOnRails-Deploying-a-Rails4-App-on-CentOS7-production-server-with-Apache-and-Passenger-2.php

The issues are mostly the ones that happened during production deployment.

I hope this material will help for you to use the repo: Rails4-PyGoogle.

Let's start with our favorite one:

The change you wanted was rejected.
Maybe you tried to change something you didn't have access to.

This could be anything. But in my case, it happened several occasions. Noticeable one was when a new user trying to signup. The code uses postgres, memcached and redis, but they were not running. So, I got the following error in the error log in /var/log/httpd/error_log:

1. For postgres:

   App 561 stdout: PG::ConnectionBad (could not connect to server: No such file or directory
   App 561 stdout: 	Is the server running locally and accepting
   App 561 stdout: 	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
   

   The message was clear, the culprit was PG not running.

2. For memcached:

   App 23238 stdout: Processing by Users::RegistApp 20729 stdout: 127.0.0.1:11211 failed (count: 0) Errno::ECONNREFUSED: Connection refused - send
   App 20729 stdout: DalliError: No server availablerationsController#create as HTML
   

   The key was 127.0.0.1:11211 failed where 11211 is the default port for memcached.

3. Redis:

   App 23238 stdout:   Parameters: {"utf8"=>"✓", "authenticity_token"=>"sqEv74+3f0h51FUidZFWyM0ywY35d91oSzj690I/HmE=", "user"=>{"email"=>"pygoogle@aol.com", "password"=>"[FILTERED]"}, "commit"=>"Sign Up"}
   App 23238 stdout: Completed 500 Internal Server Error in 349ms
   App 23238 stdout: 
   App 23238 stdout: Redis::CannotConnectError (Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED)):
   

   This one was obvious because it says "Error connecting to Redis"

4. Another one, when I try to signup with Twitter:

   App 30999 stdout: source=rack-timeout id=081e9b2bf66c8f34bff8beb182d66abe timeout=20000ms state=ready
   App 30999 stdout: Started GET "/o/twitter?flow=signup" for 2602:306:cef8:7280:843d:b5de:5d3b:f55a at 2016-01-23 15:02:13 -0800
   App 30999 stdout: (twitter) Request phase initiated.
   App 30999 stdout: 
   App 30999 stdout: OAuth::Unauthorized (400 Bad Request):
   

This one, I did put the KEY and SECRET into env:

export AUTH_TWITTER_KEY=yX6...
export AUTH_TWITTER_SECRET=Wns...

This "lock_timeout", I cannot figure it out at all until now.

App 561 stdout: source=rack-timeout id=8fe2bf374842c58342d83daa03c7f86a timeout=20000ms service=21ms state=completed

Clearly, it create a db called "pygoogle_production"

$ bundle exec rake db:setup
psql:/var/www/pygoogle.com/Rails4-PyGoogle/db/structure.sql:6: ERROR:  unrecognized configuration parameter "lock_timeout"

We can check the postgres:

$ sudo -i -u postgres psql
Password: 
psql (9.2.14)
Type "help" for help.

postgres=# \l
                                       List of databases
        Name         |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
---------------------+----------+----------+-------------+-------------+-----------------------
 postgres            | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 pygoogle_production | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0           | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                     |          |          |             |             | postgres=CTc/postgres
 template1           | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                     |          |          |             |             | postgres=CTc/postgres
(4 rows)

postgres=# 

In production, if a page is not rendered with CSS like this:

that's most likely we do not have any assets compiled in our public folder:

$ ls public
404.html  422.html  500.html  favicon.ico

So, we need to precompile our rails app:

$ bundle exec rake assets:precompile db:migrate RAILS_ENV=production

Let's check if we have assets in our public folder:

$ ls public
404.html  422.html  500.html  assets  favicon.ico

In production, Rails precompiles images, javascripts and stylesheets to public/assets by default. The precompiled copies are then served as static assets by the web server. The files in app/assets are never served directly in production!.

It happens when I try to signup with FACEBOOK.

So, setup the env variables:

export AUTH_FACEBOOK_KEY=7897...
export AUTH_FACEBOOK_SECRET=c197e...

Then, re-compiled the rails app:

$ pwd
/var/www/pygoogle.com/Rails4-PyGoogle

$ rm -r public/assets

$ bundle exec rake assets:precompile db:migrate RAILS_ENV=production

After that I don't have any FACEBOOK signup.

Ref : Production Mode.

$ RAILS_ENV=production bundle exec sidekiq -C config/sidekiq.yml

2016-01-24T08:26:15.963Z 21211 TID-12he24 WARN: {"retry"=>3, "queue"=>"mailer", "class"=>"Sidekiq::Extensions::DelayedMailer", "args"=>["---\n- !ruby/class 'UserMailer'\n- :welcome_email\n- - 1\n"], "jid"=>"fb4c5e2a8192e8473a5e9322", "enqueued_at"=>1453623944.1512933, "error_message"=>"undefined method `email' for 1:Fixnum", "error_class"=>"NoMethodError", "failed_at"=>1453623944.2553885, "retry_count"=>1, "retried_at"=>1453623975.9594655}
2016-01-24T08:26:15.963Z 21211 TID-12he24 WARN: undefined method `email' for 1:Fixnum
2016-01-24T08:26:15.963Z 21211 TID-12he24 WARN: /var/www/pygoogle.com/Rails4-PyGoogle/app/mailers/user_mailer.rb:8:in `welcome_email'

Got this error when I tried to Facebook Sign Up:

"Not Logged In: You are not logged in. Please login and try again."

This doesn't appear to happen for Twitter. But it does for Facebook signup while the user not logged in. Odd thing is that's not always the case. So far, I do not have any clue why it's happening.

User and Devise are configured to send emails via background jobs via sidekiq. However, this hasn't been working at all from the Development stage.

As far as the setup for sending email is "delay" mode, we need to check if the following 3 key services are running:

1. redis
   To run it:

   sudo systemctl start redis.service
   

2. memcached
   To run it:

   sudo systemctl restart memcached
   

3. sidekiq
   To run it:

   RAILS_ENV=production bundle exec sidekiq -C config/sidekiq.yml
   

Several levels of pieces are involved when we want to send a confirmation email:

1. Third party smtp - in my case, it's gmail.
2. ActiveRecord (app/models/user.rb)

   class User < ActiveRecord::Base
     include Concerns::UserImagesConcern
   
     devise :database_authenticatable, :registerable,
            :recoverable, :rememberable, :trackable, :validatable,
            :confirmable, :timeoutable, :lockable, :async
   
     has_many :authentications, dependent: :destroy, validate: false, inverse_of: :user do
       def grouped_with_oauth
         includes(:oauth_cache).group_by {|a| a.provider }
       end
     end
   
     after_create :send_welcome_emails
   
   ...
   
     # Merge attributes from Authentication if User attribute is blank.
     #
     # If User has fields that do not match the Authentication field name,
     # modify this method as needed.
     def reverse_merge_attributes_from_auth(auth)
       auth.oauth_data.each do |k, v|
         self[k] = v if self.respond_to?("#{k}=") && self[k].blank?
       end
     end
   
     # Do not require email confirmation to login or perform actions
     def confirmation_required?
       false
     end
   
     def send_welcome_emails
       UserMailer.delay.welcome_email(self.id)
       # UserMailer.delay_for(5.days).find_more_friends_email(self.id)
     end
   end
   
   

3. ActionMailer (app/mailers/user_mailer.rb).

   Action Mailer now comes with a method named #deliver_later which will send emails asynchronously (our emails send in a background job). As long as Active Job is setup to use Sidekiq we can use #deliver_later. Unlike Sidekiq, using Active Job will serialize any ActiveRecord instance with Global ID. Later the instance will be deserialized.

   Mailers are queued in the queue mailers. Remember to start sidekiq processing that queue:

   $ bundle exec sidekiq -q default -q mailers
   

   Instead of inheriting our mailers directly from ActionMailer::Base, we create a parent UserMailer mailer class where we can configure default mailer properties as for example layout and "from".

   class UserMailer < ActionMailer::Base
     default from: Rails.application.config.settings.mail.from
     layout 'emails/email'
   
     def welcome_email(user)
       return false unless load_user(user).present?
       @user = user
       mail to: @user.email, subject: I18n.t('emails.welcome.subject')
     end
   
     protected
   
     def load_user(user)
       @user = user.is_a?(User) ? user : User.find(user)
     end
   end
   

4. Devise::ConfirmationsController (app/controllers/users/confirmations_controller.rb)

   class Users::ConfirmationsController < Devise::ConfirmationsController
     def new
       self.resource = (current_user or resource_class.new)
     end
   
     def create
       if user_signed_in?
         Devise.paranoid = false
         params[resource_name] ||= {}
         params[resource_name][:email] = current_user.email
       end
       super
     end
   
     protected
   
     def after_resending_confirmation_instructions_path_for(resource_name)
       user_signed_in? and edit_user_registration_path or super
     end
   end
   

5. Sidekiq - queue
   For performance and usability reasons, we send emails asynchronously using Sidekiq gem.

Make a secret key in our production server:

$ bundle exec rake secret
(in /var/www/pygoogle.com/PyGoogle)
1aefe0afb633f55b2e3547d5ff625891e60127d86eda96b2a695947d54117efdaaca2a908396a7f09bdf6f340a84de1b9089a812c7611c0a1b722630f1090066    

Then, define it in "config/secrets.yml"

/var/www/pygoogle.com/PyGoogle/config/secrets.yml:

production:
  secret_key_base: "1aefe0afb633f55b2e3547d5ff625891e60127d86eda96b2a695947d54117efdaaca2a908396a7f09bdf6f340a84de1b9089a812c7611c0a1b722630f1090066"

Restart the server:

$ sudo apachectl restart