Jekyll2024-04-09T06:12:42-04:00https://fastruby.io/blog/rss.xmlThe Rails Tech Debt BlogFastRuby.io | Rails Upgrade ServiceOmbuLabsTroubleshooting GitHub Actions with Rails and MySQL2024-04-09T06:07:13-04:002024-04-09T06:07:13-04:00https://fastruby.io/blog/troubleshooting-github-actions-with-rails-and-mysql<p>Need help executing a GitHub Actions workflow for your Rails application, especially when dealing with a MYSQL database? Whether you’re just starting or transitioning from another CI service, navigating potential pitfalls can be challenging. If you’ve found yourself nodding along, then this blog post is tailored just for you.</p> <!--more--> <h2 id="introduction">Introduction</h2> <p>In this post, we will guide you through the process of debugging a GitHub Actions workflow. The workflow is designed to meet the following requirements:</p> <ul> <li>Trigger the workflow when a pull request targeting the main branch is opened or updated.</li> <li>Provision a MySQL 5.7 database.</li> <li>Perform a bundle install for dependencies.</li> <li>Execute RuboCop for code linting.</li> <li>Run RSpec specs.</li> </ul> <p>We’ll walk you through our step-by-step debugging process that ultimately led to the successful execution of the workflow. The aim is to share this debugging journey in the hope that it might inspire new ideas for those currently grappling with their workflows. If you’re eager to jump ahead and check out the result, feel free to review the workflow below.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">CI</span> <span class="na">on</span><span class="pi">:</span> <span class="na">pull_request</span><span class="pi">:</span> <span class="na">branches</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">main</span> <span class="na">push</span><span class="pi">:</span> <span class="na">branches</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">main</span> <span class="na">workflow_dispatch</span><span class="pi">:</span> <span class="na">jobs</span><span class="pi">:</span> <span class="na">main</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Lint and Test</span> <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span> <span class="na">services</span><span class="pi">:</span> <span class="na">mysql</span><span class="pi">:</span> <span class="na">image</span><span class="pi">:</span> <span class="s">mysql:5.7</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">3306:3306</span> <span class="na">env</span><span class="pi">:</span> <span class="na">MYSQL_DATABASE</span><span class="pi">:</span> <span class="s">app_test_db</span> <span class="na">MYSQL_ALLOW_EMPTY_PASSWORD</span><span class="pi">:</span> <span class="no">true</span> <span class="na">options</span><span class="pi">:</span> <span class="pi">&gt;-</span> <span class="s">--health-cmd "mysqladmin ping"</span> <span class="s">--health-interval 10s</span> <span class="s">--health-timeout 5s</span> <span class="s">--health-retries 5</span> <span class="na">steps</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v4</span> <span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span> <span class="na">with</span><span class="pi">:</span> <span class="c1"># ruby-version: '2.7.8' # Not needed with a .ruby-version file</span> <span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Lint</span> <span class="na">run</span><span class="pi">:</span> <span class="s">bundle exec rubocop -c .rubocop_with_todo.yml</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup DB, Run tests</span> <span class="na">env</span><span class="pi">:</span> <span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">test</span> <span class="na">APP_DB_HOST</span><span class="pi">:</span> <span class="s">127.0.0.1</span> <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span> <span class="s">bin/rails db:create db:schema:load</span> <span class="s">bundle exec rspec --tag=~broken</span> </code></pre></div></div> <h2 id="1-cant-connect-to-local-mysql-server-through-socket">1. Can’t connect to local MySQL server through socket</h2> <p>We started with an empty workflow. We compiled our workflow by referring to examples found online that utilized MYSQL 5.7. One such example suggested that we use the <code class="highlighter-rouge">APP_DB_HOST</code> MySQL environment variable, by setting it to <code class="highlighter-rouge">localhost</code>.</p> <p>Upon running CI for the first time, we immediately encountered an error message</p> <blockquote> <p>Mysql2::Error::ConnectionError: Can’t connect to local MySQL server through socket</p> </blockquote> <p><a href="https://www.google.com/search?q=github+action+Mysql2%3A%3AError%3A%3AConnectionError%3A+Can%27t+connect+to+local+MySQL+server+through+socket&amp;oq=github+action+Mysql2%3A%3AError%3A%3AConnectionError%3A+Can%27t+connect+to+local+MySQL+server+through+socket">Our google search</a> suggested that we might need to start the MySQL service. So we modified the workflow by adding the following configuration:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">run</span><span class="pi">:</span> <span class="pi">|</span> <span class="s">sudo service mysql start</span> <span class="s">bin/rails db:create db:schema:load</span> <span class="s">bundle exec rspec --tag=~broken # exclude broken specs</span> </code></pre></div></div> <h2 id="access-denied-for-user-rootlocalhost">Access denied for user ‘root’@’localhost’</h2> <p>Starting the MySQL service appeared to be the correct adjustment, as the error message we initially encountered evolved into:</p> <blockquote> <p>Access denied for user ‘root’@’localhost’ (using password: NO)Please provide the root password for your MySQL installation</p> </blockquote> <p>At this point, we shifted our debugging strategy by exploring open-source repositories that employ the same dependencies in their CI workflows. We consider this practice essential for moments when you need hints or ideas to overcome challenges during debugging.</p> <p>We stumbled upon <a href="https://github.com/lortega/github-actions-rails-mysql/blob/master/.github/workflows/ci.yml">this example</a> demonstrating GitHub Actions with Rails and MySQL. Learning from this example, we discovered that there was no need to provide a MySQL user password. Therefore, it puzzled us when we encountered the ‘access denied for user’ error.</p> <p>We experimented with various user and password combinations, including providing root user credentials and attempting to proceed without any user or password. Unfortunately, none of our attempts proved successful. Although altering the user and password did result in different error messages, the issue persisted. An example of one such error message is:</p> <blockquote> <p>mysql Access denied for user ‘user_test’@’localhost’ (using password: YES)</p> </blockquote> <p>We discovered that exploring not only GitHub repositories but also GitHub gists can be beneficial when tackling complex problems. In particular this <a href="https://gist.github.com/fytzzz/3851453">gist</a> caught our attention, which modified the configuration impacting user permissions used by the MySQL daemon.</p> <p>Adjusting the daemon configuration proves valuable when running a local instance of MySQL. However, in our case, we were utilizing a MySQL Docker image that inherently configured the correct user permissions</p> <h3 id="reviewing-source-code">Reviewing source code</h3> <p>As mentioned earlier, our setup involved using a MySQL Docker image. In an effort to gain more insight, we delved into the source code of the Docker image. Our aim was to identify the available environment variables and, if lucky, uncover clues on accessing the MySQL database over the containerized network.</p> <p>If you find yourself in a similar predicament, we highly recommend reviewing the <a href="https://github.com/docker-library/mysql/blob/c13cda9c2c9d836af9b3331e8681f8cc8e0a7803/5.7/docker-entrypoint.sh.">MYSQL image configurations</a> for GitHub Actions.</p> <p>After reviewing the configuration, the available environment variables and the impact of setting or not setting them became crystal clear.</p> <p>Once again, we experimented with various combinations of environment variables, drawing inspiration from the insights gained in the source code. Unfortunately, none of the following options yielded success:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">mysql</span><span class="pi">:</span> <span class="na">image</span><span class="pi">:</span> <span class="s">mysql:5.7</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="m">3306</span> <span class="na">env</span><span class="pi">:</span> <span class="na">MYSQL_DATABASE</span><span class="pi">:</span> <span class="s">app_test_db</span> <span class="na">MYSQL_ROOT_PASSWORD</span><span class="pi">:</span> <span class="s">password</span> <span class="c1"># MYSQL_ALLOW_EMPTY_PASSWORD: true</span> <span class="c1"># MYSQL_USER: user_test</span> <span class="c1"># MYSQL_PASSWORD: root</span> <span class="c1"># MYSQL_ROOT_PASSWORD: root</span> <span class="c1"># MYSQL_RANDOM_ROOT_PASSWORD: 'true'</span> </code></pre></div></div> <p>At this stage, we hypothesized that the user lacked the necessary database permissions to access the database, given our exhaustive attempts with every conceivable combination of usernames and passwords</p> <p>Even though our assumption was invalid it was supported by this <a href="https://stackoverflow.com/a/20353578">Stack Overflow post</a>. This experience emphasizes the importance of caution when relying on assumptions, even when supported by online information.</p> <blockquote> <p>Misleading information might make you think that you need to configure database permission.</p> </blockquote> <h2 id="undefined-method-strip-for-nilnilclass">undefined method ‘strip’ for nil:NilClass</h2> <p>As modifying user permissions and credentials failed to produce the desired outcome, we redirected our attention to the second part of the error message. When confronted with a persistent error, exploring different aspects of the error message might offer valuable insights.</p> <p>Given the cryptic nature of the error, we delved into the Rails source code. Specifically, we examined <a href="https://github.com/rails/rails/blob/c4d3e202e10ae627b3b9c34498afb45450652421/activerecord/lib/active_record/tasks/mysql_database_tasks.rb">the file defining MySQL database tasks</a>. Our focus on this file stemmed from the fact that the <code class="highlighter-rouge">rake db:create</code> task was failing to complete during the initialization of the workflow container.</p> <p>From our analysis of the Rails source code, we concluded that the <code class="highlighter-rouge">strip</code> command was executed after the database connection failure. Take note of this <a href="https://github.com/rails/rails/blob/c4d3e202e10ae627b3b9c34498afb45450652421/activerecord/lib/active_record/tasks/mysql_database_tasks.rb#L16-L29">section</a>, where the <code class="highlighter-rouge">create</code> task triggers the <code class="highlighter-rouge">strip</code> instruction only when the database connection fails and the user is not the <code class="highlighter-rouge">root</code> user.</p> <h2 id="unknown-mysql-server-host-mysql">Unknown MySQL server host ‘mysql’</h2> <p>Drawing from our observations in the Rails source code, we turned our attention to the possibility of a container network issue.</p> <p>While investigating potential database connection problems within the Docker container, we came across an <a href="https://kirschbaumdevelopment.com/insights/laravel-github-actions">article</a> suggesting that updating the host name to match the service name defined in the workflow might be a solution.</p> <p>Accordingly, we modified the <code class="highlighter-rouge">DB_HOST</code> environment variable to align with the service name we defined as <code class="highlighter-rouge">mysql</code>. Unfortunately, this adjustment led to a new error:</p> <blockquote> <p>Mysql2::Error::ConnectionError: Unknown MySQL server host ‘mysql’</p> </blockquote> <p>For reference, here’s what our workflow looked like at this stage</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">CI</span> <span class="na">on</span><span class="pi">:</span> <span class="na">pull_request</span><span class="pi">:</span> <span class="na">branches</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">master</span> <span class="na">push</span><span class="pi">:</span> <span class="na">branches</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">main</span> <span class="na">workflow_dispatch</span><span class="pi">:</span> <span class="na">jobs</span><span class="pi">:</span> <span class="na">main</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Lint and Test</span> <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span> <span class="na">services</span><span class="pi">:</span> <span class="na">mysql</span><span class="pi">:</span> <span class="na">image</span><span class="pi">:</span> <span class="s">mysql:5.7</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="m">3306</span> <span class="na">env</span><span class="pi">:</span> <span class="na">MYSQL_DATABASE</span><span class="pi">:</span> <span class="s">app_test_db</span> <span class="na">MYSQL_ROOT_PASSWORD</span><span class="pi">:</span> <span class="s">password</span> <span class="na">options</span><span class="pi">:</span> <span class="pi">&gt;-</span> <span class="s">--health-cmd "mysqladmin ping"</span> <span class="s">--health-interval 10s</span> <span class="s">--health-timeout 5s</span> <span class="s">--health-retries 5</span> <span class="na">steps</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v4</span> <span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span> <span class="na">with</span><span class="pi">:</span> <span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup DB, Run tests</span> <span class="na">env</span><span class="pi">:</span> <span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">test</span> <span class="na">DB_HOST</span><span class="pi">:</span> <span class="s">mysql</span> <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span> <span class="s">sudo service mysql start</span> <span class="s">bin/rails db:create db:schema:load</span> <span class="s">bundle exec rspec --tag=~broken</span> </code></pre></div></div> <blockquote> <p>Tip: Save yourself time and remove linting while you debug. Only run the bare essentials required to debug the workflow.</p> </blockquote> <h2 id="mysql2error-host-1721801-is-not-allowed-to-connect-to-this-mysql-server">Mysql2::Error: Host ‘172.18.0.1’ is not allowed to connect to this MySQL server</h2> <p>Despite our efforts, the persisting errors strongly suggested a failure to establish a connection with the MySQL instance, indicating potential issues with the host or port settings. For local MySQL use with Docker, specifying <code class="highlighter-rouge">127.0.0.1</code> as the database host is often necessary due to internal Docker image quirks, as highlighted in this <a href="https://dev.to/risafj/how-to-use-mysql-on-docker-for-rails-local-development-switching-mysql-versions-made-easy-4ihf">resource</a>.</p> <p>Recognizing this, we considered applying a similar approach in our CI environment. Initially, updating the <code class="highlighter-rouge">DB_HOST</code> didn’t yield the desired outcome, prompting us to also modify the <code class="highlighter-rouge">MYSQL_ROOT_HOST</code> environment variable. However, this adjustment resulted in a new error:</p> <blockquote> <p>Mysql2::Error: Host ‘172.18.0.1’ is not allowed to connect to this MySQL server</p> </blockquote> <p>In a subsequent attempt, we explicitly set the port number to 3306, yet we still encountered a connection failure:</p> <blockquote> <p>Mysql2::Error::ConnectionError: Can’t connect to MySQL server on ‘127.0.0.1:3306’”</p> </blockquote> <h2 id="success-">Success 💪</h2> <p>Our exploration led us to a helpful <a href="https://dev.to/lortega/setting-up-github-actions-with-rails-6-1-and-mysql-5-7-23lg">online article</a> that explicitly demonstrated the effectiveness of specifying the port number as <code class="highlighter-rouge">3306</code>. Interestingly, we noted that the article didn’t include a line to start the MySQL service (i.e., <code class="highlighter-rouge">sudo service mysql start</code>).</p> <p>In our quest for clarity on whether the MySQL instance needed explicit startup, we discovered a key insight: GitHub includes MySQL in their Ubuntu images. <a href="https://ovirium.com/blog/how-to-make-mysql-work-in-your-github-actions/">This article</a> prompted the realization that we might inadvertently be attempting to access a running MySQL instance other than the one we intended.</p> <blockquote> <p><a href="https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2204-Readme.md#mysql">GitHub includes MySQL in their Ubuntu images.</a>.</p> </blockquote> <p>We initiated the MySQL instance following the guidance from <a href="https://alexrs.me/2021/github-actions-mysql-rails">this article</a>. However, unbeknownst to us, we unintentionally started the MySQL instance that was included in the Ubuntu image. This inadvertent action hindered our ability to access the MySQL instance specified in our action configuration.</p> <p>Upon realizing our mistake, we promptly removed the line (<code class="highlighter-rouge">sudo service mysql start</code>) responsible for starting the local MySQL instance. And finally, we were able to successfully connect to the intended MySQL instance.</p> <h2 id="conclusion">Conclusion</h2> <p>In this blog post, we embarked on a detailed exploration of the challenges encountered while setting up a GitHub Actions workflow for a Rails application with MySQL integration. We faced various problems, like errors in the workflow and issues with Docker settings and starting MySQL.</p> <p>Each problem was carefully looked into and solved. By checking source code, reading online articles, and seeking help from the community, we successfully resolved each hurdle, culminating in a clear understanding of how to establish a reliable CI/CD pipeline for Rails applications with MySQL.</p> <p>In navigating the complexities of GitHub Actions and MySQL integration, we’ve shared our debugging journey to offer insights and solutions. Even though this post describes an application that made use of MySQL, the same issue can arise with application that make use of PostgreSQL, since it is also included but not started by default in the ubuntu images used by Github Actions. If you find yourself grappling with similar challenges in your CI configuration, don’t hesitate to reach out.</p> <p>Need to streamline your CI/CD processes? <a href="/#contactus">Contact us today</a> to propel your project forward!</p>fbuysNeed help executing a GitHub Actions workflow for your Rails application, especially when dealing with a MYSQL database? Whether you’re just starting or transitioning from another CI service, navigating potential pitfalls can be challenging. If you’ve found yourself nodding along, then this blog post is tailored just for you.Getting to the Root of Slow Page Loads: Our Database Optimization Story2024-04-03T12:50:12-04:002024-04-03T12:50:12-04:00https://fastruby.io/blog/getting-to-the-root-of-slow-page-loads<p>Imagine your customers are complaining that a few specific pages in your application take forever to load and you get to work to fix the issues. But there is a problem. You don’t know where to begin. If this journey of seeking truth, I mean answers, interests you, then read the rest of the article.</p> <!--more--> <p>Recently I was facing a similar challenge. I had to understand and fix the problems that were causing slowness on a few pages of a product. To give some technical context on the product, it was running on Rails 7.1 and using Postgres 13.0 as the database of choice. For the purpose of this article, let us assume that we are working on this together. So we have narrowed down the pages that we want to fix the performance for. The next step is to figure out where to begin. The pages can be slow due to various reasons like javascript and css bloat, slow database queries, high request queue, etc.</p> <h2 id="rack-mini-profiler">rack-mini-profiler</h2> <p>The first step in this journey is to integrate <a href="https://github.com/MiniProfiler/rack-mini-profiler">rack-mini-profiler</a>. This gem will help us see insights about the page and the time it spent on different steps to load the page, so we don’t have to guess that.</p> <p>Here is a screenshot of what that data can look like:</p> <center><img src="/blog/assets/images/blog348/rmp-1.png" alt="rack-mini-profiler tab" /></center> <p>As you can see from the screenshot, it gives us information about all the database queries that were fired on the page. We can click on those mysql query links and get more information like what query is that, how much time it took, what line in the application code triggers the query. It also gives us information about the various partials that were rendered as part of the page, and how much time each partial took. In which component of the page was the majority percentage of time spent?</p> <p>For the purpose of this article, let us assume that the majority of the page load time was spent in the sql queries. So by using rack-mini-profiler, we were able to narrow down to the problem that was leading to high page load times and also the queries which took the most time.</p> <h2 id="analyzing-the-slow-queries">Analyzing the slow queries</h2> <p>Once we know which queries to debug, we are one step closer to a better performing page. A query could be slow due to various reasons like lack of or wrong indexing, fetching extra columns, lack of joins, etc. One of the most common reasons which hurts the most is lack of or wrong indexing.</p> <p>But how do we even find out what are these indexes and which ones are being used or simply put, how do we know how the database is running a query? The answer to these questions lies in query plans. We can use <code class="highlighter-rouge">explain</code> or <code class="highlighter-rouge">explain and analyze</code> on the query.</p> <p>This is how to run the commands from the psql terminal:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>explain select * from users limit 1; explain analyze select * from users limit 1; </code></pre></div></div> <p>The query plan for this query looks like this:</p> <center><img src="/blog/assets/images/blog348/qp1.png" alt="query plan for the query" /></center> <p>The query plans give us information about what kind of scanning the database had to do to fetch the right data. In this case, the database performed a sequential scan on the users table. We can see the associated cost of performing an index scan. If it was using any index, it would show that in the query plan.</p> <p>When we see that the query is doing a sequential scan, and then filtering on a column, it nudges us to try and add an index to the table on that column, in this case, index on the provider column of the users table. Once we add the index, we should run the <code class="highlighter-rouge">explain analyze</code> command on the query again to see what kind of difference it makes. We should benchmark queries before and after applying optimisations.</p> <p>Also when I look at the query:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>explain select * from users limit 1; </code></pre></div></div> <p>I wonder, do we need all the fields from the users table? This is another area we should look into when we are debugging queries. The less data points we fetch, the faster we will get the query results.</p> <p>But there is a big problem that lies ahead of us. Let us assume that we added the index on provider to the users table and we ran the query again. Upon running it with explain analyze, we see that it is still doing a sequential scan and not using the newly created index. What does this mean? This brings us to the next section.</p> <h2 id="test-data">Test data</h2> <p>Using rack-mini-profiler in the earlier section, we had narrowed down the slow queries. But when I ran these queries in our local environment, we realised that they are not slow at all. When we are running these queries, and running explain analyze on them, we do not see a high associated cost in running the query, even the sequential scans are fast.</p> <blockquote> <p>Sometimes, when the data in the table is too small, then doing a sequential scan is faster than looking for that data in the index and then fetching those selected rows from the table.</p> </blockquote> <p><code class="highlighter-rouge">rack-mini-profiler</code> was being run on the production application, while the same queries and pages are not slow in our local environment. One of the reasons for this is because the database and the data in it are not of similar scale when compared with the production environment. Then, how do we solve this problem?</p> <p>One way is to take a database dump from the production environment and restore that dump locally. This way we can at least work on the same amount of data. The server configuration still remains widely different from what we have running on production, but that seems far-fetched to have the same configuration locally. There is a problem with this though. What if the database on production is too big to download and restore locally? What if the database contains personal information about the users that we are not allowed to download without their consent? This brings us to the second way.</p> <p>The second way is to generate fake data locally. For this, we use the <a href="https://github.com/faker-ruby/faker">faker</a> gem to generate the fake data. But we wonder how much data is enough data to test the slowness in our queries? Sometimes it won’t be possible to generate the same amount of data as there is production. And there is no right answer to how much data is enough. For example, when I first ran the query:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>explain select * from users limit 1; </code></pre></div></div> <p>The associated cost was very slow. The total number of records in the users table was 20 at the time. There is just not enough data in the database to perform and benchmark any kind of optimisation.</p> <p>So I used the faker gem to add 200,000 records in the users table. This is the script I used:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>200000**.times do |index| u = User.new u.email = Faker::Internet.email u.provider = 'github' u.save end </code></pre></div></div> <p>And now when I run <code class="highlighter-rouge">explain</code> on the select query, I see a much higher associated cost, and applying optimisations shows some change in the performance of those queries. This is one way to generate test data locally.</p> <h2 id="database-server-configurations">Database Server configurations</h2> <p>Sometimes we can fix our queries with all the right techniques, and yet we do not see a big improvement in the performance of our queries. One of the reasons for that could be the configurations of the database. To get a view of some of the database settings, we used a gem called <a href="https://github.com/ankane/pghero">pghero</a> to analyze that and also get some of the other additional insights it gives us about our database.</p> <p>After integrating the gem with our Rails application, it can look like this:</p> <center><img src="/blog/assets/images/blog348/pghero1.png" alt="pghero overview page" /></center> <p>As we can see in the above screenshot, it gives us an overview of our database. It also suggests indexes to apply based on the queries it fires, tracks slow queries for us and more. The tab we are interested in is the tune tab on the left side panel.</p> <p>This is what it looks like:</p> <center><img src="/blog/assets/images/blog348/config.png" alt="postgres current configurations" /></center> <p>This does not mean the only way to figure out the configuration is to integrate this gem. Another way is to log into the <code class="highlighter-rouge">psql</code> terminal and query the <code class="highlighter-rouge">pg_settings</code> table. For example, if we want to find the <code class="highlighter-rouge">shared_buffers</code> value, we can run this query:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT name, setting::numeric * current_setting('block_size')::numeric AS shared_buffers_bytes FROM pg_settings                                                                WHERE name = 'shared_buffers'; </code></pre></div></div> <p>But by using <code class="highlighter-rouge">pghero</code> not only do we get insights on the configuration, but also on various other things like number of connections, CPU load, slow queries, suggested index etc. So we chose to integrate this gem.</p> <p>Now we have seen the values of various configuration settings, but the next question is, how do we know if these values are what we want or if we should change them to something else? To answer this question, we will use another tool called <a href="https://pgtune.leopard.in.ua/">pgtune</a>. This is what it looks like when we use the tool and fill some information about our database:</p> <center><img src="/blog/assets/images/blog348/pgtune.png" alt="postgres suggested configurations by pgtune" /></center> <p>While we are using this tool to get suggestions around various other configuration values for the database, it is always a good idea to read through the database official documentation and find out more about what values should be set in the configuration.</p> <p>One thing to consider when applying these changes is to first apply them on a staging database instance which is of the same / similar configuration as the production database and benchmark the impact on the staging database.</p> <h2 id="conclusion">Conclusion</h2> <p>After applying these configuration changes, and applying the optimisation techniques, we should see an improvement in the page load times for our slow pages. This is how we would approach a problem of high page load times when the bottleneck is the database and the queries.</p> <p>Having issues with your page load time? Want to optimise your application’s performance? Check out our <a href="https://www.fastruby.io/tune">Tune Report</a> and <a href="/#contactus">talk to us today!</a></p>rishijainImagine your customers are complaining that a few specific pages in your application take forever to load and you get to work to fix the issues. But there is a problem. You don’t know where to begin. If this journey of seeking truth, I mean answers, interests you, then read the rest of the article.Speeding Up Assets Precompilation2024-03-27T06:14:35-04:002024-03-27T06:14:35-04:00https://fastruby.io/blog/speed-up-assets-precompile<p>There has been a lot of conversations on social media about the “<a href="https://world.hey.com/dhh/you-can-t-get-faster-than-no-build-7a44131c">NoBuild</a>” approach: using native browser features and plain CSS+JavaScript to avoid a precompilation step for our assets.</p> <p>In many cases, it’s not easy to move to a “NoBuild” setup (and in some cases it’s not even possible depending on the application’s needs), and we can still aim to make the <code class="highlighter-rouge">assets:precompile</code> task as fast as possible if we can’t eliminate it.</p> <p>In this article we’ll explore some areas for optimization using one of our applications.</p> <!--more--> <h2 id="the-starting-point">The Starting Point</h2> <p>We’ll be using a Rails 7.0 app, with <a href="https://github.com/rails/sprockets-rails">Sprockets</a> to handle the CSS and <a href="https://github.com/rails/webpacker">Webpacker</a> to handle the JavaScript. We are using the <code class="highlighter-rouge">rails assets:precompile</code> command as this is the command we run during deployment in Heroku.</p> <p>For every iteration, we have to clear the already compiled assets with the <code class="highlighter-rouge">rails assets:clobber</code> task before running the precompile task. This is essential because the different bundlers are smart enough to avoid extra work when possible if files are already compiled and that can skew the results. We use Heroku to host this app, and the assets are compiled from scratch when deploying.</p> <p>When talking about improving performance, the first step we must always do is to profile and measure what we are trying to improve.</p> <p>This has 2 main benefits:</p> <ol> <li>We have a base number to compare with (to know when something is improving or not)</li> <li>With a detailed profiling result we can identify which parts of the process are slow to focus on those areas (to not waste time on changing what’s already fast).</li> </ol> <p>The first idea was to use the <code class="highlighter-rouge">time</code> command to wrap the task (<code class="highlighter-rouge">time rails assets:precompile</code>). This provides information about how much time it took to run the command but it doesn’t provide enough information to know where the time is going.</p> <p>In addition of the <code class="highlighter-rouge">time</code> command, we decided to also print different timestamps in some parts of the process to have more granular data.</p> <h3 id="boot-sequence-for-a-rails-task">Boot sequence for a Rails task</h3> <p>It is important to understand the boot sequence of the app to identify what we want to measure and to find opportunities. Running the <code class="highlighter-rouge">rails assets:precompile</code> command follows this sequence:</p> <ul> <li> <p>The <code class="highlighter-rouge">rails</code> command is executed with the <code class="highlighter-rouge">assets:precompile</code> argument. This internally calls the <code class="highlighter-rouge">Rails::Command.invoke</code> method with <code class="highlighter-rouge">assets:precompile</code> as the command to execute. (<a href="https://github.com/rails/rails/blob/v7.1.2/railties/lib/rails/commands.rb#L18">source</a>)</p> </li> <li> <p>Eventually, Rake is used to load our application using the <code class="highlighter-rouge">Rakefile</code></p> </li> </ul> <p>This is what a <code class="highlighter-rouge">Rakefile</code> typically looks like (and this is our case):</p> <div class="language-rb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require_relative</span> <span class="s2">"config/application"</span> <span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">load_tasks</span> </code></pre></div></div> <p>Note that, in this step, we are loading our app, up to the point of running all the initializers too.</p> <p>After the application and the tasks are loaded, Rake will continue processing the actual <code class="highlighter-rouge">assets:precompile</code> task</p> <p>Multiple tasks can be specified, so Rake will loop through them invoking them one after the other. (<a href="https://github.com/ruby/rake/blob/master/lib/rake/application.rb#L138">source</a>)</p> <h3 id="measuring">Measuring</h3> <p>We want to start measuring 4 parts initially (then we can go deeper in each area if we need to):</p> <ol> <li>How long it takes from pressing <code class="highlighter-rouge">Enter</code> on the terminal to start processing the <code class="highlighter-rouge">Rakefile</code> file</li> <li>How long it takes for the application source to be loaded</li> <li>How long it takes for tasks to be loaded</li> <li>How long it takes after loading the tasks to be back to the terminal (the actual assets compilation)</li> </ol> <p>We picked these first 4 areas because they are the boundaries of our application code and the first lines from our source that are executed.</p> <p>Once we decide what we need to optimize we can go deeper patching Rails’ and Rake’s source code or measuring other files to have more details, but this is good as a starting point.</p> <p><strong>Start and End Times</strong></p> <p>The first thing we do is to print the timestamps before and after the command:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">time</span> <span class="o">(</span><span class="nb">echo</span> <span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%s.%N'</span><span class="si">)</span> <span class="o">&amp;&amp;</span> rails assets:precompile <span class="o">&amp;&amp;</span> <span class="nb">echo</span> <span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%s.%N'</span><span class="si">)</span><span class="o">)</span> </code></pre></div></div> <p>This will first print the current timestamp, then execute the task, and then print the timestamp at the end. We are also adding a <code class="highlighter-rouge">time</code> call wrapping these calls for extra information to double check our results, but the timestamps are enough.</p> <p><strong>Rake Reading the Rakefile</strong></p> <p>Next, we are printing the current timestamp at the beginning of the <code class="highlighter-rouge">Rakefile</code> with <code class="highlighter-rouge">puts "#{Time.now.to_f} - loading Rakefile"</code>. With this information and the initial timestamp we can get our first number.</p> <p><strong>Rakefile Lines</strong></p> <p>To measure how long it takes to load the application and the tasks, we have two options:</p> <ul> <li>We can use something like the <a href="https://ruby-doc.org/stdlib-2.7.1/libdoc/benchmark/rdoc/Benchmark.html#method-c-measure"><code class="highlighter-rouge">Benchmark#measure</code></a> method (or any similar tool)</li> <li>Or we can print the current timestamp after each line and then compare with the previous number.</li> </ul> <p><strong>Processing the Task</strong></p> <p>Since the final timestamp is printed in the terminal with the last <code class="highlighter-rouge">echo $(date '+%s.%N')</code> command, we need the timestamp at the end of parsing the <code class="highlighter-rouge">Rakefile</code> file. If we used timestamp printing for the previous measures we already have this.</p> <h3 id="final-code">Final Code</h3> <p>This is how our Rakefile looks like:</p> <div class="language-rb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="no">Time</span><span class="p">.</span><span class="nf">now</span><span class="p">.</span><span class="nf">to_f</span><span class="si">}</span><span class="s2"> - loading Rakefile"</span> <span class="nb">require_relative</span> <span class="s2">"config/application"</span> <span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="no">Time</span><span class="p">.</span><span class="nf">now</span><span class="p">.</span><span class="nf">to_f</span><span class="si">}</span><span class="s2"> - config/application loaded"</span> <span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">load_tasks</span> <span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="no">Time</span><span class="p">.</span><span class="nf">now</span><span class="p">.</span><span class="nf">to_f</span><span class="si">}</span><span class="s2"> - tasks loaded"</span> </code></pre></div></div> <p>And now we are ready to get some numbers.</p> <h2 id="webpacker">Webpacker</h2> <p>After running the command we get these output:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1705326728.2913189 1705326729.4916348 - loading Rakefile 1705326732.9132965 - config/application loaded 1705326732.9966373 - tasks loaded 1705326737.6213546 real 0m9,333s user 0m8,855s sys 0m2,520s </code></pre></div></div> <p>We then ran the command multiple times and calculated the averages.</p> <p>From this we can see that:</p> <ul> <li>It takes <strong>1.16s</strong> to start loading the application</li> <li>It takes <strong>3.38s</strong> to load the application</li> <li>It takes <strong>0.08s</strong> to load the tasks</li> <li>It takes <strong>4.83s</strong> to compile the assets</li> </ul> <p>Some important things to note:</p> <ul> <li> <p>Our first idea was to optimize the assets compilation, but we found out that this phase accounts for ~51% of the total time, so we have other areas to focus too. The other ~49% of the time affects all other commands, not just the assets compilation, so improving the time for that can also impact the time it takes for the other commands list starting the console or the server.</p> </li> <li> <p>We can see that loading the tasks is really quick compared to the other numbers (less than 1% of the total time) and it’s not worth analyzing at this point for us, so we’ll skip it in the rest of the article.</p> </li> </ul> <h2 id="rails-vs-rake">Rails vs Rake</h2> <p>We’ll first focus in the initial phase of the process. There is not much we can do there since it’s internal code by Rails and Rake, but what happens if we use <code class="highlighter-rouge">rake assets:precompile</code> instead of <code class="highlighter-rouge">rails assets:precompile</code>? Let’s see the numbers:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1705327358.4954120 1705327359.1051338 - loading Rakefile 1705327362.9320574 - config/application loaded 1705327363.0320442 - tasks loaded 1705327367.6751084 real 0m9,183s user 0m8,236s sys 0m2,511s </code></pre></div></div> <blockquote> <p>Note that we can’t simply replace <code class="highlighter-rouge">rails</code> with <code class="highlighter-rouge">rake</code> always, to be safe we use <code class="highlighter-rouge">bundle exec rake</code> here. You can <a href="https://www.ombulabs.com/blog/ruby/learning/understanding-bundler.html">learn why here</a>.</p> </blockquote> <p>By replacing <code class="highlighter-rouge">rails</code> with <code class="highlighter-rouge">bundle exec rake</code>, we can see a reduction from <strong>1.16s</strong> to <strong>0.63s</strong> (using averages from multiple runs). This difference doesn’t look big, but compared to the original <strong>9.3s</strong> total this is a <strong>6/7%</strong> reduction of the total time.</p> <h2 id="loading-the-application">Loading the Application</h2> <p>We will cover optimization for this phase in another article in more depth with more measures, but here are some tips on how to optimize this:</p> <ul> <li> <p>We can use the <code class="highlighter-rouge">Rake.application.top_level_tasks</code> variable to know which tasks are going to be executed. With this information, we can conditionally initialize parts of the application to not load code or don’t do actions that won’t be needed for the assets. For example: if we have an initializer that performs a request to an external API, we could make a decision to skip that call when <code class="highlighter-rouge">Rake.application.top_level_tasks == ["assets:precompile"]</code>.</p> </li> <li> <p>We can use <a href="https://github.com/nevir/Bumbler">Bumbler</a> to know the time it takes to require each of our dependencies for a given environment: <code class="highlighter-rouge">RAILS_ENV=development bumbler --all</code>. Then we can analyze the results and see which gems are not needed for a given environment and rearrange the <code class="highlighter-rouge">Gemfile</code> putting each gem in the right group.</p> </li> <li> <p>We can measure each part of the boot process with <code class="highlighter-rouge">Benchmark.measure</code> or <code class="highlighter-rouge">puts</code> statements to identify the slowest parts to focus.</p> </li> </ul> <h2 id="assets-precompilation">Assets Precompilation</h2> <p>We finally got to the actual phase we wanted to optimize initially. We have 2 areas to focus here: how much work needs to be done (how many assets need to be compiled using different compilation steps) and what tool does the work (which bundler we use)</p> <h3 id="bundlers">Bundlers</h3> <p>The first idea we decided to try was to replace Webpacker with another bundler. We decided to try with Vite, Esbuild, and Bun to compile our JavaScript.</p> <blockquote> <p>We’ll use the <code class="highlighter-rouge">rails</code> command and not <code class="highlighter-rouge">rake</code> for these tests to compare with the initial metrics.</p> </blockquote> <p><strong>Vite</strong></p> <p>Our first test was to migrate to <a href="https://vitejs.dev">Vite</a> using the <a href="https://github.com/sergii/vite_rails?tab=readme-ov-file#installation-"><code class="highlighter-rouge">vite_rails</code> gem</a>. We won’t go over the steps of the migration since that is out of the scope of this article.</p> <p>These are the measurements while using Vite to compile the assets:</p> <ul> <li><strong>4.52s</strong> for the assets precompilation phase</li> <li><strong>9.12s</strong> for the whole process to execute</li> </ul> <p>The number is slightly better, only around <strong>300ms</strong> faster. This change is not that significant, and based on other reports online we were expecting a better number.</p> <p><strong>Esbuild</strong></p> <p><a href="https://esbuild.github.io">esbuild</a> is known to be fast, and faster than both <a href="https://rollupjs.org">Rollup</a> (used by Vite) and Webpack. For this, we migrated the assets pipeline from the <code class="highlighter-rouge">webpacker</code> gem to the <a href="https://www.fastruby.io/blog/esbuild/webpacker/javascript/migrate-from-webpacker-to-esbuild.html"><code class="highlighter-rouge">jsbundling-rails</code> gem with esbuild as the bundler</a>.</p> <p>After the setup was done, we ran the <code class="highlighter-rouge">rails assets:precompile</code> task and the average of the phase that compiles the assets is now <strong>1.99s</strong>, and the total time for the command is now <strong>6.53s</strong>.</p> <p>This means a reduction of more than <strong>50%</strong> compared to the original <strong>4.82s</strong> with Webpacker for that specific phase of the process and a reduction of over <strong>30%</strong> for the complete command.</p> <p><strong>Bun</strong></p> <p>We couldn’t leave <a href="https://bun.sh">Bun</a> out of this, known for being <code class="highlighter-rouge">blazingly fast</code>. We also used the <code class="highlighter-rouge">jsbundling-rails</code> gem to set this up.</p> <p>After a few runs and calculating averages, the assets compilation step of the process took an average of <strong>1.95s</strong>, slightly faster than esbuild by <strong>40ms</strong>, but not really significant (each run varies for more than than).</p> <p>Both Bun and Esbuild provide a similar improvement in speed.</p> <h3 id="reducing-the-assets">Reducing the Assets</h3> <p>It is expected that, the more work the bundlers have to do, the longer it will take for this process to finish. So the next optimization we can work on is removing unused JavaScript and CSS code, replacing big dependencies with smaller ones (or eliminate them), removing unused images, fonts, and any other type of assets that is no longer needed, replacing JS code and images with modern CSS features (when possible), replacing steps in the pipeline, etc.</p> <p>For example: instead of using SASS for our CSS, we can try to move to native CSS nesting and custom properties if that’s what we need, removing this step during the build process. Instead of using a plugin like PurgeCSS to remove unused CSS, we can remove the CSS from the source itself manually.</p> <h2 id="conclusion">Conclusion</h2> <p>By applying only a few of these ideas (using Esbuild/Bun and the <code class="highlighter-rouge">rake</code> command), we reduced the time from <strong>9.5s</strong> to <strong>6.5s</strong> without modifying the assets, and this is a reduction of <strong>35%</strong>. There are always more opportunities for improvements and more advanced and involved changes we can do to get the numbers even lower, and we’ll explore them in future articles.</p> <p>Is your app slow to start? <a href="/tune">We can take a look!</a></p>arieljuodThere has been a lot of conversations on social media about the “NoBuild” approach: using native browser features and plain CSS+JavaScript to avoid a precompilation step for our assets. In many cases, it’s not easy to move to a “NoBuild” setup (and in some cases it’s not even possible depending on the application’s needs), and we can still aim to make the assets:precompile task as fast as possible if we can’t eliminate it. In this article we’ll explore some areas for optimization using one of our applications.How Do You Know When Your App is Not Compliant?2024-03-12T14:53:58-04:002024-03-12T14:53:58-04:00https://fastruby.io/blog/how-do-you-know-if-app-is-noncompliant<p>Ensuring that your company’s website is current with compliance standards is extremely important and essential for any Rails application. Operating with a compliant application guarantees security that can help with handling sensitive data and maintaining users’ trust. The more compliant your website is, the more secure it will be against data breaches, which helps users feel safe when they’re using it.</p> <p>So what does it take to be compliant? In this article, we will focus on security and cover some indicators to help identify if your Rails app might not be compliant anymore.</p> <!--more--> <h3 id="understanding-compliance-requirements">Understanding Compliance Requirements</h3> <p>Compliance requirements relevant to Rails applications include but are not limited to common standards and regulations such as GDPR, PCI DSS, HIPAA, and accessibility guidelines like WCAG. This can involve various aspects such as security, privacy, accessibility, and legal requirements.</p> <h3 id="security-vulnerabilities">Security Vulnerabilities</h3> <p>Security is an essential component for compliance. If your Rails app is not regularly updated with the latest security patches or if vulnerabilities are discovered in the libraries or dependencies it uses, it could become non-compliant with security standards.</p> <p>Rails itself is really good at prioritizing security and provides many built-in security features. However as an application continues to evolve and grow with the introduction of new features, third party libraries or dependencies, chances are that vulnerabilities could have been introduced.</p> <p>Regularly updating dependencies, implementing secure coding practices, and conducting thorough security assessments are essential steps to mitigate vulnerabilities and comply with industry standards like OWASP (Open Web Application Security Project) guidelines. This can save a company from any possible data breaches.</p> <p>Resources listed below are helpful in identifying what type of security risks can exist and how to handle them:</p> <ul> <li><a href="https://guides.rubyonrails.org/security.html">Rails Guides on Securing applications</a></li> <li><a href="https://owasp.org/www-project-top-ten/">Top ten security risks to be aware of</a></li> <li><a href="https://cheatsheetseries.owasp.org/cheatsheets/Ruby_on_Rails_Cheat_Sheet.html">Rails Security Tips Cheatsheet</a></li> </ul> <h3 id="legal-and-regulatory-requirements">Legal and Regulatory Requirements</h3> <p>Depending on the nature of the app and the targeted industry, compliance with specific legal and regulatory requirements may be necessary. This could include regulations such as HIPAA or FINRA. If your application has not been audited more frequently, there is a chance that some of the rules and regulations for compliance with these standards have changed. This could indicate that the app is not compliant and this needs to be addressed immediately.</p> <h3 id="data-privacy">Data Privacy</h3> <p>If your app handles sensitive data (such as personal information or payment details) and fails to comply with data privacy regulations like GDPR, CCPA or PCI DDS, it could result in non-compliance. This includes inadequate data encryption, improper handling of user consent, or insufficient data access controls. As your application and user base expands, the risk of mishandling user data and sensitive information will increase as well. Periodic assessments and audits can help identify potential vulnerabilities before they escalate.</p> <h3 id="accessibility">Accessibility</h3> <p>Compliance with accessibility standards such as the Web Content Accessibility Guidelines (WCAG) ensures that the app is usable by individuals with disabilities. Signs of accessibility issues include inaccessible forms, missing alternative text for images, and lack of keyboard navigation support. Automated accessibility testing tools and manual testing with assistive technologies can help identify and fix these issues.</p> <h3 id="conclusion">Conclusion</h3> <p>Identifying non-compliance in your Rails app is crucial for maintaining data security, protecting user privacy, and upholding legal and regulatory requirements. By understanding common signs of non-compliance and implementing proactive measures to address these issues, you can ensure that your Rails app remains compliant and trustworthy for users.</p> <p>Need help keeping your Rails applications secure? <a href="https://www.fastruby.io/security-audit">Contact us for a security audit!</a></p>aisayoEnsuring that your company’s website is current with compliance standards is extremely important and essential for any Rails application. Operating with a compliant application guarantees security that can help with handling sensitive data and maintaining users’ trust. The more compliant your website is, the more secure it will be against data breaches, which helps users feel safe when they’re using it. So what does it take to be compliant? In this article, we will focus on security and cover some indicators to help identify if your Rails app might not be compliant anymore.Ruby &amp; Sinatra Compatibility Table2024-03-08T10:14:03-05:002024-03-08T10:14:03-05:00https://fastruby.io/blog/sinatra-and-ruby-compatability-table<p><a href="https://sinatrarb.com/">Sinatra</a> is known in the Ruby world for being a lightweight framework for building Ruby web applications with minimal effort.</p> <p>Over time Sinatra has been through many versions, and sometimes it gets complicated keeping track of which versions of Sinatra are compatible with which versions of Ruby. Therefor, we made a handy chart!</p> <!--more--> <p>If you find yourself or your company struggling on an upgrade for either Sinatra or Ruby let us know, our company specializes in upgrades! <a href="/our-services">We have all different packages, for all different size companies</a>.</p> <table class="ruby-compatibility-table"> <thead> <tr> <td><a href="https://rubygems.org/gems/sinatra" target="_blank">Sinatra Version</a></td> <td><a href="https://www.ruby-lang.org/en/downloads/releases/" target="_blank">Required Ruby Version</a></td> <td>Recommended Ruby Version</td> </tr> </thead> <tbody> <tr> <td>4.0</td> <td>&gt;= 2.7.8</td> <td>3.3.Z</td> </tr> <tr> <td>3.2.0</td> <td>&gt;= 2.6.0</td> <td>3.3.Z</td> </tr> <tr> <td>3.1.0</td> <td>&gt;= 2.6.0</td> <td>3.2.Z</td> </tr> <tr> <td>3.0.X</td> <td>&gt;= 2.6.0</td> <td>3.1.Z</td> </tr> <tr> <td>2.2.X</td> <td>&gt;= 2.3.0</td> <td>3.0.Z</td> </tr> <tr> <td>2.1.0</td> <td>&gt;= 2.3.0</td> <td>2.7.1</td> </tr> <tr> <td>2.0.X</td> <td>&gt;= 2.2.0</td> <td>2.4.1</td> </tr> <tr> <td>1.4.x</td> <td>&gt;= 1.8.7</td> <td>2.0.0</td> </tr> </tbody> </table> <p><br /> To find more information about the most recent Ruby releases check out this page: <a href="https://www.ruby-lang.org/en/downloads/releases/">Ruby Releases</a></p> <h2 id="feedback-wanted-updates">Feedback Wanted: Updates</h2> <p>Note that this table goes to Sinatra 1.4.X, before that we were not able to find information on specific Ruby versions, but let us know if you have find incompatibilities for earlier versions.</p> <p>Feel free to get in contact with us through social media, on <a href="https://ruby.social/@FastRuby">mastadon</a> or <a href="https://twitter.com/fastrubyio">Twitter</a>.</p> <p>We will continue to update this article as new versions of Sinatra are released.</p>fionadlSinatra is known in the Ruby world for being a lightweight framework for building Ruby web applications with minimal effort. Over time Sinatra has been through many versions, and sometimes it gets complicated keeping track of which versions of Sinatra are compatible with which versions of Ruby. Therefor, we made a handy chart!The Evolution of ActiveModel::Error in the Rails Framework2024-03-05T11:23:46-05:002024-03-05T11:23:46-05:00https://fastruby.io/blog/the-evolution-of-activemodel-error-in-rails-framework<p>Rails progression emphasizes simplicity and productivity. Through versions, Rails integrated tools, enhanced performance, and adapted to industry standards, keeping a focus on developer happiness and efficiency. <code class="highlighter-rouge">ActiveModel::Error</code> is an example of that. On this blog post, we’ll dive into the evolution of this object.</p> <!--more--> <p>If you find yourself trying to upgrade your Rails application, it’s very likely that you will have to deal with changes that were made to <code class="highlighter-rouge">ActiveModel::Error</code> over time.</p> <p>There are some key differences between versions 5.0 and 6.0 of Rails on how it deals with validation errors in the model layer.</p> <p>Back in Rails 3.0 <code class="highlighter-rouge">ActiveModel::Error</code> was inheriting from the <code class="highlighter-rouge">ActiveSupport::OrderedHash</code> class, and it responded to <code class="highlighter-rouge">Hash</code> methods. It already included methods like <code class="highlighter-rouge">full_messages</code>, <code class="highlighter-rouge">details</code>, getters and setters.</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Rails 3.0 - activemodel/lib/active_model/errors.rb</span> <span class="k">class</span> <span class="nc">Errors</span> <span class="o">&lt;</span> <span class="no">ActiveSupport</span><span class="o">::</span><span class="no">OrderedHash</span> </code></pre></div></div> <p>This <code class="highlighter-rouge">ActiveSupport::OrderedHash</code> inheritance was changed to an initializer later on Rails 3.1 to instantiate the error’s messages attribute.</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Rails 3.1 - activemodel/lib/active_model/errors.rb</span> <span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">base</span><span class="p">)</span> <span class="vi">@base</span> <span class="o">=</span> <span class="n">base</span> <span class="vi">@messages</span> <span class="o">=</span> <span class="no">ActiveSupport</span><span class="o">::</span><span class="no">OrderedHash</span><span class="p">.</span><span class="nf">new</span> <span class="k">end</span> </code></pre></div></div> <p>Rails 4.0 changed the <code class="highlighter-rouge">messages</code> attribute initialization from <code class="highlighter-rouge">ActiveSupport::OrderedHash</code> to use the <code class="highlighter-rouge">Hash</code> class, changing the behavior of the <code class="highlighter-rouge">to_xml</code>, <code class="highlighter-rouge">as_json</code>, and <code class="highlighter-rouge">to_hash</code> methods. This change was made to adjust to Ruby version 1.9 or higher as Rails 4 had a minimum requirement of Ruby 1.9++ by that time. You can check <a href="https://github.com/rails/rails/pull/4930">the pull request here</a>.</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Rails 4.0 - activemodel/lib/active_model/errors.rb</span> <span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">base</span><span class="p">)</span> <span class="vi">@base</span> <span class="o">=</span> <span class="n">base</span> <span class="vi">@messages</span> <span class="o">=</span> <span class="p">{}</span> <span class="k">end</span> </code></pre></div></div> <p>Lately, in Rails 5.0 they included the <code class="highlighter-rouge">details</code> attributes in the initializer to determine what validator has failed. You can check the <a href="https://github.com/rails/rails/pull/18322">pull request here</a>.</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">base</span><span class="p">)</span> <span class="vi">@base</span> <span class="o">=</span> <span class="n">base</span> <span class="vi">@messages</span> <span class="o">=</span> <span class="p">{}</span> <span class="vi">@details</span> <span class="o">=</span> <span class="no">Hash</span><span class="p">.</span><span class="nf">new</span> <span class="p">{</span> <span class="o">|</span><span class="n">details</span><span class="p">,</span> <span class="n">attribute</span><span class="o">|</span> <span class="n">details</span><span class="p">[</span><span class="n">attribute</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span> <span class="p">}</span> <span class="k">end</span> </code></pre></div></div> <p>Once <a href="https://github.com/rails/rails/commit/6ec8ba16d85d5feaccb993c9756c1edcbbf0ba13">this commit</a> got in the Rails repository, they started deprecating <code class="highlighter-rouge">getters</code>, <code class="highlighter-rouge">setters</code> and <code class="highlighter-rouge">[]=</code> methods because of inconsistent behavior when dealing with then:</p> <blockquote> <p><code class="highlighter-rouge">errors.messages[:key]</code> and <code class="highlighter-rouge">errors.get(:key)</code> can be accessed only by symbol, but <code class="highlighter-rouge">errors["key"]</code> can be access by both string or symbol</p> </blockquote> <blockquote> <p><code class="highlighter-rouge">errors.set(:key, ["error"])</code> is overwriting all errors, but <code class="highlighter-rouge">errors[:key] = "error"</code> (which should do the same thing) pushes error to existing ones.</p> </blockquote> <blockquote> <p><code class="highlighter-rouge">errors.set("key", ["error"])</code> will not convert key to symbol, but <code class="highlighter-rouge">errors["key"] = "error"</code> and <code class="highlighter-rouge">errors.add("key", :invalid)</code> will convert it.</p> </blockquote> <p>You can check the discussion <a href="https://github.com/rails/rails/pull/18631">in this pull request</a>.</p> <p>Rails 5.0 also deprecated the <code class="highlighter-rouge">add_on_empty</code> and <code class="highlighter-rouge">add_on_blank</code> methods in a following <a href="https://github.com/rails/rails/commit/259d33db8cae4f139c4646077f1637a8224dfdb2">commit</a>.</p> <p>Rails 6.0 brings new methods to <code class="highlighter-rouge">ActiveModel::Error</code>:</p> <blockquote> <p><code class="highlighter-rouge">slice</code>, that was already provided to <code class="highlighter-rouge">Hash</code> via <code class="highlighter-rouge">ActiveSupport</code></p> </blockquote> <blockquote> <p><code class="highlighter-rouge">of_kind?</code> that allow us to check presence of a specific error</p> </blockquote> <blockquote> <p>and adds a new configuration option to customize format of the <code class="highlighter-rouge">full_message</code> output</p> </blockquote> <p>Rails 6.1 changes <code class="highlighter-rouge">ActiveModel::Error</code> to improve its object orientation behavior in <a href="https://github.com/rails/rails/pull/32313">this pull request</a>. The changes include a query interface, enable more precise testing, and access to error details.</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">model</span><span class="p">.</span><span class="nf">errors</span><span class="p">.</span><span class="nf">where</span><span class="p">(</span><span class="ss">:name</span><span class="p">,</span> <span class="ss">:foo</span><span class="p">,</span> <span class="ss">bar: </span><span class="mi">3</span><span class="p">).</span><span class="nf">first</span> </code></pre></div></div> <p>Also making it easier to find the message with corresponding details of one particular error:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># model.errors.details</span> <span class="p">{</span><span class="ss">:name</span><span class="o">=&gt;</span><span class="p">[{</span><span class="ss">error: :foo_error</span><span class="p">,</span> <span class="ss">count: </span><span class="mi">1</span><span class="p">},</span> <span class="p">{</span><span class="ss">error: :bar_error</span><span class="p">},</span> <span class="p">{</span><span class="ss">error: :foo_error</span><span class="p">,</span> <span class="ss">count: </span><span class="mi">3</span><span class="p">}]}</span> </code></pre></div></div> <p>This change also opened the possibility of advancing modding, as errors are now objects so we can add functionality on top of them. For example, we can create custom methods to disable global attribute prefixes on an error’s full messages.</p> <p>Some methods like <code class="highlighter-rouge">as_json</code>, <code class="highlighter-rouge">add</code>, and <code class="highlighter-rouge">include?</code> remain unchanged after this Rails 6.1 change. Other methods were deprecated (<code class="highlighter-rouge">full_message</code>, <code class="highlighter-rouge">generate_message</code>, <code class="highlighter-rouge">has_key</code>) and new methods were added to this class: <code class="highlighter-rouge">messages_for</code>, <code class="highlighter-rouge">where</code>, and <code class="highlighter-rouge">import</code>.</p> <p>The change tries its best at maintaining backward compatibility, however some edge cases won’t be covered. For example, <code class="highlighter-rouge">errors#first</code> will return <code class="highlighter-rouge">ActiveModel::Error</code> and manipulating <code class="highlighter-rouge">errors.messages</code> and <code class="highlighter-rouge">errors.details</code> hashes directly will have no effect.</p> <p>Rails 7.0 removes all previous deprecated methods from <code class="highlighter-rouge">ActiveModel::Error</code> from the code.</p> <h3 id="conclusion">Conclusion</h3> <p>In conclusion, the <code class="highlighter-rouge">ActiveModel::Error</code> emerges as an essential component within the Rails environment, playing a crucial role in the robustness and adaptability of modern Rails applications. Its ability to seamlessly handle and manage validation errors not only enhances the user experience by providing clear and concise error messages but also contributes significantly to the overall maintainability of the code. By encapsulating error details within this specialized object, developers can streamline error handling processes, ensuring a more efficient and organized approach to troubleshooting and debugging.</p> <p><code class="highlighter-rouge">ActiveModel::Error</code> stands as a testament to Rails’ commitment to simplicity, convention over configuration, and the creation of applications that are not only powerful but also resilient and developer-friendly.</p> <p>Running an end-of-life Rails version in production and in need of an upgrade? <a href="https://www.fastruby.io/#contactus">Send us a message, we can help!</a></p>hmdrosRails progression emphasizes simplicity and productivity. Through versions, Rails integrated tools, enhanced performance, and adapted to industry standards, keeping a focus on developer happiness and efficiency. ActiveModel::Error is an example of that. On this blog post, we’ll dive into the evolution of this object.Dual-Boot Ruby2024-02-27T09:14:13-05:002024-02-27T09:14:13-05:00https://fastruby.io/blog/dual-boot-ruby<p>As we mentioned many times, at FastRuby.io we like to use the <a href="/blog/tags/dual-boot">Dual-Boot technique</a> during upgrades to quickly test the same code with the current and the next version of what we are upgrading. We usually talk about dual-booting Rails versions but this can be used to upgrade Ruby itself too. We have to make some changes to adapt the technique, and we’ll explain the basic changes in this article.</p> <!--more--> <h2 id="dual-boot-gems-vs-dual-boot-ruby">Dual-Boot Gems vs Dual-Boot Ruby</h2> <p>When we talk about the Dual-Boot technique, the main concept is to have 2 different <code class="highlighter-rouge">Gemfile.lock</code> files, one with the dependencies for the current version of the app, and one with the dependencies for the next version.</p> <p>We typically include a snippet like this at the top of the <code class="highlighter-rouge">Gemfile</code>:</p> <div class="language-rb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">next?</span> <span class="no">File</span><span class="p">.</span><span class="nf">basename</span><span class="p">(</span><span class="kp">__FILE__</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"Gemfile.next"</span> <span class="k">end</span> </code></pre></div></div> <p>And we use the <code class="highlighter-rouge">next?</code> helper method to set different versions of gems like this:</p> <div class="language-rb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="k">next</span><span class="p">?</span> <span class="n">gem</span> <span class="s2">"rails"</span><span class="p">,</span> <span class="ss">github: </span><span class="s2">"rails/rails"</span> <span class="k">else</span> <span class="n">gem</span> <span class="s2">"rails"</span><span class="p">,</span> <span class="s2">"~&gt; 7.1.0"</span> <span class="k">end</span> </code></pre></div></div> <p>Then, the application can be run with different commands to run one version or the other: <code class="highlighter-rouge">rails s</code> will run Rails 7.1, <code class="highlighter-rouge">BUNDLE_GEMFILE=Gemfile.next rails s</code> will run Rails’ main branch.</p> <p>In order to dual-boot Ruby, we can use the same technique but with a few considerations.</p> <h2 id="considerations">Considerations</h2> <p>Versions of the gems are generally only defined in the <code class="highlighter-rouge">Gemfile</code> files, but the Ruby version is defined in many places. The most common are:</p> <ul> <li>.ruby-version file (used by some version managers)</li> <li>.tool-versions file (used by some version managers)</li> <li>Gemfile (used by bundler, RVM, Heroku, and other tools)</li> <li>Gemfile.lock (the Ruby version in this file is mostly informative)</li> <li>Dockerfile (used by Docker to prepare the container)</li> <li>CI setup steps</li> </ul> <p>And applications can also have the expected Ruby version defined in other not-so-common places.</p> <p>We have to make sure that a dual-boot setup is compatible with all of them to avoid a negative impact on the developer experience.</p> <p>It’s important to note that, when dual-booting a gem, we only need to run the same command with different <code class="highlighter-rouge">BUNDLE_GEMFILE</code> env variables to quickly test the different version. However, when dual-booting Ruby, we need to first switch the version that is currently active using our Ruby version manager before running the app with the desired version.</p> <h3 id="gemfile-files">Gemfile files</h3> <p>It is really common to specify the required Ruby version in the <code class="highlighter-rouge">Gemfile</code> like this:</p> <div class="language-rb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ruby</span> <span class="s2">"3.2.2"</span> </code></pre></div></div> <p>This is used by Bundler to ensure that we are using the expected Ruby version when running the app, but it’s also used by other tools like RVM or Heroku to pick which version of Ruby to set up.</p> <p>To support the dual-boot, we can change that line to this:</p> <div class="language-rb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ruby_version</span> <span class="o">=</span> <span class="k">next</span><span class="p">?</span> <span class="p">?</span> <span class="s2">"3.3.0"</span> <span class="p">:</span> <span class="s2">"3.2.2"</span> <span class="n">ruby</span> <span class="n">ruby_version</span> </code></pre></div></div> <p>With this code, Heroku will install Ruby 3.2.2 when building the application, and at the same time it allows us to run the app with either Ruby 3.3.0 or 3.2.2 without failing.</p> <blockquote> <p>Note that the version displayed in the <code class="highlighter-rouge">Gemfile.lock</code> file is not a problem for us: the <code class="highlighter-rouge">Gemfile.lock</code> file will show the current Ruby version, and the <code class="highlighter-rouge">Gemfile.next.lock</code> file will show the next. Those files are not meant to be used with the other Ruby.</p> </blockquote> <h3 id="version-files">Version files</h3> <p>Different Ruby version managers handle the <code class="highlighter-rouge">*version</code> files in different ways. Managers like <code class="highlighter-rouge">asdf</code> or <code class="highlighter-rouge">rbenv</code> will update their version files when switching rubies, so there’s nothing to change for these files to make them compatible with a dual-boot setup. It’s important to not commit changes in these files when switching between Ruby versions though. We want these files in the repository to always show the current Ruby version and not the next one.</p> <h3 id="docker-configuration">Docker configuration</h3> <p>If the setup includes using Docker to run the application, we have a complete article on <a href="todo:add-link-when-published">how to dual-boot both Ruby and Rails using docker</a>.</p> <h3 id="ci-setup">CI setup</h3> <p>Finally, we need to update the CI files to run the different jobs with both versions of Ruby. Most CI solutions support a <code class="highlighter-rouge">matrix</code> feature to define multiple values of a given tool and they will automatically execute one job for each of those values.</p> <p>Here’s an example using a matrix to setup <a href="https://github.com/fastruby/next_rails/blob/main/.github/workflows/main.yml#L16">multiple Ruby versions in GitHub Actions</a>.</p> <p>This is not enough though, since we also need to specify the <code class="highlighter-rouge">BUNDLE_GEMFILE</code> variable.</p> <p>To do that we have multiple options:</p> <p><strong>Duplicate jobs</strong></p> <p>One option is to duplicate the original job and change the values to set up Ruby and the environment variables. You can re-use the original information using <a href="https://www.linode.com/docs/guides/yaml-anchors-aliases-overrides-extensions/">YAML anchors, aliases, and overrides</a> if the CI service supports them.</p> <p><strong>Jobs Matrix</strong> Another option is to use the matrix feature with extra configuration for each Ruby version if supported.</p> <p>GitHub Actions, for example, supports matrix configurations using the <code class="highlighter-rouge">include</code> property to set other variables for a given matrix value, and the parameters will include the extra keys:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">strategy</span><span class="pi">:</span> <span class="na">matrix</span><span class="pi">:</span> <span class="na">ruby</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">3.2.2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">3.3.0"</span><span class="pi">]</span> <span class="na">include</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">gemfile</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gemfile.lock"</span> <span class="na">ruby</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3.2.2"</span> <span class="pi">-</span> <span class="na">gemfile</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gemfile.next.lock"</span> <span class="na">ruby</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3.3.0"</span> </code></pre></div></div> <p>Check the documentation on <a href="https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#expanding-or-adding-matrix-configurations">Expanding or adding matrix configurations</a> for more information.</p> <p>CircleCI supports a different pattern by defining all the possible values and excluding the combinations we don’t want to run:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">workflows</span><span class="pi">:</span> <span class="na">workflow</span><span class="pi">:</span> <span class="na">jobs</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">build</span><span class="pi">:</span> <span class="na">matrix</span><span class="pi">:</span> <span class="na">parameters</span><span class="pi">:</span> <span class="na">ruby</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">3.2.2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">3.3.0"</span><span class="pi">]</span> <span class="na">gemfile</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">Gemfile.lock"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">Gemfile.next.lock"</span><span class="pi">]</span> <span class="na">exclude</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">ruby</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3.2.2"</span> <span class="na">gemfile</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gemfile.next.lock"</span> <span class="pi">-</span> <span class="na">ruby</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3.3.0"</span> <span class="na">gemfile</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gemfile.lock"</span> </code></pre></div></div> <p>Check the documentation on <a href="https://circleci.com/docs/configuration-reference/#excluding-sets-of-parameters-from-a-matrix">Excluding sets of parameters from a matrix</a> for more information.</p> <p>TravisCI offers a different pattern to define the different configurations explicitly instead of using implicit combinations:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">jobs</span><span class="pi">:</span> <span class="na">include</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">rvm</span><span class="pi">:</span> <span class="s">3.2.2</span> <span class="na">gemfile</span><span class="pi">:</span> <span class="s">Gemfile</span> <span class="pi">-</span> <span class="na">rvm</span><span class="pi">:</span> <span class="s">3.3.0</span> <span class="na">gemfile</span><span class="pi">:</span> <span class="s">Gemfile.next</span> </code></pre></div></div> <p>Check the documentation on <a href="https://docs.travis-ci.com/user/build-matrix/#listing-individual-jobs">Listing individual jobs</a> for more information.</p> <h2 id="conclusion">Conclusion</h2> <p>Dual-booting Ruby is similar to any other dependency, but there are some important differences that we need to address to do it right. There are more ways to do this but we try to minimize the disruption to the normal development workflow as much as possible. With these ideas you can default to use the current Ruby version while also being able to use the next version on demand with a few commands.</p> <p>It’s important to note that, in some cases, dual-booting Ruby is not needed when the upgrade is simple enough. An initial test using the new Ruby version can be done without adding the complexity of the dual-boot to make an informed decision.</p> <p>Running behind and need to upgrade to Ruby 3.3.0? <a href="/#contact-us">Let us help you!</a></p>arieljuodAs we mentioned many times, at FastRuby.io we like to use the Dual-Boot technique during upgrades to quickly test the same code with the current and the next version of what we are upgrading. We usually talk about dual-booting Rails versions but this can be used to upgrade Ruby itself too. We have to make some changes to adapt the technique, and we’ll explain the basic changes in this article.The Rails Developer’s Reference to PostgreSQL indexes2024-02-21T07:16:36-05:002024-02-21T07:16:36-05:00https://fastruby.io/blog/the-rails-developers-reference-to-postgresql-indexes<p>In a <a href="/blog/common-problems-in-rails-performance">previous article</a>, we listed down common culprits that led to a sub-optimal performance in Rails applications. One of the culprits was missing or incorrect indexes.</p> <p>Therefore we thought it would be very useful to have a handy reference to the different kinds of indexes, when you should use them and maybe even when <em>not</em> to use them.</p> <!--more--> <ul> <li><a href="#different-kinds-of-indexes">Different kinds of indexes</a> <ul> <li><a href="#b-tree-indexes">B-Tree Indexes</a></li> <li><a href="#hash-indexes">Hash Indexes</a></li> <li><a href="#gist-and-sp-gist-indexes">GiST and SP-GiST Indexes</a></li> <li><a href="#gin-indexes">GIN Indexes</a></li> <li><a href="#brin-indexes">BRIN Indexes</a></li> </ul> </li> <li><a href="#conclusion">Conclusion</a></li> </ul> <h2 id="different-kinds-of-indexes">Different kinds of indexes</h2> <h3 id="b-tree-indexes">B-Tree Indexes</h3> <p>The Binary Tree index, or B-Tree index, is the default index type we get from a <code class="highlighter-rouge">CREATE INDEX</code> statement.</p> <p>It’s probably also the most common index type we’ll use as developers. The use case is straightforward. You should use a B-Tree index when:</p> <ol> <li>Your queries retrieve small datasets relative to the table size.</li> <li>The column you want to index has high cardinality, which is to say, their data isn’t replicated too many times in different rows.</li> </ol> <blockquote> <p>For example, if you have a products table that has a category column, it’s natural that many products will have the same value for their category. Therefore, category has <em>low cardinality</em>. Inversely, if the data is unique for every row (say, the product id), that column has <em>high cardinality</em></p> </blockquote> <p>The unique id column in tables are usually the more obvious candidates for this index.</p> <p>Generally speaking, PostgreSQL will use this index for most of the commonly used comparison operators (<code class="highlighter-rouge">&gt;</code>,<code class="highlighter-rouge">&gt;=</code>, <code class="highlighter-rouge">&lt;</code>, <code class="highlighter-rouge">&lt;=</code>, <code class="highlighter-rouge">=</code>, <code class="highlighter-rouge">BETWEEN</code> and <code class="highlighter-rouge">IN</code>) as well as the null checking operators (<code class="highlighter-rouge">IS NULL</code> and <code class="highlighter-rouge">IS NOT NULL</code>). It can even use B-Tree indexes for pattern matching operators, <a href="https://www.postgresql.org/docs/16/indexes-types.html#INDEXES-TYPES-BTREE">under certain conditions</a>.</p> <p>There are a few ways to add an index in an ActiveRecord migration:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># In a create_table block</span> <span class="n">t</span><span class="p">.</span><span class="nf">index</span><span class="p">(</span><span class="ss">:column_name</span><span class="p">)</span> <span class="c1"># or outside of the block</span> <span class="n">add_index</span><span class="p">(</span><span class="ss">:table_name</span><span class="p">,</span> <span class="ss">:column_name</span><span class="p">)</span> </code></pre></div></div> <p>See the docs for <a href="https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/SchemaStatements.html#method-i-add_index"><code class="highlighter-rouge">add_index</code></a> for all the possible options.</p> <p>Finally, as a SQL statement:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">column_name_idx</span> <span class="k">ON</span> <span class="k">table_name</span><span class="p">(</span><span class="k">column_name</span><span class="p">);</span> </code></pre></div></div> <h3 id="hash-indexes">Hash Indexes</h3> <p>For practical purposes, a hash index can be considered to have a specialized use case for columns that could also be indexed with a B-Tree but that we know for a fact it will always be queried using the <code class="highlighter-rouge">=</code> operator.</p> <p>The way hash indexes work is that they create a 32-bit hash for the value in the indexed column. Which brings us to the first drawback for this type of index: they only work for equality comparisons. Another drawback to be considered is that they have a maintenance overhead during data modifications since the database must solve hash collisions by rehashing the data.</p> <p>Even with all these drawbacks, they are faster for equality comparisons than B-Tree indexes.</p> <p>The query planner will consider using hash indexes whenever the indexed column is involved in a comparison with the equal operator.</p> <p>It’s worth noting that it is <em>very rare</em> that using a hash index will ever be needed. However, if your use case fits into its requirements and you need to get more performance than what you have with a B-Tree index, it’s worth testing some queries using a hash index and see if there is some real benefit.</p> <p>To create a Hash index using ruby and ActiveRecord, pass the <code class="highlighter-rouge">using: 'hash'</code> option to <code class="highlighter-rouge">index</code> or <code class="highlighter-rouge">add_index</code>:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># In a create_table block</span> <span class="n">t</span><span class="p">.</span><span class="nf">index</span><span class="p">(</span><span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'hash'</span><span class="p">)</span> <span class="c1"># or outside of the block</span> <span class="n">add_index</span><span class="p">(</span><span class="ss">:table_name</span><span class="p">,</span> <span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'hash'</span><span class="p">)</span> </code></pre></div></div> <p>And as a SQL statement:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">column_name_idx</span> <span class="k">ON</span> <span class="k">table_name</span><span class="p">(</span><span class="k">column_name</span><span class="p">)</span> <span class="k">USING</span> <span class="n">HASH</span><span class="p">;</span> </code></pre></div></div> <h3 id="gist-and-sp-gist-indexes">GiST and SP-GiST Indexes</h3> <p>GiST (Generalized Search Tree) and SP-GiST (Space Partitioned) indexes aren’t a specific kind of index but rather an infrastructure that allows database users to define indexing for complex data types.</p> <p>Out of the box, PostgreSQL comes with implementations for several geometric shapes and text search. These data classes that implement the required GiST functions are called <em>operator classes</em></p> <p>What this means, practically speaking, is that such indexes have a very diverse yet <em>specialized</em> set of applications and use cases, ranging from finding some point within a given distance from some coordinate to full text search.</p> <p>Usage of these indexes is application dependent so it’s best to read PostgreSQL’s own documentation on the matter and determine if your use case might fit: <a href="https://www.postgresql.org/docs/16/gist.html">GiST chapter</a> and <a href="https://www.postgresql.org/docs/16/spgist.html">SP-GiST chapter</a></p> <p>To create a GiST or SP-GiST index in Rails:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># In a create_table block</span> <span class="n">t</span><span class="p">.</span><span class="nf">index</span><span class="p">(</span><span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'gist'</span><span class="p">)</span> <span class="n">t</span><span class="p">.</span><span class="nf">index</span><span class="p">(</span><span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'spgist'</span><span class="p">)</span> <span class="c1"># or outside of the block</span> <span class="n">add_index</span><span class="p">(</span><span class="ss">:table_name</span><span class="p">,</span> <span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'gist'</span><span class="p">)</span> <span class="n">add_index</span><span class="p">(</span><span class="ss">:table_name</span><span class="p">,</span> <span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'spgist'</span><span class="p">)</span> </code></pre></div></div> <p>And with SQL:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">column_name_idx</span> <span class="k">ON</span> <span class="k">table_name</span><span class="p">(</span><span class="k">column_name</span><span class="p">)</span> <span class="k">USING</span> <span class="n">GIST</span><span class="p">;</span> <span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">column_name_idx</span> <span class="k">ON</span> <span class="k">table_name</span><span class="p">(</span><span class="k">column_name</span><span class="p">)</span> <span class="k">USING</span> <span class="n">SPGIST</span><span class="p">;</span> </code></pre></div></div> <h3 id="gin-indexes">GIN Indexes</h3> <p>GIN stands for Generalized Inverted Index. These indexes are appropriate for data that contain multiple component values, such as arrays. An inverted index contains a separate entry for each component value and efficiently handles queries that test for these component values.</p> <p>Hence the “Inverted” in the name. A normal B-Tree index will have one place to represent the data for one row, whereas a GIN index can have the same row referenced in multiple places. It’s analogous to the table of contents at the back of many books: the same term (or component value of a row) is referenced in multiple different pages.</p> <p>Like GiST and SP-GiST indexes, the exact way how a GIN index maps a column of a given data type, depends on the GIN operator class.</p> <p>It’s important to note that GIN indexes have one big downside: They’re expensive to update. Because it can reference the same row multiple times, any changes to that row means a change to all indexes referencing that row.</p> <p>To create a GIN index in Rails:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># In a create_table block</span> <span class="n">t</span><span class="p">.</span><span class="nf">index</span><span class="p">(</span><span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'gin'</span><span class="p">)</span> <span class="c1"># or outside of the block</span> <span class="n">add_index</span><span class="p">(</span><span class="ss">:table_name</span><span class="p">,</span> <span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'gin'</span><span class="p">)</span> </code></pre></div></div> <p>And with SQL:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">column_name_idx</span> <span class="k">ON</span> <span class="k">table_name</span><span class="p">(</span><span class="k">column_name</span><span class="p">)</span> <span class="k">USING</span> <span class="n">GIN</span><span class="p">;</span> </code></pre></div></div> <h3 id="brin-indexes">BRIN Indexes</h3> <p>BRIN stands for Block Range INdexes.</p> <p>BRIN indexes will divide your data into pages, according to their storage order in memory, and keep track of the beginning and ending values of the indexed columns in that page. They’ll do this for the entire table.</p> <blockquote> <p>As an example, if we index a <code class="highlighter-rouge">created_at</code> column, the index will store the value for <code class="highlighter-rouge">created_at</code> of the first row in a given page and the value for the same column for the last row of the same page, and so forth for all pages into which the table will be divided.</p> </blockquote> <p>Given this property, this index is only recommended when there is a strong correlation between the data stored and it’s physical order in memory. Building on the previous example, <code class="highlighter-rouge">created_at</code> columns are usually good candidates because rows added yesterday or earlier in the day come before the current row in memory, and subsequent insertions will also be inserted in memory closely following the value of <code class="highlighter-rouge">created_at</code>, even though there isn’t necessarily a logical rule establishing this behavior.</p> <p>Also, since it breaks the table into pages, it’s important to know how much memory each row of your table occupies. See [this blog post by Janet Carson] on ways to do this. The reason this is important is that if your table is too small and the index stores too much data into a single page, the index won’t render the speed benefits we want for queries. Once you’ve estimated your row size, knowing a page of memory has 8192 bytes, you can estimate how big your block range should be and set the <code class="highlighter-rouge">pages_per_range</code> storage parameter accordingly.</p> <p>Also, <em>BRIN indexes do not auto update by default</em>. Either the database administrator must explicitly call index maintenance functions or the index must be created with <code class="highlighter-rouge">autosummarize = 1</code>.</p> <p>Finally, in order to know if your column’s logical ordering has a good direct correlation to the storage order in memory, the <code class="highlighter-rouge">pg_stats</code> table can give you this information via the <code class="highlighter-rouge">correlation</code> column. The closer to 1, the better the fit for a BRIN index.</p> <p>To create this index in Rails:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># In a create_table block</span> <span class="n">t</span><span class="p">.</span><span class="nf">index</span><span class="p">(</span><span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'brin'</span><span class="p">)</span> <span class="c1"># or outside of the block</span> <span class="n">add_index</span><span class="p">(</span><span class="ss">:table_name</span><span class="p">,</span> <span class="ss">:column_name</span><span class="p">,</span> <span class="ss">using: </span><span class="s1">'brin'</span><span class="p">)</span> </code></pre></div></div> <p>I was unable to find a way to call these methods in a way that I could pass the <code class="highlighter-rouge">autosummarize</code> storage parameter. If anyone know, I’d be happy to hear and add it here. In the meantime, we can always execute raw SQL in a migration:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Anywhere inside the change method</span> <span class="n">execute</span><span class="o">&lt;&lt;-</span><span class="no">SQL</span><span class="sh"> CREATE INDEX developer_created_at_idx ON developers USING brin (created_at) WITH (autosummarize=1); </span><span class="no">SQL</span> </code></pre></div></div> <h3 id="conclusion">Conclusion</h3> <p>Hopefully the discussion above can help any Rails developers out there understand the different indexes available in PostgreSQL and how to properly pick the one that will help them tune their application and make it run as smooth as possible.</p> <p>In case you still need help optimizing your queries, understanding where to add indexes, what type of indexes to use or with other performance related issues, make sure to <a href="/tune">send us a message</a>. We can help!</p>mateuspereiraIn a previous article, we listed down common culprits that led to a sub-optimal performance in Rails applications. One of the culprits was missing or incorrect indexes. Therefore we thought it would be very useful to have a handy reference to the different kinds of indexes, when you should use them and maybe even when not to use them.Ruby &amp; Roda Compatibility Table2024-02-20T10:08:29-05:002024-02-20T10:08:29-05:00https://fastruby.io/blog/ruby-roda-compatibility-table<p><a href="https://roda.jeremyevans.net/">Roda</a> is a web toolkit (or framework) that focusses on simplicity, reliability, extensibility, and performance. This is a short post to show the compatibility between Roda and <a href="https://www.ruby-lang.org/en/">Ruby</a> across different versions.</p> <!--more--> <table class="ruby-compatibility-table"> <thead> <tr> <td><a href="https://rubygems.org/gems/roda/versions" target="_blank">Roda Version</a></td> <td><a href="https://www.ruby-lang.org/en/downloads/releases/" target="_blank">Required Ruby Version</a></td> <td>Recommended Ruby Version</td> </tr> </thead> <tbody> <tr> <td>3.X.Z</td> <td>&gt;= 1.9.2</td> <td>2.4.1 - 3.3.Z</td> </tr> <tr> <td>2.X.Z (&gt;= 2.10.0)</td> <td>&gt;= 1.8.7</td> <td>2.2.3 - 2.4.1</td> </tr> <tr> <td>&gt;= 0.9.0 (&lt;= 2.9.0)</td> <td>&gt;= 0</td> <td>2.1.2 - 2.2.3</td> </tr> </tbody> </table> <p><br /></p> <p>To find more information about the most recent Ruby releases check out this page: <a href="https://www.ruby-lang.org/en/downloads/releases/">Ruby Releases</a></p> <h2 id="need-to-upgrade-ruby-or-roda">Need to Upgrade Ruby or Roda?</h2> <p>Roda does not have upgrade guides per se, but the repo does provide an extensive <a href="https://github.com/jeremyevans/roda/blob/master/CHANGELOG">changelog</a> that will guide you from one Roda version to the next.</p> <p>If you don’t have the time to do it yourself you can hire our team to do it for you, send us a message over here: <a href="/#contactus">Contact FastRuby.io | Ruby and Rails Upgrade Service</a></p> <h2 id="feedback-wanted-updates">Feedback Wanted: Updates</h2> <p>If you find that this article has fallen out of date, feel free to make a comment for us to bring it up to speed. We will continue to update this article as new versions of Ruby and Roda are released.</p>fbuysRoda is a web toolkit (or framework) that focusses on simplicity, reliability, extensibility, and performance. This is a short post to show the compatibility between Roda and Ruby across different versions.Cracking the Case on Flaky Tests: Tips for Build Confidence and Seamless Upgrades2024-02-09T07:00:43-05:002024-02-09T07:00:43-05:00https://fastruby.io/blog/cracking-the-code-on-flaky-specs<p>How many times have you or someone on your team brushed off a failing build with a casual, ‘It’s fine, it’s just a flaky spec—ignore it’?</p> <p>If you’re nodding in agreement, you’re not alone. It’s a scenario familiar to many of us, especially when dealing with sprawling monolithic projects and untouched code sections.</p> <!--more--> <p>However, this attitude towards flaky tests can take a serious turn during upgrades. Upgrades rely on the capabilities of tests; if we can’t trust them, then uncertainty lingers.</p> <p>Are they genuinely flaky, or have we unintentionally introduced issues during the upgrade? Has upgrading somehow made the tests flakier? We have run into both of these issues on past upgrades.</p> <p>In this article we’ll give some tips on how to untangle the mystery behind flaky specs, with the aim of guiding you towards consistently passing builds. Ensuring your tests don’t have flakiness can help every upgrade become more likely to succeed.</p> <h2 id="the-unpredictable-nature-of-flaky-tests">The Unpredictable Nature of Flaky Tests</h2> <p>While flaky tests are often associated with integration tests using technologies like JS, Capybara-Webkit, or Selenium, other reasons are also possible. Oftentimes tests pass locally without issue, but fail CI over and over.</p> <h3 id="common-causes-of-flaky-tests">Common Causes of Flaky Tests:</h3> <p>In our experience of 100+ upgrade projects, we’ve come across flaky specs in all different types of projects. This section describes some of the most common causes of flaky specs.</p> <h3 id="race-conditions">Race Conditions</h3> <p>Flaky tests may arise from race conditions where the timing of execution influences the test outcome.</p> <h3 id="leaked-state">Leaked State</h3> <p>State leakage between tests can lead to unpredictable failures. Shared state should be prevented to minimize the impact of leaked state on the reliability of your test suite.</p> <h3 id="networkthird-party-dependency">Network/Third-Party Dependency</h3> <p>External dependencies, such as network calls or third-party services, introduce an element of unpredictability. <a href="/blog/test-doubles-testing-at-the-boundaries-of-your-ruby-application.html">Mocking and stubbing</a> should always be used to create a controlled test environment.</p> <h3 id="randomness">Randomness</h3> <p>Ironically, randomness itself can be a cause of flaky tests. It is usually a more steady option to choose specific values when testing instead of grabbing random ones, another possibility to control the randomness. For example if you want values for the age of a person, you could say random between 0 and 99.</p> <h3 id="fixed-time-dependency">Fixed Time Dependency:</h3> <p>Tests relying on fixed time values might be sensitive to time of day or system variations.</p> <h2 id="analyzing-flaky-tests-asking-the-right-questions">Analyzing Flaky Tests: Asking the Right Questions</h2> <p>Unraveling the mystery behind flaky tests often involves asking the right questions to pinpoint potential causes. By examining patterns and considering different aspects of your test environment, you can narrow down the root of the issue.</p> <p>Here are some key questions to get you started on your investigation:</p> <h3 id="timing-patterns">Timing Patterns:</h3> <p>Is there any pattern to tests failing at specific times of day?</p> <p>This could indicate a fixed time dependency issue, where the outcome of the test is influenced by the time it runs. Identifying such patterns helps uncover time-related vulnerabilities in your test suite.</p> <p>One way to check this would be to write a script that would run the tests every few hours and log the output. This could help in identifying if there is a time pattern to the flaky tests. In our <a href="/blog/rspec/debug/how-to-debug-non-deterministic-specs.html">post about debugging non-deterministic specs</a> we go into more detail on this topic.</p> <h3 id="test-order-sensitivity">Test Order Sensitivity:</h3> <p>Do certain tests consistently fail when executed in a specific order? Test order sensitivity may reveal issues with dependencies between tests. Investigating this can provide insights into shared state problems or race conditions affecting the stability of your test suite.</p> <p>A good way to investigate this is the run the tests using a specific seed number. You can grab the seed number after running the tests once, and then use it to run them in the same order again.</p> <p>This can also be really useful if something is failing in CI to see if it’s also failing locally.</p> <p>For minitest: <code class="highlighter-rouge">rails test TESTOPTS="--seed 1234"</code></p> <p>or</p> <p><code class="highlighter-rouge">rails test --seed 1234</code></p> <p>For rspec: <code class="highlighter-rouge">rspec --seed 1234</code></p> <h3 id="external-dependencies-and-network-connections">External Dependencies and Network Connections:</h3> <p>Are your tests interacting with external services or APIs?</p> <p>Do flaky tests coincide with periods of network instability?</p> <p>External dependencies often introduce variability in test outcomes. Understanding the impact of external factors on your tests is crucial for creating a more controlled and reliable testing environment. It’s <a href="/blog/testing/javascript/mocking-js-requests.html">important to use mocking and stubbing</a> instead of true API calls.</p> <h3 id="randomness-and-seeds">Randomness and Seeds:</h3> <p>Have you ensured proper seeding for tests involving randomness?</p> <p>Flaky behavior can arise if randomness is not adequately controlled. Confirming the use of seeds for random processes ensures test reproducibility and minimizes unexpected variations.</p> <h3 id="code-changes">Code Changes:</h3> <p>Have recent code changes introduced new dependencies or altered test behavior?</p> <p>Regularly reviewing code changes, especially those in proximity to failing tests, helps identify potential causes. Try to figure out when the test started failing, look back through build logs to see if something specific introduced the flakiness.</p> <p>By systematically addressing these questions, you can gain valuable insights into the nature of flaky tests and take targeted actions to enhance the stability of your test suite.</p> <p>Each question can serve as an investigative tool, bringing you one step closer to consistently green builds which will help make upgrades easier.</p> <h2 id="some-more-tips-for-keeping-everything-up-to-date">Some More Tips for Keeping Everything up to Date.</h2> <p>Asking the right questions to investigate your code base is not enough on it’s own. Team collaboration, monitoring, and staying on top of best practices all play a crucial roll in keeping the flaky specs at bay.</p> <h3 id="test-maintenance-strategies">Test Maintenance Strategies:</h3> <p>Discuss the importance of ongoing test maintenance to prevent flakiness over time. Encourage your team to regularly review and update tests, especially after significant code changes or upgrades.</p> <h3 id="monitoring-and-alerting">Monitoring and Alerting:</h3> <p>Suggest implementing monitoring and alerting systems that notify teams when a test becomes flaky. Early detection allows for timely investigation and resolution, minimizing the impact on build confidence.</p> <h3 id="documentation-and-best-practices">Documentation and Best Practices:</h3> <p>Emphasize the significance of well-documented tests. Clear documentation can help developers understand the purpose of each test, making it easier to identify potential issues and troubleshoot failures.</p> <p>Don’t make tests DRY simply for the sake of making them DRY. Think about decisions around drying up tests, and if it will make it more difficult for your team to figure out where problems are arising in the test suite.</p> <h3 id="collaboration-and-communication">Collaboration and Communication:</h3> <p>Highlight the importance of collaboration between developers and QA teams. Encourage open communication channels to promptly address flaky tests, share insights, and collectively work towards maintaining a reliable test suite.</p> <h2 id="conclusion">Conclusion</h2> <p>In the world of software development, the reliability of your test suite is so important. Flaky tests, though common, should not be dismissed.</p> <p>By implementing the strategies outlined in this article—addressing common causes, analyzing test patterns, and fostering a proactive testing culture—you can pave the way to consistently green builds and seamless upgrades.</p> <p>The journey to build confidence is an ongoing process that requires collaboration, and a commitment to the quality of your codebase.</p> <p>Take these tips, integrate them into your development workflow, and let your test suite become a rock-solid foundation for your software projects. Happy testing!</p> <p>If your team is planning to do an upgrade soon tests are so important for success. We can help you understand how to get ready for an upgrade with our <a href="/roadmap">Roadmap to Upgrade Rails</a>.</p>fionadlHow many times have you or someone on your team brushed off a failing build with a casual, ‘It’s fine, it’s just a flaky spec—ignore it’? If you’re nodding in agreement, you’re not alone. It’s a scenario familiar to many of us, especially when dealing with sprawling monolithic projects and untouched code sections.