Reporting from ephemeral containers in production

Sebastian Armano July 17, 2019

Reporting is not a particular task of business analysts anymore. Anyone at any department, at any point in time, needs a quick report based on the latest data to validate their decisions. Having the ability to create small custom reports in a couple of minutes, and in a format that can be consumed by any analysis software is a powerful resource to have in our belt. The goal is to run this kind of reports without having too much overhead that would kill our productivity, and without having to learn a new set of libraries or tools.

Platform as a service (PAAS) environments like Aptible or Heroku only give you an ephemeral container when accessing the console, which means that changes to the code or the directory structure are not be persisted. That means we need a way of extracting anything useful from that environment before we close the session or we have to start from scratch when opening another one.

At Epion Health we often face this challenge, and I decided to create a repeatable and straightforward way of achieving this.

The task

We can divide the tasks into three main phases:

  1. Processing of required data based on the report specifications
  2. Creation of the report in a format easily readable for any analysis tool
  3. Submission of the report to stakeholders

We can combine some of the features that Rails, Vim and Tmux bring to the table to swiftly complete this task.

Data processing

In our check-in application we want to run a report on how long check-ins take. The CheckIn model has start and end timestamps, and belongs to a practice. We calculate the duration in a method in the CheckIn model by subtracting those timestamps.

1
2
3
4
5
6
7
8
9

start_datetime = Date.new(2019, 05, 20).beginning_of_day
end_datetime = Date.new(2019, 05, 25).end_of_day

CheckIn.
  for_practice(1).
  where(created_at: start_datetime..end_datetime).each do |ci|
    check_in_id = ci.id
    check_in_date = ci.created_at
    check_in_duration = ci.timespan

Report Creation

CSV or comma-separated values is a format that is simple to use, flexible and supported by almost any tool around (from Business Intelligence tools like Tableau to less specific tools like MS Excel or Apple Pages). Ruby also has support for the this kind of files with its standard library class CSV.

For our report, we are taking advantage only of the CSV creation feature of the class. We open a file in wb mode (which recreates it every time we run the script), and we add contents inside the block we pass. CSV takes care of closing the file after we are done (that’s the reason behind passing a block to it in the first place).

The object passed to the block contains an array of arrays, of two levels. Ruby converts each top-level array to a row in the CSV file and each array inside it to the list of elements, separated by commas. We want headers, so we add the array of strings for that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

dir = Rails.root.join("tmp")
filename = "check_in_duration.csv"
headers_array = ["Check-in ID", "Date", "duration (sec)"]
start_datetime = Date.new(2019, 05, 20).beginning_of_day
end_datetime = Date.new(2019, 05, 25).end_of_day

CSV.open("#{dir}/#{filename}", "wb") do |csv|
  csv << headers_array

  CheckIn.
    for_practice(1).
    where(created_at: start_datetime..end_datetime).each do |ci|
    check_in_id = ci.id
    check_in_date = ci.created_at
    check_in_duration = ci.timespan

    csv << [check_in_id, check_in_date, check_in_duration]
  end
end

So with that, we’ve just generated a file in our tmp directory with the report. There’s one last thing remaining to complete the task. Extracting and submitting the file to whoever requested it.

Report submission

It would be great if we could download the file from production in some way so we can send it as an email attachment. However, Rails has better plans for us. We can do this directly in the console using ActionMailer.

1
2
3
4
5
6
7
8
9
10
11
12

mailer = Class.new(ActionMailer::Base) do
  def report_message
    attachments[filename] = File.read("#{dir}/#{filename}")
    mail(
      to: "receiver@example.com",
      from: "data@example.com", subject: "Check-ins duration",
      body: "May 20 to May 25 - Practice 111",
    )
  end
end

mailer.report_message.deliver!

We create an object of the class provided by Rails and pass a block with the attached file and a call to the mail method, which contains the needed information about the recipient, sender, and content. We complete the submission calling deliver! on it.

Great. Our script does all the needed tasks. Now, how do we run it in production?

Running the script in production

Vim, Tmux, and a plugin created by Chris Toomey are the tools that we use to complete this goal. First of all, if you don’t have it, you can install Vim Tmux Runner following the instructions in the readme. Then, these are the steps to follow:

  1. Open 2 panes of Tmux (I like to do it side by side, in a vertical split).
  2. Connect to production Rails console in one of the panes (e.g. aptible ssh --app [name_of_the_app], and then bundle exec rails console for an Aptible app)
  3. Open your script in a Vim window in the other pane
  4. In Vim, connect to the other pane running VtrAttachToPane (or your shortcut for that command). As there is only one other pane it connects automatically. If you have more than two panes opened in the same window, it asks you to pick one. Pick the one with the production Rails console.
  5. Select the range of lines you want to run (I usually run the part of the script that creates the file first, and send the email in a second step).
  6. Run VtrSendLinesToRunner which copies and runs the range selected in the other pane.

That’s it! You’ve successfully created a CSV report and sent it to an email account in a flexible, and straightforward.

Here is how the process would look like when executed:

Example video

You can save now this script in a Gist and reuse it changing only the required parts. This process has saved me from many of the pain points associated with having to run reports in an ephemeral container.

If you want to know more about how to integrate Vim and Tmux to create an excellent development environment, there is a great video on it also created by Chris Toomey for thoughtbot’s Upcase.