It’s possible to define a tensor in Ruby using the Array class of Ruby but it gets tedious when defining multi-dimensional tensors. Also, the Array object is designed to be heterogeneous which means that the elements of the array can be of different type or different classes which would seem as a plus point overall but it has a huge downside. Due to Array being heterogeneous, the memory allocation has to be in such a way that any element of any size can be added or removed from the array, which causes a lot of re-allocations. Also, the indexing and other array functions gets slower due to the heterogeneous nature.

What if there’s a scenario where there is only one type of tensor elements, which means that a homogeneous array would also do, and there are memory and speed constraints? NumRuby is the solution for such requirements.

Tensors in NumRuby

A tensor can be defined using the NMatrix object of NumRuby.

1

N = NMatrix.new [shape], [elements], :type

shape is the number of dimensions and size of each dimension of the tensor. For example, [2, 2, 2] shape tensor is a tensor with 3 dimensions and each dimension of size 2, hence number of elements is 8. A sample value of elements array for this could be [1, 2, 3, 4, 5, 6, 7, 8]. type is the data type of each of the tensor element, it could be any of :nm_bool, :nm_int, :nm_float32, :nm_float64, :nm_complex32 or :nm_complex64 depending on the requirements.

An example usage is shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

[1] pry(main)> N = NMatrix.new [2, 2], [1, 2, 3, 4], :nm_int
=>
[
 [1, 2],
 [3, 4]
]
[2] pry(main)> N.elements
=> [1, 2, 3, 4]
[3] pry(main)> N.shape
=> [2, 2]
[4] pry(main)> N[0, 0]
=> 1
[5] pry(main)> N[1, 0]
=> 3
[6] pry(main)> N.dtype
=> :nm_int

Elementwise Operations

One can also perform elementwise operations using NumRuby. Elementwise operations are broadly of 2 types, Uni-operand and bi-operand.

Uni-Operand Operations

Uni-operand operators are those that apply to just one tensor. For example, sine, cos or tan of each of element of the tensor.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

[1] pry(main)> N = NMatrix.new [2, 2], [1, 2, 3, 4], :nm_float64
=>
[
 [1.0, 2.0],
 [3.0, 4.0]
]
[2] pry(main)> N.sin
=>
[
 [0.8414709848078965,  0.9092974268256817],
 [0.1411200080598672, -0.7568024953079282]
]
[3] pry(main)> N.cos
=>
[
 [ 0.5403023058681398, -0.4161468365471424],
 [-0.9899924966004454, -0.6536436208636119]
]

Bi-Operand Operations

Bi-operand operators are those that apply to two tensor. For example, addition, subtraction or multiplication of each of the corresponding elements of the the 2 tensor.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[1] pry(main)> N = NMatrix.new [2, 2], [1, 2, 3, 4], :nm_float64
=>
[
 [1.0, 2.0],
 [3.0, 4.0]
]
[2] pry(main)> M = NMatrix.new [2, 2], [2, 2, 3, 3], :nm_float64
=>
[
 [2.0, 2.0],
 [3.0, 3.0]
]
[3] pry(main)> N+M
=>
[
 [3.0, 4.0],
 [6.0, 7.0]
]
[4] pry(main)> N*M
=>
[
 [2.0,  4.0],
 [9.0, 12.0]
]

Linear Algebra

NumRuby also supports linear algebra capabilities for 2-dimensional tensors. One can easily do operations such as matrix inverse, dot product, matrix decompositions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

[1] pry(main)> N = NMatrix.new [2, 2], [1, 2, 3, 4], :nm_float64
=>
[
 [1.0, 2.0],
 [3.0, 4.0]
]
[2] pry(main)> M = NMatrix.new [2, 2], [2, 2, 3, 3], :nm_float64
=>
[
 [2.0, 2.0],
 [3.0, 3.0]
]
[3] pry(main)> N.dot(M)
=>
[
 [ 8.0,  8.0],
 [18.0, 18.0]
]
[4] pry(main)> N.invert
=>
[
 [-1.9999999999999996,  0.9999999999999998],
 [ 1.4999999999999998, -0.4999999999999999]
]
[5] pry(main)> N.det
=> -2.0

Introduction

With GSoC 2019 coming to an end, this is my final blog which mentions all my work for the project Rubyplot.

What is Rubyplot?

RubyPlot is a plotting library in Ruby for scientific development inspired by the library Matplotlib for Python. Users can create various types of plots like scatter plot, bar plot, etc. and can also create subplots which combine various of these plots. The long-term goal of the library is to build an efficient, scalable and user-friendly library with a backend-agnostic frontend to support various backends so that the library can be used on any device.

Examples

Creating graphs in Rubyplot is very simple and can be done in just a few lines of code, for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

require 'rubyplot'

figure = Rubyplot::Figure.new(width: 30, height: 30)

axes00 = figure.add_subplot! 0,0
axes00.plot! do |p|
  d = (0..360).step(5).to_a
  p.data d, d.map { |a| Math.sin(a * Math::PI / 180) }
  p.fmt = 'ok-'
  p.marker_fill_color = :white
  p.marker_size = 0.5
  p.line_width = 2
  p.label = "sine"
end

axes00.title = "A plot function example"
axes00.square_axes = false

figure.write('example1.png')

Has the output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

require 'rubyplot'

figure = Rubyplot::Figure.new(width: 30, height: 30)

axes00 = figure.add_subplot! 0,0
axes00.candle_stick! do |p|
  p.lows = [0, 10, 20, 30, 20, 10]
  p.highs = [40, 50, 60, 70, 60, 50]
  p.opens = [10, 20, 30, 40, 30, 20]
  p.closes = [30, 40, 50, 60, 50, 40]
  p.color = :yellow
end
axes00.candle_stick! do |p|
  p.lows = [5, 5, 25, 30, 10, 10]
  p.highs = [50, 40, 65, 70, 80, 60]
  p.opens = [10, 20, 30, 40, 30, 20]
  p.closes = [35, 35, 45, 60, 75, 50]
  p.color = :blue
end

axes00.title = "A multi candle-stick plot"
axes00.square_axes = false

figure.write('example2.png')

Has the output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

require 'rubyplot'

figure = Rubyplot::Figure.new(width: 30, height: 30)

axes00 = figure.add_subplot! 0,0
axes00.bar! do |p|
  p.data [1, 2, 3, 4, 5]
  p.color = :lemon
  p.spacing_ratio = 0.2
end
axes00.bar! do |p|
  p.data [5, 4, 3, 2, 1]
  p.color = :blue
end
axes00.bar! do |p|
  p.data [3, 5, 7, 5, 3]
  p.color = :red
end

axes00.title = "A multi bar plot"
axes00.square_axes = false

figure.write('example3.png')

Has the output:

History of Rubyplot

Rubyplot started as two GSoC 2018 projects by Pranav Garg(@pgtgrly) and Arafat Dad Khan(@Arafatk) and the mentors from The Ruby Science Foundation(SciRuby), Sameer Deshmukh(@v0dro), John Woods(@mohawkjohn) and Pjotr Prins(@pjotrp). Pranav Garg worked on the GRRuby which had the GR backend and Arafat Dad Khan worked on Ruby Matplotlib which had the ImageMagick backend. The ultimate goal of combining both and creating Rubyplot. After GSoC 2018, Sameer Deshmukh combined both projects and created Rubyplot and he has maintained it ever since. Around May 2019, I started working on Rubyplot as a part of GSoC 2019.

GSoC 2019

As a part of GSoC 2019, my project had 3 major deliverables: 1. ImageMagick support(Phase 1): Support for ImageMagick back-end will be added in addition to the currently supported back-end GR, the front-end of the library will be back-end agnostic and the current overall integrity of the library will be preserved.
2. Plotting and show function(Phase 2): A new plot function will be added which plots markers (for example circles) to form a scatter plot with the points as inputs (same as plot function in Matplotlib). A new function show will be added which will allow viewing of a plot without saving it. This plot function will be back-end agnostic and hence will support both GR and Magick back-end.
3. Integration with iruby notebooks(Phase 3): Rubyplot will be integrated with iruby notebooks supporting all backends and allowing inline plotting.

As a part of GSoC 2019, I completed all the deliverables I had initially planned along with a tutorial for the library and some other general improvements.
Details of my work are as follows:

Phase 1

During Phase 1, I focused on setting up the ImageMagick backend which involved the basic functionality required for any backend of the library which are X-axis and Y-axis transform functions, within_window function which is responsible for placing the plots in the correct position, function for drawing the X and Y axis, functions for drawing the text and scaling the figure according to the dimensions given by the user. I implemented these functions using internal rmagick functions which were very useful like scale, translate, rotate, etc.
After this, I worked on the scatter plot, which was the first plot I ever worked on. This plot had a very particular and interesting problem, which was that different types of markers were internally implemented in the GR backend, but for ImageMagick backend, I had to implement everything using basic shapes like circles, lines, polygons and rectangles. To solve this I created a hash of lambdas which had the code to create different types of markers using the basic shapes.
After this I implemented all the simple plots which Rubyplot supports, these are line plot, area plot, bar plot, histogram, box plot, bubble plot, candle-stick plot and error-bar plot.

So, during Phase 1, I completed the following deliverables -
1. Set up the ImageMagick backend to have the basic functionality.
2. Implemented and tested the simple plots in Rubyplot which are scatter plot, line plot, area plot, bar plot, histogram, box plot, bubble plot, candle-stick plot and error-bar plot.

Code for Phase 1 can be found here.

Phase 2

I started Phase 2 by implementing the multi plots which are multi stack-bar plot, multi-bar plot, multi-box plot and multi candle-stick plot.
Next, I implemented the plot function which is a combination of scatter plot and line plot, using the plot function the user can easily create a scatter plot or a line plot or a combination of both. The most interesting feature of the plot function is the fmt argument which sets the marker type, line type and the colour of the plot using just characters, so instead of writing the name of the type and setting the variables, the user can simply input a string in fmt argument which has the characters for corresponding marker type, line type and colour.
Next was to implement the show function which is an alternative to write function. It draws the Figure and shows it on a temporary pop-up window without the need of saving the Figure on the device, this allows the user to test the code quickly and easily. This was done by using internal functions of the backends which are display for ImageMagick and gr_updatews for GR.

So, during Phase 2, I completed the following deliverables -
1. Implemented and tested the multi plots in Rubyplot which are multi stack-bar plot, multi-bar plot, multi-box plot and multi candle-stick plot.
2. Implemented and tested the plot function with fmt argument.
3. Implemented and tested the show function.

Code for Phase 2 can be found here and here.

Phase 3

During Phase 3, I integrated Rubyplot with the IRuby notebooks which allow the user to draw figures inside the notebook just by using the show function, through this integration the user can quickly and easily test the code step by step before running the whole codebase.
I also implemented ticks for ImageMagick backend.
Finally, I created a tutorial for the library which also contains template codes for all the plots which a user can easily get familiar with the working of the library and start using it.

So, during Phase 3, I completed the following deliverables -
1. Integrated Rubyplot with IRuby notebooks with the support for inline plotting.
2. Implemented and tested ticks for Magick backend.
3. Created the tutorial for Rubyplot.

Code for Phase 3 can be found here.

Resources(blogs, code, etc.)

Previous Work

• GSoC 2018 project GRRuby by Pranav Garg can be found here
• GSoC 2018 project Ruby Matplotlib by Arafat Dad Khan can be found here
• A talk on Rubyplot by Pranav Garg in RubyConf 2018 can be found here

My work

• Daily updates can be found here
• Proposal can be found here
• Tutorial notebook can be found here.ipynb) and can be viewed online(rendered) here
• Rubyplot Github Repository can be found here
• All my work can be found in these PRs: PR#45 and PR#52
• Other blogs can be found here:
  1. GSoC 2019 project introduction
  2. Rubyplot installation guide
  3. The Scatter plot example
  4. Simple Plots in Rubyplot
  5. Multi plots in Rubyplot
  6. The show and the plot functions
  7. IRuby integration and ticks

Future Work

I plan to keep contributing to Rubyplot and also start contributing to other projects of SciRuby.
Future work to be done for Rubyplot is to write documentation, add more tests, add more types of plots, add more backends, make the plots interactive and in future add the feature for plotting 3-Dimensional graphs which would also be interactive.

EndNote

With this, we come to an end of GSoC 2019. These 3 months have been very challenging, interesting, exciting and fun. I got to learn a lot of things while working on Rubyplot and while interacting with my mentors. I have experienced an improvement in my Software development skills and programming in general which will help me a lot in future. I would love to keep working with SciRuby on more such interesting projects and maybe even try for GSoC again next year ;)

Acknowledgements

I would like to express my gratitude to my mentor Sameer Deshmukh for his guidance and support. He was always available and had solutions to every problem I faced, I got to learn a lot from him and I hope to learn a lot more from him in the future. I could not have asked for a better mentor.

I would also like to thank Pranav Garg who introduced me to Ruby and also to the SciRuby community. During his GSoC 2018 project, he introduced me to the Rubyplot library and helped me get started with it. His suggestions were very helpful during my GSoC 2019 project.

I would also like to thank mentors from SciRuby Prasun Anand and Shekhar Prasad Rajak for mentoring me and organising the occasional meetings and code reviews. I would also like to thank Udit Gulati for his helpful insights during the code reviews.

I am grateful to Google and the Ruby Science Foundation for this golden opportunity.

GSoC 2019 proposal - https://docs.google.com/document/d/1MR01QZeX_8h7a16nmkOYlyrVt–osB1Yg9Vo0xXYtSw/edit?usp=sharing

Pull requests

• Indexing, iterating, slicing - https://github.com/SciRuby/numruby/pull/19
• Fix compiler warnings - https://github.com/SciRuby/numruby/pull/21
• Dimensional iterators - https://github.com/SciRuby/numruby/pull/22
• Broadcasting - https://github.com/SciRuby/numruby/pull/23
• Lapack wrappers - https://github.com/SciRuby/numruby/pull/26
• NumRuby::Linalg - https://github.com/SciRuby/numruby/pull/30

Blog Posts

• GSoC project introduction - https://uditgulati.github.io/blog/open-source/2019/05/21/gsoc-project-introduction-nmatrix.html
• GSoC weeks 1, 2 - https://uditgulati.github.io/blog/open-source/2019/06/10/gsoc-week-1-2.html
• GSoC weeks 3, 4 - https://uditgulati.github.io/blog/open-source/2019/06/26/gsoc-week-3-4.html
• GSoC weeks 5, 6 - https://uditgulati.github.io/blog/open-source/2019/07/10/gsoc-week-5-6.html
• GSoC weeks 7, 8 - https://uditgulati.github.io/blog/open-source/2019/07/25/gsoc-week-7-8.html
• GSoC weeks 9, 10 - https://uditgulati.github.io/blog/open-source/2019/08/16/gsoc-week-9-10.html

Future work

• As these features are core to NMatrix, they need to be tested more tightly and benchmarked. Also, since this is the first implementation of the features, some modifications can make these features more reliable and faster.
• More iterators are to be implemented in order to have a richer NMatrix API.
• The issue with slicing needs to be fixed soon. Also, need to standardize the formatting used for slice range specification.
• Broadcasting needs to be tested and benchmarked. This will let us improve it’s reliability and performance. A few APIs could be exposed which would let a user manually broadcast a matrix to a given shape.
• NumRuby::Lapack needs to have proper exception handling for LAPACK routines. This will be done by reading the info value returned by routine and raising exception accordingly.
• NumRuby::Linalg can be extended to have even more methods.

Acknowledgements

I wanna thank Ruby Science Foundation, all mentors and org admins for providing me this wonderful opportunity to enhance my knowledge and work on a project. I would also like to thank Google for organizing such a wonderful program due to which I got introduced to Open-source and Ruby Science Foundation. I especially want to thank my mentor Prasun Anand for guiding me through this period, keeping me motivated and for tolerating my procrastination.

For my summer of code Project I decided to create a plotting library.

From scratch.

In Ruby.

2. Application

The GSoC 2018 application can be found here.

3. Code

The code for the project can be found here.

RubyPlot is currently being developed here.

3. The Plotting Architecture

The plotting architecture for the library was inspired by Late Dr John Hunter’s Python Plotting Library “Matplotlib”.

The Matplotlib Architecture is be broadly divided into three layers (as shown in the masterpiece of a figure which I made below). The Backend, The Artist and the scripting layer.

The Backend layer can further be divided into three parts : The Figure Canvas, The Renderer and the Event.

Matplotlib architecture is mostly written in Python with some of its backend (the renderer AGG and Figure canvas ) written in C++ ( The original AGG backend and helper scripts, which is quite tightly bound to python). But the recently the backends are also written in Python using the renderers which have Python APIs. The finer details of the Architecture can be found here.

In interest of the time I decided to go for a iterative process to develop the Library. I decided to use an existing Artist layer. After a lot of discussion we decided to use GR Framework for the same. The only issue was that GR did not have a Ruby API.

4. Creating C extensions for GR Framework:

To create the C extensions I initially decided to use Fiddle followed by FFi. But this lead to certain issues when it came to handling arrays. Hence I decided to go with the old fashioned Ruby C API to create extensions. The code for the same can be found here.

5. Creating Scripting Layer

The Scripting Layer Is meant for high level plotting. The scripting Library created on the GR Framework wrapper has the can plot the following:

• ScatterPlots

• Line graphs

• Bar Plots

• Stacked Bar plot

• Stacked Bar plot (Stacked along z axis)

• Candlestick plots

  All the above plots have a lot of customisation options, that can be looked into in the documentation.

Each Figure can have multiple subplots, each subplot can have multiple plots

6. Working of the Library

Here is how the library works.

Figure is the class that a user instantiates this is where all the plotting take place. An instance contains the state of the figure. GR framework is used as the artist layer which does all the plotting on the figure. GR is also the backend.

GR artist layer functions are implemented in C language, we wrap the functions to ruby classes which have the call method which executes the GR function when the Object of the ruby class is called. Each of these ruby classes are called tasks which represents that they perform a task, for example ClearWorkspace performs the task of cleaning the workspace.

Now, the figure is divided into subplots. It is Subplot(1,1,1) by default. So, figure has objects of subplot, each subplot is of type bar plot or line plot etc. These plots are defined in the Plots module which is submodule of Scripting module, the Plots module has a submodule named BasePlots which defines the two bases of plots, LazyBase and RobustBase. Lazy base is for plots which are dependent on state of the figure, for example a bar graph depends on the location of axes. Every lazy plot has a unique call function rather than inheriting it from LazyBase. In LazyPlots the instances of GR Function Classes are called as soon as they are instantiated. This all is done in the call function. Robust base is for plots which are which are independent of the state of the Figure. For example: A scatter plot is independent of the location of axes. Plots which are Sub classes of RobustBase append the instances of GR function classes to tasks when initialized. These instances are called via the call method defined in RobustBase.

So, each subplot which is of type bar plot or scatter plots etc. inherits a base. Now, each subplot is just a collection of some tasks, so it has a task list which stores the tasks to be performed i.e. the Task objects, for example Scatter plot has tasks SetMarkerColorIndex which sets the color of the marker, SetMarkerSize which sets the size of the marker, SetMarkerType which sets the type of the marker and Polymarker which marks the marker of defined color, size and style. Whenever a new Subplot object is initialized, for example subplot(r,c,i), the figure is divided into a matrix with r rows and c columns and the subplot initialized with index i is set as the active subplot ans this active subplot is pushed into the subplot list. Each subplot object has a unique identity (r,c,i) so if the user wants to access a subplot which is already declared, this identity will be used. When the subplot object is called (i.e. to view or save), it first executes some necessary tasks and then pushes the tasks related to bar plot, scatter plot, etc. to the task list.

Figure is a collection of such subplots and so Figure has a subplot list which stores the subplot objects.

These tasks are just stored in the lists and are not performed (i.e. called) until the user asks to view or save the figure i.e. when the user calls view or save (which are tasks themselves) the tasks are performed (i.e. called) and the figure is plotted. This is done by using the Module Plotspace.
When the figure calls the task view or save, these tasks call the Plotspace Object and the state of figure is copied to the Plotspace Object and this Object starts executing( or performing) i.e. calling tasks from task list of each subplot in subplot list and the figure is plotted and viewed or saved.

Here is the current view of Library:

Future

The Library is currently being developed by SciRuby community here. Currently, it is a static library, after further development, it’s Architecture should look like the following:

Acknowledgements

I would like to thank Sameer Deshmukh and Prasun Anand for guiding me through every step of software design and helping me out through every decision. I would also like to thank Dr John Woods and Dr Pjotr Prins for their valuable feedback. I am glad to be a part of SciRuby community and I hope to further contribute towards it’s goal.

I would also like to thank Arafat Khan, a fellow GSoCer who worked on the library using Rmagick as the backend renderer for our fruitful debates over the architecture of the library

Finally, I would like to thank Google for giving me this opportunity.

At Sciruby, we came across the ambitious task of designing the next Matplotlib for Ruby and so we examined quite a few different plotting libraries and figured that if we could put together ideas from all of these libraries then we can make something really amazing.Primarily our main source of inspiration was Matplotlib. The Matplotlib Architecture is be broadly divided into three layers. The Backend, The Artist and the scripting layer.

Matplotlib architecture is mostly written in Python with some of its backend (the renderer AGG and Figure canvas ) written in C++ ( The original AGG backend and helper scripts, which is quite tightly bound to python). But recently the backends are also written in Python using the renderers which have Python APIs. The finer details of the Architecture can be found here.

Based on Matplotlib our initial plans for the library can be described in this visual.

We decided to build two different ruby libraries independently but with many parallels in their code. Eventually when the project gets completed we will combine them into a single repository and give users the option to use either of the libraries as a backend for construction of the plots.

For the first one, Gr plot is plotting library for ruby that uses the GR framework as a backend. And for the second one, Magick plot is a plotting library that produces quality figures in a variety of hardcopy formats using RMagick as a backend.

Magickplot is an interesting library with many features similar to GRPlot but the internal implementations of both the libraries are radically different. We believe that depending on the use cases the users can find either of them more useful than the other one.  So our next goal is to merge them together and give users a simple API to switch back ends easily from GR Plot to Magick Plot….

My work in particular dealt with building Magickplot. The library works in similar thought process to painting where you can give it an empty paper called figure canvas and then you draw upon the figure canvas using plotting features. For all drawing and plotting purposes in this library we use RMagick.

So where are our paint brushes and paints for drawing on the plot?

• draw->rectangle
• draw->circle
• draw->line
• draw->annotate

These base features will let you make Bar, Scatter, Dot, Line and Bubble plots with Magickplot with very accurate geometry. A better walk through of the construction of a single plot with this library can be found in this blog.

Application

My GSoC 2018 application for the project can be found here.

Blog Posts

The entire work for Rubyplot can be summarized in these series of blogposts:

• Introductory Blog
• Artist Base - Part 1
• Artist Base - Part 3
• Divisionism
• Scatter Plots
• Bubble Plots

Future Work

Our ultimate goal is to make this project similar to a matplotlib equivalent for Ruby with tons of really amazing customization features. I hope you find this interesting and useful.The library is currently being developed by the Sciruby community, feel free to try it out from github. Any suggestions and recommendations are welcome.

FOSS

I have been an active contributor to a few open source projects and I have started a few nice ones of my own and I feel really glad to have been introduced to the open source community. I really appreciate the effort by Google Open Source Committee for conducting GSoC every year. It is the best platform for aspiring programmers to improve their skill and give back to society by developing free and open source software.

Acknowledgements

Thanks to all my Mentors from Sciruby namely, Sameer Deshmukh, Pjotr Prins, Prasun Anand and John Woods.  Special thanks to Pranav Garg a fellow GSoCer who is the lead developer of GR-Ruby and a student in Gsoc for Sciruby

Daru-view now presents data in some more visualizations like HighMap and HighStock along with the already implemented HighCharts, GoogleCharts, DataTables and Nyaplot. It provides some more new cool features like formatting Daru::View::Table (GoogleCharts table) with different colors, pattern, etc., exporting charts to different formats, comparing different visualizations in a row and many more. Follow up with these IRuby examples to know the current features equipped in daru-view.

These figures describes the usage of some of the features implemented in daru-view during GSoC.

Application

The GSoC 2018 application can be found here.

Code

• GSoC 2018 work done summary – Progress Report
• GSoC 2018 work presentation – Advance Features in daru-view
• Discourse Discussion – daru-view discussion

The work done during this GSoC has been explained in the following eight blog posts:

1. GSoC 2018 Introduction – Goals defined
2. HighCharts and HighMaps
3. Custom Styling CSS in HighCharts
4. Exporing HighCharts | ChartWrapper | Importing data from google spreadsheet in Google Charts
5. Exporting Google Charts | Charteditor
6. Handling events in Google Charts | Multiple Charts in a row
7. Formatters in Google Charts | Loading large set of data in DataTables
8. Reduce a bunch of lines due to JS files in source HTML in rails | Rake task to add new adapter

Future Work

The future work involves removing the dependency of daru-view on gems google_visualr and lazy_high_charts by creating our own gems. Check out these new ideas that can be implemented in daru-view.

FOSS

This has been my first attempt to explore the open source community. The summer was filled with the development of open source software and definitely was a great learning experience.

I really appreciate the effort by Google Open Source Committee for conducting GSoC every year. It is the best platform for the aspiring programmers to improve their skill and give back to society by developing free and open source software.

Acknowledgements

I would like to express my sincere gratitude to Ruby Science Foundation, all the mentors and org admins for providing me this wonderful opportunity to enhance my knowledge and work independently on a project. I especially want to thank Shekhar for guiding me through the journey, helping and motivating me in every possible way.

I am very thankful to Google for organizing such an awesome program.

ArrayFire-rb now supports linear algebra on GPU and CPU. Currently only double dtype has been implemented. It supports dense and sparse matrices. It has multiple backends namely, CUDA, OpenCL and CPU.

(Note: The above benchmarks have been done on an AMD FX 8350 octacore processor and Nvidia GTX 750Ti GPU. CUDA backend of ArrayFire was used with double floating points.)

The figure shows that ArrayFire takes the least computation time of all. For elementwise arithmetic operations, ArrayFire is 2 e 4 times faster than NMatrix for Ruby whereas 2 e 3 times faster than NMatrix for JRuby.

The figure shows that ArrayFire takes the least computation time of all. ArrayFire is 3 e +6 times faster than NMatrix for JRuby and NMatrix for Ruby(not BLAS) whereas 7 e +5 times faster than NMatrix for Ruby(using BLAS).

For LAPACK routines, like calculating determinant and lower-upper factorization, ArrayFire is 100 times faster than NMatrix for JRuby whereas 6 times faster than NMatrix for Ruby(using LAPACKE).

Application

The GSoC 2017 application can be found here.

Code

ArrayFire-rb: The pull request is undergoing a review.

ArrayFire-rb Benchmarks: Codebase can be found here.

Bio::FasterLmmD : Codebase can be found here

The work on creating the bindings have been explained in the following nine blog posts:

1. Ruby C extensions for complex projects
2. Installation
3. Af_Array(see performance)
4. Test-suite and Algorithm class
5. BLAS and LAPACK routines(see performance)
6. Statistics and Random Engine routines
7. Device and Util
8. Multiple Backends: CUDA, OpenCL and CPU
9. ArrayFire-NMatrix Interface

I took a side-track working on Bio::FasterLmmD . This work is not complete and still in progress. It is an effort to call D from Ruby. The work has been explained in a previous blog post.

The work on ArrayFire-rb - JRuby has been postponed for now as I wanted to concentrate on MRI for the best results.

Future Work

The future work involves improving the ArrayFire-rb code and writing tutorials. ArrayFire is not limited to linear algebra so I will create bindings for Signal Processing, Computer Vision, etc. I will also add support for data types other than double.

The work on ArrayFire-rb - JRuby will begin as soon as ArrayFire gem is published.

FOSS

This has been my second GSoC with SciRuby. It has been more than an year contibuting extensively to FOSS.

I really appreciate the effort by Google Open Source Committee for conducting GSoC every year. It is the best platform for the aspiring programmers improve their skill and give back to society by developing free and open source software.

Last year’s GSoC work helped me to present a talk at FOSDEM 2017 and Ruby Conf India 2017. I got active in the Indian Ruby Community. Recently, I have been invited as a speaker to Ruby World Conference 2017, Matsue, Japan and RubyConf 2017, New Orleans, to talk on “GPU computing with Ruby”.

I plan to continue contributing to open source, strive for improving my skills, and help new programmers contribute to FOSS. I would be glad if I could mentor students for upcoming GSoCs.

Acknowledgements

I would like to express my sincere gratitude to my mentor Pjotr Prins, for his guidance, patience and support. I have learn a lot from him since my last GSoC and still learning. I couldn’t have hoped for a better mentor.

I am grateful to Google and the Ruby Science Foundation for this golden opportunity.

I am very thankful to John Woods, Sameer Deshmukh, Alexej Gossmann, Gaurav Tamba and Pradeep Garigipati who mentored me through the project.

What makes daru-view different ?

• daru-view is designed for interactive plotting of charts and tables.It provide different plotting tools like Nyaplot, HighCharts, GoogleCharts, DataTable. So you don’t have to write any JavaScript code from these sites and no need to shift to other language to get charts.

• It can work with any ruby web application framework like Rails/Sinatra/Nanoc/Hanami. If you want to try few examples then please look into the daru-view/spec/dummy_* examples of Rails, Sinatra, Nanoc web applications.

• Now Ruby developers are using IRuby notebook for interactive programming. daru-view support IRuby notebook as well. So if you just want to see chart for some DataFrame or Array of data, you can use daru-view.

• daru-view can generate chart images to download and save.

• daru-view adapters googlecharts, highcharts are able to geneate 3D charts as well.

• Table have some main features like pagination, search and many more to be added.It is designed to load large data set smoothly.

Introduction

Daru is doing pretty good work as the data analysis & manipulation in IRuby notebook as well as backend part of web application. Ruby web application frameworks like Ruby on Rails, Sinatra, Nanoc are popular frameworks. So if Ruby developers get the gem like daru which can do data analysis and visualization work in applications, then there is no need of shifting to another language or usage of other gem.

My project for GSoC 2017 was to “make Daru more ready for integration with modern Web framework” in terms of visualization.

To improve in terms of viewing data, daru-view, a plugin gem for daru is created. daru-view is for easy and interactive plotting in web application & IRuby notebook. It can work in frameworks like Rails, Sinatra, Nanoc and hopefully in others too.

To see a quick overview of daru-view’s features, have a look at these examples:

• IRuby Notebook Examples

• daru io and daru-view usage in Rails app

• README of daru-view

Examples

This is how we can create a Plot class object:

1

Daru::View::Plot.new(data, options)

• data can be Daru::DataFrame, data array or the format that the adapter support.

• options is a hash that contains various options to customize the chart. If you have chosen a plotting library then you must use the options according to the options the library providing. Here is the library daru-view uses. Please check the examples options, they are passing in javascript code:

  • GoogleCharts: https://developers.google.com/chart/interactive/docs/gallery

  • HighCharts: https://www.highcharts.com/demo

  • Nyaplot: https://github.com/SciRuby/nyaplot (it works same as daru)

Note: User must have some knowledge about the plotting tool(that you want to use) to use it in daru-view. So that you can pass the correct options.

GoogleCharts:

Set the plotting library to :googlecharts to use this adapter. This will load the required js files in your webpage or IRuby notebook.

1
2

require 'daru/view'
Daru::View.plotting_library = :googlecharts

Let’s create a DataFrame :

1
2
3
4
5
6
7
8
9
10
11

idx = Daru::Index.new ['Year', 'Sales']
data_rows = [
          ['2004',  1000],
          ['2005',  1170],
          ['2006',  660],
          ['2007',  1030]
]
df_sale_exp = Daru::DataFrame.rows(data_rows)
df_sale_exp.vectors = idx

# perform data manipulations, if you want.

Now time to plot it:

1
2

line_basic_chart = Daru::View::Plot.new(df_sale_exp)
line_basic_chart.chart

This will return the chart object we created using GoogleCharts. In IRuby notebook, you will see this:

You can find the IRuby notebook example in this link.

These are various charts type we can use e.g. line, area, bar, bubble, candlestick, combo, histogram, org, pie, stepped area chart, timeline, treemap, gauge, column, scatter, etc. We can find the customization options in the google charts site.

Let me try another chart type Geo :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

country_population = [
          ['Germany', 200],
          ['United States', 300],
          ['Brazil', 400],
          ['Canada', 500],
          ['France', 600],
          ['RU', 700]
]

df_cp = Daru::DataFrame.rows(country_population)
df_cp.vectors = Daru::Index.new(['Country', 'Population'])

geochart = Daru::View::Plot.new(
    df_cp, type: :geo, adapter: :googlecharts
)

Note: If you have already loaded the dependent JS files for the library then you can use adapter: :googlecharts in your Plot initialization.

HighCharts:

Set the plotting library to :highcharts to use this adapter. This will load the required js files in your webpage or IRuby notebook.

1
2

require 'daru/view'
Daru::View.plotting_library = :highcharts

Let’s pass the data as HighCharts support (we can pass a DataFrame as well):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

data = [
    ['Firefox',   45.0],
    ['IE',       26.8],
    {
       :name=> 'Chrome',
       :y=> 12.8,
       :sliced=> true,
       :selected=> true
    },
    ['Safari',    8.5],
    ['Opera',     6.2],
    ['Others',   0.7]
]
plt_pie = Daru::View::Plot.new data, type: :pie

This will return the Plot object we created. In IRuby notebook, you will see this:

You can find the IRuby notebook example in this link.

There are various charts type we can use e.g. line, area, bar, bubble, dynamic chart, pie, column, scatter, etc. We can find the customization options in the HighCharts site.

Nyaplot

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

require 'daru/view'

#  set adapter
Daru::View.plotting_library = :nyaplot

# define dataframe
df = Daru::DataFrame.new({
  a: [1, 2, 4, -2, 5, 23, 0],
  b: [3, 1, 3, -6, 2, 1, 0],
  c: ['I', 'II', 'I', 'III', 'I', 'III', 'II']
  })
df.to_category :c

# creating scatter chart
scatter_chart = Daru::View::Plot.new(df, type: :scatter, x: :a, y: :b, categorized: {by: :c, method: :color})

In IRuby notebook:

GoogleChart data table

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

require 'daru/view'

#  set adapter
# You don't need this line if you have already using google chart for plotting.
# It is just for loading the dependent js files.
Daru::View.table_library = :googlechart

# Lets use array as `data` (we can pass Daru::DataFrame as well)
data = [
  ['Galaxy', 'Distance', 'Brightness'],
          ['Canis Major Dwarf', 8000, 230.3],
          ['Sagittarius Dwarf', 24000, 4000.5],
          ['Ursa Major II Dwarf', 30000, 1412.3],
          ['Lg. Magellanic Cloud', 50000, 120.9],
          ['Bootes I', 60000, 1223.1]
  ]
galaxy_table = Daru::View::Table.new(data)
galaxy_table.table

This will return the table object we created using GoogleCharts tool. In IRuby notebook, you will see this:

We can create table using Vectors as well.

1
2
3
4
5
6
7
8
9

dv = Daru::Vector.new [43934, 52503, 57177, 69658, 97031, 119931, 137133, 154175]

# adding pagination and some customization [optional]
opts_pagination = {
  width: '100%', height: '100%' ,
  pageSize: 5,
}

table_vec = Daru::View::Table.new(dv, opts_pagination)

In Ruby Notebook:

DataTable

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

require 'daru/view'

#  set adapter.
Daru::View.table_library = :datatables

# Lets use Daru::DataFrame as `data` (we can pass Array as well)
df = Daru::DataFrame.new(
  {
    b: [11,12,13,14,15],
    a: [1,2,3,4,5],
    c: [11,22,33,44,55]
  },
    order: [:a, :b, :c],
    index: [:one, :two, :three, :four, :five]
)
df_datatable = Daru::View::Table.new(df, pageLength: 3)

Currently there is some problem to diplay it in IRuby notebook, but in web application you can see something like this using df_datatable.div :

How to use it in Ruby web application

As we know, we can get the HTML, JS code for the chart from the Daru::View::Plot or Daru:View::Table object using #div method. So just need to add that HTML, JS code in webpage in proper place.

There is few things to be noted:

1. In layout of the webpage, you have to load all the dependent JS files. So that HTML, JS code that is genearted work smoothly in that webpage. You can load the dependent js file for nyaplot library using Daru::View.dependent_script(:nyaplot), similarly for other library.

2. If you are using multiple library in one webpage then load multiple dependent JS files, in that webpage layout (generally in head tag).

We can set default adapter using Daru::View.plotting_library = :googlecharts and also we can change it for particular object while initializing object, i.e. Daru::View::Plot.new(data, {adapter: :googlecharts}). Just we have to make sure that dependent JS files are loaded for it.

To make it easy, we have defined daru_chart (that works same as Daru::View::Plot.new) , daru_table (works same as Daru::View::Table.new) for Rails application.

So you can easily use it in controller or view of the application. For reference you can check the demo Rails app.

Design of daru-view

daru-view, currently using Nyaplot, HighCharts, GoogleCharts for plotting the charts. It is also generating tables using DataTables and GoogleCharts with pagination, search and various features.

Design Pattern in daru-view

daru-view mainly uses the adapter design pattern and composite design pattern.

• Why Adapter design pattern:

  • Adapter pattern’s motivation is that we can reuse existing gems if we can modify the interface.

  • daru-view joins functionalities of independent or incompatible interfaces of different gems.

  • daru-view have Plot and Table class, which are using a adapter when adapter(library to be used for plotting) is set for Plot, Table instance.

• Why Composite design pattern:

  • To define common objects and use it for defining composite objects.

  • In daru-view we try to write common functions in a module and include it whenever needed.

Implementation

daru-view ensure that it’s functions are usable in both IRuby notebook as well as ruby web application frameworks.

The main thing we need to display something in web application or IRuby notebook is HTML code of it. daru-view generates the HTML code of the chart, table and the same can be used to display in web application & IRuby notebook.

These are the libraries which is used in daru-view currently:

Nyaplot

Nyaplot is good library for visualization in IRuby notebook only. When we use Nyaplot as the adapter in daru-view, it is usable in both IRuby notebook and web applications. Daru DataFrame or Vector is used as the data source of the chart. It works similar to the initial daru plotting system.

If user want to use the Nyaplot methods then it can be done on Nyaplot object.We can get nyplot object using daru_plot_obj.chart.

i.e.

1
2
3

daru_view_obj = Daru::View::Plot.new(
                  daru_dataframe, options={adapter: :nyaplot})
nyaplot_obj = daru_view_obj.chart

Now user can operate all the methods for Nyaplot object. Same thing is for all other adapter in daru-view.

HighCharts

To add the HighCharts features for plotting various chart types, daru-view uses the lazy_high_charts gem with additional features.

In this adapter data source can be Array of data, Daru::DataFrame, Daru::Vector or HTML table code of the data.

There are various of options in HighCharts. One can see the options that can be used in HighCharts demo link, which can be directly used in daru-view Plot.

HighCharts adaptor can work offline as well in daru-view. Developers can update the saved the JS files (in daru-view) using rake task automatically.

If you is familiar with lazy_high_chart gem and want to use it for config the chart then user can access the lazy_high_chart object using Daru::View::Plot#chart and can do necessary operations.

GoogleCharts

To add the GoogleCharts features for plotting various chart types, daru-view uses the google_visualr gem with additional features(in this module more new features are updated).

We want GoogleChart adapter to be very strong since Google chart tools always gets updated and it has amazing plotting features. Similar to the HighCharts module, here also we can use all the options described in Google Charts website.

User can access the google_visualr object using Daru::View::Plot#chart, if they want to operate google_visualr methods.

GoogleCharts as data table

One of the good thing about google chart tool is, it can be used for generating table for web application and IRuby Notebook with pagination and other features.

Daru::View::Plot can take data Array, Daru::DataFrame, Daru::Vector, Daru::View::Table as data source.

Daru::View::Table can take data Array, daru DataFrame, Daru Vector as data source.

DataTables

DataTables has interaction controls to any HTML table. It can handle large set of data and have many cool features.

To use it, daru-view uses https://github.com/Shekharrajak/data_tables gem. [Note: the gem name will be changed in near future]

It basically uses the HTML table code and add features that user want. So internally HTML table code of Daru::DataFrame and Daru::Vector is passed as data source parameter.

Future Work

daru-view will be more powerful and simple in near future. Developers can add more libraries in daru-view easily, if required. To add library follow the setups given in CONTRIBUTING.md

Conclusion

The aim of the daru-view is to plot charts in IRuby notebook and ruby web application easily, so that developers need not have to use any other gem or language for visualization.

It can work smoothly in Rails/Sinatra/Nanoc web frameworks and I hope it can work in other ruby frameworks as well, because daru-view is generating the html code and javascript code for the chart, which is basic need of the webpage.

Why not use the plotting libraries directly?

If you are using daru gem for analyzing the data and want to visualize it, then it will be good if you have data-visualization within daru and can plot it directly using DataFrame/Vector objects of daru.

daru-view will be helpful in plotting charts and tables directly from the Daru::DataFrame and Daru::Vector . daru-view using nyaplot, highcharts , google charts right now to plot the chart. So user can set the plotting library and get the chart accordingly.

Most of the plotting libraries doesn’t provide the features of plotting charts in iruby notebook. They are defined only for web applications (mostly for Rails). But daru-view can plot charts in any ruby web application as well as iruby notebook.

Acknowledgements

I would like to thank to my mentors Sameer Deshmukh ,Lokesh Sharma and Victor Shepelev for their response and support and I am very grateful to the Ruby Science Foundation for this golden opportunity.

I thank my fellow GSoC participants Athitya Kumar and Prasun Anand for their support and discussions on various topics.

Thanks to Google for conducting Google Summer of Code.

“Hello friend. Hello friend? That’s lame.” - S01E01 (Pilot), Mr.Robot

My name is Athitya Kumar, and I’m a 4th year undergrad from IIT Kharagpur, India. I was selected as a GSoC 2017 student developer by Ruby Science Foundation for project daru-io.

Daru-IO is a plugin-gem to Daru gem, that extends support for many Import and Export methods of Daru::DataFrame. This gem is intended to help Rubyists who are into Data Analysis or Web Development, by serving as a general purpose conversion library.

Through this summer, I worked on adding support for various Importers and Exporters while also porting some existing modules. Feel free to find a comprehensive set of useful links in Final Work Submission and README. Before proceeding any further, you might also be interested in checking out a sample showcase of Rails example and the code making it work.

Mark Anthony’s Speech (ft. daru)

“Rubyists, Data Analysts and Web Developers, lend me your ears;

I come to write about my GSoC project, not to earn praise for it.”

For the uninitiated, Google Summer of Code (GSoC) 2017 is a 3-month program that focuses on introducing selected students to open-source software development. To know more about GSoC, feel free to click here.

daru is a Ruby gem that stands for Data Analysis in RUby. My initial proposal was to make daru easier to integrate with Ruby web frameworks through better import-export features (daru-io) and visualization methods (daru-view). However, as both Shekhar and I were selected for the same proposal, we split this amongst ourselves : daru-io was allocated to me and daru-view was allocated to Shekhar.

“The open-source contributions that people do, live after them;

But their private contributions, are oft interred with their bones.”

This is one of the reasons why I (and all open-source developers) are enthusiastic about open-source. In open-source, one’s work can be re-used in other projects in accordance with the listed LICENSE and attribution, compared to the restrictions and risk of Intellectual Property Right claims in private work.

“So be it. The noble Pythonistas and R developers;

Might not have chosen to try daru yet.”

It is quite understandable that Pythonistas and R developers feel that their corresponding languages have sufficient tools for Data Analysis. So, why would they switch to Ruby and start using daru?

“If it were so, it may have been a grievous fault;

Give daru a try, with daru-io and daru-view.”

First of all, I don’t mean any offense when I say “grievous fault”. But please, do give Ruby and daru family a try, with an open mind.

Voila - the daru family has two new additions, namely daru-io and daru-view. Ruby is a language which is extensively used in Web Development with multiple frameworks such as Rails, Sinatra, Nanoc, Jekyll, etc. With such a background, it only makes sense for daru to have daru-io and daru-view as separate plugins, thus making the daru family easily integrable with Ruby web frameworks.

“Here, for attention of Rubyists and the rest–

For Pandas is an honourable library;

So are they all, all honourable libraries and languages–

Come I to speak about daru-io’s inception.”

Sure, the alternatives in other languages like Python, R and Hadoop are also good data analysis tools. But, how readily can they be integrated into any web application? R & Hadoop don’t have a battle-tested web framework yet, and are usually pipelined into the back-end of any application to perform any analysis. I’m no one to judge such pipelines, but I feel that pipelines are hackish workarounds rather than being a clean way of integrating.

Meanwhile, though Python too has its own set of web frameworks (like Django, Flask and more), Pandas doesn’t quite work out-of-the-box with these frameworks and requires the web developer to write lines and lines of code to integrate Pandas with parsing libraries and plotting libraries.

“daru-io is a ruby gem, and open-sourced to all of us;

But some might think it was an ambitious idea;

And they are all honourable men.”

As described above, daru-io is open-sourced under the MIT License with attribution to myself and Ruby Science Foundation. Being a ruby gem, daru-io follows the best practices mentioned in the Rubygems guides and is all geared up with a v0.1.0 release.

Disclaimer - By “men”, I’m not stereotyping “them” to be all male, but I’m just merely retaining the resemblence to the original speech of Mark Anthony.

“daru-io helps convert data in many formats to Daru::DataFrame;

Whose methods can be used to analyze huge amounts of data.

Does this in daru-io seem ambitious?”

Daru has done a great job of encapsulating the two main structures of Data Analysis - DataFrames and Vectors - with a ton of functionalities that are growing day by day. But obviously, the huge amounts of data aren’t going to be manually fed into the DataFrames right?

One part of daru-io is the battalion of Importers that ship along with it. Importers are used to read from a file / Ruby instance, and create DataFrame(s). These are the Importers being supported by v0.1.0 of daru-io :

• General file formats : CSV, Excel (xls and xlsx), HTML, JSON, Plaintext.
• Special file formats : Avro, RData, RDS.
• Database related : ActiveRecord, Mongo, Redis, SQLite, DBI.

For more specific information about the Importers, please have a look at the README and YARD Docs.

Let’s take a simple example of the JSON Importer, to import from GitHub’s GraphQL API response. By default, the API response is paginated and 30 repositories are listed in the url : https://api.github.com/users/#{username}/repos.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

require 'daru/io/importers/json'

dataframe = %w[athityakumar zverok v0dro lokeshh].map do |username|
  Daru::DataFrame.read_json(
    "https://api.github.com/users/#{username}/repos",
    RepositoryName: '$..full_name',
    Stars: '$..stargazers_count',
    Size: '$..size',
    Forks: '$..forks_count'
  )
end.reduce(:concat)

#=> #<Daru::DataFrame(120x4)>
#      Repository   Stars   Size   Forks
#   0  athityakum       0      6       0
#   1  athityakum       0    212       0
#   2  athityakum       0    112       0
#  ...    ...          ...   ...     ...

“When working with a team of Pythonistas and R developers;

daru-io helps convert Daru::DataFrame to multiple formats.

Does this in daru-io seem ambitious?

The second part of daru-io is the collection of Exporters that ship with it. Exporters are used to write the data in a DataFrame, to a file / database. These are the Exporters being supported by v0.1.0 of daru-io :

• General file formats : CSV, Excel (xls), JSON.
• Special file formats : Avro, RData, RDS.
• Database related : SQL.

For more specific information about the Exporters, please have a look at the README and YARD Docs.

Let’s take a simple example of the RDS Exporter. Say, your best friend is a R developer who’d like to analyze a Daru::DataFrame that you have obtained, and perform further analysis. You don’t want to break your friendship, and your friend is skeptical of learning Ruby. No issues, simply use the RDS Exporter to export your Daru::DataFrame into a .rds file, which can be easily loaded by your friend in R.

1
2
3
4
5
6
7
8
9
10
11
12

require 'daru/io/exporters/rds'

dataframe #! Say, the DataFrame is obtained from the above JSON Importer example

#=> #<Daru::DataFrame(120x4)>
#      Repository   Stars   Size   Forks
#   0  athityakum       0      6       0
#   1  athityakum       0    212       0
#   2  athityakum       0    112       0
#  ...    ...          ...   ...     ...

dataframe.write_rds('github_api.rds', 'github.api.dataframe')

“You all did see that in the repository’s README;

Codeclimate presented a 4.0 GPA;

Code and tests were humbly cleaned;

with help of rubocop, rspec, rubocop-rspec and saharspec.

Ambition shouldn’t have been made of humble stuff.

Yet some might think it is an ambitious idea;

And sure, they are all honourable men.”

Thanks to guidance from my mentors Victor Shepelev, Sameer Deshmukh and Lokesh Sharma, I’ve come to know about quite a lot of Ruby tools that could be used to keep the codebase sane and clean.

• rubocop : A Ruby static code analyzer, which enforces specified Ruby style guidelines.
• rspec : A unit-testing framework, which makes sure that codes of block are doing what they’re logically supposed to do.
• rubocop-rspec : A plugin gem to rubocop, that extends rspec-related rules.
• saharspec : A gem with a punny name, that extends a few features to rspec-its that are more readable. For example, its_call.

“I speak not to disapprove of what other libraries do;

But here I am to speak what I do know.

Give daru-io a try and y’all will love it, not without cause:

Does anything withhold you then, from using daru-io?”

I really mean it, when I discretely specify “I speak not to disapprove of what other libraries do”. In the world of open-source, there should never be hate among developers regarding languages, or libraries. Developers definitely have their (strong) opinions and preferences, and it’s understandable that difference in opinion do arise. But, as long as there’s mutual respect for each other’s opinion and choice, all is well.

“O Ruby community! Thou should definitely try out daru-io,

With daru and daru-view. Bear with me;

My heart is thankful to the community of Ruby Science Foundation,

And I must pause till I write another blog post.”

If you’ve read all the way till down here, I feel that you’d be interested in trying out the daru family, after having seen the impressive demonstration of Importers & Exporters above, and the Rails example (Website | Code). I’m very thankful to mentors Victor Shepelev, Sameer Deshmukh and Lokesh Sharma for their timely Pull Request reviews and open discussions regarding features. Daru-IO would not have been possible without them and the active community of Ruby Science Foundation, who provided their useful feedback(s) whenever they could. The community has been very supportive overall, and hence I’d definitely be interested to involve with SciRuby via more open-source projects.

• Convenient and efficient data wrangling for categorical data in Daru
• Visualization of categorical data
• Multiple linear regression and generalized linear models (GLM) with categorical variables in Statsample and Statsample-GLM

Lets talk about each of them in detail.

Analyzing catgorical data with Daru

Categorical data is now readily recognized by Daru and Daru has all the necessary procedures for dealing with it.

To analyze categorical variable, simply turn the numerical vector to categorical and you are ready to go.

We will use, for demonstration purposes, animal shelter data taken from the Kaggle Competition. It is stored in shelter_data.

1
2
3
4
5
6
7
8
9

# Tell Daru which variables are categorical
shelter_data.to_category 'OutcomeType', 'AnimalType', 'SexuponOutcome', 'Breed', 'Color'

# Or quantize a numerical variable to categorical
shelter_data['AgeuponOutcome'] = shelter_data['AgeuponOutcome(Weeks)'].cut [0, 1, 4, 52, 260, 1500],
    labels: [:less_than_week, :less_than_month, :less_than_year, :one_to_five_years, :more_than__five_years]

# Do your operations on categorical data
shelter_data['AgeuponOutcome'].frequencies.sort ascending: false

1
2
3
4
5
6
7
8
9

small['Breed'].categories.size
#=> 1380
# Merge infrequent categories to make data analysis easy
other_cats = shelter_data['Breed'].categories.select { |i| shelter_data['Breed'].count(i) < 10 }
other_cats_hash = other_cats.zip(['other']*other_cats.size).to_h
shelter_data['Breed'].rename_categories other_cats_hash
shelter_data['Breed'].frequencies
# View the data
small['Breed'].frequencies.sort(ascending: false).head(10)

Please refer to this blog post to know more.

Visualizing categorical data

With the help of Nyaplot, GnuplotRB and Gruff, Daru now provides ability to visualize categorical data as it does with numerical data.

To plot a vector with Nyaplot one needs to call the function #plot.

1
2
3
4
5
6
7

# dv is a caetgorical vector
dv = Daru::Vector.new ['III']*10 + ['II']*5 + ['I']*5, type: :category, categories: ['I', 'II', 'III']

dv.plot(type: :bar, method: :fraction) do |p, d|
  p.x_label 'Categories'
  p.y_label 'Fraction'
end

Given a dataframe, one can plot the scatter plot such that the points color, shape and size can be varied acording to a categorical variable.

1
2
3
4
5
6
7
8
9
10
11
12

# df is a dataframe with categorical variable :c
df = Daru::DataFrame.new({
  a: [1, 2, 4, -2, 5, 23, 0],
  b: [3, 1, 3, -6, 2, 1, 0],
  c: ['I', 'II', 'I', 'III', 'I', 'III', 'II']
  })
df.to_category :c

df.plot(type: :scatter, x: :a, y: :b, categorized: {by: :c, method: :color}) do |p, d|
  p.xrange [-10, 10]
  p.yrange [-10, 10]
end

In a similar manner Gnuplot and Gruff also support plotting of categorical variables.

An additional work I did was to add Gruff with Daru. Now one can plot vectors and dataframes also using Gruff.

See more notebooks on visualizing categorical data with Daru here.

Regression with categorical data

Now categorical data is supported in multiple linear regression and generalized linear models (GLM) in Statsample and Statsample-GLM.

A new formula language (like that used in R or Patsy) has been introduced to ease the task of specifying regressions.

Now there’s no need to manually create a dataframe for regression.

1
2
3
4
5
6
7

require 'statsample-glm'

formula = 'OutcomeType_Adoption~AnimalType+Breed+AgeuponOutcome(Weeks)+Color+SexuponOutcome'
glm_adoption = Statsample::GLM::Regression.new formula, train, :logistic
glm_adoption.model.coefficients :hash

#=> {:AnimalType_Cat=>0.8376443692275163, :"Breed_Pit Bull Mix"=>0.28200753488859803, :"Breed_German Shepherd Mix"=>1.0518504638731023, :"Breed_Chihuahua Shorthair Mix"=>1.1960242033878856, :"Breed_Labrador Retriever Mix"=>0.445803000000512, :"Breed_Domestic Longhair Mix"=>1.898703165797653, :"Breed_Siamese Mix"=>1.5248210169271197, :"Breed_Domestic Medium Hair Mix"=>-0.19504965010288533, :Breed_other=>0.7895601504638325, :"Color_Blue/White"=>0.3748263925801828, :Color_Tan=>0.11356334165122918, :"Color_Black/Tan"=>-2.6507089126322114, :"Color_Blue Tabby"=>0.5234717706465536, :"Color_Brown Tabby"=>0.9046099720184905, :Color_White=>0.07739310267363662, :Color_Black=>0.859906249787038, :Color_Brown=>-0.003740755055106689, :"Color_Orange Tabby/White"=>0.2336674067343927, :"Color_Black/White"=>0.22564205490196415, :"Color_Brown Brindle/White"=>-0.6744314269278774, :"Color_Orange Tabby"=>2.063785952843677, :"Color_Chocolate/White"=>0.6417921901449108, :Color_Blue=>-2.1969040091451704, :Color_Calico=>-0.08386525532631824, :"Color_Brown/Black"=>0.35936722899161305, :Color_Tricolor=>-0.11440457799048752, :"Color_White/Black"=>-2.3593561796090383, :Color_Tortie=>-0.4325130799770577, :"Color_Tan/White"=>0.09637439333330515, :"Color_Brown Tabby/White"=>0.12304448360566177, :"Color_White/Brown"=>0.5867441296328475, :Color_other=>0.08821407092892847, :"SexuponOutcome_Spayed Female"=>0.32626712478395975, :"SexuponOutcome_Intact Male"=>-3.971505056680895, :"SexuponOutcome_Intact Female"=>-3.619095491410668, :SexuponOutcome_Unknown=>-102.73807712615843, :"AgeuponOutcome(Weeks)"=>-0.006959545305620043}

Additionally, through the work of Alexej Grossmann, one can also predict on new data using the model.

1
2
3

predict = glm_adoption.predict test
predict.map! { |i| i < 0.5 ? 0 : 1 }
predict.head 5

This, I believe, makes Statsample-GLM very convenient to use.

See this for a complete example.

Other

In addition to the aforementioned, there are some other considerable changes:

• Improving overall structure of indexing in Daru and adding more capabilities. See this and this.
• CategoricalIndex to handle the case when index column is a categorical data. More about it here.
• Improving missing value API in Daru. Read more about it here.
• Configuring guard to enable automatic testing. More info here.

Documentation

You can read about all my work in detail here.. Additionally, my project page can be found here.

I hope with these additions one will be able to see data more clearly with Daru.

There are installation instructions in the readme and there are already some sample Kernel files in the spec folder to try out various routines on. Apart from cloning the repository, one additional thing you will need to do is download the SPICE toolkit. You may want to keep the entire compressed file for later, but you’ll only need the cspice.a file in the lib/ subdirectory. After this follow the instructions in the readme and you should be good to go. (Be sure to run bundle install to install any dependencies that you don’t already have.)

After you’re done compiling and installing, run rake pry in the gem root directory.

If you remember, almost any useful task involving the SPICE Toolkit is preceded by loading data through kernel files. The relevant routine to do this is called furnsh_c() , and the most direct way to access it through Ruby is by calling the function SpiceRub::Native.furnsh. (However, this is not recommended because SpiceRub has a specific Ruby class unifying all the kernel related methods, and also because SPICE maintains its own internal variables for both tracking loaded kernel files and unloading them.)

Below is a crude dependency sequence:

1
2
3
4
5

SpiceRub::KernelPool.load ->  SpiceRub::Native.furnsh -> furnsh_c

Ruby Interface            ->  C Accessor              -> External library

Ruby                      ->  Native Ruby-C API       -> Native 3rd Party C

That’s the basic design of the wrapper, so here are a few simple examples on using the Kernel API. => denotes the interpreted output of pry.

First of all, the main KernelPool class is a singleton class, that means it can only be instantiated with the #instance method and the usual #new is private. Any subsequent calls to #instance will produce the same object.

1
2
3
4
5
6
7
8
9
10
11
12

kernel_pool = SpiceRub::KernelPool.instance
=> #<SpiceRub::KernelPool:0x00000002db3218>

rogue_kernel_pool = SpiceRub::KernelPool.new
NoMethodError: private method `new' called for SpiceRub::KernelPool:Class
from (pry):2:in `__pry__'

rogue_kernel_pool = SpiceRub::KernelPool.instance
=> #<SpiceRub::KernelPool:0x00000002db3218>

rogue_kernel_pool == kernel_pool
=> true

This is to make sure there is only one KernelPool state being maintained at a time. Now we have a bunch of kernel files in the spec/data/kernels folder which we’ll use for this example. There is an accessor attribute called @path which can be used to point to a particular folder.

1

kernel_pool.path = 'spec/data/kernels'

From here on it’s required that you’re running pry or irb from the gem root folder in order for the paths to match in these examples.

Now let’s load a couple of kernel files, you can type system("ls", kernel_pool.path) into your console to get a list of all the test kernels available in that folder. The KernelPool object has a #load method to load kernel files. If the variable path is set, then you only need to enter the file name, otherwise the entire path needs to be provided. An integer denoting the index of the kernel in the pool is returned if the load is successful.

1
2

kernel_pool.load("naif0011.tls")
=> 0

Note that this is the same as providing the full relative path of spec/data/kernels/naif0011.tls when the path variable is not set or nil.

Let’s load two more files:-

1
2
3
4
5
6
7
8
9
10
11
12
13

kernel_pool.load("moon_pa_de421_1900-2050.bpc")
=> 1

kernel_pool.load("de405_1960_2020.bsp")
=> 2

kernel_pool
=> #<SpiceRub::KernelPool:0x00000002db3218
 @path="spec/data/kernels",
 @pool=
  [#<SpiceRub::KernelPool::SpiceKernel:0x0000000270fab0 @loaded=true, @path_to="spec/data/kernels/naif0011.tls">,
   #<SpiceRub::KernelPool::SpiceKernel:0x000000026bd698 @loaded=true, @path_to="spec/data/kernels/moon_pa_de421_1900-2050.bpc">,
   #<SpiceRub::KernelPool::SpiceKernel:0x000000024ce698 @loaded=true, @path_to="spec/data/kernels/de405_1960_2020.bsp">]>

So now if you try to view the @pool member of kernel_pool, you’ll find three SpiceKernel objects with @loaded=true and their respective file paths. There isn’t much to do with a SpiceKernel object except unload it, or check its state. Note that you can only load kernel files into a KernelPool object and unload them via the SpiceKernel object.

You can access the kernel pool by calling the #[] operator and using the index that was returned on load:

1
2
3
4
5
6
7
8
9
10
11

kernel_pool[0]
=> #<SpiceRub::KernelPool::SpiceKernel:0x0000000270fab0 @loaded=true, @path_to="spec/data/kernels/naif0011.tls">

kernel_pool.count
=> 3

kernel_pool[0].unload!
=> true

kernel_pool.count
=> 2

So here we unload the first kernel and note that the count drops to 2. If you look up kernel_pool[0], you’ll find that the kernel is still present — but its @loaded variable has been set to false, which means that it has been removed from the CSPICE internal kernel pool.

1
2

kernel_pool[0]
=> #<SpiceRub::KernelPool::SpiceKernel:0x0000000270fab0 @loaded=false, @path_to="spec/data/kernels/naif0011.tls">

To unload all kernels simultaneously and delete the kernel pool, use #clear!

1
2
3
4
5
6
7
8

kernel_pool.clear!
=> true

kernel_pool.empty?
=> true

kernel_pool.pool
=> []

And that about wraps up this blog post on basic kernel handling. Since we know how to load data but not use it yet, I’ll cover that and the various kernel types in the next blog post. Thank you for reading.

• Target: The body of interest
• Frame: A rotational frame of reference (Default is J2000 [Not to be confused with the J2000 epoch])
• Observer: An observing body whose viewpoint is used to chart the vector
• Epoch : An epoch in Ephemeris Time

SPICE has an integer-key convention for the kind of bodies that it has support for. Each body can be referenced via a string or an integer id. While there isn’t an actual strict range for integer ID classification, it is mentioned here and can be summed up in the following if and elsif clauses. (In Ruby constant strings are better off as symbols, so the constructor takes either an integer ID of a string symbol)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  if body_id > 2000000
    :asteroid
  elsif body_id > 1000000
    :comet
  elsif body_id > 1000
    :body
  elsif body_id > 10
    body_id % 100 == 99 ?
      :planet : :satellite
  elsif body_id == 10
    :star
  elsif body_id > 0
    :barycenter
  elsif body_id > -1000
    :spacecraft
  elsif body_id > -100000
    :instrument
  else
    :spacecraft

It was very tempting to involve inheritance and extend a base Body class onto these potential classes, but I simply did not see the need for it at this point. The way it is at the moment, the Body object has a reader attribute type that stores some metadata about the body for the user’s convenience. Perhaps as coverage of SPICE improves, this minor thing can be changed later on.

To create a Body object, you instantiate with either a body name or a body id. Certain bodies such as instruments will require additional kernels to be loaded. To proceed seamlessly, load a leap seconds kernel, a planetary constants kernel, and an ephemeris kernel. (All avaialable in spec/data/kernels)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

SpiceRub::Body.new(399)
=> #<SpiceRub::Body:0x00000002769da8
     @code=399,
     @frame=:J2000,
     @name=:earth,
     @type=:planet>

SpiceRub::Body.new(:earth)
=> #<SpiceRub::Body:0x000000026c73c8
   @code=399,
   @frame=:J2000,
   @name=:earth,
   @type=:planet>

SpiceRub::Body.new(:moon)
=> #<SpiceRub::Body:0x0000000214ac88
   @code=301,
   @frame=:J2000,
   @name=:moon,
   @type=:satellite>

399 and :earth map to the same body in SPICE data. The frame of reference can also be specified as a named parameter during instantiation to set a custom default frame for that particular object.

1
2
3
4
5
6

SpiceRub::Body.new(399, frame: IAU_EARTH)
=> #<SpiceRub::Body:0x000000020b1df8
     @code=399,
     @frame=:IAU_EARTH,
     @name=:earth,
     @type=:planet>

In SPICE, a state is a 6 length column vector that stores position and velocity in 3D cartesian co-ordinates

As a base case, let’s find out the the position of the Earth with respect to itself.

1
2
3
4
5

earth.position_at(SpiceRub::Time.now, observer: earth)
=>
[
  [0.0]   [0.0]   [0.0]
]

The origin as seen from itself is still the origin, so this makes sense. The methods #velocity_at and #state_at take an identical set of parameters. While there is a bit of redundancy going on, splitting them makes the API more elegant, but the basic relationship between these three vectors is the following :-

1
2
3
4

state = [
          position[0],position[1],position[2],
          velocity[0],velocity[1],velocity[2]
        ]

One thing to note is that state/velocity/position vectors will always be returned as an NMatrix object, SciRuby’s numerical matrix core, to allow for calculations via the NMatrix API.

As an example that is used in the code, one line can turn a position vector into distance from origin (here using Euclidean distance):

1
2

  position = earth.position_at(SpiceRub::Time.now, observer: observer)
  Math.sqrt( (position ** 2).sum[0] )

As a simple imprecise experiment, let’s find out how the speed of light can be “estimated” up with this data.

1
2
3
4
5
6
7
8

distance = moon.distance_from(earth, now)
=> 367441.0260814745

time = moon.light_time_from(earth, now)
=> 1.2256513340354764

distance / time
=> 299792.458

The unit of distance here is kilometers, so the speed of light by this measurement is about pretty close to the textbook figure of 3e+8 m/s.

There is also a function to check if a list of bodies are within a radial proximity from an observing body. We already calculated the distance of the moon to be about 367,000 km. The function within_proximity returns a list of all bodies that are within the specified radial distance from the calling body object.

1
2
3
4
5
6
7
8

#assuming venus and mercury are instantiated

earth.within_proximity([moon, venus, mercury], 400000, now)
=> [#<SpiceRub::Body:0x0000000191c4f8
  @code=301,
  @frame=:J2000,
  @name=:moon,
  @type=:satellite>]

Now that we’ve come to the end of the functionality, I would like to mention that there is another named argument aberration_correction which is basically an error reduction method to provide a more accurate result than the default observation. The default :none option for aberration correction basically provides the geometric observations without any corrections for reception or transmission of photons. For a list of various aberration correction methods available, have a look at the documentation for spkpos_c to find out if you need an aberration correction on SPICE data.

1
2
3
4
5
6
7

d1 = moon.distance_from(earth, SpiceRub::Time.now, aberration_correction: :none)
=> 369111.0550333138
d2 = moon.distance_from(earth, SpiceRub::Time.now, aberration_correction: :LT)
=> 369146.60640691273

d2 - d1
=> 35.55137359892251

If you want to look at it another way, no aberration correction would give you the textbook response of rigid geometry, while introducing an aberration correction would give you a somewhat more realistic output accounting for the errors that do happen when these observations are made.

Finally, if you need to generate a continuous time series for a body, then SpiceRub::Time has two functions to aid in that:

1
2
3
4
5
6
7

SpiceRub::Time.linear_time_series(now, now + 86400, 4)
=> [
    #<SpiceRub::Time:0x00000001fe8b60 @et=525180780.18277323>,
    #<SpiceRub::Time:0x00000001fe8a20 @et=525209580.18277323>,
    #<SpiceRub::Time:0x00000001fe88b8 @et=525238380.18277323>,
    #<SpiceRub::Time:0x00000001fe8750 @et=525267180.18277323>
   ]

In this case, I took a start time and an end time that was one day after and requested 4 linearly spaced epochs. This is basically an interface to NMatrix.linspace.

The other function requires you to input a start time and an end time and a step size that keeps getting added to the start time till the end time is reached. As a contrived example, we’ll take two epochs, five days apart and ask for a step size of a day, expecting six elements.

1
2
3
4
5
6
7
8
9

SpiceRub::Time.time_series(now, now + 5 * 86400, step: 86400)
=> [
    #<SpiceRub::Time:0x00000001646580 @et=525180780.18277323>,
    #<SpiceRub::Time:0x00000002f315b8 @et=525267180.18277323>,
    #<SpiceRub::Time:0x00000002f31590 @et=525353580.18277323>,
    #<SpiceRub::Time:0x00000002f31568 @et=525439980.18277323>,
    #<SpiceRub::Time:0x00000002f31540 @et=525526380.18277323>,
    #<SpiceRub::Time:0x00000002f31518 @et=525612780.18277323>
   ]

And that’s it for this blog post. I would appreciate any feedback regarding this as I’ve been juggling the design back and forth very frequently. There is large potential of expansion of the Body class, particularly creating new classes as when different Bobdy objects would have a corresponding function. (For example, the function getfov_c which returns the field of view of an instrument could be an instance function attached to the Instrument subclass of Body, but this is just potential expansions in the future.)

But what happens when you accept the fact that you’re just a speck of micro-dust adjusting time relatively for an only slightly bigger speck of dust floating in the universe? Twenty-four hours in a day and thus we reset after 2300, but consider: how would a resident of Venus know when tea-time is on Venus if he had an Earth wristwatch that reset after twenty-four hours? Barely a tenth of Venus’ day is complete in that time! (If you know anybody intent on relocating to Mars, do not gift them a clock or watch.)

So a decimal floating point representation must be the answer for uniformity. Time zones can be dealt with; we’ll just pick a convenient point in time and count the seconds from there onwards so that the location on Earth doesn’t matter henceforth. It’ll drive humans insane with the arithmetic but machines will work just fine with this. This sort of a time system is called epoch time.

And so the internal time of most UNIX machines is the number of seconds after midnight on Thursday, 1 January 1970 UTC. (And this very convention is going to open a can of worms by 2038 if there is even a small set of critical machines that haven’t moved on from 32-bit architectures.)

But we’re still not okay universally. Try going on an infinite journey to space and you’ll find that counting seconds leads to some inconsistencies with your local time when you try to synchronize with Earth. How can the number of seconds after January 1970 be different in any case? Well, your MacBook Pro has not been adjusted for … relativity! Gravity bends light and thus the perception of time. There’s a lot more mass, and thus a lot more gravitatonal fields in neighborhoods away from Earth. The exact details of how this works is beyond the scope of this blog post.

If the past few paragraphs were incessant and seemingly irrelevant, they were there to drive home the point that Earth time simply will not do when we step out of the ghetto to see what’s happening. But astronomy’s been around for way longer, and astrophysicists came forth with a time system adjusted for the relativity effects of the solar system, called Barycentric Dynamical Time, or TDB. Like our machines, it counts the seconds after a certain known reference time point, except that it adjusts for relativity and can become a standard for astronomical time.

There are many similar time scales like this, but SPICE has chosen to use TDB as the standard for most of their design. Within the SPICE API, TDB is the same as Ephemeris Time which is the main system used to specify time points of astronomical bodies. Even though spacecrafts have their clocks coordinated with UTC on Earth, you would require that time in Ephemeris Time to be able to calculate their positions and velocities using SPICE. SpiceRub::Time is built for this very purpose, to revolve around a main attribute @et for Ephemeris Time and provide many methods to convert to and from.

If you’d like to proceed with the examples, you’ll need a Leap Second Kernel file to use SpiceRub::Time. This is a generic kernel, so you can easily use naif0011.tls in spec/data/kernels of the repository folder.

So Ephemeris Time is the number of seconds elapsed after Noon, January 1, 2000, TDB. This point in time is also known as the J2000 epoch. We find that out in an instant by using the Time.parse function which is a wrapper function for SPICE’s str2et_c that converts many formats of strings to Ephemeris Time. You can have a look at the various string formats supported in its documentation here

1
2

 SpiceRub::Time.parse("12:00 Jan 1 2000 TDB")
=> #<SpiceRub::Time:0x0000000325b1f8 @et=0.0>

So as a base case, using the reference epoch gives us 0 seconds as we would expect. Now would also be a good time to find out the discrepancy in UTC as well.

1
2

SpiceRub::Time.parse("12:00 Jan 1 2000 UTC")
=> #<SpiceRub::Time:0x000000031c28e0 @et=64.18392728473108>

So right away we know that UTC was 64-ish seconds off from TDB / ET at the time of the reference J2000 epoch. What would the difference be around right now?

1
2
3
4
5

now = SpiceRub::Time.now
=> #<SpiceRub::Time:0x000000030bf8f8 @et=525173312.1827749>

(now - now.to_utc).to_f
=> 68.18277490139008

Well, here’s a surprise, it’s 68.18 now. Before I explain why that is, here is a brief overview of what the above code does:

Time.now is a convenient way to specify your current UTC timezone. It uses Ruby’s core Time.now method so this method is only good if you’re working in UTC or Earth like Timezones. For a similar purpose, the function Time.from_time let’s you create a SpiceRub Time object from a Ruby Time object.

The +/- operators return a new Time object where the right operand is added/subtracted to the left operand’s @et when it is a float or integer. If a Time object is supplied , then it does the same with the right operand’s ephemeris time instead. (Note that there really isn’t a significant meaning to having a Time object whose @et is the difference/sum of two other epochs, however you can increase a certain epoch or decrease it by a constant offset of seconds)

In our case we used #to_utc to convert from ephemeris time to UTC, and then the minus operator gave us a Time object that wasn’t really an epoch, but a difference of two epochs, so using #to_f got us exactly that.

It appears that UTC has changed by 4 seconds since 2000 with respect to ephemeris time. This is actually the adjustment of “leap seconds” that gets added to UTC to prevent it from falling too far behind other time systems. (Humans really like to hack everything, don’t they?)

To verify this yourself, if you open up the kernel naif0011.tls in your text editor and search for DELTET/DELTA_AT, you’ll find a list like representation of the following sort :-

1
2
3
4
5
6
7

DELTET/DELTA_AT        = ( 10,   @1972-JAN-1
                           ..,   ...........
                           32,   @1999-JAN-1
                           33,   @2006-JAN-1
                           34,   @2009-JAN-1
                           35,   @2012-JUL-1
                           36,   @2015-JUL-1 )

Here you can see that just before the year 2000, there were 32 leap seconds added to UTC, and in 2015 when the last leap second was added, there were 36. It’s an ongoing and indefinite process and so there really is no way to account for leap second errors far in the future for leap seconds that are yet to be added. As of now, the next scheduled addition is in December, 2016.

Coming back to our Time object, let’s look at its basic construction. One tricky task in the API was the option to specify different epochs of reference in different time scales, like International Atomic Time. As of now, Time.new requires that you have kept your word of using the J2000 epoch and allows you to use a named parameter seconds: for specifying the time scale. The use of scale as a key was avoided as it sometimes is also used to refer to the reference epoch used.

1
2
3
4
5
6

epochs = [:utc, :tdb, :tai].map
  { |scale| SpiceRub::Time.new(0, seconds: scale) }

=> [#<SpiceRub::Time:0x00000002756fc8 @et=64.18392728466942>,
    #<SpiceRub::Time:0x00000002756eb0 @et=0>,
    #<SpiceRub::Time:0x00000002756cf8 @et=32.18392727400827>]

:tai here refers to International Atomic Time. For a list of more parameters and their keyword abbreviations, have a look at this SPICE documentation for the function that the conversion is wrapped on top of.

But there is also a way to reference other epochs without doing the manual conversions yourself, you can call the class method Time.at to perform the same function as the constructor, with the option of a different reference epoch. The resultant Time object will however have its internal time referring to J2000.

A more readable way would involve step by step calculations, but that would consume runtime resources everytime Time.at is called, so I’ve basically pre-calculated the ephemeris times of the reference epochs and subtracted them from the epoch.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

  case reference.downcase
  when :j2100
    new(offset + 3155760000.0, seconds: seconds)
  when :j2000, :et
    new(offset, seconds: seconds)
  when :j1950
    new(offset - 1577880000.0, seconds: seconds)
  when :j1900
    new(offset - 3155760000.0, seconds: seconds)
  when :gps
    new(offset - 630763148.8159368, seconds: seconds)
  when :unix
    new(offset - 946727958.8160644, seconds: seconds)
  end

To quickly verify the last one with the #to_s method:

1
2

SpiceRub::Time.new(-946727958.8160644).to_s
=> "Thu Jan 01 00:00:00 UTC 1970"

It’s exactly the UNIX epoch! Let’s try out 1 day (86400 seconds) after this epoch:

1
2

SpiceRub::Time.at(86400, :unix).to_s
=> "Thu Jan 01 23:59:59 UTC 1970"

Just a second short of heading into the next day, because we’ve added 86400 TDB seconds and converted the time into a UTC string.

There are some more functions provided to work in tandem with the Body class that I’ll explain more about in the next blog post, but this more or less covers the core of SpiceRub:Time. Till then, thanks for reading.

I worked on “Port NMatrix to JRuby” in the context of the Google Summer of Code (GSoC) 2016 and I am pleased to announce that NMatrix can now be used in JRuby.

With JRuby NMatrix, a linear algebra library, wraps Apache Commons Math for its most basic functionalities. NMatrix supports dense matrices containing either doubles or Ruby objects as the data type. The performance of JRuby with Apache Commons Maths is quite satisfactory (see below for performance comparisons) even without making use of JRuby threading capabilities.

I have also ported the mixed_models gem, which uses NMatrix heavily at its core, to JRuby. This gem allowed us to test NMatrix-JRuby with real-life data.

This blog post summarizes my work on the project with SciRuby, and reports the final status of the project.

The original GSoC proposal, plan and application can be found here. Until merging is complete, commits are available here.

Performance

I have benchmarked some of the NMatrix functionalities. The following plots compare the performance between NMatrix-JRuby, NMatrix-MRI, and NMatrix-MRI using LAPACK/ATLAS libraries. (Note: MRI refers to the reference implementation of Ruby, for those who are new.)

Notes:

1. LAPACK and ATLAS aren’t involved in most element-wise operations, such as addition and subtraction.
2. NMatrix-MRI relies on LAPACK/ATLAS for calculating determinants and LU Decomposition (lud).

Result

1. For two-dimensional matrices, NMatrix-JRuby is currently slower than NMatrix-MRI for matrix multiplication and matrix decomposition functionalities (calculating determinant and factoring a matrix). NMatrix-JRuby is faster than NMatrix-MRI for other functionalities of a two-dimensional matrix — like addition, subtraction, trigonometric operations, etc.

2. NMatrix-JRuby is a clear winner when we are working with matrices of arbitrary dimensions.

Implementation

Storing n-dimensional matrices as one-dimensional arrays

The major components of an NMatrix are shape, elements, dtype and stype. When initialized, the dense type stores the elements as a one-dimensional array; in the JRuby port, the ArrayRealVector class is used to store the elements.

@s stores elements, @shape stores the shape of the matrix, while @dtype and @stype store the data type and storage type respectively. Currently, I have nmatrix-jruby implemented only for :float64 (double) and Ruby :object data types.

NMatrix-MRI uses struct as a type to store dim, shape, offset, count, src of an NMatrix. ALLOC and xfree are used to wrap the NMatrix attributes to C structs and release the unrequired memory.

Slicing and Rank

Implementing slicing was the toughest part of NMatrix-JRuby implementation. NMatrix@s stores the elements of a matrix as a one-dimensional array. The elements along any dimension are accessed with the help of the stride. NMatrix#get_stride calculates the stride with the help of the dimension and shape and returns an Array.

1
2
3
4
5
6
7
8
9
10

def get_stride(nmatrix)
  stride = Array.new()
  (0...nmatrix.dim).each do |i|
    stride[i] = 1;
    (i+1...dim).each do |j|
      stride[i] *= nmatrix.shape[j]
    end
  end
  stride
end

NMatrix#[] and NMatrix#[]= are thus able to read and write the elements of a matrix. NMatrix#MRI uses the @s object which stores the stride when the nmatrix is initialized.

NMatrix#[] calls the #xslice operator which calls #get_slice operator that use the stride to determine whether we are accessing a single element or multiple elements. If there are multiple elements, #dense_storage_get returns an NMatrix object with the elements along the dimension.

NMatrix-MRI differs from NMatrix-JRuby implementation as it makes sure that memory is properly utilized as the memory needs to be properly garbage collected.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def xslice(args)
  result = nil

  if self.dim < args.length
    raise(ArgumentError,"wrong number of arguments\
       (#{args} for #{effective_dim(self)})")
  else
    result = Array.new()
    slice = get_slice(@dim, args, @shape)
    stride = get_stride(self)
    if slice[:single]
      if (@dtype == :object)
        result = @s[dense_storage_get(slice,stride)]
      else
        s = @s.toArray().to_a
        result = @s.getEntry(dense_storage_get(slice,stride))
      end
    else
      result = dense_storage_get(slice,stride)
    end
  end
  return result
end

NMatrix#[]= calls the #dense_storage_set operator which calls #get_slice operator that use the stride to find out whether we are accessing a single element or multiple elements. If there are multiple elements #set_slice recursively sets the elements of the matrix then returns an NMatrix object with the elements along the dimension.

All the relevant code for slicing can be found here.

Enumerators

NMatrix-MRI uses the C code for enumerating the elements of a matrix. Just as with slicing, the NMatrix-JRuby uses pure Ruby code in place of the C code. Currently, all the enumerators for dense matrices with real data-type have been implemented and are properly functional. Enumerators for objects have not yet been implemented.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

def each_with_indices
  nmatrix = create_dummy_nmatrix
  stride = get_stride(self)
  offset = 0
  #Create indices and initialize them to zero
  coords = Array.new(dim){ 0 }

  shape_copy =  Array.new(dim)
  (0...size).each do |k|
    dense_storage_coords(nmatrix, k, coords, stride, offset)
    slice_index = dense_storage_pos(coords,stride)
    ary = Array.new
    if (@dtype == :object)
      ary << self.s[slice_index]
    else
      ary << self.s.toArray.to_a[slice_index]
    end
    (0...dim).each do |p|
      ary << coords[p]
    end

    # yield the array which now consists of the value and the indices
    yield(ary)
  end if block_given?
  nmatrix.s = @s

  return nmatrix
 end

Two-Dimensional Matrices

Linear algebra is mostly about two-dimensional matrices. In NMatrix, when performing calculations in a two-dimensional matrix, a one-dimensional array is converted to a two-dimensional matrix. A two-dimensional matrix is stored in the JRuby implementation as a BlockRealMatrix or Array2DRowRealMatrix. Each has its own advantages.

Getting a 2D Matrix

1
2
3
4
5
6
7
8
9
10
11
12
13
14

public class MatrixGenerator
{
  public static double[][] getMatrixDouble(double[] array, int row, int col)
  {
    double[][] matrix = new double[row][col];
    for (int index=0, i=0; i < row ; i++){
        for (int j=0; j < col; j++){
            matrix[i][j]= array[index];
            index++;
        }
    }
    return matrix;
  }
}

Convert a 2D-matrix to 1D-array

1
2
3
4
5
6
7
8
9
10
11
12
13
14

public class ArrayGenerator
{
  public static double[] getArrayDouble(double[][] matrix, int row, int col)
  {
    double[] array = new double[row * col];
    for (int index=0, i=0; i < row ; i++){
        for (int j=0; j < col; j++){
            array[index] = matrix[i][j];
            index++;
        }
    }
    return array;
  }
}

Why use a Java method instead of Ruby method?

1. Memory Usage and Garbage Collection: A scientific library is memory intensive and hence, every step counts. The JRuby interpreter doesn’t need to dynamically guess the data type and uses less memory, typically around one-tenth of it. If the memory is properly utilized, when the GC kicks in, the GC has to clear less used memory space.

2. Speed: Using java method greatly improves the speed — by around 1000 times, when compared to using the Ruby method.

Operators

All the operators from NMatrix-MRI have been implemented except modulus. The binary operators were easily implemented through Commons Math API and Java Math API.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

def +(other)
  result = create_dummy_nmatrix
  if (other.is_a?(NMatrix))
    #check dimension
    raise(ShapeError, "Cannot add matrices with different dimension")\
    if (@dim != other.dim)
    #check shape
    (0...dim).each do |i|
      raise(ShapeError, "Cannot add matrices with different shapes") \
      if (@shape[i] != other.shape[i])
    end
    result.s = @s.copy.add(other.s)
  else
    result.s = @s.copy.mapAddToSelf(other)
  end
  result
end

Unary Operators (Trigonometric, Exponentiation and Log operators) have been implemented using #mapToSelf method that takes a Univariate function as an argument. #mapToSelf maps every element of ArrayRealVector object to the Univariate function, that is passed to it and returns self object.

1
2
3
4
5

def sin
  result = create_dummy_nmatrix
  result.s = @s.copy.mapToSelf(Sin.new())
  result
end

NMatrix#method(arg) has been implemented using bivariate functions provided by Commons Math API and Java Math API.

1
2
3
4
5

def gamma
  result = create_dummy_nmatrix
  result.s = ArrayRealVector.new MathHelper.gamma(@s.toArray)
  result
end

1
2
3
4
5
6
7
8
9
10
11
12
13

import org.apache.commons.math3.special.Gamma;

public class MathHelper{
  ...
  public static double[] gamma(double[] arr){
    double[] result = new double[arr.length];
    for(int i = 0; i< arr.length; i++){
      result[i] = Gamma.gamma(arr[i]);
    }
    return result;
  }
  ...
}

Decomposition

NMatrix-MRI relies on LAPACK and ATLAS for matrix decomposition and solving functionalities. Apache Commons Math provides a different set of API for decomposing a matrix and solving an equation. For example, #potrf and other LAPACK specific functions have not been implemented as they are not required at all.

Calculating determinant in NMatrix is tricky where a matrix is reduced either to a lower or upper matrix and the diagonal elements of the matrix are multiplied to get the result. Also, the correct sign of the result (whether positive or negative) is taken into account while calculating the determinant. However, NMatrix-JRuby uses Commons Math API to calculate the determinant.

1
2
3
4
5
6
7

def det_exact
  if (@dim != 2 || @shape[0] != @shape[1])
    raise(ShapeError, "matrices must be square to have a determinant defined")
    return nil
  end
  to_return = LUDecomposition.new(self.twoDMat).getDeterminant()
end

Given below is code that shows how Cholesky decomposition has been implemented by using Commons Math API.

Cholesky Decomposition

1
2
3
4
5
6
7
8
9
10
11
12
13

  def factorize_cholesky
    cholesky = CholeskyDecomposition.new(self.twoDMat)
    l = create_dummy_nmatrix
    twoDMat = cholesky.getL
    l.s = ArrayRealVector.new(ArrayGenerator.getArrayDouble\
        (twoDMat.getData, @shape[0], @shape[1]))

    u = create_dummy_nmatrix
    twoDMat = cholesky.getLT
    u.s = ArrayRealVector.new(ArrayGenerator.getArrayDouble\
      (twoDMat.getData, @shape[0], @shape[1]))
    return [u,l]
  end

Similarly, LU Decomposition and QR factorization have been implemented.

LU Decomposition

Code

QR Factorization

Code

NMatrix#solve

The solve method currently uses LU and Cholesky decomposition.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

  def solve(b, opts = {})
    raise(ShapeError, "Must be called on square matrix")\
       unless self.dim == 2 && self.shape[0] == self.shape[1]
    raise(ShapeError, "number of rows of b must equal number\
       of cols of self") if self.shape[1] != b.shape[0]
    raise(ArgumentError, "only works with dense matrices") if self.stype != :dense
    raise(ArgumentError, "only works for non-integer, non-object dtypes")\
       if integer_dtype? or object_dtype? or b.integer_dtype? or b.object_dtype?

    opts = { form: :general }.merge(opts)
    x    = b.clone
    n    = self.shape[0]
    nrhs = b.shape[1]

    nmatrix = create_dummy_nmatrix
    case opts[:form]
    when :general, :upper_tri, :upper_triangular, :lower_tri, :lower_triangular
      #LU solver
      solver = LUDecomposition.new(self.twoDMat).getSolver
      nmatrix.s = solver.solve(b.s)
      return nmatrix
    when :pos_def, :positive_definite
      solver = Choleskyecomposition.new(self.twoDMat).getSolver
      nmatrix.s = solver.solve(b.s)
      return nmatrix
    else
      raise(ArgumentError, "#{opts[:form]} is not a valid form option")
    end

  end

NMatrix#matrix_solve

Suppose we need to solve a system of linear equations:

                    AX = B

where A is an m×n matrix, B and X are n×p matrices, we need to solve this equation by iterating through B.

NMatrix-MRI implements this functionality using NMatrix::BLAS::cblas_trsm method. However, for NMatrix-JRuby, NMatrix#matrix_solve is the analogous method used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  def matrix_solve rhs
    if rhs.shape[1] > 1
      nmatrix = NMatrix.new :copy
      nmatrix.shape = rhs.shape
      res = []
      #Solve a matrix and store the vectors in a matrix
      (0...rhs.shape[1]).each do |i|
        res << self.solve(rhs.col(i)).s.toArray.to_a
      end
      #res is in col major format
      result = ArrayGenerator.getArrayColMajorDouble \
         res.to_java :double, rhs.shape[0], rhs.shape[1]
      nmatrix.s = ArrayRealVector.new result

      return nmatrix
    else
      return self.solve rhs
    end
  end

Currently, Hessenberg transformation for NMatrix-JRuby has not been implemented.

Other dtypes

I have tried implementing float dtypes using FloatMatrix class provide by jblas. jblas was used instead of Commons Math as the latter uses Field Elements for Floats and it had some issues with Reflection and Type Erasure. However, using jblas resulted in errors due to precision.

Code Organisation and Deployment

To minimise conflict with the MRI codebase all the JRuby front end code has been placed in the /lib/nmatrix/jruby directory. lib/nmatrix/nmatrix.rb decides whether to load nmatrix.so or nmatrix_jruby.rb after detecting the Ruby platform.

The added advantage is that the Ruby interpreter must not decide which function to call at run-time. The impact on performance can be seen when programs which intensively use NMatrix for linear algebraic computations (e.g., mixed_models) are run.

Spec Report

After the port; this is the final report that summarizes the number of tests that successfully pass:

NMatrix

Why do some tests fail?

1. Complex dtype has not been implemented.
2. Sparse matrices (list and yale) have not been implemented.
3. Decomposition methods that are specific to LAPACK and ATLAS have not been implemented.
4. Integer dtype is not properly assigned to floor, ceil, and round methods.

Mixed_Models

Future Work

NMatrix on JRuby offers comparable speeds to MRI. For specific computations it will be possible to leverage the threading support of JRuby and speed up things using multiple cores.

Adding new functionality to NMatrix-JRuby will be easy from here. Personally, I am interested to add OpenCL support to leverage the GPU computational capacity available on most machines today.

Conclusion

The main goal of this project was to to gain from the performance JRuby offers, and bring a unified interface for linear algebra between MRI and JRuby.

By the end of GSoC, I have been able to successfully create a linear algebra library, NMatrix for JRuby users, which they can easily run on their machines — unless they want to use complex numbers, at least for now.

I have mixed_models gem simultaneously ported to JRuby. Even here, NMatrix-JRuby is very close to NMatrix-MRI, considering the performance .

Acknowledgements

I would like to express my sincere gratitude to my mentor Pjotr Prins for the continuous support through the summers, and for his patience, motivation, enthusiasm, and immense knowledge. I could not have imagined having a better advisor and mentor, for this project.

I am very grateful to Google and the Ruby Science Foundation for this golden opportunity.

I am very thankful to Charles Nutter, John Woods, Sameer Deshmukh, Kenta Murata and Alexej Gossmann, who mentored me through the project. It has been a great learning experience.

I thank my fellow GSoC participants Rajith, Lokesh and Gaurav who helped me with certain aspects of my project.

Working Repository : GitHub Repo

List of Commits : List of commits to the above repository

Example iRuby Notebooks : A bunch of iRuby Notebook examples of the Ruby API

Firstly, I must admit that I was unable to meet all the goals of my proposal. In retrospect, perhaps it was a bit too ambitious for my skill level at the time, and I undoubtedly spent a lot of time learning, about Ruby-C extensions, the SPICE Toolkit, and good Ruby code and API.

General software requirements such as a shippable gem with an install script that downloads the correct binary dependencies and a dataset fetcher were among the targets in my proposal that I could not meet. If you’d like to read the full proposal, you can have a look at it here.

With that said, I am happy with what I did complete, and along with most of the functions in the proposal list being ported successfuly, I have added 3 Ruby classes to provide a better abstracted experience while using the Ephemerides subsystem of SPICE. Given the vastness of SPICE itself, this seems like meagre coverage, but it is a stepping stone that I (and I hope others) would build upon.

The SpiceRub API

Please follow the blog links below to read the posts concerning the Ruby classes, the posts basically demonstrate the simple API that can be used for various tasks involving ephemerides. It would help if you followed the order of the links below as they build on top of each other.

KernelPool :- A Singleton class to handle the loading and unloading of SPICE Data (Kernels)

Time :- A class that references ephemeris time and has flexible construction functions

Body :- A class that represents a body in space whose motion can be observed with respect to another body.

I spent majority of the post mid-term period on the latter two classes, while most of the time before was spent on writing C extensions to port the SPICE API to Ruby. (and learning good spec manners, something that was not that quick to grow on me but towards the end got ingrained into me)

I ran into a lot of bugs, learnt more about compiler flags and other building options, and it gave me surprisingly good exposure to low level language trivia for a high level Ruby project. Honestly, I can type SpiceRub faster than most words on my keyboard now =)

Future Road Map

Things that I’ll tackle (after a short break) include :-

1) Installation Integrity so that gem install does everything needed for installation.

2) Complete Documentation

3) The test coverage of SpiceRub::Time is currently lacking because I changed the API at the last minute, but as most of these functions wrap around Native functions that have already been tested, this won’t be too high a priority, but it will be done.

4) Kernel Fetcher Script : Having to crawl the web for relevant Kernels was a massive headache during this project, and adding something that directly refers to NAIF’s FTP servers would be a neat addition.

5) Expand the API : There is a lot of SPICE concept I am not well versed with, and there are a bunch of functions ported that would work very well with a better API. (Most immediate task I can see is making a better API for the Geometry Finder subsystem, of which many functions have already been ported in /ext/spice_rub/spice_geometry.c)

Acknowledgements

I’d like to thank my mentors for this project, Dr. John Woods, Shaun Stewart, and Victor Shepelev for their guidance and knowledge. John and Shaun with their expertise in Astro research and Victor’s vast knowledge about professional Ruby code has helped keep this whole ship together, and I’m looking forward to sailing it a few more journeys.

Also, John made NMatrix which is sort of the backbone for a lot of SpiceRub’s functionality.

Finally, I would like to thank Andrew Annex who wrote SpicePy, and Philip Rasch who wrote spiceminer , two Python wrappers for the SPICE Toolkit. SpicePy has a high port and test coverage (I lifted a lot of tests from here that weren’t available in the SPICE documentation and SpiceMiner had an OOP style API which I referred to while designing the Body class.

It’s been an incredible summer, thank you for reading :)

Linear mixed models

My GSoC project is the Ruby gem mixed_models. Mixed models are statistical models which predict the value of a response variable as a result of fixed and random effects. The gem in its current version can be used to fit statistical linear mixed models and perform statistical inference on the model parameters as well as to predict future observations. A number of tutorials/examples in IRuby notebook format are accessible from the mixed_models github repository.

Linear mixed models are implemented in the class LMM. The constructor method LMM#initialize provides a flexible model specification interface, where an arbitrary covariance structure of the random effects terms can be passed as a Proc or a block.

A convenient user-friendly interface to the basic model fitting algorithm is LMM#from_formula, which uses the formula language of the R mixed models package lme4 for model specification. With the #from_formula method, the user can conveniently fit models with categorical predictor variables, interaction fixed or random effects, as well as multiple crossed or nested random effects, all with just one line of code.

Examples are given in the sections below.

Implementation

The parameter estimation in LMM#initialize is largely based on the approach developed by the authors of the R mixed models package lme4, which is delineated in the lme4 vignette. I have tried to make the code of the model fitting algorithm in LMM#initialize easy to read, especially compared to the corresponding implementation in lme4.

The lme4 code is largely written in C++, which is integrated in R via the packages Rcpp and RcppEigen. It uses CHOLMOD code for various sparse matrix tricks, and it involves passing pointers to C++ object to R (and vice versa) many times, and passing different R environments from function to function. All this makes the lme4 code rather hard to read. Even Douglas Bates, the main developer of lme4, admits that “The end result is confusing (my fault entirely) and fragile”, because of all the utilized performance improvements. I have analyzed the lme4 code in three blog posts (part 1, part 2 and part 3) before starting to work on my gem mixed_models.

The method LMM#initialize is written in a more functional style, which makes the code shorter and (I find) easier to follow. All matrix calculations are performed using the gem nmatrix, which has a quite intuitive syntax and contributes to the overall code readability as well. The Ruby gem loses with respect to memory consumption and speed in comparison to lme4, because it is written in pure Ruby and does not utilize any sparse matrix tricks. However, for the same reasons the mixed_models code is much shorter and easier to read than lme4. Moreover, the linear mixed model formulation in mixed_models is a little bit more general, because it does not assume that the random effects covariance matrix is sparse. More about the implementation of LMM#initialize can be found in this blog post.

Other existing tools

Popular existing software packages for mixed models include the R package lme4 (which is arguably the standard software for linear mixed models), the R package nlme (an older package developed by the same author as lme4, still widely used), Python’s statmodels, and the Julia package MixedModels.jl.

Below, I give a couple of examples illustrating some of the capabilities of mixed_models and explore how it compares to the alternatives.

A usage example and discussion

As an example, we use data from the UCI machine learning repository, which originate from blog posts from various sources in 2010-2012, in order to model (the logarithm of) the number of comments that a blog post receives. The linear predictors are the text length, the log-transform of the average number of comments at the hosting website, the average number of trackbacks at the hosting website, and the parent blog posts. We assume a random effect on the number of comments due to the day of the week on which the blog post was published. In mixed_models this model can be fit with

1
2

model_fit = LMM.from_formula(formula: "log_comments ~ log_host_comments_avg + host_trackbacks_avg + length + has_parent_with_comments + (1 | day)",
                              data: blog_data)

and we can display some information about the estimated fixed effects with

1

puts model_fit.fix_ef_summary.inspect(24)

which produces the following output:

1
2
3
4
5
6

                                         coef                       sd                  z_score            WaldZ_p_value
           intercept       1.2847896684307731     0.030380582281933178        42.28983027737477                      0.0
   log_host_comments_avg        0.415586319225577     0.007848368759350875        52.95193586953086                      0.0
 host_trackbacks_avg     -0.07551588997745964     0.010915623834434068       -6.918146971979714    4.575895218295045e-12
              length   1.8245853808280765e-05    2.981631039432429e-06        6.119420400102211    9.391631916599863e-10
has_parent_with_comments      -0.4616662830553772      0.13936886611993773      -3.3125496095955715    0.0009244972814528296