Phoenix, Ecto and time zones

This is another part of my series on learning Elixir.

In the previous episode, I calculated the distance of flight. I will use it later for gathering statistics. Another metric, I would like to have is a time of flight. Usually, flight departure and arrival times are presented in local time. So it gets a bit tricky to get the time length when the journey spans across few time zones.

Elixir’s standard DateTime library doesn’t work with time zones very well. The Internet suggests I should use Timex.

But first I needed to make some changes into my database because up until now, I had only flight date. A few weeks ago I wrote a post, how to update your schema with migrations and this time I followed the same steps. Still, I haven’t figure out how to do it in a bit more automated manner.

defmodule Flightlog.Repo.Migrations.CreateFlight do
  use Ecto.Migration

  def change do
    alter table(:flights) do
      modify :arrival_date, Timex.Ecto.DateTimeWithTimezone
      modify :departure_date, Timex.Ecto.DateTimeWithTimezone
    end
    
  end
end

As you can see, I used specific Timex types. This works only with Postgre, and if you want to use timezones it needs one additional step. You’ll have to add custom type to your database:

CREATE TYPE datetimetz AS (
    dt timestamptz,
    tz varchar
);

You can read more about using Timex with Ecto on this documentation page.

I also updated my flight.ex model. It looks like that right now:

defmodule Flightlog.Flight do
  use Flightlog.Web, :model

  schema "flights" do
    field :departure_date, Timex.Ecto.DateTimeWithTimezone
    field :arrival_date, Timex.Ecto.DateTimeWithTimezone
    field :flight_number, :string
    field :plane_type, :string
    field :from, :string
    field :to, :string

    timestamps()
  end

  @doc """
  Builds a changeset based on the `struct` and `params`.
  """
  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:departure_date, :arrival_date, :flight_number, :plane_type, :from, :to])
    |> validate_required([:departure_date, :arrival_date, :flight_number, :plane_type, :from, :to])
  end
end

After that, I walked along the path that’s proven to work in the flight distance part. I added a new function in my math.ex library, making use of Timex diff function:

    def flightTime(earlier, later) do
        hours = Timex.diff(later, earlier, :hours)
        minutes = rem(Timex.diff(later, earlier, :minutes), 60)
        "#{hours}:#{minutes}"
    end

And I’m calling it from the view in another function, so it’s easily accessible from the template:

  def time(time1, time2) do
    Flightlog.Math.flightTime(time1, time2)
  end

And that was it. Although it took me few hours because I was struggling a bit with Timex types. I didn’t read carefully the documentation and for example missed the step with creating new Postgre type. Good lesson here, to look into docs carefully :)

The effect:

 

Screen Shot 2017-05-07 at 23.26.43.png

As you can see, I made some approach at formatting dates. Unfortunately, I didn’t manage to show them in local time. They’re in UTC in here. This will be next step most likely.

That’s all for today. Next week we’ll try to test this new module. In the meantime, check previous episodesAnd if you’re interested in machine learning, look into my weekly link drop.

Doing math in Elixir – calculating Great Circle Distance

In last two parts, I was setting up and consuming quick API to get additional airport data. One of this information was exact airport location on the Earth. In this part, I’ll use that information to calculate the distance of the travel and show how you can use standard math library in Erlang VM.

Calculating distance of air travel based on location is not super hard, but it needs to accommodate the fact that Earth is not flat. We’ll use something called “Great-circle distance“, as that’s how planes fly. So when you take most commonly used Mercator projection of the map, the shortest path between two points won’t be a straight line. It’s gonna have more parabolic-like shape. It’s best visible if you pick longer flights. Just go to flightradar24.com and check any plane overflying the Atlantic. They seem to be all flying towards north first and then turn south as the flight progresses. In fact, they fly a straight line, but because Eart is a sphere (in simplified model), meridians are not really parallel to each other.

After this geography primer, let’s get to math. The formula for calculating great-circle distance looks like that:

\Delta\sigma=\arccos\bigl(\sin\phi_1\cdot\sin\phi_2+\cos\phi_1\cdot\cos\phi_2\cdot\cos(\Delta\lambda)\bigr).

Where \phi _{1},\lambda _{1} and \phi _{2},\lambda _{2} are latitude and longitude of two points on Earth. This formula takes in radians and outputs radians. The result is angle difference between two points. To get the distance in kilometres, we have to multiply it by the radius of the Earth, which in the metric system is around 6370kms.

d=r\,\Delta \sigma .

To the code! I changed my functions in view, so instead of returning separately latitude and longitude, they give back a pair of coordinates as a tuple.

https://gist.github.com/mlusiak/0c934302b100a8b0f8899010fdd8422b

Then I pass the tuples to newly created function distance, that calls newly created Flightlog.Math module. I put the module in /lib folder and it all worked by the first run!

https://gist.github.com/mlusiak/031d81a94c3302bb6be9068c8bc29345

:math is a reference to Erlang VMs math library. One thing that’s needs explanation is changing degrees into radians. Radian is a different mathematical representation of the angle. 1 radian means, that the length of the arc of the part of the circle, that this degree describes equals the length of the radius of that circle. So the full circle is slightly over 6 radians (2 * Pi), as a formula for the length of the circumference of the circle is 2 * Pi * radius.

That’s all for today. Next week we’ll try to test this new module. In the meantime, check previous episodesAnd if you’re interested in machine learning, look into my weekly link drop.

Extending your Ecto model

Last week I used Ecto models to quickly created the database and I was very surprised how all got generated for me. But the model I build was very simplistic, and now I need to extend it. I started working on first serious functionality for the project, and I’ll have to do some changes. For start, I’m just testing by adding one field.

I generated new migrations file in /priv/repo/migrations folder and a bit by trial and error I end up with file like that:

defmodule Flightlog.Repo.Migrations.CreateFlight do
  use Ecto.Migration

  def change do
    alter table(:flights) do
      add :plane_type, :string
    end

  end
end

And after running mix ecto.migrate, I actually got some results:

Michals-MBP:flightlog michal$ mix ecto.migrate
01:01:37.663 [info]  == Running Flightlog.Repo.Migrations.CreateFlight.change/0 forward
01:01:37.663 [info]  alter table flights
01:01:37.666 [info]  == Migrated in 0.0s

This worked for adding fields to the database but didn’t automagically update all the access layers. There’s probably some way to it, but this time I did it manually.

First I updated the views, for example added following into show.html.eex:

https://gist.github.com/mlusiak/0979cb46187ee75cd724190e09aa96c1

This solved the visuals but still didn’t work. The crucial were changes in the model:

  defmodule Flightlog.Flight do
  use Flightlog.Web, :model

  schema "flights" do
    field :date, Ecto.DateTime
    field :flight_number, :string
    field :plane_type, :string
    field :from, :string
    field :to, :string

    timestamps()
  end

  @doc """
  Builds a changeset based on the `struct` and `params`.
  """
  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:date, :flight_number, :plane_type, :from, :to])
    |> validate_required([:date, :flight_number, :plane_type, :from, :to])
  end
end

I added new field both in schema part and in changeset. Cast is responsible for things being updated. I didn’t have to add it to be validates as required.

So this was actually a bit tedious, but I’m probably missing something here. Hopefully I’ll figure it out by the time I’ll need to make bigger changes.

I also found this post, that’s deal with similar problem.

That’s all for today. Tune in next week for another part. Also, check previous episodesAnd if you’re interested in machine learning, look into my weekly link drop.

Connecting to database with Ecto

In the last episode of my Elixir adventures, I messed with own-made Views and Controller to display hardcoded flight information for my FlightLog. This week I’ll try to show data based on the content of the database.

Phoenix doesn’t have built-in data access capabilities. But there’s awesome Ecto project, that’s sort of beefed up ORM. It reminds me Entity Framework (or good parts of it) in .NET or Rails. It supports multiple databases, although the default is Postgres. If you generated your Phoenix project by default and didn’t exclude Ecto, you should have everything you need to start. If no, refer to this guide

To add data access to FlightLog, I started the way, I would usually do in a web project. I installed the database (I went with Postgres), created some table, inserted some sample data. It was proven later, that it wasn’t a necessary step.

To build your first model, go to your project root and type for example:

$ mix phoenix.gen.html Flight flights date:datetime flight_number:string from:string to:string
* creating priv/repo/migrations/20150409213440_create_flight.exs
* creating web/models/flight.ex
* creating test/models/flight_test.exs
* creating web/controllers/flight_controller.ex
* creating web/templates/flight/edit.html.eex
* creating web/templates/flight/form.html.eex
* creating web/templates/flight/index.html.eex
* creating web/templates/flight/new.html.eex
* creating web/templates/flight/show.html.eex
* creating web/views/flight_view.ex
* creating test/controllers/flight_controller_test.exs

Add the resource to your browser scope in web/router.ex:

    resources "/flights", FlightController

and then update your repository by running migrations:

    $ mix ecto.migrate

As you can see, this gave me a lot of stuff – a migration, a controller, a controller test, a model, a model test, a view, and a number of templates. It also instructs as to add new Controller to the router. Let’s do that.

I also removed recent additions, because they served the same purpose. I initially though I’ll just edit them, but Ecto surprised me by doing everything for me. Updated route file looks like that:

defmodule Flightlog.Router do
  use Flightlog.Web, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", Flightlog do
    pipe_through :browser # Use the default browser stack

    resources "/flights", FlightController
  end

  # Other scopes may use custom stacks.
  # scope "/api", Flightlog do
  #   pipe_through :api
  # end
end


After that I run mix ecto.migrate which executed migration file:

defmodule Flightlog.Repo.Migrations.CreateFlight do
  use Ecto.Migration

  def change do
    create table(:flights) do
      add :date, :datetime
      add :flight_number, :string
      add :from, :string
      add :to, :string

      timestamps()
    end

  end
end


This created tables for me as defined in phoenix.gen.html. I noticed that database has some extra fields for basic journaling like inserted_at and updated_at. I logged into the database and added some data just to have something to show and fired up my project again:

mix phoenix.server

When you browse to the site, you’ll notice that not only I have a list of flights, but also buttons for actions like Edit, Delete or Add a new item. All generated for me and even with decent styling. That was a very nice surprise.

Screen Shot 2017-04-06 at 13.00.42.png

So first look at Ecto is very positively surprising. It created all the boilerplate code for me, but it is also a code that’s understandable and easy to edit. There seem not to be any underlying magic. If I need to change default behaviour I feel I wouldn’t have much problem doing that. We’ll see in future if practice will support those claims.

I haven’t covered any of the internals how Ecto works, so if you’re interested I suggest you peek into this guide and official documentation.

That’s all for today. Tune in next week for another part. Also, check previous episodesAnd if you’re interested in machine learning, look into my weekly link drop.

Adding new routes in Phoenix

This is another part of my series on learning Elixir.

Last week I mixed new Phoenix project and looked how different parts of framework fit together. This week I started adding my own routes and pages. I followed the official guide but altered it a bit to better fit my application.

Adding a Route
First things to change, when you’re adding new pages, is the router.ex file, that sits in the web folder in your project. All of the work today will be contained within web folder. To add a new route, simply add a new line within the existing scope. We’ll talk more about scopes in one of the future episodes.

  scope "/", Flightlog do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
    get "/flights", FlightsController, :index
  end

What the new Route says is, when browser hits /flights path, index action of FlightsController controller will be executed. Or in other words, this routes maps GET request for /flights to the index action of FlightsConroller. This won’t work because this controller doesn’t exist yet. Let’s fix it!

Adding a Controller
The Controller is very similar to the one I created last week. Nothing new here. We have index function that renders index.html view.

defmodule Flightlog.FlightsController do
  use Flightlog.Web, :controller

  def index(conn, _params) do
    render conn, "index.html"
  end
end


Adding a View and a Template
As you can see below, I added some code to the Template. I’m quite sure that it should be handled in View, but for now, I hard coded it this way. Below flights_view.ex and index.html.eex:

defmodule Flightlog.FlightsView do
  use Flightlog.Web, :view
end

https://gist.github.com/mlusiak/059a840dcf17493e50c215b54f2dcdc4

You can also see reference to the path /flights/1 that has not been declared. Let’s do it now.

Adding another Route, Controller and Template
I pasted below the code for all three files, marking changes in bold.

defmodule Flightlog.Router do
  use Flightlog.Web, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", Flightlog do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
    get "/flights", FlightsController, :index
    get "/flights/:id", FlightsController, :show
  end

  # Other scopes may use custom stacks.
  # scope "/api", Flightlog do
  #   pipe_through :api
  # end
end
defmodule Flightlog.FlightsController do
  use Flightlog.Web, :controller

  def index(conn, _params) do
    render conn, "index.html"
  end

  def show(conn, %{"id" => id}) do
    render conn, "show.html", id: id
  end
end

https://gist.github.com/mlusiak/91f2ab3d668f89d2f8056eb0582a32ca

There were no changes in flights_view.ex.

There are some important things happening here. In Router, I pattern match against the params passed into the show function so that the id variable is bound to the value we put in the :id position in the URL. For example, if our URL is http://localhost:4000/flights/1, the id variable would be bound to 1.

In Template, I again hard coded some logic. There are for sure better ways to do it, but it works for now. It’s worth mentioning, that id passed is a string! I debugged this for an hour while comparing to integer 1 and couldn’t figure out why it doesn’t work ;). Everyday you learn something new. And the effects:

http://localhost:4000/flights/1

Screen Shot 2017-03-21 at 23.51.09.png

http://localhost:4000/flights/2

Screen Shot 2017-03-21 at 23.51.29.png

As you can see it’s all rendered within parent template. I’m not gonna get bothered by this at this point.

If you want to read more about Templates, again official guide is fantastic. I also stumbled upon this very interesting blog post that explains that Templates are just functions.

Thanks for reading this another part of my adventures with Elixir and Phoenix. Come next week for more. And if you’re interested in machine learning, look into my weekly link drop. on that topic.

First look at Phoenix

This week I decided to push some things in the project. Instead of looking into the internals of Elixir and Erlang VM, let’s actually do something. My project for DSP competition is a web application, and most popular framework for web development in Elixir is called Phoenix. Phoenix use as it’s foundation Erlang’s HTTP server – Cowboy. I’m not gonna look into it right now, I don’t think it will be necessary at this point.

More important for now will be knowledge what is Plug. It is a specification for constructing modules, that can be composed together and configure how web applications behave. Modules and functions that are built to this specification are called Plugs. Plugs can be chained together to create pipelines and can handle nearly everything, i.e. authentication or parameter pre-processing. To learn more look into this guide.

Thanks to Plugs, Phoenix doesn’t have a need for a monolithic middleware. And you can customise every step of handling a request. Let’s look at the elements that handle a request up to rendering, step by step.

The Endpoint is responsible for handling request until the moment that routes take over. Which includes starting server, applying the configuration and also applying a set of plugs that are common for all requests.

The Router is responsible for a few things. First of all, it parses incoming requests and sends them to appropriate Controller and Action. You can also create Pipelines here, which are sequences of Plugs, that can be easily applied to routes. So on top of common plugs, you can add your own. For example, to differently handle requests that will be rendered in a browser, vs. API requests.

Controllers do what they usually do in MVC model. They provide actions, which may prepare data for views, render stuff (through Views) on screen or do redirects.

Views are presentation layer. They’re rendered based on Templates. Templates in Elixir are precompiled, which makes them very fast.


Code
Armed with those basics, let’s look at some code that’s generated when you start Phoenix project.

First what I did is mix new Phoenix project instead of previous one.

$ mix phoenix.new hello_phoenix

This caused some havoc in my repository but  I managed to clean it up. After this and few other steps, like setting up a database, I could start it up.

$ mix phoenix.server

After that I could navigate to http://localhost:4000/ to see my first Phoenix application:

phoenix after start.png

Let’s take a look, how default Router looks like:

defmodule Flightlog.Router do
  use Flightlog.Web, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", Flightlog do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
  end

  # Other scopes may use custom stacks.
  # scope "/api", Flightlog do
  #   pipe_through :api
  # end
end

You can notice declared separate pipelines for browser and API. The one for browsers has accept-headers for HTML, handles session and adds anti-forgery tokens. One for API is much simpler. Then inside of scope (which is out of the scope of this post :P) you can see this pipeline applied and route for “/” declared that will trigger PageController’s action named :index.

defmodule Flightlog.PageController do
  use Flightlog.Web, :controller

  def index(conn, _params) do
    render conn, "index.html"
  end
end


This renders a view based on template located in file index.html.eex.

Next week I’ll start working on the code and adjust the routes for my own needs.


More resources

If you interested into looking Phoenix yourself, their documentation is very good, especially the guides. I also watched this video from 2015 NDC Oslo by the creator of Phoenix, Chris McCord. It gave me good overview how things work in Phoenix.

Thanks for reading this another part of my adventures with Elixir and Phoenix. Come next week for more. And if you’re interested in machine learning, look into my weekly link drop.

Mixing a new Elixir

Wow, 1.5 years since last post passed quickly. Time to do something about it!

I’m taking part in Maciej Aniserowicz’sDaj się poznać” contest. “Get noticed” in English. It’s a competition for programmers aiming at promoting technical blogging and building open source projects. Throughout next 12 weeks, participants have to write at least 2 blog posts each week. And at least one of them should be related to the project they’re building and hosting on github.

I’ll be writing one technical post related to the project at the beginning of week, usually around Monday or Tuesday. On Fridays I’ll be posting news and links to interesting resources in area I’ve been exploring last few months – machine learning. I’ve been thinking for a long time about some way of gathering interesting use cases, scientific papers, presentations and online courses. This way it can be beneficial to others. First round on Friday.

My project won’t make world a better place, but hopefully I will learn something. My main goal is to play around with Elixir, Phoenix (web framework) and some frontend technology (haven’t decided yet if it’s gonna be React or Elm). I will be building clone of the flightdiary.net. It’s a web app, where you can log flights you’ve taken, and you get nice map and statistics.

screen-shot-2017-02-25-at-23-28-32

 

For start, I created repository on github, installed elixir and setup IDE with plugins. I’m using Visual Studio Code, as it became my go-to editor. I do all my F# and Javascript stuff there, and there’s also great plugin for Elixir written by Mat McLoughlin.

Then I initialised project with this simple command:

mix new flightlog

This created some files and initial project structure:

screen-shot-2017-02-26-at-00-23-48

We’ll look into how it all works in next episode.

As I’m new to Elixir, I’ll need some learning. The resources I’ve been using so far are:
This talk by Bryan Hunter
Rob’s Connery book
Gutek’s series of articles [Polish]

That’s it for today. See you on Friday for some ML news and next week for more Elixir!