dsp2017 Archives - Michał Łusiak

Weekly ML drop #11

I’ve become more and more interested in machine learning during last year. This is my way of collecting and sharing interesting reads on the topic I stumble upon. Those posts are published each Friday, they are divided into few categories and the format is constantly evolving. Last week I didn’t have time to prepare a weekly drop, so some “news” today may be a bit older.

News
In this part, I share interesting news from machine learning and artificial intelligence world. Those are mostly not very scientific articles about interesting applications, predictions and controversies that AI causes.

Data is the new oil
In this very interesting article in The Economist, author bring arguments how in the 21st century, data will be (and already is) the main resource to fuel the economy (compared to 20th-century oil)

Sent to Prison by a Software Program’s Secret Algorithm
A man charged with fleeing police in a car was sentence 6 years in prison. One of the information used against him was set of bar charts analysing risks and threats he’s posing to the society. It wasn’t the main proof, but it is a little bit unsettling.

The parts of America most susceptible to automation
Short article discussion which areas of US and why will be most hit by the advent of automation. There’s interesting map there also.

Harnessing automation for a future that works
In this article, on the other hand, the author claims we’re not quite there, and we’ll need close cooperation between people and machines to progress automation.

6 areas of AI and ML to watch closely
If you’re interested what’s really hot in the industry, this article lists 6 most interesting technology areas.

Learning materials
Here I’m sharing material for learning ML that I found useful – online courses, blogs, books etc. This is usually rather technical stuff.

Deep learning simplified
Series of short videos explaining basic concepts of Machine Learning. Recommended rather for total beginners.

This is it for today, thanks for reading. If you liked the post, let me know and please check other parts of the series.

Phoenix, Ecto and time zones

This is another part of my series on learning Elixir.

In the previous episode, I calculated the distance of flight. I will use it later for gathering statistics. Another metric, I would like to have is a time of flight. Usually, flight departure and arrival times are presented in local time. So it gets a bit tricky to get the time length when the journey spans across few time zones.

Elixir’s standard DateTime library doesn’t work with time zones very well. The Internet suggests I should use Timex.

But first I needed to make some changes into my database because up until now, I had only flight date. A few weeks ago I wrote a post, how to update your schema with migrations and this time I followed the same steps. Still, I haven’t figure out how to do it in a bit more automated manner.

defmodule Flightlog.Repo.Migrations.CreateFlight do
  use Ecto.Migration

  def change do
    alter table(:flights) do
      modify :arrival_date, Timex.Ecto.DateTimeWithTimezone
      modify :departure_date, Timex.Ecto.DateTimeWithTimezone
    end
    
  end
end

As you can see, I used specific Timex types. This works only with Postgre, and if you want to use timezones it needs one additional step. You’ll have to add custom type to your database:

CREATE TYPE datetimetz AS (
    dt timestamptz,
    tz varchar
);

You can read more about using Timex with Ecto on this documentation page.

I also updated my flight.ex model. It looks like that right now:

defmodule Flightlog.Flight do
  use Flightlog.Web, :model

  schema "flights" do
    field :departure_date, Timex.Ecto.DateTimeWithTimezone
    field :arrival_date, Timex.Ecto.DateTimeWithTimezone
    field :flight_number, :string
    field :plane_type, :string
    field :from, :string
    field :to, :string

    timestamps()
  end

  @doc """
  Builds a changeset based on the `struct` and `params`.
  """
  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:departure_date, :arrival_date, :flight_number, :plane_type, :from, :to])
    |> validate_required([:departure_date, :arrival_date, :flight_number, :plane_type, :from, :to])
  end
end

After that, I walked along the path that’s proven to work in the flight distance part. I added a new function in my math.ex library, making use of Timex diff function:

    def flightTime(earlier, later) do
        hours = Timex.diff(later, earlier, :hours)
        minutes = rem(Timex.diff(later, earlier, :minutes), 60)
        "#{hours}:#{minutes}"
    end

And I’m calling it from the view in another function, so it’s easily accessible from the template:

  def time(time1, time2) do
    Flightlog.Math.flightTime(time1, time2)
  end

And that was it. Although it took me few hours because I was struggling a bit with Timex types. I didn’t read carefully the documentation and for example missed the step with creating new Postgre type. Good lesson here, to look into docs carefully :)

The effect:

Screen Shot 2017-05-07 at 23.26.43.png

As you can see, I made some approach at formatting dates. Unfortunately, I didn’t manage to show them in local time. They’re in UTC in here. This will be next step most likely.

That’s all for today. Next week we’ll try to test this new module. In the meantime, check previous episodes. And if you’re interested in machine learning, look into my weekly link drop.

Weekly ML drop #10

Neuralink and Brain’s Magical Future
Tim Burton from Wait But Why wrote another lengthy article on one of the Elon Musk’s project. This time it’s about Neuralink, some sort of direct interface to the brain. Tim also tackled other AI-related topics, like Superintelligence, which is also a worthy read.

The Myth of Superhuman AI
Kevin Kelly, on the other hand, thinks that we’ll never get to superintelligence for those five reasons. Kevin was also mentioned in two weeks ago edition. I am currently reading his book and if you’re interested in future, it’s a must-read.

Software Predicts Cognitive Decline Using Brain Images
We talked several times about advancements in medicine, that Machine Learning brings. Image recognition algorithms are getting or surpassing human levels of detecting various threats to our health on diagnostic imagery. This article treats the topic of using a neural network to early detection of Alzheimer’s disease.

The first wave of Corporate AI is doomed to fail
The author of the article compares current wave of AI-advances to first booms of the internet and cloud computing, that failed miserably. Only after backing off and some advances those technologies hit a home run.

Waymo’s Self-driving cars will take first raiders
Alphabet’s self-driving cars company is going to run tests in Phoenix, including people outside’s of Google. People can sign up, and the rides will be for free. The goal is to check how people use and react to self-driving cars.

Learning materials
Here I’m sharing material for learning ML that I found useful – online courses, blogs, books etc. This is usually rather technical stuff.

Videos from ICLR conference has been published
International Conference on Learning Representations which took place in Toulon, France in late April, just published their video on Facebook.

Learning AI if you suck at math
This is a good article for total beginners, who not only do not have much experience in Machine Learning but also feel they’re lacking in math. It links to several good resources to jump up your algebra and calculus.

This is it for today, thanks for reading. If you liked the post, let me know and please check other parts of the series.

Weekly ML drop #9

We need tools to track AI impact on jobs market
An expert panel composed mainly of economists and computer scientists said in a new report, that The World needs a way to measure how technology impacts job market. As when we started measured our economies in the 1930s, which greatly improved government’s awareness of issues to address.

Series of articles on how AI is used in biggest tech companies
Backchannel throughout last year visited major tech companies to interview them, how they use Machina Learning and AI. Apple, Google and Facebook.

Machine learning will be a great help in detecting cancer
The article from Google’s research blog shows how assistance of machine learning algorithms will greatly improve detecting cancer.

Will democracy survive Big Data and Artificial Intelligence?
This is a longer read from Scientific American that analyses various impacts of Big Data and growth of Machine Learning will have on future societies. It also tries to answer the question what we should do now, to secure our future.

Learning materials
Here I’m sharing material for learning ML that I found useful – online courses, blogs, books etc. This is usually rather technical stuff.

Intel’s Deep Learning 102
Continuation of the webinar I linked last week. In this part, they’re taking an overview of more advanced topics, like convolutional neural networks and recurrent neural networks.

This is it for today, thanks for reading. If you liked the post, let me know and please check other parts of the series.

Doing math in Elixir – calculating Great Circle Distance

In last two parts, I was setting up and consuming quick API to get additional airport data. One of this information was exact airport location on the Earth. In this part, I’ll use that information to calculate the distance of the travel and show how you can use standard math library in Erlang VM.

Calculating distance of air travel based on location is not super hard, but it needs to accommodate the fact that Earth is not flat. We’ll use something called “Great-circle distance“, as that’s how planes fly. So when you take most commonly used Mercator projection of the map, the shortest path between two points won’t be a straight line. It’s gonna have more parabolic-like shape. It’s best visible if you pick longer flights. Just go to flightradar24.com and check any plane overflying the Atlantic. They seem to be all flying towards north first and then turn south as the flight progresses. In fact, they fly a straight line, but because Eart is a sphere (in simplified model), meridians are not really parallel to each other.

After this geography primer, let’s get to math. The formula for calculating great-circle distance looks like that:

$\Delta\sigma=\arccos\bigl(\sin\phi_1\cdot\sin\phi_2+\cos\phi_1\cdot\cos\phi_2\cdot\cos(\Delta\lambda)\bigr).$

Where $\phi _{1},\lambda _{1}$ and $\phi _{2},\lambda _{2}$ are latitude and longitude of two points on Earth. This formula takes in radians and outputs radians. The result is angle difference between two points. To get the distance in kilometres, we have to multiply it by the radius of the Earth, which in the metric system is around 6370kms.

$d=r\,\Delta \sigma .$

To the code! I changed my functions in view, so instead of returning separately latitude and longitude, they give back a pair of coordinates as a tuple.

https://gist.github.com/mlusiak/0c934302b100a8b0f8899010fdd8422b

Then I pass the tuples to newly created function distance, that calls newly created Flightlog.Math module. I put the module in /lib folder and it all worked by the first run!

https://gist.github.com/mlusiak/031d81a94c3302bb6be9068c8bc29345

:math is a reference to Erlang VMs math library. One thing that’s needs explanation is changing degrees into radians. Radian is a different mathematical representation of the angle. 1 radian means, that the length of the arc of the part of the circle, that this degree describes equals the length of the radius of that circle. So the full circle is slightly over 6 radians (2 * Pi), as a formula for the length of the circumference of the circle is 2 * Pi * radius.

That’s all for today. Next week we’ll try to test this new module. In the meantime, check previous episodes. And if you’re interested in machine learning, look into my weekly link drop.

Weekly ML drop #8

German retailer Otto allows algorithm to order their supplies
To shorter delivery times, Otto allows their Machine Learning based system to automatically resupply the stock, which led to shorter deliveries, fewer returns and less overall losses.

Nobody understands the Deep Learning
There are certain fields, where it is required, that algorithm used to find results must be explainable. Unfortunately, it’s close to impossible answer the question “what exactly caused this network to give this answer” given its complexity – hunders of layers with thousands of neurones. It also makes debugging and finding errors very hard.

AI can aquire biases against race, gender, etc.
There’s an old saying about computer systems: Garbage in, garbage out. It also works with ML systems. What kind of data we feed to learning algorithms, will impact how their models work. That’s why AI based on human generated data, can be not that democratic as some people promise.

Fast Drawing for everyone
This google post talks about the AutoDraw experiment, that figures out what you wanted to draw, and propose you a better representation of it. There are some limitations though. It won’t offer you a cat in similar shape, just bunch of predefined cats. And also a number of recognised objects is quite limited. Still, an interesting toy to play. And if you’re interested in science, behind it, there’s this article on research blog.

Google’s neural networks duel against each other
Most of today’s Machine Learning is so-called supervised learning. It does mean, that somebody needs to feed the algorithm with data, that’s supervised (for example labelled images). One overly simplified case could be that one network is generating cat pictures, and the other one is recognising cats and they get better by feeding data to each other.

Video
I pick one or two videos every now and then that touches an interesting subject in AI and ML field. Sometimes it’s more scientific and the other it’s about real life applications.

In the future, everything will be smart
In this short video Kevin Kelly, author of “The Inevitable“, talks how AI will be a commodity, as electricity became in XIX century and we’re on the brink of another revolution.

Learning materials
Here I’m sharing material for learning ML that I found useful – online courses, blogs, books etc. This is usually rather technical stuff.

Deep Learning 101 from Intel
This one hour webinar goes trough basic concepts of Deep Learning and how those type of algorithms perform on Intel’s stack.

This is it for today, thanks for reading. If you liked the post, let me know and please check other parts of the series.

Setting up quick API with F# and Azure Functions

As mentioned in my last week Elixir blog post, I produced some quick fake API based on Azure Functions. I thought it’s gonna take a couple of minutes, but it turned out to be a whole adventure in itself.

The creating of a function is a breeze.

Go to Portal, click big green “+” sign and search for “Function App”
Pick Function App published by Microsoft
Fill all the necessary fields like App name (must be globally unique) or location. For hosting plan I used “Consumption plan” which means, I pay only for the time that function is running. I also like to pin my stuff to the dashboard, so it’s easier to find.
It will take several minutes to deploy.
Now you can create your functions for the app. F# is hidden in small print just above “Create this function” button. So click “create your own custom function”.
Then with Language drop-down, pick “F#” and for Scenario – “API & Webhooks”. There should be on the F# function triggered by HTTP request. That’s the one you want for API.
You’ll get premade piece of code with a simple function that is triggered by HTTP POST with name object and responses “Hello “.

Then I started writing the logic I wanted. I made an array of hard coded airport data. I made the function to accept only GET requests (you can change it in function.json file). In code, I parse query strings and get the airport IATA code. If I have this airport in my array, I response 200 with JSON containing the data. Otherwise, I return 404. If there’s no parameter in the query string, function answers with 500.

It’s relatively simple and straightforward F# code. I just struggled a lot with debugging. The small editor on Azure doesn’t give you static analysis, nor type information and no squigglies. You need to run the function and check for compilation errors or runtime errors. There was also some weird scoping behaviour, that forced me to declare the Airports array within the function. Anyways, after 2hrs I had an API that did what I wanted. You can see the code below. It’s not bulletproof, but it does the job. And I got to play with Azure Functions a bit.

https://gist.github.com/mlusiak/5053aabbc1c6e76db082dc8daa952c81

If you want to read more about other types of F# Azure Functions, Mathias Brandewinder wrote recently two posts about timer and queue triggered functions.

That’s all for today. Tune in next week for another part. Also, check previous episodes. And if you’re interested in machine learning, look into my weekly link drop.

Calling an external API from Phoenix app

Last week I wrote about how to extend your Ecto model. I wanted to figure it out because I thought this week, I’ll add to mother other flight related data. Turns out, I did it in completely different way. Not sure if this is the correct approach, so feel free to bash mi in comments (in a constructive way!).

One of the data I have on my flights so far is a departure and destination airport IATA code. Those are 3 letters code, commonly used in civil aviation (SFO = San Francisco, CPH = Copenhagen etc.). I wanted to use some simple API to get more data based on this code. So I decided to write my own using Azure Functions and F#. It became a separate adventure. I’ll write another post about it.

When I had my API ready, I started figuring out, how to put it in Phoenix app. My first approach was a bit like you do stuff in ASP.NET MVC – adding those extra fields to the model, and trying to update them from within the controller. Unfortunately, I couldn’t make it work. I couldn’t even access fields of the model.

The solution I ended up doing, is putting all the web calls and logic for parsing them within the view file. It feels dirty for me. It would be very dirty in ASP.NET MVC. So my intuition is, that it’s not the best way to do it. But so far the only one I made working. After those changes my Flight view file looks like that:

https://gist.github.com/mlusiak/b00a756b891e701354cba66fae55bd78

And then I use those functions within the template that renders the view:

https://gist.github.com/mlusiak/ecb3acf26baaad25d32f78af89080319

Immediately you can see another problem with the code – for every information that I want to show, I’m making a separate web request. But it works, which is a huge improvement. Done is better than perfect.

Screenshot to prove that it does, what it should:

Screen Shot 2017-04-16 at 01.52.21.png

That’s all for today. Tune in next week for another part. Also, check previous episodes. And if you’re interested in machine learning, look into my weekly link drop.

Weekly ML drop #7

Exploring the mysteries of Go with AlphaGo and China’s top players
AlphaGo together with Chinese Go Association is bringing together the most talented Go players and computer scientist to explore deeper into the game. Turns out, that last year victory of AlphaGo over human player totally changed how humans play this game.

Who will pay insurance in the era of self-driving cars?
This article discusses impact self-driving cars may have on the insurance industry. As it is suggested currently by analysts, and some first legislation follows, car makers may be responsible for that cost.

Why deep learning is suddenly changing your life?
Just another article that discusses what exactly happened in recent years, that machine learning seems to be everywhere and changing everything. It also explains basic terms and methods used in the field.

AI based hedge fund created a new currency
The article, with a bit sensationalist title, discuss new fintech startup Numerai, that uses open market for AI algorithms to make its trading decisions and rewarding authors of the best algorithms.

How predictive AI will change shopping
The author of this texts brings up few examples how AI revolution impacts the retail world. Thanks to better data collections, connected devices and putting this data together, retail companies can offer better suggestions, increasing their profits.

Learning materials
Here I’m sharing material for learning ML that I found useful – online courses, blogs, books etc. This is usually rather technical stuff.

Artificial Intelligence and the Growing Importance of Soft Skills
Very interesting read about which skills are endangered and which not by AI. If you want to prepare for robots taking your job, that’s a good start.

Introduction to Machine Learning
This short video from Intel Nervana AI academy explains basic terms of Machine Learning and most popular types of it.

Stanford series on Natural Language Processing with Deep Learning
Another academic source of knowledge. I haven’t had time to dig into it, but quick skimming suggests, that’s another amazing 20hrs of the content of ML knowledge, this time targeted at NLP.

This is it for today, thanks for reading. If you liked the post, let me know and please check other parts of the series.

Extending your Ecto model

Last week I used Ecto models to quickly created the database and I was very surprised how all got generated for me. But the model I build was very simplistic, and now I need to extend it. I started working on first serious functionality for the project, and I’ll have to do some changes. For start, I’m just testing by adding one field.

I generated new migrations file in /priv/repo/migrations folder and a bit by trial and error I end up with file like that:

defmodule Flightlog.Repo.Migrations.CreateFlight do
  use Ecto.Migration

  def change do
    alter table(:flights) do
      add :plane_type, :string
    end

  end
end

And after running mix ecto.migrate, I actually got some results:

Michals-MBP:flightlog michal$ mix ecto.migrate
01:01:37.663 [info]  == Running Flightlog.Repo.Migrations.CreateFlight.change/0 forward
01:01:37.663 [info]  alter table flights
01:01:37.666 [info]  == Migrated in 0.0s

This worked for adding fields to the database but didn’t automagically update all the access layers. There’s probably some way to it, but this time I did it manually.

First I updated the views, for example added following into show.html.eex:

https://gist.github.com/mlusiak/0979cb46187ee75cd724190e09aa96c1

This solved the visuals but still didn’t work. The crucial were changes in the model:

  defmodule Flightlog.Flight do
  use Flightlog.Web, :model

  schema "flights" do
    field :date, Ecto.DateTime
    field :flight_number, :string
    field :plane_type, :string
    field :from, :string
    field :to, :string

    timestamps()
  end

  @doc """
  Builds a changeset based on the `struct` and `params`.
  """
  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:date, :flight_number, :plane_type, :from, :to])
    |> validate_required([:date, :flight_number, :plane_type, :from, :to])
  end
end

I added new field both in schema part and in changeset. Cast is responsible for things being updated. I didn’t have to add it to be validates as required.

So this was actually a bit tedious, but I’m probably missing something here. Hopefully I’ll figure it out by the time I’ll need to make bigger changes.

I also found this post, that’s deal with similar problem.

That’s all for today. Tune in next week for another part. Also, check previous episodes. And if you’re interested in machine learning, look into my weekly link drop.