Author: Adilet Maratov

BDL Accelerate Hackathon in Beirut, Lebanon

A month ago or so, some of my friends told me that there will be a hackathon in Beirut, Lebanon. The hackathon was hosted by Garage48 team and they offered three spots for Kyrgyzstan IT people, all expenses paid! I filled in the form without knowing what criteria they use for the selection. Later I asked Ragnar Sass, the co-founder of Garage48 about it and he told me that it was passion about technology and experience, well, I’m glad I was selected. So, the journey begins.

Day One

Our aircraft landed at 9 pm by the local time and we were kindly met by the SmallVille hotel worker and he took us to the hotel. In the morning we realized that the hackathon starts at 5:30 pm. That was awesome news and we went out to see what Beirut looks like. Well, in a few words, I found Beirut pretty nice city. Loved walking next to the beach and looking at the Mediterranean Sea. The only ouch was that there is a very huge difference between the poor and the rich. At the downtown, there are no people and if you see some, they look not like the poor. When you go further and go out of the downtown, Beirut changes gradually. The traffic of old cars and lots of motorcycles. People cross roads without looking at traffic lights and etc. I saw car stores of very expensive cars and a few of them in the downtown, and non of them in the sides of the city. That made me quite sad.

There were six of us having fun and traveling around. They were awesome guys and we saw lots of thing that day, because everyone had some ideas where to go next.

At 5:30 we were at the de Forum building and the hackathon started. We were listening to pitches that team leaders prepared. After that, the pitchers were in rush to find somebody to join their teams. If a team consists of less than four people, that team is broken and the members should join other teams. I joined Dr.Jones team. It’s team leader wanted to create a platform, very similar to Uber, but in the niche of medical care. I thought that it will be very good product. Other five guys joined us and we were a team of 7 members. It was very promising. After that every team discussed what they will do and what will be on the demo at the end of the day tree.

Day two

At 9 in the morning, I was in the de Forum building with my team and we started the project. Since I am a Python/Django developer, I was responsible for the backend. The other guy (forgotten his name) was responsible for the iPhone app. We platform connects doctors and tourists so that when tourists are far from the home, they can easily find some doctors around and get help. The system implies that the user uses it before the trip and adds some medical and surgical history. That would help the local doctors to know the patient better. Most of the backend was ready at that day and iPhone app needed some little fixes. When we went back to the hotel, we already finished almost everything.

This day was full of new connections and talks. It was my first trip connected with IT, and it was great. I was so glad to see that all the people around are eager to create something new using technology and help the sick. Well, the atmosphere was indescribable.

Well, the hackathon was not standalone by itself. It was a part of the greater event – BDL Accelerate 2016, The biggest conference on the Miditerrenian.  Steve Wozniak, Tony Fadell and many other speakers were there. There were about 20000 people on the conference from more than 50 countries. I’m glad I took participation in such a great event. Unfortunately, The Woz did not impress me. His speech was not something extraordinary. The only thing he was talking about was how he started using transistors. I could read about it on the Internet. I hoped his speech will be something like Steve Jobs’ speech.

Day three

This day was the most important for us – this day determines the winner of the hackathon. This day brought me a very good lesson – do not try to do things with production quality for hackathons, it will steal your time. My mistake was that I was trying to do everything like if I would do it for production. When I was done with backend part on my local machine, for some reasons, I decided to deploy the project on the server using gunicorn and nginx. That would be okay, but I did not have all the permission on that machine and problems came pretty fast. The right solution for this situation was to run the project on Django’s built-in server. It is not production ready, but it would certainly okay for the demo at the end of the day. I killed four hours to be up and running with nginx and gunicorn. Never do this, if there is a small timeframe and a few users of the backend.

When we were done with the project, the presentation began. It was 6 pm already and everybody presented their projects. The projects were awesome. I wish I took participation in all of them.

We did not win. I think that is because the investors always look not only how cool you project is, but also to whether the project is doable by the team that started it. And if not, they are not going to invest money to the project. I think our project was too huge for our project and it would take a couple of year to make this business profitable, which is not the goal of this hackathon.

The winner team developed Android application, that help people with face muscular problems. The app detects your face and shows you how to move your eyebrows and etc. This project is awesome. It is pretty easy to implement, help lots of people and it is possible to make money from it. So, I am not too surprised by the jury decision.

Conclusion

Lesson1: Never write or implement code in hackathon for production use. Do minimum movements to get up and running. Hack it, it is hackathon.

Lesson2: Make new connections. Do not be shy. Meet new people and talk to them, create new connections so that you have somebody after the event is done.

 

“A mind for numbers” book review

I’ve just read A mind for numbers by Barbara Oakley. I really liked the book and the way the information is presented there. In this post, I’m just going to provide a little summary on what this book is about. What I am writing here is just what I liked and had impression of.

States of mind. So the main idea of the book is that humans have two types of thinking – focused and diffused. The focused thinking implies that you intentionally think about some problem and try to solve it. Diffused thinking is opposite, you do not intentionally think about the problem and make no attempt to solve it.

These two states of mind always change each other and have their own pros and cons. For example, focused thinking is required in order to start solving a problem. You cannot solve a problem before you focus on it and intentionally try to solve it. But there are so many tough mathematical problems, that you cannot stay focused on it all the time and diffused thinking comes in.

In order to solve mathematical (and not only) problems, one should use both states of mind. The focused thinking collects information to process and does some processing on it too, but diffused thinking makes only processing using all the resources of the brain. It is important to start with focused thinking and and push it really hard, so that diffused state has required resources and can process it in the background.

Portions of information. When reading a technical book, always try to go through the whole chapter and read statements, look at the pictures and graphs, citations and other highlighted staff. This will help you to form portions of information so that when you start reading it from the beginning of the chapter, your mind can construct the full picture of what is going on.

When you are done with the chapter, always close the book and try to pull everything you can from your memory without looking into the book. It is proven that remembering works better not when you reread what you want to remember, but when you pull it from your memory. If you cannot do it for the first time, just reread it and try again. It works like a charm.

Procrastination. Habits are the actions that we used to do. They have three phases:

  1. Signal. It is when you receive a signal from the outer world.
  2. Action. This is the action you do after the signal is received.
  3. Compensation. (not about money) After the action is done, we feel good or bad, which determines whether we will continue to do this action next time.

The problem of procrastination is that the Compensation part is very pleasant. Lots of people scared of doing math and they even feel pain when thinking about it. What do you feel when thinking about your favorite youtube channel? I can guess, it is pleasure. That is why it is hard to do math and easy to procrastinate watching youtube videos or surfing the web after funny pics and etc.

In order to stop procrastinating, one should make compensation part more pleasant. Stating a goal to finish a task is a good way to increase compensation saying that you are done with a big part of the task. Changing the reaction to a signal is also helpful. For example, you’ve got a pic from your friend on WhatsApp and instead of chatting, you should keep working on the task. It is always good to turn off the Internet and sound on your phone while working.

Other tips. 

  • When you do not want to do something, but you know that it is important, just start doing it and you will see that it is easier than you thought.
  • When remembering something, always try to visualize it, because human’s brain works better with graphical staff than written staff.
  • If you think that you do not know about something because of someone else, you are mistaken. You are the only one who is responsible for your knowledge.
  • Tests and exams are good for you. As they make you to pull the information from you memory, what helps you to remember better.
  • Never give up 🙂

Central Authentication Service (CAS) implementation using Django microservices

Microservices are popular these days and a lot of companies tend to move from monolithic architecture to distributed ones.  I’ve just finished implementing CAS for Django microservices at BeSmart. I tried to find some tutorials on the subject, but didn’t find anything except for the official documentation on readthedocs.org, so I will try to make other devs’ lives a little bit easier.

Prerequisites

In this tutorial, I will be using Django framework for microservices. And then I will use django-mama-cas and django-cas-ng packages. So before beginning, make sure you go through the official documentation of the packages and get familiar with them.

django-mama-cas is CAS server package adapted to Django framework.

django-cas-ng is CAS client package adapted to Django framework.

The Central Authentication Service (CAS)

Before beginning, we need to understand what CAS is and what problems it solves.

The Wikipedia says: The Central Authentication Service (CAS) is a single sign-on protocol for the web. Its purpose is to permit a user to access multiple applications while providing their credentials (such as user id and password) only once. It also allows web applications to authenticate users without gaining access to a user’s security credentials, such as a password. The name CAS also refers to a software package that implements this protocol.

In other words, CAS consists of two parts – the server part and the client part. I have a very good analogy of what CAS is. Say, your are going to a live concert of your favorite band. So, your application you want the users to authenticate to using the CAS protocol is the concert itself. There are ticket-agents that sell and validate tickets to the concert. The ticket-agent is the CAS server that gives the users permissions to access your application and validates their access credentials. And of course, people that go to the concert are CAS clients that get tickets and go to the concert.

Django microservices with CAS

Say you are building online shopping web application. So microservices will probably go like django-shop, django-billing, django-auth.

django-shop will present the products to users.

django-billing will accept users’s money for the products.

django-auth will authorize users, so that they have access to other two microservices.

Django-shop and django-billing both need provide users with authentication interface. You do not want users to provide their login and password to the first app and then repeat the same when purchasing goods. You want users login one time and have access to all your microservices.

I will not explain how to create django projects. You can find tons of tutorials on the subject if you google it. Just create the above three projects using virtualenv and continue here.

So, we have two CAS clients (django-shop and django-billing) and one CAS server (django-auth).

 

Django-auth project.

We will setup django-auth project first. Activate the environment and then do the following:

pip install django-mama-cas                # install CAS server package

Add django-mama-cas package to the list of installed apps in your settings.py file.

INSTALLED_APPS = (
    # ...existing apps...
    'mama_cas',
)

Once added, run ./manage.py migrate to create the required database tables.

Now we need to setup urls:

urlpatterns = [
    # ...existing urls...
    url(r'', include('mama_cas.urls')),
]

This makes the CAS server available at the top level of your project’s URL (e.g.http://127.0.0.1:8000/login)

MAMA_CAS_ENABLE_SINGLE_SIGN_OUT = True

Now add the following setting to your settings.py file. This will make sure that when you logout from one microservice, the CAS server will send logout requests to the rest of microservices.

MAMA_CAS_SERVICES = [
    {
        'SERVICE': 'http://127.0.1.1:8000',
        'CALLBACKS': [
            'mama_cas.callbacks.user_name_attributes',
        ],
        'LOGOUT_ALLOW': True,
        'LOGOUT_URL': 'http://127.0.1.1:8000/accounts/callback',
    },
    {
        'SERVICE': 'http://127.0.2.1:8000',
        'CALLBACKS': [
            'mama_cas.callbacks.user_name_attributes',
        ],
        'LOGOUT_ALLOW': True,
        'LOGOUT_URL': 'http://127.0.2.1:8000/accounts/callback',
    },
 ]

This setting is important. MAMA_CAS_SERVICES setting contains the information about CAS clients that will be able to authenticate to the CAS server.

As you probably have noticed, the SERVICE setting contains 127.0.1.1 and 127.0.2.1 as hosts. It is important to keep hosts for every microservice different since the authentication cookies will be saved to hosts, not ports.

The LOGOUT_URL setting contains the url where a request will be sent when user logs out from any service connected to the CAS server.

That is it for the CAS server.

Django-shop project.

Now we will setup our CAS client Django project. Activate the corresponding environment and run the following command:

pip install django-cas-ng                # this will install CAS client

And add this app to the list of INSTALLED_APPS:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_cas_ng',
    ...
)

Also, we need to add it’s middleware to our settings.

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    ...
)

And in order to authenticate from CAS server, add AUTHENTICATION_BACKEND

AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend',
    'django_cas_ng.backends.CASBackend',
)

And finally, add the following settings.

CAS_SERVER_URL = '127.0.0.1:8000'
CAS_VERSION = '3'

CAS_SERVER_URL contains the url of the CAS server, so that this app know where to redirect when user wants to authenticate.

CAS_VERSION is the version of the CAS Protocol. We will use the third version since it is tested well by django-mama-cas and django-cas-ng packages.

Now go to the main urls file and add the following urls so that proper authentication form is loaded.

import django_cas_ng

url(r'^accounts/login$', django_cas_ng.views.login, name='cas_ng_login'),
url(r'^accounts/logout$', django_cas_ng.views.logout, name='cas_ng_logout'),
url(r'^accounts/callback$', django_cas_ng.views.callback, name='cas_ng_proxy_callback'),

It is clear what first two urls do, they just use login and logout views of the django-cas-ng package. The last url processes logout request that come from the CAS server. (Look at the LOGOUT_URL setting of the django-auth settings.

Once added, run ./manage.py migrate to create the required database tables.

Django-billing project.

Do the same as for django-shop project.

 

Conclusion

Hopefully, it did not took too much time. I would recommend you to look at the official documentation and check for other functionalities of these packages. Also, make yourself familiar with the CAS protocol itself so that you can build more with this protocol.

And do not be shy to ask question!

 

Acceptance testing database hack in Django

A couple of months ago, I started working on a web application project with a lot of acceptance tests. Because the project was two years old, everything I expected to do was to add some new features and keep the project running.

So there are a lot of techniques that can be applied in order to shorten running time – people try to optimize their code and use Sqlite in memory.

It’s been a while since I develop web applications using Django. This is clear that testing is one of the most important parts of the development process. And everybody knows how time-consuming it is to run those tests either unittests or acceptance ones over and over again.

So I what I am going to here is to show how I tried to make tests running time even shorter and I assume that your project is setup and ready to go. In this post, I am going to use Lettuce testing framework for testing.

The prerequisites to this hack:

  1. Django project.
  2. Lettuce testing framework installed and fully functioning (or any other like Behave).
  3. Lettuce’s terrain.py file (or any other file that contains test runner configurations)
  4. Behavioral (acceptance) tests already written using Lettuce.

So when you run:

./manage.py harvest

Django creates a new database for you and runs all acceptance tests in the system. Everything is good while you do not have that much acceptance tests, but what if your project is huge and it has up to two thousand testing scenarios already written and you have somehow figure out how to make running time shorter.

The first thing what comes to mind is to keep testing database in memory and you probably added that line ‘memory’ config in your testing database settings.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': ":memory:",
    }
}

This is what Django’s official documentation offers. Now, let us try to redo this in a different way.

So here is the list of actions I am going to do:

  1. create a RAM memory storate directory
  2. place our sqlite db to that directory
  3. modify the Lettuce’s terrain.py file to efficiently handle sqlite file.

Create a RAM storage directory

For this purposes run the following commands:

sudo mkdir /media/ramdisk # it will create a ramdisk directory
sudo mount -t tmpfs -o size=500M,mode=0777 tmpfs /media/ramdisk # mount to RAM
sudo gedit /etc/fstab
tmpfs /media/ramdisk tmpfs size=500M,mode=777 0 0

The last two lines save changes so that ramdisk directory is available even after system reboot.

Now we can change our database settings shown above:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': "/media/ramdisk/mytest_db.sqlite3",
    }
}

So, if I run tests with this settings, Django will create mytest_db.sqlite3 database in RAM. But if we leave it without further actions, this would not be clever.

The next step is to add a few lines of code into our terrain.py file.

I usually call it from terrain.py file’s function that run before each scenario. The code is below:

def handle_db_recovery():
    db_name = settings.DATABASES['default']['NAME']
    db_copy = db_name + 'template'
    try:
        if os.path.isfile(db_copy):
            call_command('migrate', interactive=False, verbosity=1)
            shutil.copyfile(db_copy, db_name)
            return
        os.remove(db_name)
    except OSError:
        pass # this is not good actually, but I didn't found a better way
    call_command('migrate', interactive=False, verbosity=1)
    shutil.copyfile(db_name, db_copy)

So, let us look what the snippet is actually doing. The db_name is the name of database from settings files that you use and db_copy is the copy of original database plus concatenation of ‘template’ to the end of the name.

Then we check whether there is a database with the db_copy name in ramdisk directory, which is in RAM. If yes, simply copy that database into original one and if not, create a new database from scratch.

What it gives is that when you test a web application, you need your test to be independent from each other. So the first step to achieve that is to clean the database after every testing scenario.

The code snippet serves exactly that purpose. If you call the function before every testing scenario run, it will save your time and will not create a new db or run migrations and etc, but simply keep a copy of ready-to-use database file for you.

But keep in mind that if you have some new migrations, you need to go to ramdisk directory and manually delete that copy sqlite file and rerun your tests. Or you would better add a code snippet that can handle such situations for you.

Good luck!