James Morris - JMOZ

I pretend I know PHP and internet stuff.

Organically Increase Your Followers With the Twitter API and a Little Python

| Comments

There are many reasons why you may want to increase your Twitter followers: for status, engagement and for more reach are a few that come to mind. I’m certainly in the latter - more reach == more potential hits to my blog == web domination.

If you’re in the status camp you’ll most likely have gone down the route of paying for followers. You know the drill - pay x amount of dollars for x amount of followers. The majority, if not all of these followers are trash accounts; accounts solely created to follow others for no other reason than to increase their follower count. They won’t provide traffic to your blog or engage with you.

If you’re after organic followers then it requires a little more effort than simply paying for them. Real, organically acquired followers can be valuable as they may retweet your tweets, engage with you, or follow links back to your blog. You can painstakingly follow or retweet users in the hope that they’ll check you out and follow you. You could @message them but that’s a little spammy.

Or you could favourite certain tweets and automate the process large scale. I’d favour favouriting over retweeting as it doesn’t pollute your timeline. Less pollution and spam on your timeline means more room for relevant content the user may be interested in, potentially resulting in a new follower.

We’re going to use a mix of simple Python and the Twitter API.

The method

  • Search for tweets by related users or content, e.g. tweets containing ‘James Morris’ or tweets with the hashtag ‘#php’.
  • Favourite the tweets.
  • User gets a notification that their tweet has been favourited. Our timeline stays the same (unlike retweets, favourites don’t get pushed to our timeline).
  • User checks out profile of the favouriter and if they see something they like, hopefully we get a new organic follower!
  • Rinse and repeat for more followers.

The results

So when I first started using a script I think my follower count was somewhere in the 400s. After a few weeks to a month of using it - when I remembered (there’s a 24hr limit) - my follower count is up about 75% to the near 700s.

Interactions are up also - some good some pointless. I get users retweeting and favouriting my tweets and some engaging and @messaging me. A lot of users don’t realise it’s just a script blanket favouriting all mentions of certain terms; these users tend to @reply back with thanks or engage in conversation.

I’ve noticed a lot of new related followers, e.g. developers and programmers, so all in all the script has been pretty useful for me. The reason this has worked is that I’ve selected my keywords and hashtags accordingly, rather than blanket spamming random unrelated tweets. I’m looking to build on it and create a simple web app others can sign up and use also.

You need to be careful with this, as if not used honestly but in a spammy manner, there could be consequences; however, you could eliminate the build-up of favourites by writing another script that loops over your favourites list and simply calls the API’s favorites.destroy.

The code

So first off we need a way to search for tweets. We set up the Twitter API with keys (which you’ll need to get from their developer site first).

1
2
3
4
5
6
7
8
9
from twitter import Twitter, OAuth, TwitterHTTPError


OAUTH_TOKEN = 'foo'
OAUTH_SECRET = 'bar'
CONSUMER_KEY = 'baz'
CONSUMER_SECRET = 'bat'

t = Twitter(auth=OAuth(OAUTH_TOKEN, OAUTH_SECRET, CONSUMER_KEY, CONSUMER_SECRET))

And we create a function that hits the search endpoint.

1
2
def search_tweets(q, count=100, max_id=None):
    return t.search.tweets(q=q, result_type='recent', count=count, lang="en", max_id=max_id)

Here’s an example of some of the data you can get. The tweets are all under the key ‘statuses’ and contain the important tweet ‘id’.

1
2
3
4
5
6
7
8
9
10
11
12
[(t['text'], t['id']) for t in example.search_tweets('web development', 5)['statuses']]

[(u'No Title: ...the latest web development and internet marketing techniques adopted nowadays.articles play a ver... http://t.co/INHI5uPfWM',
  363044527784726529),
 (u'Please ReTweet: Contact us for the best prices for Print Design, Web Design & Development.',
  363044464652075010),
 (u'Interested in becoming a Web Developer, Programmer or Software Developer? Check out our Development Professsional Programme...',
  363042887107227648),
 (u'MIT, India based company focusing on development and testing of commercial software products. Web application, web software,desktop apps etc',
  363042582982430720),
 (u"Come meet the man behind our graphics, web development, and film projects! Jarrod Bruner's Employee Spotlight: http://t.co/OohMVtRaEp",
  363041474885070849)]

Riveting tweets I’m sure you’ll agree.

Next we create a function that favourites a tweet and plug that into a function that searches and the loops over the results and favourites each tweet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
def favorites_create(tweet):
    try:
        result = t.favorites.create(_id=tweet['id'])
        print "Favorited: %s, %s" % (result['text'], result['id'])
        return result
    except TwitterHTTPError as e:
        print "Error: ", e
        return None


def search_and_fav(q, count=100, max_id=None):
    result = search_tweets(q, count, max_id)
    first_id = result['statuses'][0]['id']
    last_id = result['statuses'][-1]['id']
    success = 0
    for t in result['statuses']:
        if favorites_create(t) is not None:
            success += 1

    print "Favorited total: %i of %i" % (success, len(result['statuses']))
    print "First id %s last id %s" % (first_id, last_id)

For favorites_create() we try to catch the exception that is thrown when you favourite an already favourited tweet (this happens a lot for retweets). For search_and_fav() we store some useful tweet id’s that could be used for the max_id param and also count how many successful favourites we create.

Here’s a gist of the full code:

And you can run it like so:

1
2
3
4
5
6
7
8
9
10
11
import twitter_favouriter

twitter_favouriter.search_and_fav('web development', 5)

Favorited: Attorney Web Design - Foo Web Development http://t.co/mpO18BBsHB, 363739308114460672
Favorited: Full stack web development http://t.co/nkLYoufo7W (via Instapaper), 363738790067568640
Favorited: Complete #Mobile #Web #Development #Solutions & #Optimizations http://t.co/y7U5i6ptsR, 363738298411253760
Favorited: Also, I will never recommend any of my web development clients to use mediatemple. @mediatemple, 363737857111764992
Favorited: 10 Best Joomla Development Companies Announcd by bestwebdesignagenciescom 4 July 2013 http://t.co/HFSZ1QcBV8 #development #JmExperts #web, 363737532694925312
Favorited total: 5 of 5
First id 363739308114460672 last id 363737532694925312

PHPUnit Mocking and Method Chaining

| Comments

I’ve been given the task of unit testing Symfony2’s security layer, which at first seems daunting, but in reality with a clever bit of PHPUnit mocking, it’s actually quite simple.

Symfony2 makes heavy use of method chaining. In the security listener it’s common to see such code as:

1
2
3
4
<?php
if (!$this->securityContext->getToken()->isAuthenticated()) {
    throw new UnauthorizedHttpException('Need an authenticated token');
}

To mock this you might think to do something like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<?php
$token = $this->getMockBuilder('App\FooBundle\Security\Authentication\Token\UserToken')
    ->disableOriginalConstructor()
    ->getMock();
$token->expects($this->any())
    ->method('isAuthenticated')
    ->will($this->returnValue(true));

$securityContext = $this->getMockBuilder('Symfony\Component\Security\Core\SecurityContext')
    ->disableOriginalConstructor()
    ->getMock();
$securityContext->expects($this->any())
    ->method('getToken')
    ->will($this->returnValue($token));

$this->assertTrue($securityContext->getToken()->isAuthenticated());

This is valid and should pass.

But there’s a better way to do this without having to create the stub UserToken object. We can add a method to the mocked SecurityContext - isAuthenticated() which will return true. We then tell the call to getToken() to return an instance of it’s self - the mocked SecurityContext which has our stub method isAuthenticated():

1
2
3
4
5
6
7
8
9
10
11
12
13
<?php
$securityContext = $this->getMockBuilder('Symfony\Component\Security\Core\SecurityContext')
    ->disableOriginalConstructor()
    ->setMethods(array('getToken', 'isAuthenticated'))
    ->getMock();
$securityContext->expects($this->any())
    ->method('getToken')
    ->will($this->returnSelf());
$securityContext->expects($this->any())
    ->method('isAuthenticated')
    ->will($this->returnValue(true));

$this->assertTrue($securityContext->getToken()->isAuthenticated());

Using the at() matcher

In the process of digging into PHPUnit’s internals I found a couple of discrepancies with the docs which I’ve documented below.

The matcher PHPUnit_Framework_MockObject_Matcher_InvokedAtIndex at(int $index) documentation states:

Returns a matcher that matches when the method it is evaluated for is invoked at the given $index.

Which to me implies that the following code would behave as expected:

1
2
3
4
5
6
7
8
9
<?php
$mock = $this->getMockBuilder('Foo')
    ->disableOriginalConstructor()
    ->getMock();
$mock->expects($this->at(1))  // on index 1, 2nd call to foo()
    ->method('foo')
    ->will($this->returnValue('bar'));

$this->assertEquals('bar', $mock->foo()->bar()->baz()->foo();

When in reality, the correct way to use at() is not on a per-method basis, but on all methods:

1
2
3
4
5
6
7
8
9
<?php
$mock = $this->getMockBuilder('Foo')
    ->disableOriginalConstructor()
    ->getMock();
$mock->expects($this->at(3))  // on index 3, 4th method call
    ->method('foo')
    ->will($this->returnValue('bar'));

$this->assertEquals('bar', $mock->foo()->bar()->baz()->foo();

Unit test demonstrating chaining

For reference, here is the unit test I was working with when testing the behaviour of PHPUnit.

A Websockets, Ratchet, Silex and Redis Pubsub Implementation

| Comments

I was approached by a betting/gambling development company who potentially needed a middleware building that would pull from an existing gambling web service and basically transmit to connected iPhone clients the changes from the web service.

At first, the obvious answer might be to create another REST web service that the iPhone clients could just ping for changes. However, one of the devs explained that this wouldn’t be fast enough, or scale - they’d need changes to be transmitted as soon as possible, as the app would be a real-time betting app and there’d be thousands of connections to the server.

I’d recently been using Redis which has a publish/subscribe feature and also knew about Websockets and node.js - but never had the chance to implement them. I figured these could all fit together somehow and suggested a building a proof of concept.

Once I started to look around I discovered a PHP event driven programming solution, Ratchet. I decided to use this in conjunction with Silex, and Predis-async to build the middleware, the idea being that anything can use Redis publish to communicate with the middleware Ratchet app, sending messages (in the case of the potential betting solution, changesets to the betting odds) to the front end. I would use Websockets and autobahnJS on the front end (iOS apps can happily use Websockets to communicate with a server).

There was no documentation on using Ratchet or React’s event loop with Predis-async so I’ve decided to make the solution available online, on github jmoz/silex-test. Here’s a sample of the daemon where you can see the Predis-async integration:

The proof of concept app is running at silex-test.jmoz.co.uk/pubsub.

Below is the text from the README.

Websockets, Ratchet, Redis pubsub proof of concept

This is a proof of concept HTML5 websocket app showing how ratchet and websockets can be used along with Redis’ pubsub publish and subscribe.

You can use the form below to subscribe/unsubscribe to a channel. When you subscribe you also subscribe to the corresponding Redis channel. So if using the cli you connect to Redis and do a PUBLISH channel:jmoz foo, the message will be published from Redis to the Websocket server then to the browser.

You can publish via websockets to the WS server which will then broadcast to connected clients. This does not touch Redis.

You can make an AJAX request to this app which will then use Redis and PUBLISH which will be picked up by the WS server and broadcast to clients. This is a way to demo the raw Redis pubsub without using the cli.

The following tech is used:

  • Ratchet - the core WAMP app that uses an event loop, handles websockets and subscribes to Redis.
  • Predis-async - the client library for Redis that implements the react event loop.
  • Silex - used as the framework and provides the AJAX endpoint to publish via Redis.
  • AutobahnJS - the JS library to implement websockets.

Your Files Aren’t Safe on Soundcloud

| Comments

I use Soundcloud to listen to and download free mixes from DJs and events I’m into. Recently I’ve noticed a trend where artists are putting up their EPs and tracks in full form as a preview, and provide their own download links where the user can buy the track and have the Soundcloud download disabled. In theory this should provide a user with a taster, then they’ll head to the itunes link and pay for the download.

You can circumvent the missing/disabled Soundcloud download feature and download Soundcloud for free.

If you are on one of the pages where the Soundcloud download is not available, this means you will have access to a 128kbps mp3 file. If you’re on one of the pages where the Soundcloud download is available, then use that as the downloadable file is usually a higher quality than the hacked method.

For example, let’s take a freely available Jackmaster Boiler Room mix.

First, you will need Google Chrome. Load the page. Press CMD+ALT+I (or probably CTRL+ALT+I on Windows) to load the developer tools bar. Click the tab ‘Network’, then at the bottom of the screen click ‘XHR’. At the bottom left of the screen, to the right of the magnifying glass, click the icon that says ‘Use large resource rows’. Now that you’ve got your toolbar set up and the page has loaded all it’s assets, click the icon at the bottom left of the screen that clears the logged data.

Now click the Soundcloud play button, you should see some requests logged in the toolbar.

Look at the 3rd file in that screenshot. You can see the type is audio/mpeg and the file name ends with 128.mp3. Obviously, this is the 128kbps mp3 file.

To download the file, the easiest way is to open it in a new tab then save the file. First, right click the 128.mp3 file and you’ll see ‘Open Link in New Tab’, click this.

Next a new black tab will open and the mp3 should start playing. Now go to File > Save Page As.. and there’s your mp3.

Interestingly enough, if you look at the console tab in your developer toolbar, you’ll see a message from Soundcloud ;)

For this example, a download link is already available so you don’t need to use this method. The 128kbps file this method produces is around 55Mb whereas the builtin Soundcloud downloaded file is a higher quality and weighs in at 140Mb.

All of these steps also apply to Soundcloud pages where there is no free download link. This means if you upload some new tracks you think are private, unfortunately a user can still download them for free.

Producers and DJs, your files aren’t safe on Soundcloud.

Ps. I’ve looked a little bit into their app and all of this can be achieved programmatically. Oh and this method also applies to Mixcloud.

Infinite Scroll Featuring JQuery, Flask, Twitter Bootstrap and a Cameo From Redis

| Comments

The idea behind jQuery infinite scroll is actually quite a clever one - it employs the principal of ‘graceful degradation’ - that is to add JavaScript and jQuery functionality to already working static HTML, so that if a user doesn’t have JS enabled (which is rare nowadays), the JS enhanced functionality can still be achieved but in a less impressive, less dynamic, old school way.

Think about pagination for a moment. Pagination ultimately exists to allow you to iterate over, or view, an entire recordset. Infinite scroll achieves the end result, but in a different manner.

jQuery infinite scroll is built on top of regular, static HTML pagination. It looks at the pagination links on a page and dynamically loads a page in the background, then pulls out specified items in the HTML and appends them onto the current page. So if you have pagination on your page already, or easily provided by your framework, you can get up and running with infinite scroll in about 10 minutes.

A Python pagination class

I’m using a Python micro framework, Flask, which doesn’t come with built in pagination. There’s probably some library available on pip that can provide it but I wanted to write it myself. I like Test Driven Development, in fact, I love TDD. This, coupled with the fact that I’m learning Python atm made me decide to build the class from scratch for fun. I wrote my tests along with it so I’m certain it works correctly for my case.

Here’s the Pagination class I came up with.

You can see the usage in the doc. The unit test I used is below.

Putting it together with Flask

The only thing you need to change in Flask is to set up a route that takes a current page parameter and to pass the Pagination object to the view so it can display a list of links.

I use Redis in my app so I’ve added some client code with a bonus bit of Redis’ ZREVRANGE; you can see how I’ve used the Pagination object and offsets to get my result set back.

Twitter bootstrap pagination

Here’s the HTML to display a nice bootstrap pagination widget.

You can see what the end result looks like on the bootstrap pagination page.

This all works great so it’s time for the jQuery.

jQuery infinite scroll

So after all that, the easiest part is to include the infinite scroll JavaScript file, then a small bit of JS sets it all up.

.container is my main div with my div.items inside which hold each piece of amazing content from Redis. The only thing I’ve tweaked is the bufferPx value as I found whilst scrolling on my MacBook Air, I’d hit the bottom of the page quite easily and needed a larger pixel amount to buffer at.

That’s it. The most complicated part of this was writing the Pagination class and unit tests - but that’s my favourite part (I’m a backend dev). Hopefully your framework will have pagination built in, and if so, implementing infinite scroll should be a breeze.

The Hacker News Effect

| Comments

Last week I submitted a post to Hacker News which managed to make it to #5 on the front page. The post in question, Learning Python the pragmatic way, is by far the most successful post I’ve launched. You can view it on HN here.

I submitted it at around 1pm and bit by bit watched the number of real-time visitors in analytics increase to a peak of around 220 concurrent viewers. The popularity of the post peaked at around 3pm with the majority of 2007 hits coming from HN and Twitter.

Hacker News

I expected the post would get picked up a little bit, due to the recognisable (if you know about Python) title, but not to the extent it did. You can see the performance at HN on this graph.

  • It made it to #4 on the HN front page.
  • The post got 73 points.
  • There were 28 comments, largely positive with people relating their learning and coding experiences.
  • From the day of submission (29th Nov) till today (4th Dec), the post has had 13758 pageviews with an average of 5 minutes on page.
  • About 7000 pageviews came from HN.

I was most surprised by the level of conversation on HN. Lots of positive comments but with a couple of pedantic criticisms. I was also surprised and appreciative that a couple of people attempted to back me up over the criticism!

Twitter

Once it gained traction on HN, the post quickly spread to Twitter. My phone started going off with people @mentioning me with a link to the post. The original tweet that my blogging app sent out received no retweets and was hardly noticed; most of it stemmed from a HN tweet and people sharing it using their own methods.

  • The tweet by @newsycombinator was a Top Tweet and received 24 retweets and 83 favourites.
  • Searching for the url on Twitter showed 78 tweets.
  • I was @mentioned by other users with a link to the article 14 times.
  • Gained around 13 related or interested followers.
  • About 1000 pageviews came from Twitter.

My blog

On average I get around 150 visits to my blog a day. From the 30th - 4th I’ve had around 13k visits and the visits per day are now tailing off and returning near to normal.

I received 6 comments, all positive on the actual post page, but the majority were on HN.

What I’ve learnt

When I came up with the post, everything revolved around the title - I knew it would be recognisable and slightly provocative. The problem was relating the content to the title which I tried to do as much as possible, but ultimately I just wanted to write about my stack and show some code examples.

I expected the criticism from the nerds but was blown away by the amount of positive feedback and Tweets directed at me.

This was the most effort I’ve put into a post; previous ones have just basically been code examples with descriptions. This shows if you put some work in and relate to people your post could do well!

Learning Python the Pragmatic Way

| Comments

Around 5 months ago I was on a motocross bike making my way to the top of the Bokor mountain range in Cambodia. I’d chosen to take time off as a contract PHP developer in London and travel around south east Asia - Thailand, Cambodia and Malaysia. I spent the first half of the year there and discovered, partied, relaxed and forgot about work. Whilst there I decided that I needed something new to learn when I arrived back in the UK; I’d recently just learnt and professionally used Symfony2 for PHP, so I decided to change things up a bit and learn either Python or iOS development. I chose Python.

After arriving back in the UK and just managing to catch the summer, I was in no rush to go back to work so spent around 3 months building a Python app I would eventually use as my homepage, jmoz.co.uk. I tried to spend a few hours a day on it, but to be honest, I spent a lot of time enjoying my life and free time; I even went back to Thailand (I got bored of working in my kitchen and in coffee shops around Shoreditch, besides the flight was a deal) for a month where I properly relaxed and kept fit, training muay Thai. Whilst there I’d do a few hours coding here and there in coffee shops, in bars and restaurants and even beaches! (It’s amazing what you can do when you’re really into something).

I’d say with a month of solid work on a project you’re passionate about, and with previous programming experience, you could learn Python from scratch to an intermediate level and gain knowledge of the ecosystem and stack options.

This post is about what I’ve learnt, how I did things and how my app works (and also an explanation of what the hell I’ve been doing since December 11).

Learning Python the pragmatic way

There’s a well known online book, Learn Python The Hard Way. I was advised by a fair few people to read this at the start of my quest to learn Python. It has 52 exercises. I think I managed to look at a few then decided against going through the entire book. Personally, I don’t have the time and the patience to sit through pages and pages of code and syntax and then expect to remember it, never mind put it all together and start developing.

For example, exercise 37: symbol review. There’s nearly 100 different symbols to look at and the exercise suggests “Spend about a week on this, but if you finish faster that’s great.” - too much, too long.

For me, the easiest and most pragmatic way to learn Python is not by reading through pages of reference documentation and syntax, but to just start coding.

Pick a task, an app or project, or something you’re passionate about and take the bold step of building it entirely in the new language. You’ll use your existing knowledge of programming paradigms and design patterns from your previous language to get through it.

For the first week or so of transitioning from PHP to Python my history in Google would look something like, “how to do x from php in python”:

  • foreach loop in Python
  • Python concatenate string
  • Python array

Each core concept you need to learn will rapidly branch out to other language constructs and features, e.g. discovering the equivalent of PHP’s array opens you up to Python’s world of dicts, lists, sets, tuples.

I will admit, at first I found the learning curve rather steep. Reflecting on it now, it was due to the masses of reference and API documentation and long articles or books such as “Learn Python the hard way”. There was just too much information to take in at once. I couldn’t find any posts which gave a really good lo-down of the Python ecosystem (I found this once I’d finished my project). Once I decided to just start coding, I found the pace I learned at increased rapidly. And there was a noticeable point where everything ‘just clicked’ into place; for me this was once I’d figured out data structures, loops and the object/class implementation.

Use Stackoverflow!

Stackoverflow is undoubtedly my go-to resource for help on Python. Countless times I’ve scoured the official Python docs for detailed info and hammered Google, but then turn to SO and find someone with the exact same problem and more often than not, a solution to go with it.

Use your friends and colleagues

I’m fortunate enough to have worked with some clever people that know more things than I. Often it’s far easier for me to throw a quick tweet out asking for advice, or Skype someone. I’ll favour advice from friends over posts or articles I find. So thanks to @rtt, @glenswinfield and anyone else that gave me tips.

A prime example is one of the first Python packages I used - requests. I tweeted something along the lines of “Something analogous to PHP’s HTTPRequest, in Python?”. I got a one word reply of “requests” which at first I thought was a piss-take purely because of who it came from. Turns out it’s really simple to make HTTP requests using a nice API.

Here you can see how some first hand advice saved me time and introduced me to a nice api; first the PHP which I’ve used before, followed by the Python:

Get Pythonic

Coming from PHP to Python is kinda weird. PHP started off really messy and bad but then started making movements towards more structured code, inheritance, interfaces, composition, vast nested directory structures and namespaces. The language, library and framework design started getting more complex and over-engineered (cough Symfony2 cough); it started to look and behave like Java.

Python throws that out the window. Once you start looking at other peoples work you realise coders use Python very dynamically and less structured as the modern PHP community does. And it kinda makes sense. Are all those interfaces really necessary?

I found myself trying to do things the Python way, and not how I would do it in PHP. I duck-typed. I liked list comprehensions and generators. I started googling for ’Pythonic’ examples. import this:

My project

For a long time I have owned my domain but no homepage or app was present. I am an avid user of social media services and apis such as Foursquare, Posterous, Instagram and Twitter. I decided that I would use these services rather than create my own, and aggregate them into one single service which would be a representation of me. People come to my homepage to find out about me and so I’ll give them the public things I share (this could be a bad idea).

So I had project I was passionate about. Learning Python is also something I’d wanted to do for some time. I’d also been hearing about Redis and NoSQL so decided to give that a spin.

First steps

One of the first things I knew I would need for my app was to hit Foursquare’s api and pull my checkins. I’ve done this before in PHP so it was just a matter of finding a decent client library for Foursquare. I went with the library recommended by Foursquare, conveniently named foursquare and easily installed with pip install foursquare. Pulling my checkins was as easy as:

This taste of Python and Foursquare’s API pushed me on to writing some simple parsing of the response and transforming the core parts of the checkin data (id, location name) into ‘items’ for my backend to save then display later in a timeline.

Using Redis as a MySQL replacement

I’ve grown tired of setting up databases, using yaml schemas to create tables, performing complex joins and SQL to get data out the db. This is also partly due to the fact the last couple of contracts I’ve done have been web service based and made extensive usage of REST apis.

I looked and played with Redis and found I could replace an item table or schema with an item hash. And that hash could be dynamic and easily updated or changed.

Oh and also you could just start adding hashes into the db without any setup code. Easy.

I knew that a core part of my app would be a timeline on the index page, of items in descending date created order. Initially I used a list and did some sorting then rpush’d item ids onto the list. This worked well but seemed rigid. Later on, I found out about, and refactored to a sorted set implementation where the score was the date created and the member was the item id. Now I could pull out the items by date created in descending order with ZREVRANGE.

The library I used, redis-py also allowed pipelining which greatly sped up Redis.

Choosing a framework

When I first started with Python, Django was the de facto and only framework I’d heard of. I started to play with it and got as far as a ‘hello world’ from a controller (view).

The further I delved into Django, I started to realise that my app would be mainly domain logic and would use little features from Django - all I really needed was some simple page controlling and a template layer. I had read that Django was pretty heavy and was starting to find this out. It’s about this time that I was made aware of a micro framework named Flask.

Flask looked perfect for what I needed, and I kept hearing it mentioned so decided to give it a blast. With literally a few lines of code I’d managed the same as what I’d got in Django, in far less time and with a lot more ease.

It also looked incredibly similar to Silex (or the other way round) - a PHP micro framework I had just been using to create an API.

If you’re looking to get into Python and need a starting framework, I’d highly recommend you try Flask, below is the minimal code that powers the index of jmoz.co.uk.

Flask also supports extension, such as Flask-OAuth, which I’m using to provide login via Twitter. Below you will find an example of my callback for Twitter’s OAuth process.

My stack lo-down

So after getting some example code working and playing with Redis, I’d delved far enough into Python to start implementing my stack. Through the advice of friends and colleagues I ended up with the following stack:

For everything.

Allows each project to use it’s own environment with a specific Python version and all packages bundled into the venv. Makes it very easy to deploy the same environment on your live box.

For version control and deployment.

The micro framework that puts everything together.

I use Redis as the NoSQL database store. I store items as hashes and use sorted sets for a timeline.

Nginx proxies http requests to Gunicorn.

Gunicorn is the WSGI server that runs the app.

Controls the Gunicorn process and allows easy restarting and updating of the app.

Nginx proxying to Gunicorn

Nginx is very easy to use in combination with Gunicorn - just proxy all requests to the Gunicorn server running on port 8000. In the example configuration below you can also see how I redirect www. requests to the canonical domain jmoz.co.uk.

Supervisor config for Gunicorn

Supervisor can be used to monitor the Gunicorn process; starting it at boot and making it easy to restart with supervisorctl restart gunicorn-jmoz. The configuration below specifies the path to the gunicorn bin file in my venv. Without the path to the venv the Python environment and dependant packages will not load correctly.

I store this file inside my project then create a symlink like so:

web@sagat:~$ ll /etc/supervisor.d/
total 8.0K
4.0K drwxr-xr-x  2 root root 4.0K Nov 22 18:46 .
4.0K drwxr-xr-x 88 root root 4.0K Nov 26 09:50 ..
   0 lrwxrwxrwx  1 root root   49 Nov 22 18:46 gunicorn-jmoz.conf -> /home/web/sites/jmoz.co.uk/gunicorn-jmoz.conf

For this directory.d/ style of configs to work, make the end of /etc/supervisord.conf look like:

[include]
files = supervisor.d/*.conf

Mac (dev) environment vs Debian (live) environment

I develop on my Mac using brew, pip and virtualenv. This all works great and is a pretty standard setup. The problem I have is that my Linode VPS is running Debian stable (Squeeze). Debian packages are always out of date due to their strict testing and acceptance process so my Mac’s Python 2.7.1 does not match up with Squeeze’s 2.6.6.

This may not be a problem for you, but for me it broke my unit tests. When I deployed my code, I ran my tests on the live VPS to confirm everything was working but I got errors in the unittest package due to new features introduced in Python 2.7.

I found a solution using pythonbrew which lets you install different Python versions (environments) in your home directory, and toggle between versions on the fly. On my VPS I managed to get 2.7.3 running (I still need to update locally) which fixed all my tests.

On my Mac I use virtualenv directly, with my venv inside my project directory (it could be moved to home) which all works fine. To activate my Mac venv I do:

james on moz-air in ~/code/jmoz.co.uk-flask (master)
 $ . venv/bin/activate
(venv)james on moz-air in ~/code/jmoz.co.uk-flask (master)
 $

But on Debian I have to go through pythonbrew and activate the venv (stored in home) like so:

web@sagat:~$ pythonbrew venv use jmoz
# Using `jmoz` environment (found in /home/web/.pythonbrew/venvs/Python-2.7.3)
# To leave an environment, simply run `deactivate`
(jmoz)web@sagat:~$

These are the 2 main differences between environments. Once the venv is activated, everything behaves the same (so far).

Deploying using Git, installing, then restarting Gunicorn

I do a git push live master then a git post-receive hook takes care of checking out my master branch and running an install script contained in the project.

Here’s the post-receive hook which deploys the master branch only:

The install script first has to set up pythonbrew (Debian specific, I don’t use pythonbrew on my mac) so that the pythonbrew command will work - this is because the web users .bashrc is not executed during the git push and post-receive hook. If you were to look at my web users .bashrc, you’d see:

[[ -s "$HOME/.pythonbrew/etc/bashrc" ]] && source "$HOME/.pythonbrew/etc/bashrc"

Which you are told to add as part of the pythonbrew installation process.

The install script then looks like this:

This will activate the virtualenv so we get the correct version of Python, then we install all dependencies into said venv using pip and a requirements file (which is generated during dev with pip freeze > requirements.txt after installing any new packages).

Once pip has finished installing any new packages or updating them we restart the Gunicorn process using supervisorctl so that any changes to Python files are picked up.

Once the restart has finished we should have a successful deployment of our Python app all from a simple git push live master!

Further reading

There were a handful of posts and articles that I found really useful, not just one line code examples.

Use the tutorial first - find a section you need more info on, e.g. classes.

Use the library (or API) for detailed functions and methods, e.g. find out what operations a dict supports.

You can learn Python without reading a book. A lot of them are dated; Google and Stackoverflow are better options. The post above is a great example, and also worth reading purely for the use of the phrase, “neckbeard ass”.

This is a great resource if you want to learn from scratch. Personally I’d suggest to just start coding then consult certain exercises when needed.

Check out Flask, a nice micro framework.

Learn about virtualenv, pip and Python’s ecosystem. That last link seems like an absolute godsend, and I’d recommend you read it first; unfortunately I only found it after I’d spent countless hours researching and learning things and after I’d finished my app!

What I’ve learnt

The easiest and most practical way to learn a language is to build an app, that you’re passionate about. That’s how I learned PHP about 5 years ago - I built some crappy name-in-a-chuck-norris-fact image machine thingy which ended up on a dedicated server with over 1000 visitors a day. It’s crazy to think I started there and ended up somehow at BSkyB getting paid to write PHP.

I’m happy with how I progressed with Python. I know enough now not to be scared at the sight of unknown Python code. I’m actively discovering and learning new things every day and I love it. I might prefer it to PHP.

I intend to develop all personal projects in Python and improve my knowledge to a level where I can be professionally employed as a Python developer. Fancy testing me? Get in touch.

Oh, and visit my app, James Morris.

Update: My core intention with this was to show a basic stack and examples for newcomers. The surplus references to ‘just start coding’ and LPTHW etc is an attempt to reference the recognisable title (which I hoped would generate interest ;) More on hacker news. Follow me for sort of tech related tweets.

A Django-nonrel Testapp Blog With Tagging

| Comments

If you’re working with django-nonrel and appengine, there’s a django-testapp barebones application you can work off as a starting point (there’s no application code, just setup code and files for appengine).

I decided to have a play with it and come up with a basic blogging application. I scoured the web for example simple django-nonrel apps but couldn’t find too many; because of this I’ve decided to release this small example so as to point people in the right direction when starting out with appengine and nonrel.

I had a few troubles trying to figure out the best way to implement simple tagging for a blog post. The solution I ended up with used a ListField from django-toolbox (a django ManyToMany field is not supported by django-nonrel).

I also integrated django-bootstrap-toolkit so it comes with a frontend out the box.

I forked the repo and uploaded my changes to django-testapp so go there and clone the project to get a basic django-nonrel blogging application with tagging, and some bootstrap thrown in for free.

Git Push to Multiple Remotes

| Comments

I’ve been in Thailand for the past month but whilst there I was working on an app and regularly committing to my local git. The fear of getting my Mac stolen or lost prompted me to push my work to my remote vps for a backup.

It’s always better to have a couple of separated backups so I could potentially use Github, but my app is private and I don’t have a Github pro account with private repos. Fortunately Bitbucket can provide unlimited private repos so I decided to give that a try.

Git push to multiple remotes

If you add 2 remotes in git, when you do a git push the code will only go up to one remote. The trick is to set up a pseudo origin ‘merged-remote’ so that when you git push origin master the code will go up to both remotes.

First make sure you’ve added both your remotes in the regular fashion, then edit the config so there are 2 url items for your new merged-remote origin:

 $ cat .git/config 
[remote "my_vps"]
    url = ssh://my_vps/home/web/jmoz.co.uk.git
    fetch = +refs/heads/*:refs/remotes/my_vps/*
[remote "bb"]
    url = https://jmoz@bitbucket.org/jmoz/jmoz.co.uk.git
    fetch = +refs/heads/*:refs/remotes/bb/*
[branch "develop"]
    remote = bb
    merge = refs/heads/develop
[remote "origin"]
    url = ssh://my_vps/home/web/jmoz.co.uk.git
    url = https://jmoz@bitbucket.org/jmoz/jmoz.co.uk.git

Now when I do a git push origin develop my branch goes up to both of them:

(venv)james on moz-air in ~/code/jmoz.co.uk-flask (develop)
 $ git push origin develop 
Counting objects: 42, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (33/33), done.
Writing objects: 100% (33/33), 4.92 KiB, done.
Total 33 (delta 16), reused 0 (delta 0)
To ssh://my_vps/home/web/jmoz.co.uk.git
   a567eac..97f5758  develop -> develop
Counting objects: 42, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (33/33), done.
Writing objects: 100% (33/33), 4.92 KiB, done.
Total 33 (delta 16), reused 0 (delta 0)
remote: bb/acl: jmoz is allowed. accepted payload.
To https://jmoz@bitbucket.org/jmoz/jmoz.co.uk.git
   a567eac..97f5758  develop -> develop

This is perfect for my current workflow where I want a backup of my work in multiple locations. Later on in development I could look at using the bitbucket remote as a backup and my vps box as my deployment box making use of git push and hooks to deploy the code.

Python: Convert Datetime to Timestamp and Back Again

| Comments

It’s pretty easy to get a datetime object from a timestamp in Python. However the same can not be said the other way around. Just when I start to think “oh Python’s pretty cool with a nice API” the further you look into the language and library you start to notice some cracks.

Convert a timestamp to a datetime object

1
2
3
4
5
from datetime import datetime

print datetime.fromtimestamp(1346236702)

#2012-08-29 11:38:22

Convert a datetime object to a timestamp

Why oh why did they not just implement a datetime.totimestamp() method on datetime?

1
2
3
4
5
6
7
8
from datetime import datetime
import time

dt = datetime.fromtimestamp(1346236702)

print time.mktime(dt.timetuple())

#1346236702.0