xenji.com

Book review - PHPStorm Starter (Packt Instant)

The nice people at Packt recently asked me to review on of their new books from the INSTANT series, a focussed line of books concentrating on one single topic. Although PHPStorm can become a big one, this book keeps things straight.

I am talking about: Instant PhpStorm Starter

The target group for this book are beginners and I mean real beginners. After the typical "how to download" and "how to install" and The author creates a short introduction of about five steps to bootstrap the reader and get her/him ready to rock. This intro is quite OK, as far as you have never used anything else than notepad.exe before. Most of the stuff is very common and feels a bit like a transcript of the manual.

The chapter "Top features you need to know about" is mostly about code templates, file templates and shorthand operations in the editor. Those hints are useful and even a power user might feel surprised to find the one or the other little trick that she/he did not know about.

The part about the "High-level programming operations for the PHP language" sounds better than it (sadly) really is. Accessing the documentation, instance diagrams and the autoformatter do not count as high level programming functions for me. One thing I use every day in PHPStorm is missing: The function-, file- and symbol search

The last big chapter is about using a VCS with PHPStorm. This is another chapter that I feel unsure about. I am a very intense console user when it comes to VCS, esp. git. This makes it hard for me to judge if this part of the book is helpful for beginners. I think it might be, esp. if they come from the eclipse world. From my point of view, the VCS chapter should have been replaced with a chapter about the plugin architecture of PHPStorm, a careful selection of helpful plugins and some more "navigating through your code" hints. The hint for the structure- and hierarchy view is totally missing.

When I try to imagine the point of view of someone who comes e.g. from the eclipse world, I must say, that it would have been of my very interest, that PHPStorm can handle eclipse project files (this is because IntelliJ Idea, the Java IDE from JetBrains, is a commercial competitor of the eclipse Java Development Tools) very well and that I could use both at the same time without forcing me to choose one over the other.

Conclusion: If you never used an IDE before you will learn a lot of new things from this book. If you are an experienced IDE user of any flavor, this book will show you some low-level tricks in PHPStorm, but the, from my personal point of view, important things like code navigation are missing.. If you used any other JetBrains IDE product before (like RubyMine, PyCharm, WebIDE, etc), most of this book is useless for you.

MySQL analysis with AnalyzeMySQL

I recently joined the scalability team at trivago and my first job was to get into the actual MySQL schema to discover which parts can be improved regarding performance and space consumption.

trivago's database consists of about 230 tables, many of them having a history of more than four years, some of them up to 7 years. A few of them do have some historic flaws, like having a signed bigint as primary key (because nobody could imagine what space will be needed in a few years), others do have indexes that are not longer used, etc. There are many tasks to do until you get to know the database scheme very well.

In order to support my work I started to write a litte tool, which was meant to perform some basic tasks on the scheme. The first one was to see what col definitions are much bigger than the actual maximum value it keeps.

What came out of this approach was a small framework-ish script collection that parses the table structures and gives programatic access to it. The framework is designed to use a plugin structure to achieve a simple extensibility.

I'll try to write another post to explain the creation of such a plugin as soon as possible. In the meantime I would like to invite you to have a look at the (yet unfinished and unpolished) code and provide me some feedback.

You can find the code on github: https://github.com/xenji/AnalyzeMySQL

Visualizing trivago's search traffic on a heatmap

While evaluating a message queue for trivago's architecture I wrote a litte demo to present RabbitMQ in one of our internal meetings. The demo should display live searches from the trivago platform and display them on a map using a heatmap layer. The demo should show that the MQ is able to cope the amount of traffic that would hit the queues and this type of technology solves some of our current near realtime problems.

As I was not allowed to touch the live systems code for it, the data needed to come from somewhere else. The accesslog seemed a proper source of information as our scribe logger delivers it in near-realtime. We use two different parameters to determine the location the visitor wants to search for, one is an integer, the other one is a list of integers. A "nice to have" feature is to show the visitors location. The scripts show this feature already, but the screen recording just shows the destinations. Log parsing? AWK to the rescue!

To resolve the IDs to a geo coordinate we did some pre-caching using Redis and a simple ID => geo construct.

The AWK script writes it's result to the STDOUT where a simple node.js scripts gets the content, resolves the geo data from Redis and passes the result to the RabbitMQ exchange. The script already shows the capability to use the geo-lite module and resolve the senders location via geoip lookup.

After sending it to the exchange, the messages are collected from the queue by a simple node.js server daemon. This daemon does only a simple job and pushes the messages from the RabbitMQ via broadcast to all connected websocket clients.

The frontend is as simple as the rest of the stack. On the client-side we use a bit of jQuery, the wonderful gmap3 library and socket.io.

The only limitation with this small toy is the limitation that the browser gives us. I've tried to raise the number of simultaneous data points displayed at the same time, but my MacBook Air (late 2012) went hot on more than 5000 points at the same time. This is related to the interaction with the event-list that controls the heatmap layer.

The average message rate in the video was between 300 and 600 messages per second but my screen recorder is just capable of 10fps, sorry ;).

trivago search destinations on a heatmap from Mario Mueller on Vimeo.

Using tunnelbroker with Apple's Airport Extreme

Inspired by the IPv6 episode from ChaosRadioExpress I tried tunnelbroker.net as a my 6in4 provider, because Unity Media cannot get native IPv6 running for long-term customers (ridiculous: new customers get it without any problem). tunnelbroker enables you to build a static tunnel, which is quite problematic if you have a DSL connection or something similar that changes it's IPv4 frequently. I use my Raspberry-PI, running on ArchLinux ARM, together with this litte ruby script to keep my public IP updated.

X2EZWJNRVT4Q

Release process - from startup to professional

The release process is one of the most underestimated processes in a growing company. Most startups choose between two versions of release management: none or fully automatic. The latter one mostly fits ruby/rails driven startups due the availablity of capistrano or git based hosting companies. The "none" variant means, in our case, deployment via svn checkouts on the server(s). You could say this is release management too, but if you are honest to yourself ...

Startups tend to grow and so you will encounter some point in time when you need to scale the processes within the startup. The release process is one of them, as you get more servers, more developers and more features per development cycle (I tend not to say sprint or release cycle for getting the agile and classically managed in one boat). In addition to the management complexity the software complexity will grow with every feature you do. Phrases like rollback, dependency management and multi-environment come up in tech meetings and the "management by in-flight magazine" comes up with continous deployment to lower the time to market dramatically (sorry, no link or reference for the anti-pattern - the original article seems to be gone, but I guess you can imagine what it means? The CEO or Product Developer reads something in a magazin while traveling and enters the tech department to announce the immediate resource allocation on the new hot topic XYZ without any preperation or even sense). I tend to call this DDD (dream driven development) resulting in RDD (rage driven development) on the tech side.

When you come from a web agency (like me) and start designing a new process for such a company you might tend to rely on a classic release management process. Starting from the way how feature tasks are assigned up to the moment where you build and deploy the piece of software on the server cluster. I have been proven wrong since I stated the attempt to build such a process. The new goal should be to mix up the two worlds, aiming for maximum flexibility within a given level of process security. Cutting it into smaller pieces to take the changes step by step makes it easier for the developers and for the management.

I'll do another post about choosing a software stack to support the release management in a few days.

Indexing octopress article in elasticsearch

I was not satisfied with the simple search plugin that came with octopress (although it is suitable for many blogs), so I started looking for a simple search solution without any branding and with all the features I wanted. I came across Solr, but it was oversized for my concerns. I did not want to create a complex scheme and push the documents with brackground jobs or something similar. This is where elasticsearch came into play. The very dynamic nature fitted very well and the easy to use http interface was exactly what I needed.

I've hacked a simple indexer plugin and a simple javascript file to query the index and display the results in a badly styled overlay (I'm no CSS guy at all).

Here comes the plugin, which is a simple jekyll generator. I does not check for anything, but if you are able to blog with octopress you should be able to cope with the sources and customize them to fit your needs. You'll need to add those two config entries for your index url (which should be protected by http-basic auth) and your index name.

The javascript is also dead simple. You'll need a jsonp request if your elasticsearch server is hosted on another domain or port (which is very likely). The script is nothing crazy. Most of the code is rather bad, using jQuery selectors directly and not using any scope related stuff. The code might work better as a jQuery plugin or something more reusable. Anyway, it works for me as a first step. Feel free to enhance it and drop me a line over twitter so I can link to your solution.

bind function missing in iOS 5 JavaScript

While developing a new, jQuery based, frontend, we encountered some strange behavior while testing a release of the app on the iPad 2 running iOS 5.1.1. Between a bunch of commits the iPad's JavaScript just stopped to do anything. Not a single event was fired, nothing happend.

The debugging tools on the iPad are not that great and so was the debugging process. We found out, that the Konquerer Browser uses the same engine as iOS and the bug was reproducable in Konqueror, too. Git bisect came into play and we found the commit that killed the iPad. We changed the modenizr full build to a custom slim build. This kicked out the "Touch Event" part of the build and this was the reason for all the trouble. The "Touch Event" part defines the bind() function if it is missing in the JavaScript engine. We borrowed the function from modenizr and made the iPad work again.

Sidenote: iOS 6 beta 4, which has a new JavaScript Engine, works without any problems.

Hetzner vServer on ArchLinux (german)

This post is written in german language, as most people who are interested in it are customers of the german hosting provider "Hetzner".

Lange Suche

Ich bin erst seit Kurzem eine Fan von ArchLinux. Vorher war ich bei SuSE, Fedora, Debian, Ubuntu, Gentoo (Stage 1) und PC-BSD. Das letzte Ubuntu Release, 12.04, hat mich enttäuscht und dazu getrieben mir eine echte Alternative zu suchen. Ich war vorher viel im Gentoo und BSD Umfeld unterwegs und mochte den Ansatz der vollkommenen Anpassbarkeit sehr. Der Schritt zu ArchLinux war damit nicht so weit entfernt. Doch nun begann die Suche erst wirklich. Einen Hosting Provider, am liebsten mit deutschem Standort, und vertretbaren Preisen war nicht so schnell zu finden. AWS, Linode und Rackspace bieten ArchLinux direkt an, jedoch sind alle mit 19,99 USD für 512Mbyte RAM vServer mehr einfach viel zu teuer und wenig attraktiv.

Leider bieten viele deutsche Hosting Provider nur Virtuozzo-basierte Lösungen an, so z. B. Strato und Hosteurope. Die Shared-Kernel Architektur verbietet es dann irgendwas Anderes als die angebotenen Distributionen zu verwenden.

Hetzner to the rescue

Hetzner war der erste (und soweit ich weiss auch einzige) Anbieter mit KVM/QEMU Architektur. Zusätzlich bietet Hetzner seinen Kunden an, den Server im sog. Rescue System auszuliefern. Damit bekommt man ein schlankes Debian gebootet, von dem man seinen Server wunderbar bearbeiten kann.

Leider stimmen die Tutorials auf https://wiki.archlinux.de/title/Arch_Linux_auf_einem_Root-Server und http://www.uninformativ.de/?ndo=single&newsid=138 nicht mehr zu 100%, so dass ich meine Erfahrung hier noch mal schnell zusammen schreibe.

Der Arch Linux Wiki Artikel arbeitet mit einem Software Raid, was meiner Meinung nach für einen vServer wenig Sinn macht, und einer pacman basierten Installation. Hetzner verfügt jedoch über eine Web-KVM Konsole, die es uns erlaubt eine normale Installation durchzuführen. Die Installationsanweisung auf uninformativ geht eben diesen Weg.

Das uninformativ Tutorial kann man bis zur Installation von grub gut durcharbeiten. Leider ist der klassische grub unter dem aktuellen Rescue System von Hetzner nicht mehr verfügbar. Da die aktuelle netinstall Version von Arch keine alte Version von grub mehr supportet und die Version 0.9 nun nur noch im AUR verfügbar ist, sollte man hier besser auf syslinux setzen oder sich mit der manuellen Einrichung von grub2 anfreunden.

Der Rest läuft wie im Artikel von uninformativ beschrieben.

Stateful web applications pt.2

This is a followup of my first article about stateful web applications, which you can find here, you may want to read it first.

Push state - poll state

We are still talking about stateful web applications. In my previous article I've talked about a challenge - response method to keep client and server side in sync. This method had some drawbacks but was simple to achieve.

The push state - poll state method is more complex concerning the setup, but way simple when it comes to the development part. Instead of using a single, serial channel of communication, we break up things and use two separate channels for our needs. This simple change gives us the possibility to acquire more change state requests as we could to with the sequential version (meaning the challenge - response method).

The setup

You will need something that accepts a high number of concurrent request. I would not recommend an Apache httpd for that. I've tried node.js and it worked. (nginx might fit as well, depending on your software stack.) This server, let's call it "change-bucket", gets all the change state requests from all clients via PUT request. It responds with some 2xx state, maybe 204 or 201. This is just the info, that the put-request reached the server. As you might imagine, this might lead to an eventual consistency state. The change-bucket passed the change request to a shared queue, which could be a redis server or something like ApacheMQ, RabbitMQ, or similar. The queue server maintains one list per application (of which multiple per user can exists side by side). By the way: redis is awsome for such a job, because it supports list operations natively - even blocking ones - and produces less overhead than a real message queue. The last thing you need is something like memcache. You can use redis or a real memcache server - depending on your setup. The key is: it must be fast and at least persistent as long as it has a power supply.

Now, that we have all change requests of the user in a queue, we need to get them out and change the server-side application state and reflect those changes to the client. This is also a point in this setup, where you can decide which way to go.

We will now switch perspectives and take a view from the client side now. This helps us to identifiy the point where those two ways will come together again.

The client

The client need a poll mechanism. You know, something like setTimeout() with a propper tail recursion. I would not recommend using call(), as it is of lower priority in the most JavaScript Engines. The intervall depends on your technology stack, but I recommend to make it possible to set it to 250ms or less. This enables you to fake a real-time feeling on the customer's side. This poll request must go somewhere and it needs to receive something. What does it receive? The full application state, where does it go? Now we reached the other side of our two way split.

Path 1 - blocking list operations

This way is of moderate complexity and lacks of scalable performance. You will need only one additional piece for your architecture puzzle. You need an endpoint for the client to ask for the latest application state and you will need to make this endpoint fetch the stack of change requests from redis. This fetch operation must be blocking to prevent a concurrency issue. The concurrency issue comes from requesting the same endpoint twice. The first requests has not persisted the new application state yet and the second request loads the old state from the memcache server. The last one who persists the state wins. In this case we would get a not recoverable, inconsistent state. Another problem: you will need to solve the unsuccessful put-requests in the change-bucket with e.g. a retry loop in the client.

Path2 - everything non-blocking

In this setup you will need another, daemon like instance to solve the problem of concurrency. We need to decouple the polling process from persisting the new state. We could do this by a cronjob or a node.js server or something completly different. The important aspect is, that this cronjob just takes data from the queue and merges it into the new state. The polling mechanism may see the same state twice, but that is OK as far as we got rid of our concurrency issues.

Conclusion

We've build a circle, a round-trip for our application state. You might now have the feeling, that this is a shit load of overhead regarding the task of "just" persisting an application state, but that is OK. It is a rather complex setup with it's own issues and I must admit, that it works only in theory at the moment.

Future

One of the big advantages of this setup is the possibility to scale it. If the polling mechnism reached it's scale, you can easily exchange the polling with persistent socket connections. If you do so, you might not even need a single change-bucket, but you could use the same socket to do both parts of the communication over a single architectual piece.

Stateful web applications pt.1

Preface

In modern web application we mostly use two sided setups. One is a client - which will be a browser in this case - the other is some sort of server with some sort of application. The languages used on the server side do not matter at all, but the client side language will be JavaScript.

It is all about state

The application has a state. This might mean some sort of dataset which the client works on or something which is brought to the client by the server and is fully or partly persisted to enable the client further requests in the same data context. This topic is highly connected to caching those datasets and working on those cached results rather than working on live datasets.

HTTP is stateless, which means that we must rely on other techniques to synchronize the server and the client state. You might call for (web-)sockets now, but not all companies are able to cope with a change of their infrastructure to use sockets for the number of users that are concurrently browsing the app.

Challenge - Response

One way of getting this done is a challenge - response communication protocoll between the server and the client. The client always adds a token to identifiy itself on the server side. In addition, when the client sends a request it adds a ticket id to it. The server processes the response either in realtime or asynchronous. When doing it in realtime, the server responds directly to the client request. The async request must trigger polling mechanism where the server answers at some time with the requested ticket id. This type of communication works well, as long as your requests do not depend on each other. If you need state changes in a certain order, you need to add some kind of counter to it. The server needs to keep up with the counter and cannot process counter number 5 before counter number 4 was processed successfully.

This might lead to a deadlock situation, where counter 4 is lost in space for some reason and the server got stuck at holding back anything above counter 4.

Another drawback might be the danger to slip into some sort of interpretation of the client state. The server gives you a bunch of values to set some of the states that were asked for in the last response. You might be forced to start guessing the state of former components based on the partial response.

Too abstract? Ok, here is an example. You open some sort of info box and it's content must be loaded from a server via an ajax call. The response comes back and you display your text. This box is one of three in a left column UI element. The UI concept says, that not more than one box should be open at the same time. OK, what happens next is common sense at the moment, but from my point of view it is a great danger. The client decides, that, on opening another box, the actual box has to be closed. This state never reaches the server. The client interpreted, that he must close it, because another box want be gain the state "open".

Lets get one step further. We have another, dependent view element that also requires a server request. The answer to that request could possibly open one of the three boxes. As the server's answer enforces the client to do that, the actual state of the client (which is unknown to the server) gets lost. The client need to interpred the result again and concludes, that the active open box must be closed to achieve the requirements of the former server response.

Where is the drawback now, you might ask. Try to let the client send a server request based on data that cannot be interpreted correctly be the client:

• You have a entity to show via an id.
• You need to show meta data based on the id.
• You have a default setting on the root level of your application ("/")
• You do not want to show the meta-data to the default id, but another one

If you just deliver the id, the client is not able to decide if it's default or not. The amount of "please client: if default do that, if not do this" raises and will kill you/your app some day.

The next article will be about using a "Push / Poll State Sync" approach.