Monday, 12 December 2016

7 challenges and myths about going microservices

1) This is a cultural change in organisation

True. Microservices are like a mental change how we do software. There are going to be many, smaller projects, a layer of network communication, releasing and versioning issues. If you have a team of people who never wrote any automation scripts, never released their own projects in production environments, never did any CI/CD - it will be really hard. You can expect a lot of resistance, learning curves will vary depending on personal ops experiences, collaboration in team and willingness to learn new, different approach.

2) Microservices are easier to develop

Yes and no. It is easier to develop such services on its own - projects are smaller, it is easier to jump and get the understanding of business logic inside a single project, fix a bug, upgrade libraries. On the other hand - you need a well functioning deployment pipelines and good inter-services communication patterns to get all the things going. The power of microservices is because they operate together - but this is also a big challenge itself, as networks can fail, messages can be dropped, responses could be randomly timing out, testing is different and may feel harder, teams may not communicate well and so on. You need to decide on communication (sync/async), develop patterns for graceful failing, handling errors, startup/initialisation and so on.

3) You need devops culture

Yes, true. For a working solution, every development team should be able to test and deploy. Deployments should be simple and easy to repeat. Think about one-click deployment pipelines. Teams should be cross-functional. Operational excellence and understanding of devops and continuous integration principles comes also from understanding production environments. Monitoring, metrics, logging, tracing requests, dumping messages. Troubleshooting issues at 3am, reverting to previous versions, fixing corrupted data, all these things helps you to develop culture of operational excellence. "Bad behavior arises when you abstract people away from the consequences of their actions"

4) You need to have a good TDD/BDD

Not per se, but it helps a lot. Technically you don't need to have automated tests to prove your software works, but you know it helps a lot to move faster, make important changes safer and it is usually cheaper when you can quickly prove new version doesn't break existing functionality.

5) You need continuous delivery

A simple, working solution to push new changes easily helps a lot. Actually it is more like a cause-and-effect loop - a microservice project is small so it is faster to change it and deploy more frequently than a big, monolithic app. It has to be easy and every developer should be able to release changes.

6) You should (not) use REST

It depends. Communication patterns like Rest tends to be simple and well-known to start with, but with more complex environments, they can be terrible in terms of error-handling, decoupling and fail-over. With synchronous patterns like rest, one service depends on current availability of another, even if the job flow doesn't really require it. That makes e.g. failing gracefully with partial outage much harder. Async communication on the other hand has higher initial cost, it could be also harder to follow such execution paths in the code.

7) You need to use containers like docker

What you really need is to have manageable, versioned, repetitive environments where you can easily deploy your services. You can archive it in many ways - good orchestration tools like ansible/salt/chef/puppet or whatever is hot and tasty today.
While technically it is possible to build images/machines every time, caching some pre-build stages as e.g. docker images and adding only your changed service may speed up this process a lot.
You need a quick release cycle, so if docker helps you to archive it - go for it.

Sunday, 10 April 2016

Graph database Neo4j behind Panama Leaks

A graph database Neo4j was in a centre of toolbox used to power investigative journalism in what seems to be the biggest-ever financial leak history.

"Through its Data & Research unit, ICIJ provided the data analysis expertise to make the documents exploitable by reporters. They extracted the metadata of documents using Apache Solr and Tika, then connected all the information together using the leaked databases, creating a graph of nodes and edges. The data was stored in the Neo4j graph database edited by our partner Neo Technology. The result provides unique insights into the offshore banking world showing the relationships between banks, clients, offshore companies and their lawyers."

Sunday, 14 February 2016

Keep your SBT projects' dependencies up to date

There is a simple SBT plugin to check if there are never versions of libraries used by SBT projects.

You may add it to your project (why not?), but you can also keep it separately, especially when you quickly want to check several projects.

Create a file:


and add such content there:

addSbtPlugin("com.timushev.sbt" % "sbt-updates" % "0.1.10")

After that you can run

sbt dependencyUpdates

or just type "sbt" and later, in the console, use keyword expansion to save on typing...

> dependencyUpdates
[info] Found 4 dependency updates for minimal-akka-scala-seed
[info]   com.typesafe.akka:akka-actor        : 2.3.11 -> 2.3.14 -> 2.4.1
[info]   com.typesafe.akka:akka-testkit:test : 2.3.11 -> 2.3.14 -> 2.4.1
[info]   org.scala-lang:scala-library        : 2.11.6 -> 2.11.7      
[info]   org.scalatest:scalatest:test        : 2.2.4  -> 2.2.6       
[success] Total time: 1 s, completed 14-Feb-2016 00:27:55