North East Blog Directory

November 17, 2014

SuperMondays

North East Tech Videos

Last month’s talks are now available thought the Internet Archive.

We had a few technical issues but we cobbled together most of the event in some form.

Jason’s slides are available through slidedeck.io.

Sadly we don’t have Paul’s presentation but you can fine the amazing facts in Paul’s post on Medium.

 

by supermondays at November 17, 2014 10:22 PM

November 12, 2014

Alistair's Blog

What version of Ubuntu am I on?

This is a question that I ask myself a lot and keep having to look up almost every time. Sometimes I am not certain what distribution I am actually using. Here is the summery version that hopefully will save me having to look the command up each time, and perhaps help you as well.

To find out all the version information about the Linux kernel you are running run the command…

uname --all

To find out the OS release (such as the Ubuntu version) have a look in the os-release file…

more /etc/os-release

…and on most systems you can look at that same information neatly formatted using…

lsb_release

by Alistair MacDonald at November 12, 2014 11:46 AM

November 10, 2014

SuperMondays

Super Mondays – Digital Institute: Clouds, Sensors and Big Data

Eventbrite - Super Mondays - Digital Institute: Clouds, Sensors and Big Data

This month we’ll focus on the work of the Digital Institute at Newcastle University. There will be 3 talks:

1. The TMS Platform for Analysing Streaming and Historic Twitter Data
Rebecca Simmonds, Newcastle University

Analysis of social media data has the potential to provide useful insights in a wide range of domains including social science, advertising and policing. Performing social media analysis in real-time on streaming data can give insights into events as they occur. Similarly, low latency querying of historic data is also valuable. However, the rate at which new data is generated makes it a real challenge to design a system that can achieve these two goals. This talk will describe and demonstrate such a system. It is cloud-based, and exploits both continuous query, and NoSQL database technology (Cassandra). Evaluation results are presented which show that the system can scale to process queries on data arriving at the rate of the full Twitter firehose.

2. The Urban Observatory
Phil James, Newcastle University

Understanding how the city works now is the first part of the challenge if we are to make it work better in the future. Cities are complex, with interactions happening across many scales and sectors. The Urban Observatory is collecting and managing data from across Newcastle, integrating sensor platforms with citizen based data collection, geospatial and satellite data. It includes a new framework for integrating data into workflows, models and other applications. The Urban Observatory provides a baseline for gaining a richer understanding of how the city operates, and in the process repurpose and reuse the data across many different disciplines and applications. It is a key part of the £50m investment by Newcastle University in Science Central.

3. Big Data Analytics in the Cloud with e-Science Central
Hugo Hiden and Mark Turner, Newcastle University

e-Science Central is an open-source, cloud platform for Data Analytics. It supports the storage, sharing and analysis of bid data. Analysis is through scalable workflows that can combine services written in a variety of languages including Java and R. e-Science Central is portable across internal clusters of servers, as well as a variety of Public Clouds, including Azure, Amazon and OpenShift. Developed over the past 6 years, it now supports £M projects in academia and industry. This Talk will describe its design, give examples of its use, and demo its capabilities.

Eventbrite - Super Mondays - Digital Institute: Clouds, Sensors and Big Data

by supermondays at November 10, 2014 09:35 AM

November 08, 2014

Ryan Tomlinson

Consul: Service Discovery in a Microservice World

What is Service Discovery?

Service discovery is the concept of separating how processes and services find and communicate with one another. Typically this involves each service/node announcing themselves, some form of storage of the name and host/port of the service, and the ability to query (discover) this data from other services. In effect and in short, decoupling services.

Don't let "microservice" in the title of the post fool you. Service discovery is important on any form of the SOA continuum, whether you have 2 or 200 services.

Why Service Discovery? What's wrong with current approaches?

There are two most common forms of approaching service reachability.

  1. Directly contacting the host (Diagram 1)
  2. Directly contacting a load balancer or proxy (Diagram 2)

Typical inter-service communication

The first has the obvious disadvantage of being a single point of failure and lacking any form of resilience and is absolutely discouraged in any form.

The second is slightly better. You still directly address a host but your application services sit on nodes that the load balancer will route traffic to. If a node goes down then most load balancers will remove it from the pool and traffic will not be routed to it (hopefully). The load balancer will track the health of the nodes and manage their routing. Importantly, however, case two still suffers from a single point of failure and it still tightly couples Service A to Service B. If you decide to move Service B or change its location you have to update all consuming services (including Service A) with the new address.

What's so special about Consul?

Consul is a service discovery implementation built by the guys at Hashicorp (who also created a lot of other awesome tools including Vagrant and Packer).

The particularly genius part though is how they've implemented it. Most service discovery systems are invasive to an application in that each application must intermittently announce itself to a central cluster to tell it that's it's alive and available for traffic. When it wishes to remove itself from discovery it issues an unannounce command. Services will store a cache of other service definitions and therefore understand how to contact those services. The diagram below shows a typical topology:

Consul takes a different approach by building on top of DNS. In practice this means that each service node in your system contains the Consul agent (a process running on each box). The Consul agent is configured (flat JSON file) to know how to query the status/health of the service. Using the diagram above that may be, as an example "service-b-01.example.com:3000/service-status". If DNS doesn't suit you then it also provides a HTTP API. The agents are also responsible for reporting into the Consul server cluster.

Consul is also highly fault tolerant. If the whole service cluster for Consul goes down then this doesn't stop discovery. It does this by using a Gossip Protocol to manage membership and broadbast to the cluster. This means that Consul isn't just a typical client-to-server system, it's also client-to-client. In addition the server nodes use the Raft concensus algorithm to provide consistency for leader election.

Consul is datacenter aware by design. In fact the DNS schema in consul is addressable by datacenter. Service lookups are of the format:

[tag.]<service>.service[.datacenter][.domain]

So we might do:

$ dig @127.0.0.1 -p 8600 service-a.service.dc1.consul. SRV

Which would return

;; ANSWER SECTION:
service-a.service.dc1.consul.  0   IN  SRV 1 1 3000 service-a-01.node.dc1.consul.

;; ADDITIONAL SECTION:
service-a-01.node.dc1.consul. 0   IN  A   10.1.10.12  

You'll notice above, because it's just a DNS query, I can query directly for SRV records in case the service is running on a non-standard port 80.

Does it have a shiny Web UI?

Of course. You can have a play with the online demo here http://demo.consul.io/ui/

Consul Demo UI

Conclusion

This post really just touches the surface of what Consul is and how it works. Consul Template was recently released which is also incredibly useful and interesting if you're using HAProxy, Nginx or similar.

There's a lot of community tools gaining traction, particularly around Docker with both docker-consul and registrator looking interesting projects in the world of Docker.

by Ryan Tomlinson at November 08, 2014 08:45 PM

# This site is managed by the team at SuperMondays, is hosted by the team at Consilience and is maintained on github.