Hands-on experiment building microservices and cloud native applications
View project on GitHub
There are lots of new technologies and application patterns associated with microservices. If you're like us, learning from reading someone's sample just isn't enough. Game On! is a throwback text adventure game whose purpose is to give intrepid developers who like to experiment "something to build". This blog describes our own misadventures as we do the same.

Advanced KubecCtl

Advanced Kubectl

Over at Game On! Dev Central recently, we’ve been heads down figuring out how to take the work we did for Game On! on Kubernetes, and make it even simpler for users to get started.

So our resident techno-wizards have come up with a pile of scripts to help orchestrate the deploy onto K8S, and do some of the config and setup and boring backgroundy stuff that’s required to get a polyglot microservice app up and running in your cluster. (Apparently the ‘techno-wizards’ prefer other titles like “Software Engineer”, but that doesn’t sound anywhere near as much fun! so.. Techno-Wizards they shall be!)

Anyway, one of them came wandering over and started describing ancient incantations he’d recently learned that allowed him to check the status on a bunch of pods. I had him write down some notes to share with the wider world, because we all know that sharing ancient magics never leads to the ressurection of sand gods bent on destroying the modern world, but instead leads to enlightenment and happiness.

The tale here starts with a simple requirement…

  • “Find out when Game On! has finished launching in the cluster, and is ready to accept requests”

The simplest approach here is kubectl get pods, and read the output, and wait until the appropriate pods are ready. Of course, Game On! deploys to it’s own namespace, so the command becomes kubectl get pods -n gameon-system.

As you can see from the output, we can easily see which pods are ready by just looking at the READY column, and looking for the 1/1 entries

kubectl get pods -n gameon-system
NAME                      READY     STATUS    RESTARTS   AGE
auth-1816639685-xejyk     1/1       Running   0          26m
webapp-1816639685-xejyk   1/1       Running   0          26m
mediator-1816639685-xejyk 1/1       Running   0          26m

We can read the output with grep, and just look for 0/ to know when a pod isn’t ready yet.

It’s a tad crude, but it works, or rather, worked. Right up until we added istio.

Once istio was part of the application, each pod now has two containers. One as the app itself, and another as the istio sidecar. Unfortunately, this means our trick for checking 0/ fails, because 1/2 doesn’t start with 0/ so we believe the pod is started. And since the istio sidecar starts blazingly fast, and our Game On! service needs to boot a JVM, it’ll stay in 1/2 for quite a while before the app is actually working.

So, we could improve our grep..

kubectl get pods -n gameon-system | tail -n +2 | awk '{print $2;}' | awk -F "/" '{if($1==$2){print "OK";}else{print "BAD"}}' | grep BAD | wc -l

As a first pass, this seems almost sane ;)

  • get the pod list
  • lose the headers
  • grab the ‘ready’ column only
  • test if the values each side of the ‘/’ char are equal
  • check if any lines had values that didn’t match

You could improve it a little if you know about --no-headers, but the concept is the same, it just uses awk to process the output to create something we can use grep with.

If only there was a better way…

Kubectl supports “jsonpath” a neat little option that on the surface, allows you to retrieve metadata fields even if they are not supported by kubectl’s --field-selector argument. eg. kubectl get pod auth -n gameon-system -o jsonpath='{}' will return just the name of the pod

You can use it with lists of items too, eg kubectl get pods -n gameon-system -o jsonpath='{}' will output the names of all the Game On! pods. Jsonpath is used by a bunch of stuff, and even has some awesome online jsponpath evaluators where you can test out syntax. Jsonpath includes filters, that allow you to reduce lists to only items matching a filter.

“Great!” went the Techno-Wizard, I will use that to get the list of all Game On! pods, but filter it to only the Pods that are not ready.

Sadly, there are some minor differences between the implementations, and one of those ends up being quite the drawback for K8S.

The issue is that the status we want to query is inside a list inside the pod declaration. And the pods themselves are in the list we are trying to filter. If we wanted to express this in jsonpath, it would end up a little like this…


Here we are saying take the list of items, where the list of containerStatuses contains an instance of ready being false. Except that involves processing a nested array, which the K8S JSONPath can’t handle.

So instead, we’ll use the {range} support in kubectls jsonpath support.

kubectl get pods -n gameon-system -o jsonpath='{range .items[*]}{" "}{}{"="}{range .status.containerStatuses[?(@.ready==false)]}{""}{.ready}{end}{end}

This will use range to iterate over the pods (in the items array), outputting the name of each pod, then using a nested range to iterate over the statuses, selecting only those where ready is false.

For pods that are ready, we’ll see output like auth= and for not ready pods we’ll see web=false or possibly web=falsefalse (because false will be output for each time a container stbut atus is false, and with istio-sidecar & the app, there are two possibilities). The output will all be on a single line, but with a little work we can fix that.

If we throw in a liberal sprinkling of tr commands, and a little grep magic, and a dash of sed.. tada..

kubectl get pods -n gameon-system -o jsonpath='{range .items[*]}{" "}{}{"="}{range .status.containerStatuses[?(@.ready==false)]}{""}{.ready}{end}{end}' | tr ' ' '\n' | grep "=false" | sed -e 's/^\([^-]*\).*=false.*/\1/g' | tr '\n' ' '

Here we take the space separated string, swap the spaces for linefeeds, then use grep to filter to only the “not ready” pods, then use sed to convert the container names back to their gameon short service names (eg. auth-deploy-c54875487-fhre9s becomes auth), before finally using tr to swap the linefeeds back into spaces!

But why?

Because that gives us a command that everytime we run it will give us back the names of the services that are not ready yet.

And when we wrap that into a script that calls it at regular intervals, we can meet our initial goal with a little style, by listing the services we are still waiting for.


while [ true ]; do
  NOTREADY=$(kubectl get pods -n gameon-system -o jsonpath='{range .items[*]}{" "}{}{"="}{range .status.containerStatuses[?(@.ready==false)]}{""}{.ready}{end}{end}' | tr ' ' '\n' | grep "=false" | sed -e 's/^\([^-]*\).*=false.*/\1/g' | tr '\n' ' ')
  if [ -z $NOTREADY ]; then
  echo -en "\rWaiting for [ $NOTREADY]$DOTS"
  sleep 5

(Extra points if you can spot what’s wrong with the DOTS handling ;) )

And that concludes our tale, as the Techno-Wizard returns once more from whence he came, mumbling about PodSecurityPolicies and allowedCapabilities, but that will have to be a tale for another time.

Moving GameOn to Kubernetes

Game On! and Kubernetes

Here at Game On! central, we’ve been considering how to run on Kubernetes for quite some time, and during a pause in the recent Holiday Season, we had a chance to add Kubernetes support allowing you to run the core services locally.

This work partly builds on an earlier attempt to bring Game On! to Kubernetes back in May last year. We learnt a few lessons from that attempt, and like to think this attempt is a little cleaner.

Running Game On! in Kubernetes

Want to try it out? we’ve tested on minikube, and IBM Cloud Private (ICP).

It’s as simple as;

  • Setup your minikube/icp, and have kubectl installed locally, and configured to talk to your cluster.
  • For minikube, you’ll need to enable ingress with minikube addons enable ingress
  • For ICP, this means you must execute the ‘configure client’ steps to authenticate to the cluster
  • Clone the gameon root repository
    • git clone
  • cd to the cloned project, then into the kubernetes directory
    • cd gameon/kubernetes
  • set the environment variable GAMEON_HOST to the ip of your kubernetes cluster, if you are using minikube, this is probably, if you are using ICP via Vagrant, it will default to (unless you altered the network in the Vagrantfile)
    • export GAMEON_HOST=
  • run ./ up this will do the following for you..
  • create the kubernetes namespace gameon-system
  • edit ingress.yaml and gameon-configmap.yaml to insert your cluster ip as appropriate.
  • create a new self signed certificate/keypair, and load it into a kubernetes config map
  • create the ingress, config map, couchdb, kafka, auth, mediator, map, player, and webapp deployments and services in kubernetes.

To check on progress, you can check the cluster gui, or use kubectl to check on the status of the pods in the gameon-system namespace. ( kubectl get pods --namespace=gameon-system -o wide)

After a short while, once the the pods are all up and running, you should be able to access your Game On! at


Swap for your cluster ip, the same value you set in GAMEON_HOST

If you want to remove Game On! from your cluster, run ./ down and the script will remove the artifacts it created.

Whats actually happening here?

Each of the core services has their own yaml, that contains both a kubernetes service, and a deployment. The deployment defines a replica set that will deploy the appropriate Game On! image in a pod, with the appropriate environment variables set with values from the config map defined by gameon-configmap.yaml. The certificate used by the core services to sign and validate JWTs is mapped into each container as a file, where the startup scripts are able to convert it into the keystore/truststore that are required by the Java pods.

The ingress.yaml defines a kubernetes ingress that acts as the front door for the whole setup, routing requests to the appropriate service. Requets are mapped using the path part of the URL just as happens in the proxy service when running outside of Kubernetes. ingress.yaml also defines and enables HTTPS for the front door, and if we wanted, we could configure a certificate to be used for HTTPS in the yaml here.

Lastly, we’re using which is a handy service that allows you to create hostnames that map to ip addresses, without needing to edit /etc/hosts etc. Any url ending in resolves to ipaddress, so in our case will resolve to, this is great, because ingress definitions require a hostname, not an ip address, and this lets us create one for testing, without any setup.

TIP: your router may be “helpful” and block dns resolution for ip addresses in private networks, such as 192.168.x.x or 10.x.x.x, if so, look in your router for if you can configure a Domain Whitelist for for RFC1918 responses. This is required at least for OpenWRT/LEDE. If you can’t unblock that, you’ll have to resort to editing /etc/hosts or equivalent for your platform.

Pods of fun!

Now it’s running inside Kubernetes, in future, we’ll be able to scale the number of services, for better availability, and to enable graduated rollouts of newer service versions.

For now, thanks to handy tools like the IBM Cloud Private monitoring service we can look at say, how much ram each pod is using…

You can see similar, but less detailed information from minikube’s dashboard, or install heapster with minikube addons enable heapster

As becomes quite apparent from the charts, Kafka eats almost 5 times the memory of our average service! Why not have a dig around and see what you can find out!

Climbing inside your code: A unique look at the game

Long long ago, in a land far far away (also known as QCon San Francisco, 2016), I met John Voorhees and Avik Das from Primitive, a small startup building VR tools that allow you to visualize your code in an immersive way.

We were also talking about Game On!, because we’re about helping you experiment with new technologies and concepts in a differently immersive way.

They scanned and built a visualization of the core game services, and brought it to JavaOne! Check it out, it is seriously cool stuff.

Lagom gets in the game at JavaOne 2017

JavaOne was a month ago! No way! Time sure flies.

Our lab with Lightbend was an excellent adventure, complete with the exciting, heart-pounding, last-minute rush to get the materials working end to end. I don’t know about you, but that last minute hacking stuff is why I love what I do.

I enjoyed working with the Lightbenders; they’re a great team, with solid ideas around how to build reactive applications. What we demonstrated in the lab barely scratched the surface of what you can do with their framework. If you’re curious, you can check out the lab materials, but expect some new advanced adventures to grow out of this modest beginning.

Hugh and I had a blast talking to developers the booth, and generally being ridiculous. So much fun!

Devops Days in Raleigh

Venu Beyagudem, Jarek Gawor, Ranjan Kumar, and I (Heather Saunders) brought Game On! for its debut at DevOps Days in Raleigh, NC in early September.

DevOps days is a series of conferences across the world that covers topics of software development, IT infrastructure, and of course the DevOps principals that connect them and make them tick. Topics for speeches range from monitoring tool selection, to organizational culture. If you’re interested in seeing if DevOps days is coming near you, check out

IBM was a sponsor for the Raleigh conference, as well as a few others (find us in Ohio in November!), and wanted a little something more than just promotional material at the booth, so they asked if anyone might be able to present Game On! alongside the usual devops and developer tools.

As it turns out, Game On! was the perfect medium for engaging the attendees at the conference, who were mostly developers and devops professionals like us. They were interested in the game and the community around it. Once we explained that IBM was interested in helping developers learn in a fun and connected open source way, almost all the participants were interested in hearing more. A few even forgot that they had only approached the table to get their “conference passport” stamped in the first place.

I look forward to seeing how the game grows as a result of introducing it straight to developers at events like this, and am glad to have joined the Game On community myself!

Spring Room!

We have a new sample room written in Java with the Spring framework:

This post is a brief overview of the creation process.

Jumpstart with code gen

Using this guide, we made a Spring microservice starter that could be deployed to Bluemix with just a few commands:

  $ bx dev create
  ? Select a pattern:                        
  1. Web App
  2. Mobile App
  3. Backend for Frontend
  4. Microservice
  5. MFP
  Enter a number> 4

  ? Select a starter:
  1. Basic
  Enter a number> 1

  ? Select a language:
  1. Java - MicroProfile / Java EE
  2. Node
  3. Python
  4. Java - Spring Framework
  Enter a number> 4

  ? Enter a name for your project> springmsdemo                             
  ? Enter a hostname for your project> springmsdemo
  ? Do you want to add services to your project? [y/n]> y

  ? Select a service:
  1. Cloudant NoSQL Database
  2. Object Storage
  Enter a number> 1

  ? Select a service plan:               
  1. Lite
  2. Standard
  3. Dedicated Hardware
  Enter a number> 1

  Successfully added service to project.               

  ? Do you want to add another service? [y/n]> n
  The project, springmsdemo, has been successfully saved into the current directory.

Time to fill in our own code!

Borrowing from the Java room

The Java room had a lot of code we could reuse, so that’s what we did.

Notable differences:

  class WebSocketConfig implements WebSocketConfigurer {

      SocketHandler handler;

      public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
          registry.addHandler(handler, "/room");
  • We also needed to use a Spring WebSocket (org.springframework.web.socket.WebSocketSession) instead of the standard one (javax.websocket.Session). It looks roughly like this (simplified): (org.springframework.web.socket.WebSocketSession) instead of the standard one (javax.websocket.Session). It looks roughly like this (simplified):
  public class SocketHandler extends TextWebSocketHandler {

      private final HashMap<String, WebSocketSession> sessions = new HashMap<>();
      private RoomImplementation roomImplementation;

      public void afterConnectionEstablished(WebSocketSession session) throws Exception {
          sessions.put(session.getId(), session);
          session.sendMessage(new TextMessage(Message.ACK_MSG.toString()));

      public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
          Log.log(Level.INFO, this, "WebSocketSession with Id (" + session.getId() + ") closed with reason: " + status.getReason());

      public void handleMessage(WebSocketSession session, WebSocketMessage<?> message) throws Exception {
          roomImplementation.handleMessage(new Message(message.getPayload().toString()), this);

      public void sendMessage(Message message) {
          for (WebSocketSession s : sessions.values()) {
              sendMessageToSession(s, message);

      private boolean sendMessageToSession(WebSocketSession session, Message message) {
          try {
              session.sendMessage(new TextMessage(message.toString()));
              return true;
          } catch (IOException e) {

After porting some common code, we can run mvn spring-boot:run to see our Spring room running locally!

Deploying to Bluemix

After adding Travis, Docker, and JaCoCo, we were ready to deploy to Bluemix:

bx dev build


bx dev deploy

and our room is deployed as a Cloud Foundry service with a reachable endpoint!

Registering the room with Game On!


/teleport spring-sample

Simplified Docker Compose for local development

We now have a special branch in the gameon repository that contains a snapshot of a smooth end-to-end local development flow with docker-compose.

> git clone
> cd gameon
> ./ up


Yes, yes. Kubernetes is next, we promise. But with this revision of the Docker Compose path, we dropped a lot of complexity (bye, Amalgam8!) and created a simplified path for getting started. The has been updated with the new TL;DR instructions, and they are ever so much better (and more concise!).

We’ll try to cut “releases” of the gameon repository a little more frequently, so there is always a last-known-good state to work from (rather than always using whatever the most recent / experimental docker images are).


Stay tuned! Enjoy! Go play!

Headed to JavaOne 2017!

That’s right, Game On! is going back to JavaOne this year!

Frenetic activity has ensued (wow, does this feel familiar), as we prepare for the conference. If you’re a frequent visitor, there may be some bumps as we clean things up!

Both a Hands-on Lab and a general session feature the game. Be sure to put them on your schedule!

  • Introduction to Reactive Systems [HOL7896]

    In this two-hour workshop, IBM and Lightbend are teaming up to introduce you to developing reactive microservices on Kubernetes. Lagom is a microservice framework built atop Lightbend’s tried-and-proven distributed systems toolkit, Akka. This framework helps you build microservices as systems—reactive systems, to be precise—so that your microservices are elastic and resilient by nature. The reactive programming model aims to ensure systems availability and performance while processing streaming data. You will use Lagom’s development environment to avoid tedious setup and scripting when developing locally and will deploy your application to a Kubernetes cluster, so others in the workshop (and at JavaOne) can test out what you built.

    Track: Oracle Code: Containers, Microservices, DevOps, Databases, APIs, and more
    Experience Level: Advanced
    Session Type: HOL (Hands-on Lab) Session


    • Erin Schnabel, Senior Software Engineer, IBM
    • Duncan DeVore, Principle Enterprise Architect, Lightbend
  • Choosing a NoSQL API and Database to Avoid Tombstones and Dragons in Game On! [CON1756]

    Game On! (@gameontext – is an awesome throwback text-based adventure built with microservices. Completely open source, it enables everyone to choose their own adventure to learn about microservices concepts while extending the game. One of the core services is the Map, which maintains a two-dimensional map containing all the registered rooms. The Map started with a document store as a back end, but as the Map changed over time, tombstones started to accrue. And then people started to ask how to manage three dimensions, and dragons appeared. Come to this session to find out why the decision was made to change the NoSQL back end, how it was done, and the result of the change with a new NoSQL API (

    Track: Oracle Code: Containers, Microservices, DevOps, Databases, APIs, and more
    Experience Level: Intermediate
    Session Type: Conference Session


    • Leonardo Lima, CTO, V2 Tecnologia Ltda.
    • Erin Schnabel, Senior Software Engineer, IBM
    • Otavio Santana, Developer, Soujava

We have a Scala Room!

What follows is the stream-of-conciousness and research that lead up to our newest sample room!


Day 0

So for me 2017/18 is going to include adventures in polyglot.

This will include but will not be only restricted to Kotlin,Prolog, Erlang, Racket, Haskell, Closure, Scala

Day 1

Decide on tooling.

Learn to stop hating and love regexes

scala> sample
res69: String =
    "username": "username",
    "userId": "<userId>",
    "version": 1|2

scala> pattern
res70: scala.util.matching.Regex = (?s)(\w+),([^,]*),(.*)

scala> val pattern(target,id,payload) = sample
target: String = roomHello
id: String = <roomId>
payload: String =
    "username": "username",
    "userId": "<userId>",
    "version": 1|2


Room on a chip

Game On! is a fun little text adventure written using a Microservice architecture. It’s also extensible, allowing users to write their own ‘Rooms’ (locations within the text adventure world), that run as Microservices, in the cloud, on their own systems, etc.

I’ve played with Arduino’s and Raspberry Pi’s, and similar for quite some time. I tried having an Arduino read data from a Floppy Drive, which was fun, and ultimately led to a crazy floppy disk autoloader. From there, I ended up moving on from Arduino to the Maple, then from there to a Teensy 3.0, and most recently, a rather fun little collection of boards based around a Chip known as the ESP8266

esp8266 board

There are some nice things about one of the ESP8266 boards I had (Witty Cloud / GizWits):

  • you program it using the Arduino API,
  • it has built in WiFi, 4MB of flash to store your code, 64k of instruction ram, and 96k of data ram to play with.
  • it runs at 80mhz or so
  • it has an LED light that can change color
  • it has a light sensor

That’s pretty impressive for something that costs around 5$ shipped.

When you know there are Arduino JSON libraries, and Arduino WebSocket libraries that can run on it, you start wondering: is it possible to host a Room for Game On on an ESP8266 ?

I had a spare moment at a weekend, so decided to find out! =)