BigInsights Quicker Start


I’ve been taking a break from Liberty and JAX-RS recently to start tinkering with IBM’s BigInsights Hadoop distribution. To make things easier/more interesting my first attempts were using the Analytics for Hadoop service on BlueMix. In case it helps anyone, here’s what I ended up with before needing to install BigInsights myself:

And this is the script I used to upload data in the video (unfortunately I didn’t have any luck using the HttpFS API):

#!/bin/sh

BIUSER=biblumix
BIPASSWORD=password
BIURI=https://hostname:8443/data/controller/dfs

curl -iv –user ${BIUSER}:${BIPASSWORD} –insecure -X POST “${BIURI}/user/${BIUSER}/sample-data?isfile=false”

curl -iv –user ${BIUSER}:${BIPASSWORD} –insecure -X POST “${BIURI}/user/${BIUSER}/sample-data/orgdata.unl” –header “Content-Type:application/octet-stream” –header “Transfer-Encoding:chunked” -T “orgdata.unl”

curl -iv –user ${BIUSER}:${BIPASSWORD} –insecure -X POST “${BIURI}/user/${BIUSER}/sample-data/persondata.unl” –header “Content-Type:application/octet-stream” –header “Transfer-Encoding:chunked” -T “persondata.unl”

Notes:

  • since recording the demo Bluemix has added a United Kingdom region, however it looks like the Analytics for Hadoop service is currently only available on the US South region.
  • there is also now a BigInsights service on Bluemix which allows you to provision multi-node Hadoop clusters.

Modelling restful properties


I’ve recently been playing with Liberty and JAX-RS and in an effort to remember some of what I’ve discovered, I’m going to try and keep a few notes and post them here, probably along with a few questions. If anyone else finds them useful, or knows the answers, that’s a bonus!!

To start with Creating an efficient REST API with HTTP provides a nice overview of REST APIs and JAX-RS basics has a great simple sample application to get going with… and break!

JAXRSServiceModel_Main

Armed with the basics I thought it would be interesting to model the sample application to compare the working code with what Rational Software Architect (RSA) would generate for me. Obviously that’s not the most complex model in the world but it was good to have a few examples to follow:

(Those articles made much more sense when I’d tracked down all the likely looking JAXRS, REST and UML features in the RSA installer!)

Before generating any code, it was quite nice to get some basic API documentation out of the model. Not as fancy as using Swagger UI but certainly better than no documentation!

systemproperties-apidoc

It’s only a start but that’s all for part 1. Hopefully there’ll be more posts at some point when I get further, probably along these lines:

  • Creating an OSGi Web project and generating some code
  • Using a REST client to check it works, including a puzzling Wink problem
  • Adding some debug
  • Expanding the sample with some awkward long running processing
  • Deploying to Bluemix
  • Anything else I encounter along the way!

If you know of any good articles/books/fancy new media that would help, please leave recommendations below. If there are better ways to design REST services (Swagger looks interesting but I haven’t had a chance to investigate), please share them. And any other tips, comments, or questions of your own are also very welcome!

 

Hursley 3D Printing Expo


D’oh, looks like I missed a swarm of 3d printers in Hursley recently! I wonder if anyone has printed a model of the house/site yet.

I’m still looking for even a vaguely plausible excuse to splash out on a 3d printer, but printing models or new 3d printers still isn’t quite enough to justify the money (or space these days)!

Java dumps


I recently had to debug a problem with the MDM Workbench where exporting a tailoring project for Information Server didn’t do anything. In fact it didn’t even report any problems!

Unfortunately the code in question likes to put a brave face on things and just reports that everything was OK, even when something goes wrong. This was the perfect opportunity to try out some of the diagnostic tools available for the IBM Java runtime, which I’ve been meaning to try for ages. I had an idea where the problem was likely to be but to find out for sure I started the workbench using the following command line:

eclipsec -vmargs -Xdump:system:events=catch,filter=java/lang/AbstractMethodError#com/ibm/mdm/tools/export/infoserver/job/MDMDatabaseDAO.queryDatabaseWithoutFilter

Sure enough the failing export produced a dump which I could check using the Memory Analyzer tool. You can get the IBM version via IBM Support Assistant but it’s probably easier to get the standard Eclipse Memory Analyzer and add the required IBM plugins from the DTFJ update site.

I’m fortunate enough to work in Hursley so I could pester someone who works on IBM Java runtime diagnostics, but there’s also a helpful article on developerWorks with details of how to trigger dumps, and how to run queries using OQL:

Debugging from dumps: Diagnose more than memory leaks with Memory Analyzer

So mystery solved- if you have an Oracle database and want to exporting tailoring projects for Information Server, make sure you set up the database connection with a more recent JDBC driver than the defaults.

Parent friendly holiday


OLYMPUS DIGITAL CAMERA

Well it’s already two weeks since we got back from our first family holiday, and I did promise to report back, so while Buzz Light-toddler appears to be sound asleep here’s a very quick review of our stay in Dorset.

The plan was to find somewhere close to home in case things didn’t go too well. There was also a slight danger that we’d need two cars/trips to get everything there, so we wanted to find somewhere already equipped for babies. We found Hastings Farm Cottages on the Child Friendly Cottages website, and it had all the big stuff (cot, highchair, pushchair, etc.) provided, giving us a fighting chance of packing everything else for our four month night stay into a relatively small car!

I have no idea if all the Child Friendly Cottages are the same standard but Cowslip Cottage was excellent and came with everything we needed. Well, except for electricity for a bit but we were looked after with some hot water for an emergency cup of tea! (Entertainingly there was a small field of solar PV panels nearby but those were no use when the main supply is off, to protect the inverter- luckily they also had a generator to plug the kettle in to!)

It may not be far from home but Dorset has plenty of things to keep a toddler entertained, and Hastings Farm was a fabulous spot. We arrived with a baby, who decided it was a perfect place to start walking, so we had our own entertainment- chasing a new toddler!

Fortunately everyone survived and we now know that we can manage holidays, and we probably don’t need a bigger car. (They need less stuff when they get older, right?) So, any suggestions for where to go next?

Certifiable


I recently switched to a new ISP, who have so far been excellent, however they use certificates signed by CAcert. While I generally agree with the principle behind that decision, it does make life difficult. They cheerfully say, “You can check the certificate is signed by CAcert, if you like, before accepting it.” But how?

Warning: the following approach to checking the certificate is signed by CAcert is quite likely to be rubbish, so it’s probably not a good idea to follow it! In my defense, it seemed like a reasonable balance between just accepting some random certificate and complete paranoia but if you know a better way, please let me know.

They aren’t on Windows but the CAcert root certificates are already included in various places, so it turns out that the simple answer might be to grab the certificate from a suitable Linux distribution. Just to be on the safe side, I wanted to find a distribution I could download securely. The best option I found was Tails, which has a secure download and, for extra peace of mind, can be verified with OpenPGP.

My chosen method for trusting the tails signing key was a tad more interesting on Windows due to the lack of an sha256sum command. Luckily it seems you can do anything in PowerShell, so with a little help from Brian Hartsock’s blog, this did the trick instead:

$ha = [System.Security.Cryptography.HashAlgorithm]::Create(“SHA256″)
$stream = New-Object System.IO.FileStream(“tails-signing.key”, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read)
$sb = New-Object System.Text.StringBuilder
$ha.ComputeHash($stream) | % { [void] $sb.Append($_.ToString(“x2″)) }
$sb.ToString()

 

All good, certificate verified. I would still rather Andrews & Arnold just used a proper certificate though: there are clearly problems with trusting all the certificate authorities that are included in browsers/operating systems by default but CAcert doesn’t exactly look like a fantastic example either, and normal users really don’t have any chance of making a more informed choice.