Recently I’ve been writing some automation code for Atlassian’s JIRA project management / bug tracking product. Think of JIRA as a generic AGILE-aware tool that can provide everything from project tracking all the way down to task tracking, trouble ticketing and so on.

The task I had set myself was to automatically create a ticket of a particular problem type (one that arises a lot) so that I could save a few minutes every time by not having to do so manually. That aim quickly became something that could plug into another script which could feed identified problems into my script so that it could generate the problem ticket on behalf of that tool.

What I learned was an interesting lesson in refactoring as I tried to optimize my code.

That API, Though

JIRA offers a REST API using JSON, so it has been a good experiment to shift gears from my recent NETCONF/XML scripting and play with REST/JSON instead. So roughly speaking, this is the sequence of events that needed to occur in order to generate a ticket correctly:

1. Create a new problem ticket
2. Add a specific user as a “watcher” (who will receive notifications on the ticket updates and transitions)
3. Change the ticket state to be “Acknowledged” (because we know about it; this isn’t for action, just for tracking).

Coding Process

As I’ve said before, I’m not a programmer. That means my scripts tend to get developed in steps as I build confidence in the methods I’m using. In this case, my first issue was to figure out how to access the REST API. I actually started off using curl to test out my theoretical process, with a JSON file for the POST data. This let me rapidly test out the data format and get feedback pretty quickly until I had a JSON file that would complete step 1 of the process. With that much in hand, I looked to turn this into a “proper” script.

Falling back on perl as usual (my go to language for quick development) I decided to use the LWP::UserAgent module as my HTTP/HTTPS mechanism and added a few other modules like JSON that I knew I would need to decode the output I was being sent. As my POST data, I took the JSON I had developed for use with curl and put it in a string in my script. After some playing around, I had code in place that would send that JSON data to JIRA and successfully create a new problem ticket. The JSON code was static – it didn’t change each time a ticket was created – but it was a good proof of concept, and provided a jumping off point for steps 2 and 3.

The Next Steps

With Step 1 working, I started working on Steps 2 and 3. Again I prototyped using curl then pulled the code into the perl script. The perl script ended up looking something like this (in pseudo-code form):

# Step 1
set up the request object
attach the JSON POST data to it
POST the data to URL1
check the output
grab the ticket# from the output

# Step 2
set up the request object
attach the JSON POST data to it
POST the data to URL2 (which includes the ticket#)
check the output

# Step 3
set up the request object
attach the JSON POST data to it
POST the data to a URL3
check the output (which includes the ticket#)

Why This Sucks

It’s probably obvious, but this code needs some (a lot of) work. For a start, I’m repeating tasks in the code – which screams out ‘make this a subroutine.’ So my first set was to take the request setup and functionalize that, and now my code looks like this:

# Step 1
$output = sendRequest(URL1,$json1);
check the output
grab the ticket# from the output

# Step 2
$output = sendRequest(URL2,$json2);
check the output

# Step 3
$output = sendRequest(URL3,$json3);
check the output (which includes the ticket#)

That looks a bit better, doesn’t it? I’m still repeating the “check the output” step, but right now that’s unavoidable because I need to extract information from some elements of that output in order to feed into the next step’s URL. As I worked on this I realized that I hated having the JSON defined within the script – it’s not exactly friendly to anybody else in the future who might need to tweak the JSON data after a JIRA update. And so I decided to move both the JSON and the URL into a template file.

Template Files

My template files ended up pretty simplistic. Comment lines use a # at the start and are ignored, with two exceptions (URL: and METHOD: lines), then the data (JSON in this case) follows:

# Template to create a new issue
#
# URL: http://jira.mycorp.net/Rest/Api/2/Whatever/Blah
# METHOD: POST 
{
    "some JSON"
}

By the way, if you change the METHOD (say, to GET), the program will issue a GET request instead. This has to be in the template to match the kind of request required for a function. Because all the key information is now kept in template files, my “calling code” starts looking like this instead:

# Step 1
$output = sendRequest('create-problem-ticket');
check the output
grab the ticket# from the output

# Step 2
$output = sendRequest('add-watcher');
check the output

# Step 3
$output = sendRequest('acknowledge-problem-ticket');
check the output (which includes the ticket#)

Variables

My JSON code, though in an external file now, is still static. I need to be able to insert data in real time. For example, when raising a problem ticket, I need to be able to insert the current time as the “Reported Time” field. So, ever one to reinvent the wheel, I added a token system and populated some of those tokens automatically within the program. For example, the current time can be inserted in the templates dynamically by using %%CURRENTTIME%% in the appropriate place, e.g.

{
    "reported-time": "%%CURRENTTIME%%"
}

I was asked recently “Why don’t you use one of the existing templating systems out there?” Ah. That would be because I don’t know about them, so I went about this in my own way. Probably in retrospect I’ll wish I had used an existing templating system, but that’s the fun of developing for yourself in blissful ignorance!

Checking the Output

After posting a query to a URL, I’ll get some kind of response. Sometimes the response is simply a “200 OK” to say sure, gotcha. Other times (like when you create the problem ticket), you’ll get back a brief response including data you’re going to want to keep and reuse, like the ticket number. Initially, I moved the “Check we got a 2xx response” code to the request subroutine, because it either worked or it didn’t, and we’d stop if it failed. So now my pseudocode looks like this:

# Step 1
$output = sendRequest('create-problem-ticket');
grab the ticket# from the output

# Step 2
$output = sendRequest('add-watcher');

# Step 3
$output = sendRequest('acknowledge-problem-ticket');

It’s starting to look a bit neater now, but I’d really like to clean up the one bit that remains hard-coded: extracting the ticket number from the returned JSON.

And so the template file got two more “reserved comments” as it were:

# TOKENS
{
    "TICKETNUMBER": "$.result.ticketnum"
}

# CONTENT
{
    json data
}

Now the POST data is found after the “# CONTENT” marker, and we have a new section “# TOKENS” after which we have some JSON that tells the script what data to extract and where to store it. In this case, I’m defining a JSON Path (the JSON equivalent of XML’s XPath) and indicating that the data found at this JSON Path should be stored in a token called “TICKETNUMBER”. That will then allow me to extract and store the ticket number, and the request in step 2 which references %%TICKETNUMBER%% will dynamically insert the number of the newly created ticket. Best way to do it? I have no idea, but now my “calling code” looks like this:

# Step 1
$output = sendRequest('create-problem-ticket');

# Step 2
$output = sendRequest('add-watcher');

# Step 3
$output = sendRequest('acknowledge-problem-ticket');

I like this. We now have a consistent way to call the three steps. In fact though, now I’m irritated that the sequence of three steps are hard coded in my script too. What if another step is needed later?

Recipes

Brutally stealing terminology from elsewhere, I decided to create a “recipe” that defined all the steps necessary to create a problem ticket, where each of those steps in turn was defined in its own template file. My recipe looks suspiciously like the pseudocode:

# Recipe for creating a new problem ticket
{
    "create": [
                "create-problem-ticket",
                "add-watcher",
                "acknowledge-problem-ticket"
    ]
}

But now I have a recipe called “create” that my program can load up, and it now knows the templates to call, in order, and each template will define any tokens that need to be extracted along the way. How do I avoid having “create” hard coded in my program? Put it on the command line. After all, I’m already going to give a filename to the program in order to include information about the problem ticket that will be attached when we create this, so why not also say which recipe to use?

This has a second benefit; now when I want to add a new function, say to mark a ticket as resolved, I can simply add a new recipe to the recipes file:

{
    "create": [...],
    "resolve": [
                 "mark-as-resolved"
    ]
}

See where I’m going with this? By abstracting each step of the process, I ended up with a script that contains none of the needed processes within it; they’re all template and recipe files, which means they can be easily edited and updated later. In fact I’ve left it so that you can call any recipe on the command line that it can find in the recipes file, so future updates don’t require any perl coding in order to support new recipes.

Calling the script (including specifying a filename for ticket content) looks like this:

jsontool -id JIRA -action create -file problem.txt

The program ID in this command line is something I haven’t mentioned. This provides an indication of what recipes and templates to pick, After all, if I have multiple scripts calling my jsontool each of which wanting an action called “create”, but each needing a slightly different ticket created, how do I distinguish? Simply put, in the templates directory, if you create a subdirectory named after a program id (e.g. “JIRA”) you can make a version of any template that is specific to the JIRA application and that overrides a template with the same name in the default directory. For example, the way I create a new problem ticket is very specific to the original JIRA script, so that has a template called ‘create-problem-ticket.json’ within the JIRA subfolder that overrides the default template with the same name in the default folder. However, adding a watcher and marking a ticket as resolved are done the same way whether it’s for a problem ticket or any other kind of ticket. Therefore, those templates only exist in the default template directory. Why do this? Because now I can define the recipe “create” once, and apply it to multiple applications simply by overriding any application-specific elements with their own templates.

Output

Oh wait, there’s more. In my original script, after executing the three steps the program would print the new ticket number to STDOUT so that the calling program could log the ticket that was generated on its behalf. How can I do that now? Well, the recipe file sounds like the ideal location for that, so I added a new section along these lines:

# OUTPUT 
{
    [
         "%%TICKETNUMBER%%\n"
    ]
}

In this case I want to print the ticket number followed by a newline. Bingo. I can output multiple lines if needed, and because I’m using the %%TOKEN%% format and substituting for that, I’ll be able to include any information that either the program creates or more like that I defined as TOKENS in my template file.

Reinventing the Wheel?

In creating this script I probably went over a whole number of areas where about a million people before me have already solved the problem (and better than I have). However, the learning process was valuable.

For example, while I originally tried to parse the JSON output by mapping it to a perl data structure, I realized pretty quickly that this became challenging for some more buried information, and in particular that referencing those elements from a JSON recipe file was tricky in the extreme. The solution was a JSON::Path module which provides a similar functionality to xPath (for XML). Now I can reference deeply-buried elements as a single string and let the module deal with it. I mentioned in a post recently that I’d had this revelation and suddenly saw why Jeremy Schulman had gone with xPath definitions in the Junos PyEZ library. I now have an idea of why in addition to what.

Similarly the idea of creating a recipe containing smaller steps is far from new. Nonetheless, the iterative development process led me to a solution where I could easily add additional functions. In fact, this is the script I turned towards an A10 (as I mentioned in my last post). I created a new recipe called getconfig to grab the running configuration from the A10, that included the necessary steps of authorization (from which I extract a session_id), showing the running configuration, then closing the session out, each of which has its own template file. To use this, I can issue a command like this:

jsontool -id A10 -action getconfig -object a10.mycorp.net

Refactoring

Ivan Pepelnjack – if I understood him correctly in our Software Gone Wild podcast – referred to this process of revision and making code generic, as “refactoring”. So I guess I am refactoring this code, and I’m pleased with the end result. From a script that originally had a single purpose, I’ve ended up with a generic REST/JSON engine that I can use across multiple systems. I like this; I’m getting to reuse my code!

Is it perfect? Of course not. Remember: IANAP (I Am Not A Programmer)! I am 100% a scripter and hacker, at best. But I mean well and generally can make things work. I simply know that “real” programmers would likely laugh at what I do. Nonetheless, since I suspect there are many other folks out there who are also not programmers, and I hope that by sharing my own process it might prove useful to others or, even better, that I’ll learn something from the comments! Be gentle 🙂