Not content to merely play with NETCONF, I have taken a side step briefly to complete another little tool that relies on a REST API and JSON. I’ll come back on the NETCONF script soon, and explain how I’m using YAML for configuration files, but evidently I’m a sucker for punishment as in the last two weeks I’ve been building queries and parsing data structures in XML, YAML and JSON.
Hang in there – the acronyms aren’t nearly as bad as they seem.
XML
XML (eXtensible Markup Language) is pretty simple in its own way. If you create good HTML (by which I mean you actually close your tags properly), XML should be obvious:
<?xml version="1.0" encoding="UTF-8"?>
<email type="insecure">
<to> Alice</to>
<from>Bob</from>
<Subject>Encryption</Subject>
<body>I'd like to tell you something, but this is an insecure channel.</body>
</email>
Pretty simple, right? It’s a clean hierarchical data structure.
JSON
JSON (JavaScript Object Notation) is a little cleaner than XML (certainly there’s less typing), though it’s still highly dependent on correctly-closed tags. The same information as above could be represented in JSON as:
{
"email": {
"type": "insecure",
"to": "Alice",
"from": "Bob",
"Subject": "Encryption",
"body": "I'd like to tell you something, but this is an insecure channel."
}
}
Same idea as XML (although the attribute ‘type = “insecure”’ doesn’t translate directly); it’s a hierarchical structure, although this time you simply have to close the braces, which means a little less typing.
YAML
YAML. What does YAML stand for? Supposedly it originally was “Yet Another Markup Language” (with a gentle nod, I suspect to YACC). More recently though it is accepted to be a somewhat recursive acronym, “YAML Ain’t Markup Language”. Yeah. So, YAML is very similar to JSON, but similar to Python, indenting is semantically important. Our example above in YAML would be:
---
email:
type: insecure
to: Alice
from: Bob
Subject: Encryption
body: "I'd like to tell you something, but this is an insecure channel."
Seems simple enough, right? Certainly it’s cleaner than JSON, and much nicer to read than XML, although with all the above, small details turn out to be critical, and with all the above, there’s more to it than can be conveyed in a quick example.
Changing Flavors
My challenge in the last week has been that I have had to flip between all three formats for different purposes.
I’m using YAML to store configurations for my NETCONF script, specifically a list of all the devices and their types. I chose YAML because it seems to be the most human-friendly configuration format so should be simplest for others to maintain in the future.
For my NETCONF queries and responses, I’ve been working with XML to send requests and to interpret the results sent back.
Finally for my latest diversion with a REST API, the data format is JSON, so I’ve been working with that format for my queries and responses.
I assure you, it isn’t pleasant jumping formats continually. Maybe better folks don’t find it a problem, but it’s hard work for my poor little brain. Still, so far it has all been successful.
XPath
One thing I learned about XML courtesy of the Junos PyEZ library is XPath. XPath makes it simpler to dig into those hierarchical XML structures in a much simpler way than if you tried to process the hierarchy yourself. The PyEZ library uses XPath to grab information from queries and turn them into tables for easy processing by a script. Using XPath libraries is much nicer than doing it yourself, believe me.
JSON Path
Working with JSON for a REST API, I realized that I needed to pull information out of a JSON structure, but sometimes that information was buried within an array. I had started out with a simple structure where I wanted to pull out individual array elements so I defined them with a list of hierarchical elements. But then I found a data structure where I hit a regular old array containing repeating elements, and I needed a list of values from the repeating elements. Wait, let me explain that better.
My original queries were simple references to values stored in a hierarchical structure. For example in this JSON:
{
"email": {
"type": "insecure",
"to": "Alice",
"from": "Bob",
"Subject": "Encryption",
"body": "I'd like to tell you something, but this is an insecure channel."
}
}
… to grab the author of the email, I might process the JSON and request $json->{’email‘}->{’from’}. But what if there were more than one email? The structure might look like this instead:
{
"email-list": [
{
"email": {
"type": "insecure",
"to": "Alice",
"from": "Bob",
"Subject": "Encryption",
"body": "I'd like to tell you something, but this is an insecure channel."
}
},
{
"email": {
"type": "secure",
"to": "Dave",
"from": "Gena",
"Subject": "Nothing Important",
"body": "AA1200291222BAAGHh8827SKDSS011263 9R8B91BV9128v81bbvvG881723_012387612361512618bbash7bbvVIBBVqu177a97VvO87V78LV"
}
}
]
}
Let’s say I want to check that Gena has sent one of these emails. How do I extract this data? Well, unfortunately “email-list” is effectively an array of data, so I would need to find the list, then look into each “email” object and the extract the from element from each one so that I could build up an array of the senders against which to compare my chosen author. JSONPath allows me to specify parts of the data in the same way as XPath – and it’s clever.
For example (using perl’s JSON::Path module) if I wanted an array of the emails, I could use JSONPath of:
@emails = '$.email-list.[*]';
How about though if I want those email authors? Well, JSONPath helps there too:
@authors = '$.email-lists.[*].email.from';
In other words, for each element within the array under email-lists, grab the ‘from’ element, then put those all in an array called @authors. The nice thing is that the JSON::Path module hides the fun required to do this (especially with more complex queries).
Why JSON Path?
I had written code to pull in a JSON file defining data to pull from a REST response. Then I had to deal with arrays, and began writing code to try and cope with finding an array in the middle of the path of elements, and as I spent time thinking about how I was going to do this, I heard the voice of Jeremy Schulman in my head saying “XPath … You want to use XPath.” Really! XPath is for XML but, I figured, surely somebody has created the same for JSON? And indeed they have. JSONPath is a clone of the XPath approach, and takes away all that coding pain for me.
What’s Your Point?
I don’t have one. Ok, maybe my point is not to reinvent the wheel. You know, as I started playing with XPath, I noted the strong similarities between what I was doing (extracting individual data elements, or arrays of data from JSON using JSONPath) with what I had seen in Jeremy’s Junos PyEZ library, where he was extracting tables of data form XML using XPath. Now I not only appreciate what he did with that, but I understand why he did it that way! As I tried to make my JSON code more generic (and applicable to more situations), I found myself using templates for the queries and templates to define information I wanted to extract. Again, I had ended up using the same approach as Jeremy’s PyEZ library did.
I emailed Jeremy this weekend with some amusement and explained this to him. It’s like I’m 2 years behind him, and I’m finally understanding not just what he did, but why he did it. My scripts are benefiting from even a relatively shallow understanding of what Jeremy did, and I now see his motivation to abstract so much of the underlying nonsense from the end user. Of course, now I’m thinking that if I were smart enough I could simply convert PyEZ to work with the other systems I need to use, but perhaps that’s a step too far right now. Either way, it’s great to have benefited from somebody else beating a new path and letting me see the light, albeit somewhat belatedly.
So thank you, Jeremy, for your vision in this area, and for putting the code out there in the public domain where idiots like me can play with it. I am by no means a career programmer, but I am loving achieving amazing things using abstractions that I first saw in Jeremy’s work – even though it took me some effort to realize that I was evidently hitting the self same issues he had hit!
Nerd on, Dude
Play with this stuff. None of us have much time, but clearly there are benefits to doing so. I’ve learned more in the last few weeks playing with NETCONF, REST API, XML, YAML and JSON than I’ve learned in months before, and maybe longer. I’m motivated again because I’m doing something new and cool. That alone, surely, justifies the time investment?
Hi John – Thank you as well. You’re on the cutting edge helping others to better understand these new concepts. It’s a community effort. I am happy to help with the tooling aspects and bring automation to a broader reach of folks.
Cheers!
— Jeremy
@nwkautomaniac