Following on from my round up of the Opengear presentation at NFD4 and my review of basic use of the ACM5004-G, I’m now keen to dig a little deeper and find out some more of what this clever little product is capable of.
Where To Start?
I decided that the first thing I wanted to play with was the Automated Remote Management and Support (ARMS) feature. Put simply, the idea is that you can define an alert condition of some sort, and that alert in turn can kick off notification and remediation actions. One of the simplest cases is to monitor the serial port for a particular phrase or pattern (regular expression), and take action accordingly. Jeremy Stretch uses this feature to detect attempts to do bad things to his online lab (e.g. formatting the flash), and once an alert is triggered, he has coded his Opengear console server to step in and cancel the command, then disconnect the user. It also then sends him an email letting him know who tried to wipe the device.
Opengear Pattern Matching
As a first try, I just wanted to match a specific phrase on the serial port (in this case, the word “goober”) to confirm that it would trigger an alert. The action I chose was to send me an email telling me that the word ‘goober’ had been typed. This is not exciting stuff, but I figured if I start with the basics and confirm that it works as expected, it’ll be easier to work on something more complex later on.
Programming the response was really simple. I went to the Auto-Response section, and added a New Auto-Response. I told it I wanted to monitor for a Serial Pattern, specifically to look for the regular expression “goober” to be transmitted on port 2 (i.e. sent to the Juniper EX3200). When detected, the Trigger Action I created was to Send Email (and the email can be customized). For this initial auto-response, I did not configure a Resolve Action. That’s it!
That Duh Moment
My first moment of idiocy is that I had not configured an SMTP server. In my defense, it’s possible that the Opengear ACM5004 might have been running sendmail and been able to directly contact remote MX servers without being told anything else, but that isn’t the case. So in the SMTP & SMS menu option, I set up my email server and the associated necessary authentication parameters. The only thing remaining then was to try the trigger and see if an email appeared.
Potential Improvements
Two things struck me as I went through this process.
- It would have been very cool if the system could have cross-checked when I created an email trigger, and notified me that I had not configured my SMTP parameters yet (and thus was creating an action that wouldn’t work). I do appreciate, mind you, that you can only protect the stupid from themselves up to a certain point 😉
- When configuring the SMTP parameters, it would be fabulous to have a “Test SMTP” button on that page so that I could trigger a test email and know that everything worked. I say that because, as I’ll explain in a moment, I was not getting the emails I expected!
Testing The Alert
I SSH’d to the specified serial port and at the login prompt typed the word goober. Checking the Opengear’s internal syslog, I found that the string had been matched and an email queued:
<14>Nov 26 00:36:14 /bin/ard[1455]: INFO /bin/ard - AR
config.autoresponse.autoresponse1 triggered. val goober
<14>Nov 26 00:36:14 /bin/ard[1455]: INFO /bin/ard - Queued
Email Action for config.autoresponse.autoresponse1
So far so good, and I checked my email. Nothing. I tested a couple more times, but again, no email. After digging around a bit I found that the emails are sent using msmtp, a utility that will send emails for you if you give it the necessary server information. In this case, all information is passed on the command line and it turned out that the alerting was being handled by two scripts located in /etc/scripts:
$ ls -al | grep mail -r-xr-xr-x 1 root root 1426 May 28 2009 alert-email -r-xr-xr-x 1 root root 418 May 28 2009 send-mail
The first script (alert-email) pulls together the SMTP server and authentication information necessary to send the email, and it then passes it all on to the second script (send-mail) which pipes it to msmtp.
By default it looks like msmtp does not log its activity, so I was not seeing anything in the syslog that would explain what had happened. And since msmtp is not a server process, there’s no ‘queue’ to look at – the program simply runs, tries to send an email, then closes. The documentation for msmtp revealed a handy command line option, “–syslog”, that would make it log details of the email it was asked to send, plus the outcome of the send attempt. Perfect! I should add that while there’s a global option to log things associated with the ARMS process (the description is pretty vague), I haven’t confirmed that it would also affect msmtp yet. For the sake of argument, let’s assume that the global option doesn’t make msmtp log its activity.
Editing Alert-Email
There’s a line in the alert-email script that makes it really easy to add in the –syslog option:
options="--connect-timeout=60"
This variable (options) gets appended as necessary by various conditional clauses and is inserted into the final command line that is passed over to send-mail. So all I needed to do to enable logging was to change that one line to read:
options="--connect-timeout=60 --syslog"
Once done, every call to alert-email would enable the syslog option, thus generating the log entries I wanted to see. Unfortunately there’s a catch when saving the updated script:
"alert-email" Read-only file system
Ah. The script I need to tweak is part of the main compressed filesystem, and I can’t touch it:
$ mount [...] /dev/root on / type squashfs (ro,relatime)
I have no idea if it’s possible to override and mount this as Read/Write, but from what I can understand about squashfs, without additional software (e.g. unionfs), it’s not possible to do so.
This gives me a problem; how do I edit a file on a readonly filesystem? I’ll cover my solution to that in the next post, but it is possible, although it’s neither fast nor ideal. However, these are the compromises that have to be made to squeeze so much out of so little hardware, so maybe it’s not so unreasonable. For this post, let’s take it as read that I managed to edit the alert-email script so that it now passes the –syslog option to msmtp.
Testing The Opengear Again
Another connection to the serial port and another spate of typing ‘goober’ and things are looking up – I have msmtp log entries showing up, and I have an explanation of what was going wrong:
<11>Nov 26 00:39:29 msmtp: host=smtp.myprovider.tld tls=off auth=on user=mylogin @myprovider.tld from=alert @opengear1. mydomain.tld recipients=john @mymaildomain.tld smtpstatus=550 smtpmsg='550 5.1.0 <alert @opengear1.mydomain.tld> sender rejected : invalid sender domain' errormsg='envelope from address alert @opengear1.mydomain.tld not accepted by the server' exitcode=EX_DATAERR
The additional spaces in email addresses are to avoid the automatic email address hiding plugin I run, and were not there in the original code. It’s not the clearest message to read, but the gist of it is that when I configured a Sender email address in the GUI, I used alert @ opengear1.mydomain.tld – i.e. the device’s FQDN. I’m guessing that my provider looked at the FQDN and discovered that, unsurprisingly, it didn’t have any MX records, so it bounced the message. Simple to fix by changing the email address to “alert @ mydomain.tld”, and a few minutes later syslog confirmed that I got it right:
<14>Nov 26 00:43:59 msmtp: host=smtp.myprovider.tld tls=off auth=on user=mylogin @myprovider.tld from=alert @mydomain.tld recipients=john @mymaildomain.tld mailsize=189 exitcode=EX_OK
Typing the word “goober” on Port 2 had successfully matched a pattern I had programmed, and triggered an email to me warning that there had been an instance of ‘goober’ on that port. It was failing previously because I’d made a mistake, and once I fixed it, it worked just as it should.
Improvements
It didn’t feel good being blind to the outcome of the email. I think I’d lean towards enabling syslog by default in future builds, but if not, it would be great if it were possible to enable msmtp logging via the GUI (assuming that the global option doesn’t do it? I’ll test more on that some time soon). As you’ll see in my next post, the mechanism I used to update the alert-email script was not especially pretty.
Caveats
This was a very basic functionality test, and there is much more to the idea of triggers and actions within ARMS; I’m planning to look at those in the future. Pattern matching on the serial port is one of ten default checks that can take place, not to mention custom checks that you can create (i.e. the Opengear can execute a script at a regular interval and the script’s response determines whether the alert condition is met). The Trigger and Resolve actions have six other options including a custom script.
It’s also worth considering that monitoring what somebody types is fairly easy to defeat. For example, if rather than just typing “goober” I were to type “goob”, then press backspace, then type “ber”, the end result (as seen by the system connected to the serial port) would almost certainly be “goober”, but to the pattern matching it is seen as “goob^hber” – which is not a match for “goober”. Similarly, if you’re trying to trap for “write erase”, what happens if somebody types “wri era” instead? This isn’t a flaw with Opengear’s software, but rather it’s something you need to design for where possible. For example, it may be better to trap on a resulting prompt or console message from the end device (in response to the command typed), as that’s more likely to be consistent, and would be hard for the user to tamper with.
What’s Next?
Next I want to use a pattern match to interfere with the serial stream – to cancel a command, log out a user, and send an email. After that, I’d like to have a look at using SMS commands to trigger events. And, as promised, I’ll look at my experiment to alter the alert-email script on a readonly filesystem.
Disclosure
Opengear was a paid presenter at Networking Field Day 4, and while I received no compensation for my attendance at this event, my travel, accommodation and meals were provided. I was explicitly not required or obligated to blog, tweet, or otherwise write about or endorse the sponsors, but if I choose to do so I am free to give my honest opinions about the vendors and their products, whether positive or negative. Opengear provided me with an entry-level console management device (ACM5004-G), as reviewed in this post.
Please see my Disclosures page for more information.
Leave a Reply