how to send html messages to hipchat from hubot ?

Comments

Hi all, our Mobile team at TripAdvisor started using HipChat mid last year for internal dev communication and it turned out to be a brilliant idea. It has simply changed the way we communicate, we no longer have to wait for other teams across the coast or continent to see an email and respond. 1-on-1 private chats/notifications prompt faster responses from fellow developers. In this post, I'm going to explain a hacky bridge that I built between XMPP based hubot and HipChat v2 API that allows one to send rich HTML messages to HipChat via hubot.

Motivation

We maintain a HipChat Bot named hbot powered by hubot for various automation/querying tasks. At the start of this year, there was a rising demand for pretty responses using rich HTML format compared to a stream of text separated by newlines. I'm completely fine with reading a stream of text but it turns out that the rest of the world isn't (including Igor Nikolaev, Justin Leider and Ryne McCall). Hence, I needed to figure out a way to send HTML messages via hubot.

Like any sane human being, you'd think this would be easy peasy. Turns out you are wrong because of hubot gitlab issue #151. This leads you to Support XHTML-IM spec in HipChat (XMPP). The latest update (as of April 2015) on the previous link states that it's something that they would support sometime in the future. Presently HipChat allows XHTML-IM body via HipChat v2 API.

For a while, I was at a dilemma about bridging the two protocols (XMPP and HipChat API) as I knew that I would be implementing a horrible hack to make this work. Truth be told, I shouldn't even be writing this tech post and let HipChat fix it for real. I know that I'll regret this hack sometime in the future but that's a thought for another day, so without further ado, let me explain to you how you can send rich html messages to HipChat from hubot.

Setting it up

I will not delve deep into this section, since I assume that you already have a working hipchat and hubot instance. However, in the interest of general public, I will list out some pointers below for setting up hubot.

Hubot Script Structure and Message Payload

Any hubot script structure will look something like below.

# Description:
#   Script does something
#
# Dependencies:
#   None
#
# Configuration:
#   HUBOT_CONFIG_VARIABLE
#
# Commands:
#   something - Does Something
#
# Author:
#   ravikiranj
somethingRegex = /something/i
module.exports = (robot) ->
  # unlike robot.respond, hear robot.hear will run your regex against any chat message
  # instead of only those directed at hubot such as "@hbot do something"
  robot.hear somethingRegex, (msg) ->
    # Determine output message
    opMsg = findSomething msg
    # Send the output message back to the hipchat room
    msg.send opMsg

In a normal scenario, you'd send the message back using msg.send, however, in our case, we need to use the HipChat v2 API to send the message back. In order to do so, we need the HipChat Room API ID from which we received the message from. Let's examine meta data present in msg.

{
"message": {
     "user": {
         "id": "123456",
         "jid": "123456_789012@chat.hipchat.com",
         "name": "Ravikiran Janardhana",
         "mention_name": "Ravi",
         "room": "hobbiton",
         "reply_to": "1_hobbiton@conf.btf.hipchat.com"
     }
}

The only useful Room ID that we can work with is the Room JID that is accessible via msg.message.user.reply_to.

Generating HipChat Room's XMPP JID to API ID mapping

As per the send_room_notification API docs, we need the API Room ID to post a room notification. As shown in the JSON payload previously, we don't have the API Room ID available via hubot but we do have the XMPP JID via reply_to. Hence we need to generate the mapping from XMPP JID to API ID. The necessary steps to do so are listed below.

Generate Personal Access Token

You can grab your account's all powerful access token at https://www.hipchat.com/account/api OR https://YOUR_HIPCHAT_INSTANCE/account/api. The auth token will be similar to a SHA-1 hash such as eacb0d1b53a6f12893e95c7c5aec16de3ff2a939. This is your auth_token that can be used with HipChat v2 API.

Dump List of Rooms

Refer to get_all_rooms API docs in order to dump the list of Rooms. You can dump the list of rooms via below curl command or the python script (useful template for paging when you have greater than 1000 rooms).

# Default YOUR_HIPCHAT_URL is https://api.hipchat.com
curl -s 'https://YOUR_HIPCHAT_URL/v2/room?max-results=1000&auth_token=YOUR_AUTH_TOKEN' | python -m json.tool > rooms.json
#!/usr/bin/env python
import requests
import json
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("dump_room_list")
if __name__ == "__main__":
    hipchat_url = "YOUR_HIPCHAT_URL" # default is "https://api.hipchat.com"
    auth_token = "YOUR_AUTH_TOKEN"
    get_all_rooms_api_url = hipchat_url + "/v2/room?max-results=1000&auth_token=" + auth_token
    output_filename = "rooms.json"
    r = requests.get(get_all_rooms_api_url)
    if r.status_code == requests.codes.ok: # HTTP 200
        with open(output_filename, "w") as f:
            json.dump(r.json(), f, sort_keys=True, indent=4, separators=(',', ': '))
            f.close()
            logger.info("Output written to %s", output_filename)
    else:
        logger.error("Failed to get all hipchat rooms, Status code = %d", r.status_code)
        logger.error("Response = %s", r.text)

The JSON output in rooms.json will be as below. The API Room ID field is what we care about the most.

{
"items": [
     {
         "id": 1, // API ROOM ID
         "links": {
             "participants": "PARTICIPANTS_URL",
             "self": "ROOM_WEB_URL",
             "webhooks": "ROOM_WEBHOOK_URL"
         },
         "name": "Default"
     },
     {
         "id": 2, // API ROOM ID
         "links": {
             "participants": "PARTICIPANTS_URL",
             "self": "ROOM_WEB_URL",
             "webhooks": "ROOM_WEBHOOK_URL"
         },
         "name": "My Room"
     }
     ...
}

Dump CSV mapping of XMPP JID, API ID

We can extract the XMPP JID via API ID using get_room API endpoint. For example, if we have a Room with API ID 100, the get_room API call returns the following response.

{
    created: "2014-08-02T23:15:09+00:00",
    guest_access_url: null,
    id: 100, // API ID
    is_archived: false,
    is_guest_accessible: false,
    last_active: "2015-05-23T00:01:25+00:00",
    links: {},
    name: "My HipChat Room",
    ...
    xmpp_jid: "100_my_hipchat_room@conf.btf.hipchat.com" // XMPP JID
}

The following script can be used to dump a CSV file of XMPP JID and API ID while adhering to HipChat API Rate Limits.

#!/usr/bin/env python
import requests
import json
import csv
import time
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("gen_xmpp_jid_to_api_id_mapping")
# Load HipChat Rooms
with open("rooms.json") as data_file:
    # Load JSON
    data = json.load(data_file)
    # Output file to write the mapping
    op = open("xmpp_jid_to_api_id_map.csv", "w")
    csv_writer = csv.writer(op, quoting=csv.QUOTE_MINIMAL)
    headers = ["xmpp_jid", "api_id"]
    csv_writer.writerow(headers)
    # Room Info API
    base_url = "https://YOUR_HIPCHAT_URL/v2/room/" # default is "https://api.hipchat.com"
    AUTH_TOKEN = "?auth_token=YOUR_AUTH_TOKEN"
    # Loop through each room
    count = 0
    for item in data["items"]:
        id_str = str(item["id"])
        api_url = base_url + id_str + AUTH_TOKEN
        logging.info("Fetching api data for %s, count = %d",api_url, count)
        r = requests.get(api_url)
        if r.status_code != 200:
            logger.info("Response = %s", r.text)
            logger.info("Error!!! Status code = %d, url = %s, continuing with next entry", r.status_code, api_url)
            continue
        op = r.json()
        if "xmpp_jid" in op:
            # XMPP_JID, API_ID
            row = [op["xmpp_jid"].encode('utf-8'), id_str]
            csv_writer.writerow(row)
            logger.info("Row = %s", row)
        # Rate Limit - 100 requests within 300 seconds (5 min) window
        time.sleep(3.5)
    # Close output file handle
    op.close()

Reviewboard Hubot Script

In order to demonstrate posting html messages to hipchat, I will use the reviewboard example. Whenever someone on our team needs "Ship It's", they post a short hand notation "!rb REVIEW_ID" in the relevant hipchat room and it is hbot's responsibility to pull out relevant details of the reviewboard entry in question and display it in the room as shown in the below screenshot.

hbot's reviewboard test example

I have already outlined the Hubot-Script-Structure, however, we need to do some preprocessing to post HTML messages. I have broken the full hubot script into sections and described each one of them below.

Load Script dependencies

The following code describes the hubot review board script and it's dependencies. I have intentionally left out Reviewboard authentication config variables and assume that the reader is familiar with it, see reviewboard web API guide for the details. Also, for the sake of simplicity, I will assume a function that will return the correct reviewboard JSON response.

# Description:
#   Retrieve Reviewboard information - v2 (HTML Rich)
#
# Dependencies:
#   https,http,fs
#
# Configuration:
#   HUBOT_HIPCHAT_API_V2_AUTH_TOKEN
#
# Commands:
#   !rb reviewId - displays information about review board entry
#
# Author:
#   rjanardhana, inikolaev
# regexp for listening to rb pattern
regexp = /!(rb)\s+([0-9]+)/i
# required for https calls
https = require 'https'
# required for http calls
http = require 'http'
# required to load xmpp_jid,api_id file mapping
fs = require 'fs'
# csv parser
parse = require 'csv-parse'
# Handlerbars templating engine
Handlebars = require 'handlebars'

Load Room XMPP JID to API ID mapping

The following code loads the xmpp_jid_api_id_map.csv file to a coffescript map/object where the key is xmpp_jid and the value is api_id. Please note that the mapping file should be placed in hubot's root folder.

# Load XMPP JID to API ID Map
xmpp_to_api_map = {}
XMPP_JID_TO_API_ID_FILENAME = "xmpp_jid_to_api_id_map.csv"
rawCSVData = fs.readFileSync(XMPP_JID_TO_API_ID_FILENAME).toString()
parse(rawCSVData, {columns: true}, (err, data) ->
    for item in data
        xmpp_to_api_map[item['xmpp_jid']] = item['api_id']
)

Handlebars Template

The following code sets up a Handlebars template that is used to generate the HTML from the reviewboard json object.

rbHTMLTemplateString = '''
<img style="display: block; margin-right: 2px;" width="16" height="16" src="https://IMAGE_HOSTING_SITE/img/emoticons/rbicon.png"></img>
<a href={{RBUrl}}><b>{{summary}}</b></a> by <b>{{submitter}}</b></br>
<span>   <b>Ship Its: </b>{{shipItCount}}</span>
<span> · <b>Open Issues: </b>{{openIssues}}</span>
<span> · <b>Resolved: </b>{{resolvedIssues}}</span>
<span> · <b>Dropped: </b>{{droppedIssues}}</span></br>
'''
rbTemplate = Handlebars.compile(rbHTMLTemplateString)

Listening to a regex pattern

The following code listens to the reviewboard regex (!rb REVIEW_ID) and tries to fetch the reviewboard entry details. It posts a HipChat Notification if we have xmpp_jid to api_id mapping, else it sends a raw string message back.

# Listen to regexp and respond
module.exports = (robot) ->
    robot.hear regexp, (msg) ->
        # Grab xmpp jid and api room id if it exists
        xmpp_jid = msg.message.user.reply_to
        api_room_id = xmpp_to_api_map[xmpp_jid]
        # Review Id
        reviewId = msg.match[2]
        # Returns an object consisting of json object and raw string
        rbResp = getReviewboardResponse reviewId
        if api_room_id?
            sendHipChatNotification msg, rbResp["json"], api_room_id
        else
            msg.send rbResp["raw_string"]

Posting the response via HipChat API

The following code block posts a hipchat room notification by making a POST request to send_room_notification endpoint with necessary auth credentials and POST data. getNotificationPayload function returns the JSON payload consisting of message, message_format and color. errorHandler function is called when we fail to post a room notification.

# sendHipChatNotification
sendHipChatNotification = (msg, rb, api_room_id) ->
    req_path = "/v2/room/" + api_room_id + "/notification?auth_token=" + process.env.HUBOT_HIPCHAT_API_V2_AUTH_TOKEN # YOUR_AUTH_TOKEN
    post_data = getNotificationPayload rb
    reqOptions =
        host: process.env.HUBOT_HIPCHAT_HOST # YOUR_HIPCHAT_HOST (e.g: hipchat.myhipchatinstance.com)
        port: 443
        path: req_path
        method: "POST"
        headers:
            "Content-Type": "application/json"
     # Construct request object and set listeners
     req = https.request reqOptions, (res) ->
         if res.statusCode != 204 # HTTP Status Code 204 = No Content
             errorHandler msg, res, post_data
             return
         data = ""
         res.on "data", (chunk) ->
             data += chunk.toString()
             return
         res.on "end", () ->
             return
         return
     req.on "error", (e) ->
         errorHandler msg, res, post_data, e
         return
     # Create and send notification
     req.write post_data
     req.end()
     console.log "Sending HipChat notification, payload = ", JSON.stringify(rb)
     return
# getNotificationPayload
getNotificationPayload = (rb) ->
    payload =
        message: rbTemplate(rb)
        message_format: "html"
        color: "yellow"
    return JSON.stringify payload
# errorHandler
errorHandler = (msg, res, post_data, error) ->
    msg.send "Failed to post notification to hipchat server, bug Ravi!"
    if res and res.statusCode
        console.log "Failed to post notification, Status code = #{res.statusCode}"
    if post_data
        console.log "Post Data = #{post_data}"
    if error
        console.log "Error = #{error}"

Putting it all together

The full hubot script is basically a concatenation of each of the sections described above and can be found at https://github.com/ravikiranj/hipchat/blob/master/rb.coffee.

Gotchas

  • If a new hipchat room was created after you generated the XMPP_JID to API_ID mapping, it will receive the raw string version of the response and not the pretty HTML format. To fix this, simply add the new hipchat room mapping by looking it up in the web UI (needs admin access) or make API calls to find out.

  • You will need to routinely regenerate the XMPP_JID to API_ID mapping to keep it in sync with the current list of hipchat rooms.

Comments

Comments powered by Disqus