(This post originally appeared on the New Relic Blog, and was co-authored by Rachel Foster)

The opening scroll: Observability for the uninitiated

For those who deal with monitoring and observability every day, the need for tools like New Relic is not only a no-brainer, it’s self-evident. It requires no further explanation.

But that’s not everyone. From the leaders in upper management all the way down to troops in the trenches, there are plenty of folks who don’t have a clear sense of what monitoring does, what observability is, and—perhaps most importantly of all—what it would even look like.

To be sure, there’s a plethora of slide decks, screen shots, and demo sites that show off dashboards and features. But to the observability newcomer, it’s easy to get lost in both the jargon and also the ‘nerd knobs’—the special purpose features that make sense when you already know about them.

In this blog, you will not only learn a few of the most common elements that observability solutions provide, you’ll also find out how to build a custom solution when the out-of-the-box configuration doesn’t suit your needs. More importantly, you’re going to have a little fun while you do it.

That’s because we are using an actual game as our sample application. You see, plenty of examples start with “here’s this fake take-out food web application” and—while they certainly show a valid real-world example of something you might want to monitor—they are usually a real yawner.

What if—hear us out—you could learn observability concepts and techniques AND PLAY A VIDEO GAME at the same time?!?

I KNOW, RIGHT?!?

That's what this example is all about. We’re going to install a game, configure New Relic to monitor it, and then—entirely for research purposes, you understand—play the game to see the stats posted on a dashboard.

Character Backstory

Dangerous Dave was a classic 1980's side-scroller style game that many folks spent hours playing when they should have been doing productive work. It was created by John Romero, the same game designer who brought to life Wolfenstein 3D, Doom, and Quake.

Along with the obvious attraction of being able to play a level or two in between examples, there’s another reason for using a game as our example: Games are not—as a class of application—particularly easy to instrument in monitoring. They aren’t generally written to share resources, emit statistics, or even run while other things are running on the same machine. By using Dangerous Dave (which we’ve dubbed “Data Driven Dave” for our purposes) we’re making the subtle point that if you can add observability here, you can add it to pretty much any business application—standard, custom-off-the-shelf, or home grown—that you can come up with.

But most of all, we’re using the Dangerous Dave game so you can have a fun way to kick the tires on New Relic monitoring and to show how easy it is to instrument non-web scripts, background worker processes, and functions, to capture and collect non-standard metrics, and to display them in a meaningful way.

Teaser trailer: Seeing is believing

We realize the best way to entice you to try out the techniques you'll see later in this blog is to show you the results now. So here it is. New Relic can take a game that looks like this,

and display the results like this.

Now remember, the point isn’t for you to ooh and aaah over how amazing this specific dashboard looks. We want you to recognize that we’re collecting and displaying a range of telemetry and data types—from simple metrics like the top score to a breakdown of individual items collected during the game with the value each contributes to the overall score. Later, we’ll even show you how this dashboard can include log data. It’s going to be good! Let’s get started.

Cheat Code: Cut (scene) to the chase

What if you want an amazing dashboard like that, but don’t want to go through every step of instrumentation we will be describing below? We’ve got good news because we’ve already done all the heavy lifting for you. If you follow these quick steps, you’ll have a dashboard of your own up and running in no time.

1. Make sure you’ve got a New Relic account. You can sign-up for free here. Free accounts include 100 GB/month of free data ingest, one free full-platform user, and unlimited free basic users.
2. Set up a system so it can run python with pygame modules on the command line. That can be a laptop, a virtual machine, a cloud instance, or even a project in an online IDE like replit.
3. Head over to Rachel’s github (aka DevRel Dragon) and create a local copy of the Data Driven Dave repo.
4. Follow the steps in the README.md file to install the game on your system and the dashboard in your New Relic account.
5. Play the game.

That’s it! Playing the game will send telemetry data into your New Relic account. You can explore your application data in the New Relic platform. Specifically, you will notice the application performance monitoring (APM) Summary, Transactions and Logs screens are now reporting data about the Data Driven Dave game. Pretty cool, right? Go ahead, take some time to explore a little bit.

Oh wow. You’re still reading—even after spinning up the game? Okay, you’re serious. Well, roll up your sleeves and let’s dig into the code to see how we pulled this off.

Ready Player One: Getting started with custom instrumentation in Python

New Relic does a great job in the guided installation process of walking you through the steps necessary to get up and running quickly with a new account. There is a bit of bias in the process, and applications without a web frontend can fall through the cracks. We are going to show you ways to solve that problem. For our purposes, because the original code behind Data Driven Dave was written in Python, we are going to take a look at how you can use the Python agent API to customize your monitoring solution.

To start, in your code you must import the newrelic.agent module and initialize it with a specified config file.

import newrelic.agent

newrelic.agent.initialize('newrelic.ini') # This is required!

The initialize call sets up the Python agent, but does not register the agent with a collector—a component that collects data from New Relic agents. You should call register_application as soon as possible—as it relates to processing order in your code—after the call to initialize.

application = newrelic.agent.register_application(timeout=5) # Force New Relic agent registration

if __name__ == "__main__":
    main()

Next, you can use Python code decorators to instrument your non-web transactions. Typically, you might use these decorators in a way that most accurately reflects the function-level segments of your transaction trace. For example:

import newrelic.agent

@newrelic.agent.function_trace()
def test_function_trace():
    pass

@newrelic.agent.datastore_trace("redis", "my_collection", "set")
def test_datastore_trace():
    pass

@newrelic.agent.external_trace("library","http://www.example.com","get")
def test_external_trace():
    pass

@newrelic.agent.background_task()
def main():
    test_function_trace()
    test_datastore_trace()
    test_external_trace()

if __name__ == "__main__":
    main()

However, for our game demonstration, we wanted to show more transactions in APM. So, we chose to use the background_task decorator exclusively. Here is an excerpt from main_fun.py:

@newrelic.agent.background_task()
def showScores(screen, tileset):
    pass

@newrelic.agent.background_task()
def savePlayerScore(player_score, screen, tileset):
    pass

@newrelic.agent.background_task()
def showCreditsScreen(screen, tileset):
    pass

def main():
    # Init pygame
    pygame.init()
    game_screen = Screen(SCREEN_WIDTH, SCREEN_HEIGHT)

Finally, at the end of your code, call shutdown_agent to shut down the agent and make a final upload of data to the collector.

newrelic.agent.shutdown_agent(timeout=2.5) # Shutdown New Relic agent

And just like that, you are sending your application telemetry data to New Relic.

Level up with Logs

What’s that you say, too easy? Good! Let’s take this to the next level.

When doing custom instrumentation in the way outlined above for backend code without a web frontend, New Relic logs in context will not automatically be configured by an agent. Instead, you must manually configure logs by first enabling log decoration and forwarding in the newrelic.ini file.

# This setting enables log decoration, the forwarding of log events,
# and the collection of logging metrics if these sub-feature
# configurations are also enabled. If this setting is false, no
# logging instrumentation features are enabled. This can also be
# set using the NEW_RELIC_APPLICATION_LOGGING_ENABLED environment
# variable.
application_logging.enabled = true

# If true, the agent captures log records emitted by your application
# and forwards them to New Relic. `application_logging.enabled` must
# also be true for this setting to take effect. You can also set
# this using the NEW_RELIC_APPLICATION_LOGGING_FORWARDING_ENABLED
# environment variable.
application_logging.forwarding.enabled = true

Next, you will need to manually decorate your code with the desired logging. Here is some sample code:

import logging

# Logging test
logging.basicConfig(level=logging.INFO)
logging.info('This is a sample info message')
logging.warning('This is a sample warning message')
logging.error('This is a sample error message')

For the Data Driven Dave example, a more meaningful log message might be to log a message at the start of a new game.

# Game start log message
logging.info('Game started')

You could also incorporate variables into your log messages. For example, you could log a message at the end of the game with the player’s final score.

# Write game final score to log
logging.info('Game ended with score %s', GamePlayer.score)

To learn more about python logging and to see more examples, check out Logging in Python.

Once you’ve decorated your code and enabled log forwarding in your newrelic.ini file, you will start to see log data showing up in the New Relic logs screen when you play the game.

Not only will you see your logs in the main log management screen along with all your other log data, you’ll see Data Driven Dave specific logs embedded within the APM UI.

Clicking on them will open a slider window showing the log details.

Take note that the attributes you see in the log details here—entity.guid, entity.name, hostname, etc.—are derived from metadata that the Python agent API is automatically adding to your log data.

Boss Battle: Adding custom events

It’s great to see that you are still with us. Let’s ramp up the challenge by introducing custom events. An event, in New Relic terms, is a discrete occurrence that is reported to a queryable data object. By creating a custom event, you may visualize and get alerts on the data that you specifically define and send to the New Relic database (NRDB).

To better understand when you might define a custom event, consider a use case in which you want to track the current level and score each time a player reaches the next level in the game. To achieve this, you might define an event_type called LevelUp, which includes current_level and player_score as parameters. Then, you would define the custom event using the record_custom_event decorator in your code.

# Record custom New Relic event
event_type = "LevelUp"
params = {'current_level': current_level_number, 'player_score': GamePlayer.score}
newrelic.agent.record_custom_event(event_type, params, application=application)

Similarly, you might also define a custom event_type called GameComplete to track the same custom attributes whenever a player completes a game.

# Record custom New Relic event
event_type = "GameComplete"
params = {'current_level': current_level_number, 'player_score': GamePlayer.score}
newrelic.agent.record_custom_event(event_type, params, application=application)

Sending that data to the NRDB, allows you to write queries in the New Relic Query Language (NRQL) like this example, which returns the maximum score achieved in the game.

SELECT max(player_score) FROM GameComplete WHERE appName = 'Data Driven Dave'

Or, like this query, which will show the maximum score achieved for each game level.

SELECT max(player_score) AS 'Top Score' FROM LevelUp WHERE appName = 'Data Driven Dave'
FACET current_level AS 'Level'

To learn more about defining custom events in your code, see the Introduction to custom events and attributes New Relic Documentation.

High Scores: Displaying your custom metrics and logs on a dashboard

Now we have come full circle. Remember at the beginning of this blog post when we showed you the end result? Later we told you there was a way to add logs to a dashboard, too. Well, let’s take a look at how you can create a dashboard with the custom metrics created in the previous steps.

You can use the dashboards capabilities in New Relic to create custom visualizations for your most important data to drive insights and better understand your applications, customers or business. There are a number of ways to get started with dashboards. For the purpose of this blog post, let’s assume that you will use the New Relic query builder to write and run NRQL queries, then add the results to a dashboard.

If you don’t feel like you're ready to dive in the NRQL deep end, that’s okay. Now is a good time to pause and go check out Rachel’s blog post Five things you need to know to get started with New Relic. Specifically, #3 “learn how to query your data” will help to ease you into the process. Come back here when you are ready to move on.

You’ve already seen some examples of using NRQL to report game metrics based on the LevelUp and GameComplete custom events. Let’s take a look at how you might use a custom event to add a table on your dashboard to show the total number of each item collected and its associated total value.

First, you would need to create a custom event called CollectedItem in the classes.py file.

def collectItem(self, item_pos, level):
    x = item_pos[0]
    y = item_pos[1]
    item = level.getNode(x, y)
    itemIsEquipment = False

    # If the item is an equipment, add it to the inventory
    if isinstance(item, Equipment):
        itemIsEquipment = True
        self.inventory[item.getId()] = 1

    # Increment score
    self.score += item.getScore()

    # Record custom New Relic event
    event_type = "CollectedItem"
    params = {'item.id': item.getId(), 'item.score': item.getScore(), 
    'item.type': item.getType(), 'item.isEquipment': itemIsEquipment, 
    'levelNumber': self.currentLevelNumber, 'item.gfx_id': self.gfx_id}
    newrelic.agent.record_custom_event(
        event_type, params, application=newrelic.agent.application())

    level.clearNode(x, y)

Then, you would be able to query that custom event to get an item count and sum of the item score.

SELECT count(*) AS '# Collected', sum(`item.score`) AS 'Total Value' FROM CollectedItem 
FACET if(`item.id` = 'items', item.type, item.id) AS 'Item Collected' SINCE 1 hour AGO

Now, let's take a look at how you could add log data on your dashboard. You have the option of adding charts and tables to a dashboard directly from the logs UI. Look for the NRQL and Add to dashboard options on the screen. Or, within the query builder you could select from the Log event type.

SELECT count(*) FROM Log WHERE entity.name = 'Data Driven Dave' FACET `level` SINCE 1 hour AGO TIMESERIES

To explore other query ideas for your game dashboard, complete the prerequisite steps for the Data Driven Dave repo to import a sample dashboard into your account. You will then be able to view the queries behind each dashboard widget.

End Credits

Congratulations, you made it all the way to the end! While we don’t have any epic song to scroll up the screen or weird plot twist that causes you to question whether this was all a dream. What we can offer is unlimited opportunities for further adventures. In fact, you could think of this entire blog post as more of an extended training side quest like you find at the start of the game, just to learn the ropes. Let’s summarize what you learned.

First, you learned that the New Relic guided install process is not the only way to ingest data. More importantly, you learned that you can use tools like the Python agent API to manually decorate your code and customize the APM transaction details you send to the NRDB.

Next, you learned how to manually send log details about your backend application to New Relic. We showed you how to configure the newrelic.ini file to enable log decoration and forwarding, as well as how to decorate your code to report log messages for multiple log levels.

From there we moved on, and you learned how to define your own custom event details and write those to a queryable object in the NRDB. We showed you a couple of use cases to help you understand when it makes sense to report custom events to New Relic.

Finally, you learned how to pull all of this information together into a dashboard that highlights key application data you might like to track. It’s important to note here that those same queries we showed you could also be used to create alerts. For example, you could set an alert to let you know when a new high score is achieved. But, we’ll have the alerts discussion for another blog post. We’ve done enough here today, don’t you think?

Most important of all, you discovered that learning about observability can be fun! Now that you’ve proven you can capture and collect metrics, events, logs, and transactions inside Data Driven Dave, you have a complete inventory of skills and tools to tackle REAL opponents: all the applications and services running in your production environment!