Updating data files, commits vs. pull requests

For once, I'm wondering a bit if this post can be helpful to somebody else. I believe my context is pretty specific. Anyway, just in case it might be the case, here it is.

My Jet Train project makes use of GTFS. GTFS stands for General Transit Feed Specification. It models public transportation schedules and their associated geographic information.

GTFS is based on two kinds of data, static data, and dynamic data. Static data may change but do so rarely, e.g., transit agencies and bus stations. They are available as static files that you need to download now and then. Before, I had to download and overwrite them every time I run the demo.

As a developer, I'm lazy and wanted to automate this task. I used GitHub Actions for that:

name: Refresh Dataset
on:
  schedule:
    - cron: '12 2 * * 1'                                                     # 1
jobs:
  build:
    name: Refresh Dataset
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2                                            # 2
      - name: Fetch dataset archive
        un: curl -o archive.zip https://api.511.org/transit/datafeeds\?api_key\=${ {secrets.FIVEONEONE_API_KEY} }\&operator_id\=RG  # 3
      - name: Extract archive
        run: unzip -o -d ./infrastructure/data/current/ archive.zip          # 4
      - name: Add & commit
        uses: stefanzweifel/git-auto-commit-action@v4                        # 5
        with:
          commit_message: Update to latest data files
          add_options: '-u'

Run the action weekly
Checkout the repository
Get the static data files archive
Extract files from the archive
Use the git-auto-commit action

It's not an issue to commit directly. Indeed, it's not code but data. The code should already have all built-in safeguards to prevent unexpected data from causing exceptions at runtime. I already had a couple of surprises previously and applied a lot of defensive programming techniques.

Yet, I was not happy with the above automation:

Commits happen every week, regardless of whether I need to run the demo or not. It creates a lot of unnecessary commits. That's the reason I scheduled the action weekly and not more often.
The action is scheduled on Mondays. If I run the demo on a Friday, I'll need to update the data files anyway.

Hence, I decided to switch to an alternative approach. Instead of committing, I updated the script to open a Pull Request. If I need to run the demo, I'll merge it (and pull locally); if not, it will stay open. If an opened PR already exists, the action will overwrite it. Now, I can schedule the action more frequently.

name: Refresh Dataset
on:
  schedule:
    - cron: '12 2 * * *'                                                     # 1
jobs:
  build:
    name: Refresh Dataset
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2                                            # 2
      - name: Fetch dataset archive
        run: curl -o archive.zip https://api.511.org/transit/datafeeds\?api_key\=${ {secrets.FIVEONEONE_API_KEY} }\&operator_id\=RG  # 3
      - name: Extract files of interest from the archive
        run: unzip -o -j archive.zip agency.txt routes.txt stop_times.txt stops.txt trips.txt -d ./infrastructure/data/current  # 4
      - name: Remove archive
        run: rm archive.zip                                                  # 5
      - name: Create PR
        uses: peter-evans/create-pull-request@v3                             # 6
        with:
          commit-message: Update to latest data files
          branch: data/refresh
          delete-branch: true
          title: Refresh data files to latest version
          body: ""

Run the action daily
Checkout the repository
Get the static data files archive
Extract only required files from the archive
Remove the archive file for cleanup
Use the create-pull-request action. The action creates a PR that automatically contains all new and updated files; that's the reason why I only extract some files and remove the archive.

As I mentioned in the introduction, I'm not sure this post can help many people. If it does, please don't hesitate to comment to let me know about your use case.

The complete source code for this post can be found on Github:

hazelcast / jet-train

Jet Trains

This project is a demo of Hazelcast Jet, a data streaming engine based on Hazelcast IMDG.

It displays the position of public transports in the Bay Area in near real-time.

Note	It used to showcase Switzerland’s public transports Unfortunately, the Swiss data provider doesn’t provide the GTFS-RT feed anymore

Figure 1. Demo screenshot (click to watch a recording sample)

The technology stack consists of:

Hazelcast Jet
Hazelcast IMDG
Kotlin for the code
Spring Boot for the webapp
Maven for the build system

Overall structure

The project contains several modules with dedicated responsibilities:

Name	Description
`common`	Code shared across modules
`infrastructure`	Contain the static data files, as well as configuration files for Docker Compose and Kubernetes
`local-jet`	As an alternative to the previous module, start a local Jet instance to be able to debug inside the IDE
`load-static`	Load GTFS-RT static data from files in memory Those…

View on GitHub

Originally published at A Java Geek on August 15^th, 2021