Measuring Deployment Success with GitHub Actions

Last week, we took a look at how shareable activities - specifically the link made available to a particular activity’s logs in the management console - could be surfaced as a part of a GitHub integration using GitHub actions. It was a nice bit of fun playing around with GitHub actions for the first time, and it let me make something new and useful for my team's work.

But I caught a little bit of the curiosity bug. What else could I link up to my workflow, now that I can trigger events in response to a Platform.sh environment deployment? Well, since I linked an activity’s logs to a failed deployment, it only makes sense to trigger some things in response to a successful one.

Using GitHub actions to monitor Platform.sh deployment status

Take a moment to check out the previous post centered around GitHub actions. It’ll provide a little context, plus go into much more detail about how the status of a Platform.sh deployment can actually be used to trigger workflows on GitHub. Plus, we’re going to build additional jobs in the same post-deploy.yaml file we started in that post.

Let’s take a look at the file, ignoring the displaylogs job we made previously, and instead focusing on the build job that sets everything else up:

---
name: Post-deploy
on:
  push:
    branches-ignore:
      - "master"
env:
  PLATFORMSH_CLI_TOKEN: ${{ secrets.CLI_TOKEN }}
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

jobs:
  build:
    runs-on: ubuntu-latest
    name: "Get environment URL"
    outputs:
      commit_status: ${{ steps.status.outputs.env_status }}
      env_url: ${{ steps.url.outputs.env_url }}
      integration_status: ${{ steps.wait.outputs.integration }}
    steps:
      - uses: actions/checkout@v2
      - name: "Await deployment"
        id: wait
        run: |
          COMMIT_STATUS="pending"
          sleep 10
          STATUSES=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r 'length')
          if [ $STATUSES == 0 ]; then
              echo "Not on a Platform.sh integrated environment. Skipping."
              echo "::set-output name=integration::none"
          else
              until [ "$COMMIT_STATUS" == "success" ] || [ "$COMMIT_STATUS" == "failure" ]; do
                sleep 10
                ENV_URL=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].target_url')
                COMMIT_STATUS=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].state')
              done
              echo "Environment deployed. Finished."
              echo "::set-output name=integration::platformsh"
          fi
      - name: "Pass status"
        id: status
        if: steps.wait.outputs.integration == 'platformsh'
        run: |
          COMMIT_STATUS=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].state')
          echo "::set-output name=env_status::$COMMIT_STATUS"
      - name: "Pass URL"
        id: url
        if: steps.wait.outputs.integration == 'platformsh'
        run: |
          ENV_URL=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].target_url')
          echo "::set-output name=env_url::$ENV_URL"

This job (build):

• Is triggered by all push events that don’t occur on the default branch.
• Places requests on the Status API, until Platform.sh informs GitHub that the environment has either succeeded or failed to deploy the new commits.
• Passes both that final status and the environment’s URL to any subsequent jobs that define needs.build in their configuration.

Github actions status checks ensures deployment ease and success

One of the great things about GitHub actions is that I don’t have to write them. If I, for example, wanted to make a quick status check on the environment, I can leverage the generous work of another published action in my workflow pretty easily.

test:
  name: "Status tests"
  runs-on: ubuntu-latest
  needs: build
  if: needs.build.outputs.commit_status == 'success'
  steps:
    - name: "Application should deploy successfully"
      uses: lakuapik/gh-actions-http-status@v1
      with:
        sites: '["${{ needs.build.outputs.env_url }}"]'
        expected: "[200]"

In this case, I’ve written a new job called test which will:

• Only run if the output of the build job (and the status of my Platform.sh environment) is successful.
• Will use the lakuapi/gh-actions-http-status public action to place a request on the environment URL also retrieved from the build job.
• Apply an expected status to that request. If the statuses match, the step will pass, adding another check to my pull request. Should it receive a 404 or 503, the step fails and we can go investigate where the deployment fell short.

This snippet only covers the homepage of the development environment, but I could also include more endpoints in the sites array, with matching expected statuses that will show up as separate steps in my final workflow for the push.

Configuring Lighthouse with GitHub actions to monitor user experience

Status checks for GitHub actions are great, but what I really want to know is whether a pull request is modifying the experience a user has when they visit my application in some negative way. Once again, I can add a job that uses a publicly available GitHub action - in this case, to perform a Lighthouse audit on my environment URL.

To start, we can add a few workflow-level environment variables to post-deploy.yaml

env:
  PLATFORMSH_CLI_TOKEN: ${{ secrets.CLI_TOKEN }}
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  ACC_MIN: 0.90
  BPR_MIN: 0.90
  PRF_MIN: 0.90
  SEO_MIN: 0.95

These variables are going to set a few thresholds for the four main audits we’d want to keep high on our homepage: Accessibility, Best Practices, Performance, and SEO.

Here’s what the Lighthouse report looks like for our Platform.sh homepage:

Not too shabby!

Let’s add a new job called audit which uses the jakejarvis/lighthouse-action GitHub action, first focusing on the Accessibility audit.

audit:
  name: "Lighthouse Audit"
  runs-on: ubuntu-latest
  needs: build
  if: needs.build.outputs.commit_status == 'success'
  steps:
    - name: Audit live URL
      uses: jakejarvis/lighthouse-action@master
      with:
        url: ${{ needs.build.outputs.env_url }}
    - name: Upload results as an artifact
      uses: actions/upload-artifact@master
      with:
        name: report
        path: "./report"
    - name: Accessibility test
      run: |
        REPORT=$(ls report | grep \.json$)
        SCORE=$(cat report/$REPORT | jq -r '.categories.accessibility.score')
        if [ 1 -eq "$(echo "${SCORE} < ${{ env.ACC_MIN }}" | bc)" ]; then
            echo "Threshold Failed:"
            echo "  - Score:     $SCORE"
            echo "  - Threshold: ${{ env.ACC_MIN }}"
            exit 1
        else
            echo "Passed:"
            echo "  - Score:     $SCORE"
            echo "  - Threshold: ${{ env.ACC_MIN }}"
        fi

This audit job will:

• Only run on pushes to a branch associated with a pull request integrated with a Platform.sh environment that has successfully deployed. (See build above)
• Use the jakejarvis/lighthouse-action action with the environment URL passed along from the build job.
• Upload the full report as a workflow artifact, so that we can inspect individual recommendations in detail later.
• The action will also save a local copy of the report in two formats - html and JSON - in a report subdirectory. We can then use the jq tool to retrieve the accessibility score.
• Run a simple test based on the Accessibility audit using the ACC_MIN threshold we set previously. In this case, if the accessibility score for our environment is below 90%, the step and workflow will fail for the push.

Using this same logic we can add tests for Best Practices:

- name: Best practices test
  run: |
    REPORT=$(ls report | grep \.json$)
    SCORE=$(cat report/$REPORT | jq -r '.categories."best-practices".score')
    if [ 1 -eq "$(echo "${SCORE} < ${{ env.BPR_MIN }}" | bc)" ]; then
        echo "Threshold Failed:"
        echo "  - Score:     $SCORE"
        echo "  - Threshold: ${{ env.BPR_MIN }}"
        exit 1
    else
        echo "Passed:"
        echo "  - Score:     $SCORE"
        echo "  - Threshold: ${{ env.BPR_MIN }}"
    fi

Performance

- name: Performance test
  run: |
    REPORT=$(ls report | grep \.json$)
    SCORE=$(cat report/$REPORT | jq -r '.categories.performance.score')
    if [ 1 -eq "$(echo "${SCORE} < ${{ env.PRF_MIN }}" | bc)" ]; then
        echo "Threshold Failed:"
        echo "  - Score:     $SCORE"
        echo "  - Threshold: ${{ env.PRF_MIN }}"
        exit 1
    else
        echo "Passed:"
        echo "  - Score:     $SCORE"
        echo "  - Threshold: ${{ env.PRF_MIN }}"
    fi

and SEO.

    - name: SEO test
              run: |
                  REPORT=$(ls report | grep \.json$)
                  SCORE=$(cat report/$REPORT | jq -r '.categories.seo.score')
                  if [ 1 -eq "$(echo "${SCORE} < ${{ env.SEO_MIN }}" | bc)" ]; then
                      echo "Threshold Failed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.SEO_MIN }}"
                      echo "Download the full report under *Artifacts* at: https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
                      exit 1
                  else
                      echo "Passed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.SEO_MIN }}"
                      echo "Download the full report under *Artifacts* at: https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
                  fi

Streamline your Platform.sh deployments with GitHub actions

There’s two more things down that can be connected to Platform.sh deployments using GitHub actions. You can find the full Gist for the workflow described above on GitHub.

Goodbye for now - until I can think of something else interesting to play with here. If you have a custom workflow, or if you’d like to share tips and tricks of your setup, let us know by sharing your actions with the rest of our Community.