I like to have tangible metrics assigned to my goals. Ideally, these would be automatically tracked, and there would be a mechanism to hold me accountable.
One of my aims this year is to publish more and be more open about how I work and what I find interesting in my areas of expertise. What metrics can I attach to it? One is certainly a number of posts; the other could be how many people find them interesting enough to follow me.
To see how these metrics change over time, I decided to create a small dashboard that would track their historical values. I decided to start with X/Twitter.
Check out the dashboard created in this tutorial here: https://horosin.github.io/metrics-dashboard/
Full code: https://github.com/horosin/metrics-dashboard
You may have heard that X restricted access to their API last year. They did, but they still allow us to access our own basic metrics (contrary to platforms like LinkedIn - shame on you Microsoft; I have to scrape in order to access my data).
What We’re Going to Build
There will be a few pieces of software to write/configure:
- The code for fetching the data from X.
- A script saving the data somewhere (in this case, in the JSON file in the GitHub repository).
- Schedule to run the code periodically - every day at a given time.
- Dashboard to present the data (simple single HTML file using chart.js).
-
GitHub Pages to host the dashboard.
The best part is that we can do all of that for free (including compute).
Set-Up
Twitter Application
Setting up a Twitter application in the developer section is a prerequisite for accessing Twitter's API, which is essential for fetching data like follower counts, posting tweets, or accessing other Twitter resources programmatically. Here's a step-by-step guide to get you started.
-
Navigate to the Twitter Developer Site: Go to Twitter Developer, and sign in with your Twitter account. If you don't have a Twitter account, you'll need to create one. Complete the application/sign up.
-
Go to the Developer Dashboard: Access your Twitter Developer Dashboard.
-
Create a Project: Click on "Create Project." You will be asked to provide a project name, description, and use case. Fill these out according to your project's needs.
-
Create an App within Your Project: After creating your project, you'll have the option to create an app within this project. Click on "Create App," and fill in the necessary details like the App name.
-
Get Your API Keys and Tokens: Once your app is created, you will be directed to a page with your app's details, including the API Key, API Secret Key, Access Token, and Access Token Secret. Save these credentials securely; you'll need them to authenticate your requests to the Twitter API.
Project
Now, let’s get to coding. Create a new directory on your system, and open a console there.
mkdir metrics-dashboard
cd metrics-dashboard
Make sure to initialize a Git repository, and later, connect it to a GitHub project.
Initialize the node.js project, and install some packages that we’re going to need to authenticate with the API.
npm init
npm i dotenv oauth-1.0a crypto
Create a .env file with all the keys acquired from X before. DO NOT commit this to the repository. This is only to test the script locally.
TWITTER_API_KEY=''
TWITTER_API_SECRET=''
TWITTER_ACCESS_TOKEN=''
TWITTER_ACCESS_SECRET=''
Create the .gitignore file to avoid this. The sample below contains other paths we’d want to ignore.
node_modules/
.env
.DS_Store
dashboard/data/
Fetching the Data
First, we’ll write a Node.js script called to fetch follower statistics from your platform's API. We'll use the standard fetch library to make the API calls and oauth-1.0a to authenticate with X. After fetching the data, we will write results to a JSON file that will serve as our database.
This will be handled by a separate script. To make the output accessible to it, we will write it to an environment file available in GitHub Actions.
I am using node 20.
Create a file called x_fetch_data.js in the root of our project.
require('dotenv').config();
const OAuth = require('oauth-1.0a');
const crypto = require('crypto');
const fs = require('fs');
// Initialize OAuth 1.0a
const oauth = OAuth({
consumer: {
key: process.env.TWITTER_API_KEY, // Read from environment variable
secret: process.env.TWITTER_API_SECRET // Read from environment variable
},
signature_method: 'HMAC-SHA1',
hash_function(base_string, key) {
return crypto.createHmac('sha1', key).update(base_string).digest('base64');
}
});
const token = {
key: process.env.TWITTER_ACCESS_TOKEN, // Read from environment variable
secret: process.env.TWITTER_ACCESS_SECRET // Read from environment variable
};
const url = 'https://api.twitter.com/2/users/me?user.fields=public_metrics';
const fetchTwitterFollowerCount = async () => {
const requestData = {
url,
method: 'GET',
};
// OAuth header
const headers = oauth.toHeader(oauth.authorize(requestData, token));
headers['User-Agent'] = 'v2UserLookupJS';
const response = await fetch(url, {
method: 'GET',
headers
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const data = await response.json();
console.log(data);
// Extract the metrics
const metrics = data?.data?.public_metrics;
// Write the metrics to the environment file
fs.appendFileSync(process.env.GITHUB_OUTPUT, `METRICS=${JSON.stringify(metrics)}\n`);
};
fetchTwitterFollowerCount().catch(err => console.error(err));
To test the script, you can run:
GITHUB_OUTPUT=testoutput node x_fetch_data.js
You should see your X metrics in the output, as well as in the testoutput file:
metrics={"followers_count":288,"following_count":302,"tweet_count":1381,"listed_count":0,"like_count":591}
Saving the Data
To save the data, create another script in a file x_save_data.js . It will take the output from the environment and append it to the ./data/x.json .
Make sure to create this file first and commit it to the git repository. It should have an empty array as its content.
[]
The script also doesn’t add a duplicate record if the data was already fetched that day. It overwrites the old one instead.
const fs = require('fs');
// Parse the JSON string from the environment variable
const metrics = JSON.parse(process.env.METRICS);
const path = './data/x.json';
const now = new Date();
const today = `${now.getFullYear()}-${String(now.getMonth() + 1).padStart(2, '0')}-${String(now.getDate()).padStart(2, '0')}`;
let data = [];
if (fs.existsSync(path)) {
data = JSON.parse(fs.readFileSync(path));
}
const todayIndex = data.findIndex(entry => entry.date === today);
if (todayIndex > -1) {
data[todayIndex] = { date: today, ...metrics };
} else {
data.push({ date: today, ...metrics });
}
fs.writeFileSync(path, JSON.stringify(data, null, 2));
You can test the script by editing testouput file by adding single quotes around the JSON and then running the following. File edit as necessary, as the GitHub Actions environment behaves differently and doesn't need the quotes.
# load output from the previous script
set -a; source testoutput; set +a;
node x_save_data.js
Scheduled GitHub Action
Now, let’s create a file with GitHub action code. It will run every day at a specified time and fetch our metrics. It will then save them and commit them to the repository.
Save the following code under .github/workflows/fetch_x_data.yml .
name: Fetch X Data
on:
schedule:
# Runs at 4 AM UTC
- cron: '0 4 * * *'
workflow_dispatch: # This line enables manual triggering of the action
jobs:
fetch_data:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- name: Check out the repository
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: "20"
- name: Install dependencies
run: npm install
- name: Fetch Data from Platform X
id: fetch_data
run: node x_fetch_data.js
env:
TWITTER_API_KEY: ${{ secrets.TWITTER_API_KEY }}
TWITTER_API_SECRET: ${{ secrets.TWITTER_API_SECRET }}
TWITTER_ACCESS_TOKEN: ${{ secrets.TWITTER_ACCESS_TOKEN }}
TWITTER_ACCESS_SECRET: ${{ secrets.TWITTER_ACCESS_SECRET }}
- name: Save data
run: node x_save_data.js
env:
METRICS: ${{ steps.fetch_data.outputs.METRICS }}
- name: Commit and push if there's changes
run: |
git config --global user.email "action@github.com"
git config --global user.name "GitHub Action"
git add data/x.json
git commit -m "Update data for Platform X" || exit 0 # exit 0 if no changes
git push
Run the action manually by committing the code and then going to the “Actions” section of your project on GitHub and triggering it from there.
Dashboard
Okay, how about presenting the data? I didn’t want to mess around with simple HTML, so I asked ChatGPT to generate it for me.
Create an index.html file in the dashboard folder. We’re not using the main directory of our project in order to avoid hosting the data-fetching code alongside the HTML.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Twitter Dashboard</title>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<style>
body {
font-family: Arial, sans-serif;
margin: 0;
padding: 0;
display: flex;
flex-direction: column;
align-items: center;
}
.chart-container {
width: 80%;
max-width: 1000px;
}
canvas {
max-width: 100%;
}
h1 {
text-align: center;
margin-top: 20px;
}
h2 {
text-align: center;
margin-top: 20px;
}
</style>
</head>
<body>
<h1>Twitter Dashboard</h1>
<h2>Number of Followers</h2>
<div class="chart-container">
<canvas id="followersChart"></canvas>
</div>
<h2>Number of Tweets</h2>
<div class="chart-container">
<canvas id="tweetsChart"></canvas>
</div>
<script>
fetch('data/x.json')
.then(response => response.json())
.then(data => {
const dates = data.map(item => item.date);
const followers = data.map(item => item.followers_count);
const tweets = data.map(item => item.tweet_count);
const minFollowers = Math.min(...followers) - 100;
const minTweets = Math.min(...tweets) - 100;
const followersCtx = document.getElementById('followersChart').getContext('2d');
const tweetsCtx = document.getElementById('tweetsChart').getContext('2d');
new Chart(followersCtx, {
type: 'line',
data: {
labels: dates,
datasets: [{
label: 'Followers',
data: followers,
backgroundColor: 'rgba(54, 162, 235, 0.2)',
borderColor: 'rgba(54, 162, 235, 1)',
borderWidth: 1
}]
},
options: {
scales: {
y: {
beginAtZero: false,
min: minFollowers
}
}
}
});
new Chart(tweetsCtx, {
type: 'line',
data: {
labels: dates,
datasets: [{
label: 'Tweets',
data: tweets,
backgroundColor: 'rgba(255, 99, 132, 0.2)',
borderColor: 'rgba(255, 99, 132, 1)',
borderWidth: 1
}]
},
options: {
scales: {
y: {
beginAtZero: false,
min: minTweets
}
}
}
});
});
</script>
</body>
</html>
Commit it to the repository.
(optional) If you want to test it locally, do so by copying the data folder to the dashboard folder and launching a simple server inside it.
cp -r data dashboard/
cd dashboard
# start server with Python if you have it installed (version 3)
# otherwise, use other way e. g. https://gist.github.com/willurd/5720255
python -m http.server 8000
Dashboard Deployment to GitHub Pages
Now that we have our dashboard, it’s time to present it to the web!
If you are using a free account on GitHub, your page needs to be public, as well as the whole repository.
Create a .github/workflows/deploy_dashboard.yml file.
name: Deploy to GitHub Pages
on:
schedule:
# redeploy after data update
- cron: '0 5 * * *'
push:
branches:
- main
workflow_dispatch: # This line enables manual triggering of the action
permissions:
contents: read
pages: write
id-token: write
# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
group: "pages"
cancel-in-progress: false
jobs:
deploy:
permissions:
pages: write # to deploy to Pages
id-token: write # to verify the deployment originates from an appropriate source
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Pages
uses: actions/configure-pages@v4
- name: Copy data to dashboard folder
run: cp -r data dashboard/
- name: Update pages artifact
uses: actions/upload-pages-artifact@v3
with:
path: dashboard/
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4 # or specific "vX.X.X" version tag for this action
The action should deploy the page. You can find the URL in your GitHub project settings or in the Actions section in the workflow output.
Again, you can find mine here: https://horosin.github.io/metrics-dashboard/.
Summary
And there you have it! A complete, automated system to track your social media (X) metrics, automate data fetching, save historical data, and visualize the trends. With this setup, you can extend the functionality to other platforms and metrics, creating a comprehensive dashboard for all your analytical needs. Let me know if it’s something you’d like to read about.
Subscribe to my profile by filling in your email address on the left, and be up-to-date with my articles!
Don't forget to follow me on Twitter @ horosin, and subscribe to my blog’s newsletter for more tips and insights!
If you don't have Twitter, you can also follow me on LinkedIn.