Showing GitHub repositories automatically in my portfolio – with C#, HttpClient and my own database

My home page has a “GitHub activity” section. It should show my current repositories – with description, language, stars and the date of the last commit. At first I maintained this data by hand. That didn't work for long: every new project and every changed description meant more work, and after a few weeks the list was out of date.

The obvious idea: fetch the data where the code lives anyway – on GitHub. In this post I describe how I implemented this in my ASP.NET Core project and which pitfalls I ran into.

The most important decision: when do I query GitHub?

Before writing a single line of code, I had to think about when the data is fetched. I compared three options:

1. In the browser via JavaScript: every visitor queries GitHub themselves. That's simple, but every browser then connects to GitHub (privacy), and without authentication GitHub only allows 60 requests per hour and IP address.
2. On the server on every page view: better for privacy, but every request to my home page would trigger a request to GitHub. If GitHub goes down, my site has a problem too.
3. Synchronise and store in my own database: a button in the admin area fetches the data once and stores it. The home page then only reads from the database.

I chose the third option. Visitors never trigger a request to GitHub, the page stays fast, and it still works when GitHub is temporarily unreachable. I can also decide in the admin area which repositories appear on the home page and in what order – that wouldn't be possible with a pure live query.

Step 1: Setting up an HttpClient properly

In ASP.NET Core you shouldn't simply create an HttpClient with new, but use the built-in factory instead. I register my service as a so-called typed client:

// Program.cs
builder.Services.AddHttpClient<GitHubApiClientService>();

In the service's constructor I set up everything GitHub expects:

public GitHubApiClientService(HttpClient httpClient, IConfiguration config)
{
    _httpClient = httpClient;
    _config = config;

    _httpClient.BaseAddress = new Uri("https://api.github.com/");
    // Ohne User-Agent antwortet GitHub mit 403 Forbidden
    _httpClient.DefaultRequestHeaders.UserAgent.Add(new ProductInfoHeaderValue("AmirRezaAfshar-Portfolio", "1.0"));
    _httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.github+json"));

    // Optionales Token: kommt aus der Konfiguration, nie aus dem Quellcode
    var token = _config["GitHub:Token"];
    if (!string.IsNullOrWhiteSpace(token))
        _httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
}

With a token, the limit rises from 60 to 5.000 requests per hour. For a sync at the push of a button, the limit without a token would be enough – the important thing is that the token never ends up in the code or in Git. Locally I keep it in User Secrets, and on the server it lives in the configuration, outside the code.

Step 2: Turning the JSON response into a C# class

The list of repositories comes from the endpoint users/{benutzername}/repos. GitHub writes the field names in snake_case, for example html_url or stargazers_count. This is exactly where I made my first mistake: my properties were called HtmlUrl and StargazersCount – and simply stayed empty after deserialising, without any error message.

The solution is the [JsonPropertyName] attribute:

private class GitHubApiRepoDto
{
    [JsonPropertyName("name")]             public string Name { get; set; } = string.Empty;
    [JsonPropertyName("description")]      public string? Description { get; set; }
    [JsonPropertyName("html_url")]         public string? HtmlUrl { get; set; }
    [JsonPropertyName("language")]         public string? Language { get; set; }
    [JsonPropertyName("stargazers_count")] public int StargazersCount { get; set; }
    [JsonPropertyName("pushed_at")]        public DateTime? PushedAt { get; set; }
    [JsonPropertyName("fork")]             public bool Fork { get; set; }
    [JsonPropertyName("archived")]         public bool Archived { get; set; }
}

A DTO like this (data transfer object) contains only the fields I really need. I ignore the rest of the response.

Step 3: Synchronising instead of just reading

During the sync I go through the list from GitHub and compare it with my table:

var response = await _httpClient.GetAsync($"users/{username}/repos?sort=pushed&per_page=100&type=owner");
if (!response.IsSuccessStatusCode)
    return (0, $"GitHub API hat einen Fehler zurückgegeben ({(int)response.StatusCode}).");

var repos = JsonSerializer.Deserialize<List<GitHubApiRepoDto>>(await response.Content.ReadAsStringAsync());

foreach (var dto in repos.Where(r => !r.Fork && !r.Archived))
{
    var existing = _gitHubRepoManager.FindByName(dto.Name);
    if (existing == null)
    {
        // neu: anlegen, aber noch nicht auf der Startseite zeigen
    }
    else
    {
        // vorhanden: nur die Felder von GitHub aktualisieren
    }
}

Two rules were important to me:

• I don't show forked or archived repositories. Only my own, current work should appear in the portfolio.
• My own selection is kept. Whether a repository appears on the home page and in which position is something I decide in the admin area. The sync only overwrites the data that comes from GitHub.

The sync also cleans up: repositories that no longer exist on GitHub (deleted, archived or renamed) are removed from my table. So that this only happens when the response was really fine, I only delete something if GitHub returned at least one repository.

What I learned

• Headers are part of the interface. Without a User-Agent, GitHub rejects every request. Details like this are in the documentation – but you also have to read it.
• Empty fields are often a naming problem. If deserialised values stay empty, the first thing to check is the field names in the JSON.
• Return errors as messages. When something goes wrong, my sync returns an understandable text that the admin panel displays, instead of throwing an exception. That way I can see immediately whether, for example, the limit has been reached.
• Secrets don't belong in the code. Tokens and keys come from the configuration.
• Know the limits: per_page allows at most 100 entries. That's enough for my profile; with more repositories I would have to fetch several pages.

Conclusion

For me this small feature was good practice because it combines many topics from the everyday work of an application developer: an external REST API, asynchronous programming with async/await, JSON, my own database and a deliberate architecture decision. And in practical terms: my home page now always shows my current projects without me maintaining anything by hand.

Amir Afshar

Author of this article