Removing broken links from lists

[basharovV]

I've got a list of every dead or broken link in every awesome list, along with the error, which is probably useful to the list maintainers.

Here is the output from my cleanup script, which checks every URL in every awesome list:

Total urls: 47941  Valid urls: 41299 Invalid urls: 6642  HTTP errors: 2044  Other errors: 2406

That's 14% of all links being unavailable...

If you maintain an awesome list, feel free to find the file for your list and check if it has any dead links.

[bernardoduarte]

Here's a tool that could help check if your lists links are alive: https://github.com/dkhamsing/awesome_bot

I use it on my awesome-version-managers list .travis.yml config for CI automation.

[riderx]

in my repo i use this github action who check link and lint

name: Check Lint

on:
  workflow_dispatch:
  push:
    branches:
    - main
  schedule:
  # Run every first day at 00:00 AM every month.
    - cron: "0 0 * * *"
  pull_request:
    branches: [main]
    paths:
      - 'readme.md'

jobs:
  main:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Check links
        uses: gaurav-nelson/github-action-markdown-link-check@v1
        with:
          retry-after: 10
          use-quiet-mode: 'yes'
          use-verbose-mode: 'yes'
          config-file: '.github/workflows/markdown.links.config.json'
      - name: Awesome linter
        run: npx awesome-lint

[d-wasserman]

This was really helpful @riderx. The only change I made was I run the awesome linter first, then the check links.

[GovLeena]

🙋‍♀️

[koushik-drmgr]

Hey use my Turbo C code for this..I hope this may helpful for you

#include <stdio.h>
#include <string.h>

// Simulated function to "check" if a URL is valid
int check_url(char *url) {
// In a real scenario, network operations would be done here.
// For now, simulate that any URL containing "valid" is good, others are broken.

if (strstr(url, "valid") != NULL) {
    return 1; // Valid URL
}
return 0; // Invalid URL

}

// Function to process URLs from a file and generate a report
void process_urls(const char *input_file, const char *output_file) {
FILE *infile, *outfile;
char url[200]; // Buffer to store each URL (assuming max 200 chars)
int status;

// Open the input file in read mode
infile = fopen(input_file, "r");
if (infile == NULL) {
    printf("Error: Could not open input file '%s'\n", input_file);
    return;
}

// Open the output file in write mode
outfile = fopen(output_file, "w");
if (outfile == NULL) {
    printf("Error: Could not open output file '%s'\n", output_file);
    fclose(infile);
    return;
}

// Write header to the output file
fprintf(outfile, "URL, Status\n");

// Read each URL from the input file and check its status
while (fgets(url, sizeof(url), infile) != NULL) {
    // Remove newline character from URL, if present
    if ((strlen(url) > 0) && (url[strlen(url) - 1] == '\n')) {
        url[strlen(url) - 1] = '\0';
    }

    // Simulate the URL check
    status = check_url(url);

    // Write the result to the output file
    if (status == 1) {
        fprintf(outfile, "%s, Valid\n", url);
    } else {
        fprintf(outfile, "%s, Invalid\n", url);
    }
}

// Close both input and output files
fclose(infile);
fclose(outfile);

// Notify the user that the process is complete
printf("URL check completed. Results saved to '%s'\n", output_file);

}

int main() {
// File paths (adjust as needed)
char input_file[] = "urls.txt"; // Input file containing URLs (one per line)
char output_file[] = "report.txt"; // Output file to store the results

// Display a welcome message
printf("===== URL Checker Program =====\n");
printf("Processing URLs from '%s'...\n", input_file);

// Process the URLs and generate the report
process_urls(input_file, output_file);

// Display completion message
printf("All done!\n");

return 0;

}

[good1400]

👍