Searching for Code, Web and API differ

[FHantke]

Searching for Code, Web and API differ

Hi all,

I am not sure if this is the correct repository to open this issue, but I did not find any other better fitting one.
I started playing around with the Git rest API today to search for code snippets. However, after a few searches, I realized that the API finds less files than the normal web search. Is there an option that prevents the search from showing all results?

My code is simply a request to the /search/code endpoint.

    headers = {
        'Accept': 'application/vnd.github+json',
        'Authorization': f'Bearer {GIT_API_KEY}',
        'X-GitHub-Api-Version': '2022-11-28',
    }

    params = { 'q': '"Hello World"' }

    res = requests.get('https://api.github.com/search/code', params=params, headers=headers)
    json_data = res.json()
    print(f"Total count: {json_data['total_count']}")

The API returns 817152 results, while using the GitHub web search return 4.6M.

I would expect to find the same number of files with both methods.

Thank you
Florian

[shiftkey]

@FHantke thanks for the feedback!

Unfortunately the new web search uses Blackbird (more details in this post), and because of changes to the search syntax and search results we have not yet upgraded the REST API to use this engine.

I can't be more specific about when this will occur, but keep an eye on the issue tracker in https://github.com/github/roadmap for a preview of this new API...

[FHantke]

Thank you for the quick answer 👍
I will keep an eye on it.

[joac]

I'm here 8 month later, having the same issue...
Will clone the repos locally and search them using grep...

[paulslaby]

So you say those search syntax is different. Do you have a documentation for API search syntax? I used https://docs.github.com/en/search-github/github-code-search/understanding-github-code-search-syntax#path-qualifier which is linked from here: https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#constructing-a-search-query and it's not working.

I can live with syntax being different, but I need to know, what's possible.

Currently I am looking for query for finding files by names like: path:**/*.Dockerfile repo:XXX

[mkheffache]

+1 on this issue. I am trying to perform a simple query via search API:
org:<org_id> extension:tf
And the equivalent from the UI:
org:<org_id> path:*.tf

I am simply trying to count how many Terraform files exist within an org. The number returned from the UI is 7x the amount returned in the API.

What I would like to ask is not upgrading either of them or making a fix, but rather understand why these two queries are different and what is the source of truth?