A Push ends in: ! [remote rejected] lab -> lab (pre-receive hook declined)

[minoru7]

• Gitea version (or commit ref): 1.3.2
• Git version: 2.11.0
• Operating system: Debian Stretch (9.3)
• Database (use [x]):
  • PostgreSQL
  • MySQL (MariaDB, actually)
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist: [...io/gitea/cmd/hook.go:115 runHookPreReceive()] [F] retrieve protected branches information failed

Description

I have seen similar issues, but the solutions to those are not fixing the current issue. Example: #1685.

What's strange is that it will fail and the logs will show the above. However, if I restart the gitea service, a push happens correctly, as expected. This will be fine for a while, but somewhere after about 10 minutes or so, I would have to restart the service again to be able to push changes to my repos. This is affecting all of my repos. As I find more information, I'll be glad to post details here. Do you have any suggestions what may be causing this?

I've attached my config file for review (private info redacted). This config worked well with versions in the 1.2 branches.

app.ini.txt

[minoru7]

I have also found the following in the service logs, which may be relevant:
Completed /<my-path-to-repo.git>/info/refs?service=git-receive-pack 401 Unauthorized in 4.739506ms
Completed /<my-path-to-repo.git>/info/refs?service=git-upload-pack 401 Unauthorized in 2.956383ms

It does, however, show a "200 ok" later for each of these items. Secondly, if we were having a rights issue, why would a restart of the service fix it?

Still looking for more details to provide. Thank you for any help that could be given.

[minoru7]

And I've found this:

Dec 18 00:09:53 git01.mydomain.net gitea[21940]: [Macaron] 2017-12-18 00:09:53: Started GET / for <myIP_redacted>
Dec 18 00:09:53 git01.mydomain.net gitea[21940]: [Macaron] 2017-12-18 00:09:53: Completed / 302 Found in 681.322µs
Dec 18 00:12:58 git01.mydomain.net gitea[21940]: [mysql] 2017/12/18 00:12:58 packets.go:33: unexpected EOF

[lunny]

@minoru7 that means your config LOCAL_ROOT_URL is not corrected.

; Local (DMZ) URL for Gitea workers (such as SSH update) accessing web service.
; In most cases you do not need to change the default value.
; Alter it only if your SSH server node is not the same as HTTP node.
LOCAL_ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/

[minoru7]

@lunny I'm confused. Why does it work for a while after I restart the service? Also, we're doing a proxy per what that comment says:

    ProxyPreserveHost On
    ProxyRequests off
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/

This worked with prior versions. What has changed? Thank you for your assistance.

[lafriks]

@minoru7 because it was changed to use http internal API for operations that change DB from ssh to have less problems with sqlite database locking

[minoru7]

Ah, ok. Was that done after version 1.2.3?

So we're unable to use the same proxying we were doing before?

[lafriks]

You can do, you just need to set LOCAL_ROOT_URL so that ssh server can connect to gitea address, usually that is http://localhost....

[minoru7]

I guess I was under the impression that the following were variables pulling from other values:
LOCAL_ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/

I thought I could just place PROTOCOL=http, HTTP_ADDR=localhost, and HTTP_PORT=3000 and then that string (above) would take those values and fill in accordingly. Kind of like what happens with apache configs. That's not the case?

[lafriks]

In such case yes that should be so

[minoru7]

Ok, in that case, if you look at my config file that I attached, the values are there and should turn up as LOCAL_ROOT_URL = http://localhost:3000. Does it need to be the external address listed in ROOT_URL, or is having it "localhost" sufficient? All functions for the Gitea server are local to this server (ssh, http(s), etc.).

[minoru7]

I think I may have figured this out. We use Puppet to lay out our configurations. I noticed that every time the Puppet agent ran, it was breaking it. So then I also noticed that after I restarted the gitea service, a change was made to the /opt/gitea/custom/conf/app.ini that was removing INTERNAL_TOKEN = <some_number> under the [security] heading.

It looks like Gitea places that into the config when it starts up? And then Puppet is removing it after the fact. What is this item? And was this added after version 1.2.3?

[lunny]

@minoru7 I think it should be. The INTERNAL_TOKEN config was added in v1.2.x. Gitea web and Gitea serve/hook will share the token so that that should not be changed after Gitea web running.

[minoru7]

I never saw this in version 1.2.3, which was our previous version. We're now running 1.3.2. Is the removal of the INTERNAL_TOKEN by Puppet causing the error I listed above, you think?

[lunny]

@minoru7 I think yes.