Setup a local clone of the official git repository (Task #57)

I have some crons and post-receive hooks to fetch from/push to github and update Redmine in the process, do you already have something or would you like to have them?

I'd vote to pull the repo more often, say every 10 minutes. I find it very painful on redmine.org to see the code with such great delay. At best we should get rid of any polling and use some sort of hook (which is also possible with github) though.

Felix: All of my scripts are custom to my server so if you have something you can share that would be great.

Holger: I'd say every 30 minutes in the short term. At every 10 minutes you run the risk of loading multiple tasks and killing the server.

Github has a github services repository where we can add our own github-hook. Then we just need to add an endpoint on our side to receive that post. It's not too much work and I think there are a few github/redmine integration plugins that we can use or build on top of.

Eric Davis wrote:

Felix: All of my scripts are custom to my server so if you have something you can share that would be great.

So this is what I have in the post-receive:

#!/bin/sh

export REPOS_PATH="$PWD" 

/some/path/scm_post_hook.sh

And scm_post_hook.sh (redacted a bit, holds some branches depending on git/svn calling it) holds:

#!/bin/bash

### Initialization stuff ###
declare -A MIRROR_TO_GITHUB

### User settings ###
# Base URI of your Redmine
REDMINE_URI="some_URI" 
# The API key for the Redmine repository WS
REDMINE_API_KEY="some_key" 
# List of Repositories to push to github
MIRROR_TO_GITHUB=(["redmine_doodles"]=1 ["redmine_ical"]=1) # Format: (["reponame1"]=1 ["reponame2"]=1 ["reponame3"]=1)

### Computed settings ###
# Expects REPOS_PATH to be of the form /var/REPOS_SCM/VHOST_NAME/REPOS_NAME
REPOS_NAME=$(echo $REPOS_PATH | awk -F '/' '{print $5;}')

### Actual actions ###
# Trigger Redmine to fetch the changes for the repository's project
curl -s -o /dev/null "${REDMINE_URI}/sys/fetch_changesets?key=${REDMINE_API_KEY}&id=${REPOS_NAME}" &

# Push to github
if [[ -n ${MIRROR_TO_GITHUB[${REPOS_NAME}]} ]]; then
  git --git-dir=${REPOS_PATH} push --quiet github &
fi

This assumes all repos live in the same directory, or at least in the same depth, and that there is a github remote configured in the repo as a mirror.

The crontab line for fetching is:

*/30 * * * *   some_user  cd /some/path/to/a/git-repo && git fetch --quiet github && hooks/post-receive

There's probably nicer/better ways, has worked fine for me for quite some time though.

Felix:

It looks like your code does push and pulls. So if you push into the private repo it would update ChiliProject and then push the changes up to github. And it also has a cronjob to pull data down from github.

Maybe we could just have a cronjob that does a git pull on the local repo, when it pulls data it triggers a post commit hook to load the new commits into ChiliProject.

Eric Davis wrote:

It looks like your code does push and pulls. So if you push into the private repo it would update ChiliProject and then push the changes up to github. And it also has a cronjob to pull data down from github.

The post-receive hook pings redmine to fetch changesets for that repo, and pushes to github if the project is in the MIRROR_TO_GITHUB list, so pushing to github is optional on a per project/repo basis.

Maybe we could just have a cronjob that does a git pull on the local repo, when it pulls data it triggers a post commit hook to load the new commits into ChiliProject.

That's what the crontab line does :-)

I've setup a moonshine recipe to manage this for us now. I'll need to switch the cloning repo once our repo is setup (#29) but it's working good so far.

Code: https://gist.github.com/9278e19f8e28b05a831b

Documentation¶

It installs curl which is used by the git hook.

It makes sure there is a /srv/git_mirrors directory for our user account. This is so if we want to have multiple mirrors, for example official plugins. (file "/srv/git_mirrors")

It then runs the git clone command (exec('git-mirror-chiliproject')) but only if the target doesn't exist (creates => target_path).

Then it sets up some variables and creates the script post-receive hook as a string (update_repo_view_hook). This is written out to the disk with a mode of 755 so it can be executed.

Finally a cronjob is added to run the git fetch and post-receive hook every 15 minutes (cron).

The actual git hook uses curl and the SysController to fetch the changesets for our project.

So overall, the code flow would be:

1. We push to github
2. Within 15 minutes, the cron kicks off
3. cron triggers the git fetch which loads the new changesets
4. Then the post-receive hook is called
5. Which hits the SysController, which loads the new changesets

We should be able to reuse this code on other mirror too, it's pretty simple.