Draft: yolo github mirroring

#540

added 1 commit

• f793e3fe - yolo github mirroring

Compare with previous version

added 1 commit

• fcd621f2 - fixup! yolo github mirroring

Compare with previous version

marked this merge request as draft from klausenbusk/infrastructure@fcd621f2

added 1 commit

• 10597ee9 - fixup! yolo github mirroring

Compare with previous version

added 1 commit

• 627a3802 - fixup! yolo github mirroring

Compare with previous version

If we consider something like this, we should run it from u privileged users and not root (even tho there is a systemd layer). It's rarely a good idea to let root processes do network facing operations if not absolutely unavoidable. Can we try not to use a privileged user?

Indeed. I was already considering if we could use DynamicUser=. Let me take a stab at it.

I think the best we can do is creating a dedicated user and use setfacl to gain access to the files, alternatively we can perhaps do some idmapping. Neither is ideal IMHO.

@anthraxx how this look? Ignore the very bad code quality, but I hope the concept is understandable.

Basically a dedicated user and group (github-mirror), /srv/gitlab/data/git-data/repositories/ read-only bind mounted into /var/lib/github-mirror/repositories/ with idmapping (so the github-mirror user can read the files).

added 1 commit

• 5db49de9 - poc idmap

Compare with previous version

Should use --force.

Should perhaps only take the specified refs for the push into account.

Should be moved to the state directory.

While researching #626 (closed) I thought "perhaps we can solve this the same way" with a ruby script.

Fooling around in the rails console, this may be doable.

Some "notes":

irb(main):028:0> p = Project.find_by_full_path("archlinux/infrastructure")
=> #<Project id:7 archlinux/infrastructure>>
irb(main):029:0> r = p.remote_mirrors[0]
=> 
#<RemoteMirror:0x00007733ff9e5688
...
irb(main):046:0> p.remote_mirrors.create()
=> 
#<RemoteMirror:0x00007733f841dce8
 id: nil,
 project_id: 7,
 url: nil,
 enabled: false,
 update_status: "none",
 last_update_at: nil,
 last_successful_update_at: nil,
 last_update_started_at: nil,
 last_error: nil,
 only_protected_branches: false,
 remote_name: nil,
 encrypted_credentials: nil,
 encrypted_credentials_iv: nil,
 encrypted_credentials_salt: nil,
 created_at: nil,
 updated_at: nil,
 error_notification_sent: nil,
 keep_divergent_refs: nil,
 mirror_branch_regex: nil,
 credentials: nil>

This would of course not fix the issue, that gitlab is not deleting deleted branches on the mirror.

Something is working:

irb(main):008:0> r = p.remote_mirrors.create(enabled: true, url: "ssh://github.com/archlinux2/infrastructure.git", aut
h_method: "ssh_public_key")
=> 
#<RemoteMirror:0x00007f29ab96f088
...
irb(main):009:0> r.ssh_public_key
=> "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0iyBF8/bOepc4knhjbjzo7Nin/bx5FPhFyTyY4HAFX5Tx2F3IGy/9+4TD6T4Mv0bjWT1YjXbpdTAqD0+xj56oFHMlVLGF27znkoc6/1Fa5cjoPx1haqgHpaZpVGXZytoe3rSMMZy0OeKuFYckZWDo5AbxjjgyR5U+rd5jqI3iZLdTCWY+6MFOTZIIZwevrofKK72baGq5/9QjBsJ/OfSscw5Ha7rAtc5ZyCheTkef2N9hdBYzq6ha3RFQ9B+QlUVIRRQHyoEz8TUt3/p+FF03GHRkF4Wffp2SFKAQxxqFZlSikCXA8c0BhodEktOt4UfrpYvLeUGslaYNZxrNCjrc+LtxJuQCTmr0EkTAkqZmxLdA47gYDmsM9i5+V4Tm6Rt9JRbxTcKzcF83I4M2pjQkA57ZT1jDWGoUEmkA1kU5/d2NzSq9Olv9fNoXaAVA7k26IfW+fFfNlQT6kNhtFFCxscBMOubqgqCXRHAY9XNu7MsK5xVgE8d1ihpteSKNOOiws12GM4tXk4Z2jFPndYkDM4DXVZltqvR9vq5X6/g3HLZEp1w2XHrt9CJMDRTZefqywSajSZJwij2uLBm2GIOqVswzNNQ5EKJGUWE/8AbgDF0gTGZwurQOswY5XEaNRTsqa+dgAHUsTK+qXyORSd8XtuFkeITKEBcuPKV4+sMRZw== git@gitlab.archlinux.org"

mentioned in issue #540

We should look further into the ruby possibilities, before gong this route, so closing this.

closed