1. 29 Jul, 2021 6 commits
    • Evangelos Foutras's avatar
      Merge branch 'syncarchive-drop-delay-updates-option' into 'master' · ceea8ba8
      Evangelos Foutras authored
      syncarchive: Drop --delay-updates; breaks incremental scan
      
      See merge request !472
      ceea8ba8
    • Evangelos Foutras's avatar
      syncarchive: Drop --delay-updates; breaks incremental scan · 9fd5d458
      Evangelos Foutras authored
      The --delay-updates option results in 6G memory usage per archive mirror
      for a total of ~18G memory used on gemini when all three archive mirrors
      are syncing. Less important (but still revelant!) is the memory usage on
      each mirror, which climbs to about 11G during each synchronization.
      
      Removing the --delay-updates option should be OK considering the archive
      hosts data that almost never changes. Without this option, rsync is able
      to do a sequential scan which uses 90M of memory (per archive mirror) on
      gemini and about 250M on each mirror individually.
      9fd5d458
    • Kristian Klausen's avatar
      86225110
    • Evangelos Foutras's avatar
      Merge branch 'syncrepo-updates' into 'master' · a0762ecb
      Evangelos Foutras authored
      syncrepo tweaks now that rsync has fixed the stale .~tmp~ issue
      
      See merge request !471
      a0762ecb
    • Evangelos Foutras's avatar
      syncarchive: Tweak rsync options and drop temp dir · 49ba41bf
      Evangelos Foutras authored
      This reflects the recent changes made to syncrepo.
      49ba41bf
    • Evangelos Foutras's avatar
      syncrepo: Drop --temp-dir option as unneeded · 4fe427a2
      Evangelos Foutras authored
      Using a temporary directory outside of /srv/ftp was meant to protect
      against incomplete files from being synced by downstream mirrors. It
      does not achieve this to much effect though; each file gets uploaded
      to the temporary directory but then immediately moved under a .~tmp~
      directory at its target location (.~tmp~ because of --delay-updates,
      otherwise the file would be renamed to its final path).
      
      The `--delay-updates` option by itself sufficiently protects against
      temp files being transferred to downstream mirrors; when used by the
      receiver, it automatically adds an exclude rule for ~.tmp~, behaving
      exactly like we want it to. As such, the `--temp-dir` option doesn't
      provide any further benefit and can be removed.
      4fe427a2
  2. 28 Jul, 2021 9 commits
  3. 27 Jul, 2021 1 commit
  4. 25 Jul, 2021 1 commit
  5. 24 Jul, 2021 4 commits
  6. 23 Jul, 2021 1 commit
  7. 20 Jul, 2021 2 commits
  8. 18 Jul, 2021 5 commits
  9. 17 Jul, 2021 1 commit
    • Evangelos Foutras's avatar
      Use RandomizedDelaySec=30min in Borg TextCollector · 3aa4d49f
      Evangelos Foutras authored
      Doing this in an attempt to be kind to our Borg hosts in cases where the
      prometheus-borg-textcollector.timer is restarted on all hosts and avoids
      having all machines querying the Borg hosts within the same minute. Only
      downside is that the timers will trigger every 75-ish minutes instead of
      exactly every hour, but this should not be a problem.
      3aa4d49f
  10. 16 Jul, 2021 8 commits
  11. 15 Jul, 2021 1 commit
  12. 14 Jul, 2021 1 commit