• Matt The Horwood@lemmy.horwood.cloud
    link
    fedilink
    English
    arrow-up
    5
    ·
    2 days ago

    It took longer than planned as we are in the process of getting everything sorted for quick rebuilds, most of the time was waiting for GitHub to not poo it’s self.

    If we had everything in ansible it would be 5 minutes

    • corsicanguppy@lemmy.ca
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 day ago

      If we had everything in ansible it would be 5 minutes

      Ansible at the day-job takes about 45 min to do a patch. As someone who’s used a fair number of alternatives, maybe consider anything else on the planet?

      But, seriously. The response to this particular sploit wasn’t really “config management”-y. It was a simple process that cron generally handled anyway, and network segregation and firewalling allowed us to prioritize easily. My worry is that there are some core architectural decisions you may need to revisit in the post-mortem on this, because there may be some fundamental changes required to get sensible wins in the future.

      (If you’re on containers, I weep for your validation challenges; also, get bazel)

      • Matt The Horwood@lemmy.horwood.cloud
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        We find ansible to be the best way to manage our fleet, patching and keeping config in sync.

        That’s not to say it can’t be slow sometimes, but it works for us.