From some blogs I’ve written in the past you might already know that we are using AWS SSM to patch and maintain the complete EC2 and onprem instances at one of our customers. The previous posts about that topic are here:
- Avoiding patching madness by using AWS SSM
- Avoiding patching madness by using AWS SSM
- A simple Terraform script to create an AWS EC2 playground
- Schedule reboots of your AWS instances and how that can result in a hard reboot and corruption
While that in general is working fine and fully automated we ran into an issue lately which forced us to create our own CentOS 7 repositories and use them with SSM to apply the patches to the CentOS machines.
To describe the issue: We have two patch baselines per operating system. One for all development and test systems that applies all patches that are released up until the date the patch baseline is running. Then we have second one for the production systems with an approval delay of 14 days. As we run production patching 2 weeks after we patched the development and test systems that should guarantee that we get the same patches applied to production. And exactly here is the issue: “if a Linux repository doesn’t provide release date information for packages, Systems Manager uses the build time of the package as the auto-approval delay for Amazon Linux, Amazon Linux 2, RHEL, and CentOS. If the system isn’t able to find the build time of the package, Systems Manager treats the auto-approval delay as having a value of zero.”. That basically means: As you never know when CentOS will release their patches, which are based on the RedHat sources, you can never be sure that you get the same patches applied to production as they were applied 14 days before to development and test. Lets do an example: Our patching for development and test happened the 10th of April. The kernel package v3.10.0-1127 was released for CentOS on April 27th and was therefore not applied to the development and test systems. When production patching happened two weeks later that kernel package was available but also satisfied our auto approval rule of 14 days. So we basically had a patch installed on the production which never made it to the development and test systems. This is why we decided to go for our own repositories so we can decide when the repositories are synced.
Setting up a local yum repository is quite easy and you can find plenty of howtos in the internet, so here is just a summary without much explanation. We deployed a new CentOS 7 EC2 instance, then installed a webserver and the epel repository:
[centos@ip-10-47-99-158 ~]$ sudo yum install epel-release nginx -y [centos@ip-10-47-99-158 ~]$ sudo systemctl start nginx [centos@ip-10-47-99-158 ~]$ sudo systemctl enable nginx [centos@ip-10-47-99-158 ~]$ sudo systemctl status nginx
As yum gets the packages over http or https adjust the firewall rules:
[centos@ip-10-47-99-158 ~]$ sudo systemctl start firewalld [centos@ip-10-47-99-158 ~]$ sudo systemctl enable firewalld [centos@ip-10-47-99-158 ~]$ sudo firewall-cmd --zone=public --permanent --add-service=http [centos@ip-10-47-99-158 ~]$ sudo firewall-cmd --zone=public --permanent --add-service=https [centos@ip-10-47-99-158 ~]$ sudo firewall-cmd --reload
Update the complete system and install the yum utilities and the createrepo packages:
[centos@ip-10-47-99-158 ~]$ sudo yum update -y [centos@ip-10-47-99-158 ~]$ sudo yum install createrepo yum-utils -y
Prepare the directory structure and synchronize the repositories:
[centos@ip-10-47-99-158 ~]$ sudo mkdir -p /var/www/html/repos [centos@ip-10-47-99-158 ~]$ sudo chmod -R 755 /var/www/html/repos [centos@ip-10-47-99-158 ~]$ sudo reposync -g -l -d -m --repoid=base --newest-only --download-metadata --download_path=/var/www/html/repos/centos-7/7/ [centos@ip-10-47-99-158 ~]$ sudo reposync -l -d -m --repoid=extras --newest-only --download-metadata --download_path=/var/www/html/repos/centos-7/7/ [centos@ip-10-47-99-158 ~]$ sudo reposync -l -d -m --repoid=updates --newest-only --download-metadata --download_path=/var/www/html/repos/centos-7/7/ [centos@ip-10-47-99-158 ~]$ sudo reposync -l -d -m --repoid=epel --newest-only --download-metadata --download_path=/var/www/html/repos/centos-7/7/
Create the repositories from what was synced above:
[centos@ip-10-47-99-158 ~]$ sudo createrepo /var/www/html/repos/centos-7/7/base [centos@ip-10-47-99-158 ~]$ sudo createrepo /var/www/html/repos/centos-7/7/extras [centos@ip-10-47-99-158 ~]$ sudo createrepo /var/www/html/repos/centos-7/7/updates [centos@ip-10-47-99-158 ~]$ sudo createrepo /var/www/html/repos/centos-7/7/epel
… and set the selinux context:
[centos@ip-10-47-99-158 ~]$ sudo semanage fcontext -a -t httpd_sys_content_t "/var/www/html/repos(/.*)?" [centos@ip-10-47-99-158 ~]$ sudo restorecon -Rv /var/www/html/repos
Configure nginx to point to the repositories:
[centos@ip-10-47-99-158 ~]$ sudo vi /etc/nginx/conf.d/repos.conf
## add the folling section
server {
listen 80;
server_name 10.47.99.158;
root /var/www/html/repos/;
location / {
index index.php index.html index.htm;
autoindex on; #enable listing of directory index
}
}
… and restart the webserver:
[centos@ip-10-47-99-158 ~]$ sudo systemctl restart nginx
From now on you should see the directory structure when you point your browser to the IP of the EC2 instance:
To regularly synchronize the repositories depending on your requirements create a small script that does the job and schedule that with cron, e.g.:
#!/bin/bash
LOCAL_REPOS="base extras updates epel"
##a loop to update repos one at a time
for REPO in ${LOCAL_REPOS}; do
if [ "$REPO" = "base" ]; then
reposync -g -l -d -m --repoid=$REPO --newest-only --download-metadata --download_path=/var/www/html/repos/centos-7/7/
else
reposync -l -d -m --repoid=extras --newest-only --download-metadata --download_path=/var/www/html/repos/centos-7/7/
fi
createrepo /var/www/html/repos/centos-7/7/$REPO
semanage fcontext -a -t httpd_sys_content_t "/var/www/html/repos(/.*)?"
restorecon -Rv /var/www/html/repos
done
Test the repository from another CentOS 7 instance:
Using username "centos". Authenticating with public key "imported-openssh-key" [centos@ip-10-47-98-80 ~]$ sudo bash [root@ip-10-47-98-80 centos]$ cd /etc/yum.repos.d/ [root@ip-10-47-98-80 yum.repos.d]$ ls CentOS-Base.repo CentOS-CR.repo CentOS-Debuginfo.repo CentOS-fasttrack.repo CentOS-Media.repo CentOS-Sources.repo CentOS-Vault.repo [root@ip-10-47-98-80 yum.repos.d]$ rm -f * [root@ip-10-47-98-80 yum.repos.d]$ ls -la total 12 drwxr-xr-x. 2 root root 6 Jun 25 06:39 . drwxr-xr-x. 77 root root 8192 Jun 25 06:36 .. [root@ip-10-47-98-80 yum.repos.d]$ cat local-centos.repo [local] name=CentOS Base baseurl=http://10.47.99.158/centos-7/7/base/ gpgcheck=0 enabled=1 [extras] name=CentOS Extras baseurl=http://10.47.99.158/centos-7/7/extras/ gpgcheck=0 enabled=1 [updates] name=CentOS Updates baseurl=http://10.47.99.158/centos-7/7/updates/ gpgcheck=0 [epel] name=CentOS Updates baseurl=http://10.47.99.158/centos-7/7/epel/ gpgcheck=0 [root@ip-10-47-98-80 yum.repos.d]# [root@ip-10-47-98-80 yum.repos.d]$ yum search wget Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile ======================================================================= Name Exactly Matched: wget ======================================================================= wget.x86_64 : A utility for retrieving files using the HTTP or FTP protocols Name and summary matches only, use "search all" for everything. [root@ip-10-47-98-80 yum.repos.d]#
… and you’re done from a repository perspective.
Now it is time to tell SSM to use the local repositories with your patch baseline. If you don’t know how SSM works or how you can apply patches using SSM check the previous post.
All you need to do is to adjust the patch baseline to include your repositories as “Patch sources”:
Schedule your patching and then check the logs. You should see that SSM is now using the local repositories:
...
u'sources': [{u'configuration': u'[local]\nname=CentOS Base\nbaseurl=http://10.47.99.158/centos-7/7/base/\ngpgcheck=0\nenabled=1', u'products': [u'*'], u'name': u'base'}, {u'configuration': u'[extras]\nname=CentOS Extras\nbaseurl=http://10.47.99.158/centos-7/7/extras/\ngpgcheck=0\nenabled=1', u'products': [u'*'], u'name': u'extras'}, {u'configuration': u'[updates]\nname=CentOS Updates\nbaseurl=http://10.47.99.158/centos-7/7/updates/\ngpgcheck=0', u'products': [u'*'], u'name': u'updates'}
...
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-Base.repo
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-CR.repo
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-Debuginfo.repo
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-Media.repo
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-Sources.repo
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-Vault.repo
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-fasttrack.repo
06/25/2020 10:00:28 root [INFO]: Moving file: CentOS-x86_64-kernel.repo
06/25/2020 10:00:28 root [INFO]: Executing lambda _create_custom_repos
06/25/2020 10:00:28 root [INFO]: Creating custom repo base
06/25/2020 10:00:28 root [INFO]: Creating custom repo extras
06/25/2020 10:00:28 root [INFO]: Creating custom repo updates
06/25/2020 10:00:28 root [INFO]: Creating custom repo epel
Loaded plugins: fastestmirror
That’s it. Doing it this way you have full control about which packages will be installed. The downside, of course, is, that you need to maintain your own copy of the repositories.
Cet article Attaching your own CentOS 7 yum repository to AWS SSM est apparu en premier sur Blog dbi services.