Research / By

Automated Self Repairing Server Using Jenkins Automation

The original project was designed around the idea of having a training environment for new employees to the Red Team, the idea was to give them a safe place to practice skills in a variety of challenges like a Capture The Flag Platform.

What came from the original design and build went in a completely different direction, to build a platform that could not be destroyed, and if it was, that it could repair itself. This project was designed and built using VMware virtualization Technologies and specifically the vSphere application, a PHP web application for the website and scripts, and a Java application called Jenkins WebUI that facilitates full server automation. the platform design allows the system to be 100% self-sufficient at any time without the need of the systems designer to control it, thus the platform may continue its usage within the company after the student had returned to education, or if there are software related issues that may cause the system to go offline

For the simplistic sake, I won’t be discussing the Website, mainly because this post is about server repairing. Even tho this was done on a paid platform “vSphere” it could be easily replicated with the use of Proxmox and plugins. I will also be assuming that you understand the uses of vSphere and Jenkins if not check out these two locations

Jenkins – https://www.jenkins.io/

VMware vSphere – https://docs.vmware.com/en/VMware-vSphere/

The WebServer

This is a simple windows 7 box on the VMware ESXI server as a VM running the following software. Why windows 7, we had multiple licenses for windows 7 so it was not a concern at the time

 

  • XAMPP “Apache, for the Web application”

  • Xlight FTP “to copy files over… lazy mode activated there”

Auto-Start Scripts

These scripts were put in the startup folder to allow for them to run on startup. It was a simple Batch file that starts each service as needed for usage on the system

Here you can see the Jenkins User interface with build projects sitting ready to be launched by the user
 
 
 
 
 
 
 

Here you can see a simple script that can be found on my GitHub in both batch and shell form if you need it.

The script in batch just opens a browser page to request the URL and the closes, the shell version uses curl to do the same function

http://[USER]:[API TOKEN]@[SERVER IP]/job/[BUILD NAME]/build?token=[API TOKEN]

The build works by the API calling the project via the URL very simple tool

What you see here is the build as it is fully laid out from start to finish, it is a very simple process to build a process using the plugin for the vSphere that can be downloaded from the Jenkins plugin store and set up easily.

 

It brakes down like this

  1. The build is requested via the API
  2. Connect to the vSphere System
  3. Clone an Image for a Prebuilt Image to A set IP
  4. Start the Image
  5. Wait for the Set time of 30 seconds
  6. Curl webpage
  7. If the page can be curled Go Back to Wait
  8. If the page cant be curled continue
  9. Connect to the vSphere System
  10. Attempt to stop vSphere VM
  11. Delete VM on vSphere
  12. Clone an Image for a Prebuilt Image to A set IP
  13. Start the Image
  14. Go Back to Wait

 

Yeap that simple, the server will attempt to check for the service every 1 hour, if it fails to reach the service, destroy and rebuild itself. Now, this becomes self Repairing when you have an exact clone of this Jenkins VM, doing the exact same thing on the Main, so you constantly have one checking the other and if one detects the other is down its attempts to rebuild itself.

Of course, this can be stopped by stopping both systems being stopped in the time frame before checks and again this is a very basic level system. I thought this was something interesting I worked on while at college and decided to share this idea

Scroll to Top