EC2 Disk Space Optimization: Server Tuning Guide

When running test automation frameworks using stack like Selenium, Pytest, Java, Jenkins, JMeter, and others on an EC2 instance, disk space can quickly get exhausted due to:

• Large log files from test executions
• Accumulated browser cache (Chrome/Firefox)
• Old Jenkins workspaces and builds
• Redundant Docker containers and images
• Unused dependencies and libraries

I ran into this exact issue while debugging an overloaded EC2 instance running Selenium-based test automation.

After executing du -ah /, I discovered GBs of unnecessary data piling up. Here’s how I optimized disk space without affecting test executions.

---

1. Clear Test Execution Logs & Reports

Pytest & Allure Reports Cleanup

Pytest, Selenium, and Allure generate test reports that can quickly accumulate. Remove old reports to free up space:

rm -rf /home/ec2-user/reports/*
rm -rf /home/ec2-user/.pytest_cache

Alternatively, automate cleanup with a cron job:

crontab -e

Add this to clean reports every week:

0 0 * * 7 rm -rf /home/ec2-user/reports/*

Jenkins Console Logs & Old Builds

Jenkins stores old job logs and workspaces that can consume GBs of space. Run:

rm -rf /var/lib/jenkins/jobs/*/builds/*
rm -rf /var/lib/jenkins/jobs/*/workspace/*

To configure automatic cleanup:

1. Go to Jenkins → Manage Jenkins → Configure System
2. Set “Discard Old Builds” to retain only the last 10 builds
3. Use logrotate for old logs:

sudo logrotate /etc/logrotate.conf

---

2. Clean Up Selenium & Browser Cache

Chrome/Firefox WebDriver Cache Cleanup

Selenium tests generate cache, logs, and temporary files. Remove them periodically:

m -rf /home/ec2-user/.cache/google-chrome/
rm -rf /home/ec2-user/.mozilla/firefox/

To avoid excessive cache accumulation, start Chrome in incognito mode using:

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--incognito")
driver = webdriver.Chrome(options=options)

Selenium Debug Logs Cleanup

If you enable debug logs, ensure they don’t pile up. Remove logs older than 7 days:

find /home/ec2-user/selenium-logs -type f -name "*.log" -mtime +7 -exec rm {} \;

---

3. Docker Cleanup for Test Automation Containers

If you’re running Selenium Grid, JMeter, or Jenkins in Docker, old containers and images consume significant space. Run:

docker system prune -a
docker volume prune

Check the largest Docker images and remove unnecessary ones:

docker images --format "{{.Repository}}:{{.Tag}} {{.Size}}"
docker rmi <image_id>

---

4. Remove Unused Python & Java Dependencies

Python Virtual Environments & Unused Packages

Check and remove old virtual environments and unused Python dependencies:

rm -rf /home/ec2-user/.venvs/
pip freeze | xargs pip uninstall -y

Java JARs & Dependencies Cleanup

If you run TestNG, Selenium, JMeter, or Cucumber, old .jar dependencies can take up space. Remove unnecessary ones:

find /home/ec2-user/.m2/repository -type f -name "*.jar" -size +100M -delete

---

5. JMeter Test Data & Result Cleanup

JMeter generates huge log files and reports after test executions. Delete old test data:

rm -rf /home/ec2-user/jmeter/results/*
rm -rf /home/ec2-user/jmeter/logs/*

To automate cleanup, modify jmeter.properties:

jmeter.save.saveservice.output_format=xml
jmeter.save.saveservice.assertion_results=none

---

6. Set Up Automated Monitoring for Disk Usage

Install ncdu for Disk Space Analysis

Instead of running du -ah /, use ncdu for a graphical disk usage breakdown:

sudo yum install -y ncdu
ncdu /

Set Up AWS CloudWatch Alarms for Disk Usage

If your EC2 instance runs out of disk space frequently, create a CloudWatch alarm:

1. Open AWS CloudWatch
2. Go to Alarms → Create Alarm
3. Set Metric: EC2 → Per-Instance Metrics → Disk Space Utilization
4. Configure an alert when disk usage exceeds 80%

---

7. Move Large Files to S3 or EFS

For test execution videos, logs, and archived reports, consider offloading them to AWS S3:

aws s3 sync /home/ec2-user/reports s3://your-bucket-name/

To make JMeter save test results directly to S3:

jmeter -n -t test.jmx -l s3://your-bucket-name/results.jtl

---

8. Remove Orphaned Processes & Zombie Files

Sometimes, test automation scripts leave behind orphaned processes or temporary files.

Kill Stuck Processes

Find and kill processes consuming high memory/disk:

ps aux --sort=-%mem | head -n 10  # Show top memory-consuming processes
kill -9 <PID>

Use htop for real-time process monitoring:

sudo yum install -y htop
htop

---

9. Compress Large Files & Logs Instead of Deleting

Instead of deleting critical log files, compress them to save space:

gzip /var/log/*.log
tar -czf /home/ec2-user/jmeter-reports.tar.gz /home/ec2-user/jmeter/results/

For automatic compression, set up logrotate:

sudo nano /etc/logrotate.d/custom_logs

/var/log/myapp/*.log {
    weekly
    rotate 4
    compress
    missingok
}

---

10. Use a Dedicated Storage Volume (EBS) for Test Data

If test artifacts (logs, screenshots, reports) take up space, mount a separate EBS volume:

Attach an EBS Volume

1. Go to AWS EC2 Console → Elastic Block Store → Create Volume
2. Attach it to your EC2 instance
3. Mount it:
   • sudo mkfs -t xfs /dev/xvdf
   • sudo mkdir /mnt/testdata
   • sudo mount /dev/xvdf /mnt/testdata

Move test reports to this volume:

mv /home/ec2-user/reports /mnt/testdata/

---

11. Optimize Memory Swap Usage

If your EC2 instance runs low on RAM, swap usage increases disk writes. Optimize swap space:

Check Swap Usage

free -m

Disable Swap for Performance (if enough RAM available)

sudo swapoff -a

Reduce Swappiness to Minimize Disk Usage

sudo sysctl vm.swappiness=10

---

12. Delete Temporary Build Files (Java, Node.js, Python, etc.)

Build processes generate temporary files that aren’t cleaned up automatically.

Java & Maven Build Cleanup

rm -rf ~/.m2/repository
rm -rf /home/ec2-user/workspace/*.class

Node.js Build Cleanup

rm -rf node_modules/.cache
rm -rf dist/

Python Bytecode Cleanup (__pycache__)

find . -name "__pycache__" -type d -exec rm -r {} +

---

13. Use a More Efficient Filesystem (XFS Instead of ext4)

If your EC2 instance has high I/O activity, switching to XFS can improve disk efficiency.

Check Current Filesystem

df -Th

If using ext4, convert the volume to XFS:

sudo mkfs -t xfs /dev/xvdf
sudo mount /dev/xvdf /mnt/testdata

---

Final Thoughts

Optimizing disk space on an EC2 instance used for test automation is crucial for performance and stability. By cleaning up test reports, Selenium logs, Docker containers, and old dependencies, you can free up storage and ensure smooth test execution.

Got any other cleanup strategies? Drop a comment—I’d love to hear them! 🚀

Bookmark