Designing a Secure Code Execution Engine for Programming Assessments

Problem

Evaluating programming assignments at scale is challenging. Manual grading is slow, error-prone, and not scalable. Basic automated systems often lack security, flexibility, and cannot safely execute arbitrary code from students.

Core Challenge

The main challenge is executing untrusted code submissions. Without proper isolation, malicious or buggy code can compromise the system, access sensitive data, or abuse resources.

Architecture

The system flow:

1. User submits assignment in Moodle
2. Moodle calls backend REST API
3. API provisions a VirtualBox VM
4. Untrusted code is executed in the VM
5. Result is collected and returned to Moodle

Technical Approach

• REST API orchestrates assignment submission, execution, and result retrieval
• VirtualBox VMs are provisioned on demand for each code execution
• Execution pipeline manages code transfer, execution, and teardown

Security Design

• Isolation: Each submission runs in a dedicated VM, preventing code breakout
• Blast radius control: Resource limits and network isolation reduce risk

Trade-offs

• VMs provide strong isolation but are slower to start than containers
• Security prioritized over raw performance for initial prototype

Outcome

• Working prototype with safe code execution
• Integrated with Moodle for real assignment workflows
• Demonstrated secure, automated grading

Future Improvements

• Switch to containers for faster startup and better resource efficiency
• Add a queue system for scalable, parallel execution
• Support for multiple programming languages