ODWai - Object Detection on The Web Application Interface using Deep Learning - Applying for Web Testing

Abstract

In web application automation testing, AI is widely used to analyze the UI and recognize elements as manual testers do. Using AI, we can check whether elements are present at the appropriate places or not. Now, AI will enable testers using the testing tools, created with the help of Computer Vision and Deep Learning, to perform better quality testing. In this study, we propose an approach using the Deep Learning method (using the Faster R-CNN model, OCR) to detect control elements on the web UI. The focus of the approach, called ODWai, is to detect the position of the controls/elements loaded on-screen, extract the information of the controls and generate test data for automation testing. ODWai is a PoC to prove the idea of using Deep Learning to improve test quality and support testers effectively.