For a while now, I’ve been involved in testing a complex media web application developed in Flex, and I got to the point where some help from a tool would have been much appreciated :). I have to admit that this was my first RIA project, so I decided to dig a little bit into it, especially regarding the options one would have in automating some of the functional tests.
I don’t want to argue about the ROI of an automated GUI test suite, or about the fact that maybe it would be much more useful to automate some other types of testing like unit and performance (there are specific tools for each of these).
All the info from this post can be found on the internet, but it takes a lot of time and trouble to gather and analyze it. This post will provide a list of tools I found that support Flex Automation, hoping it will help you if you’re thinking about functional testing automation for Flex based applications.
The Flash in the blackbox
I was able to identify three ways to add automation support for Flex applications:
- using the Flex Automation Framework (+ExternalInterface). Adobe has developed this framework to provide a way for developing ActionScript automation-agents that facilitate the communication between the automation tools and the Flex objects. This is possible using the ExternalInterface API. I couldn’t find a way of using the Automation Framework without the ExternalInterface, although it is not specified as the only option for establishing the communication. Even though the Flex SDK is open-source, the Automation Framework is not: the trial version can be used for scripts with up to 30 actions. One would have to pay 499€ (VAT not included) for the FlexBuilder Pro license in order to create larger scripts.
- by developing your own framework in ActionScript and make it visible through the ExternalInterface – more info about the API can be found here.
- IAccessible interface and MSAA. Not all the objects support the IAccessible interface by default; here you can find more info on how it can be added.
Below you can find a table with the tools that could be useful. I say this because I found some tools that claim to support Flex Automation, but they are just basic macros (mouse click at a certain coordinates on the screen), i.e. iMacros from iOpus.
We plan to take a closer look at some of these tools, especially at the open-source ones but also at the ones that have a decent price. I am pretty sure that companies like IBM, Borland or HP have their own research and analysis on flex automation already available. Plus, we are not a big corporation, so we don’t plan to acquire any of them in the near future :).
Here you can find some useful info on how to setup your environment and get started with automating your applications.
Stay tuned, we’ll post some new info soon.
48 responses to “Flex/Flash Test Automation Tools”
Interesting post. Thank you!
It’s strange that you did not include FlexMonkey, which, imho, has gone a bit further than other open-source solutions you listed.
Here is the google code page:
You are right, I don’t know how I could have missed it, as it seems to be the only one fully based on ActionScript.
I updated the table, and I will look into it more closely.
Thanks for pointing it out!
I’ve found you guys on kiva. Congrats on your initiative and on growing the Ro team.
I have to tell you though, I did not expect something this technical.. I’ll be back as soon as I have more time.
I’m very happy to see a business that didn’t buy into the WordPress cult 🙂
How to test a web based application with an embedded Flex app using Flex monkey?
I’m thinking that FunFX – Watir based – or Flash Selenium/SeleniumFlex – Seleneium based – might be an easier solution as both Watir and Selenium are widely used for testing Web Apps.
If you really want to use FlexMonkey, then you need to make your test methods callable from the web app through ExternalInterface.
Please give me a solution for automation testing of web application (which is developed in java and flex)
Before I can provide some suggestions, I need a few details on what you are trying to automate.
1. What exactly do you want to automate (what do you expect to get at the end of the testing)?
2. Is it not possible to get the same results by running manual tests instead?
3. How much of your web app is developed in java and how much in flex? Is it a java based app with an integrated flex app?
4. Is your application developed with test automation in mind?
5. Wouldn’t it be easier to test the flex app separately and after the integration to do a manually system test?
These are just a few questions that came into my mind.
My point is, in order to have a solution for your problem you need to gather/provide more info related to it.
Alex could you please include the supported compilers as well (FLASH PRO / FLASH BUILDER / FLEX SDK ).
Thanks and regards,
I apologize for the late reply – it was a busy time lately.
You need the Flash Builder Pro license to use the automation framework for unlimited number of actions. If you don’t have the license, you’ll have to limit your scripts to 30 actions/script.
is it possible to do test automation of a flash application build without the use of Flex Builder? The application I want to test is build on base of ac2 (no Flex builder) and don’t us UIComponents. The flash files us MovieClips for the buttons. There is also no MXML file in use. We use QTP but I im not able to find a way to recognize Flash elements.
Here are the first things that came into my mind:
1. Did you include the test automation libraries (automation.swc, automation_agent.swc, …) when you built?
2. If not, are you using a runtime loader that was built with the automation libraries?
For more info on the setup you need to do, please see this post: http://blog.altom.ro/2009/03/setting-up-your-environment-for.html
Can you please help me out. Can you tell me that Does FlexMonkey can automate Flex Application which has Customized flex objects?
If you extended an already instrumented component (such as combo-box or button) it should work without you needing to do anything.
If your component inherits from UIComponent, then you need to instrument that class for testing capabilities – you can find more info about this here: http://livedocs.adobe.com/flex/3/html/help.html?content=functest_components2_02.html – Instrumenting Custom Components.
Either way, my advice is to use unique ids for your components, otherwise FlexMonkey will identify some unique id for each component, but I find them hard to use.
Thanks alot ALEX.. This information is very useful for me.
I’m really happy that I could help.
I’m trying ( struggling frankly ) to use flash-selenium for automating flash applications.
Now my doubt are as below
1. I have access to all .as files
Can I use this swf file locally and call these functions?
I’m trying it but it’s not working, can you please share your email id so that I can share my sample code with you
thanks in advance.
You can find additional alternative, see the following:
Thank you for your post.I am working on flex/flash testing project and I need to have an automation framework. I am trying to have most reliable tool which can help me even if application keeps changing (updates). But here are few problem or say conditions which I have to follow,
1)My application is web-based(GUI) and I do not have control on its build. So can not add any libs.
2)Product is already halfway under development and now I can not change it to make it automation friendly.
if you can help me out, it will be really great. Thx a lot again.
You left out a product. eggPlant from TestPlant (www.testplant.com) works really well on Flash/Flex applications. It uses image recognition technology so it doesn’t even care that it’s a flash or flex app, or whether it’s standalone or browser-based. I’ve been a user of the product for many years, and joined the company in a support role last year. I highly recommend you check it out. It works differently than any other tool out there.
The purpose of the post was to find ways/tools that can actually control a Flex application, for example finding controls and checking their properties, and that aren’t “so influenced” by the redesign of the UI (changing controls positions).
After watching the videos from your website, I can say that eggPlant seems to be a very interesting automation tool, so I’ll definitely give it a try.
Thanks for bringing it to our attention.
Interesting post. Thanks for the information.
Nice summary of tools. You may want to update the info for RIATest as in addition to Windows it works on Mac OS X and supports Safari too.
Thank you for commenting. You’re right, we should update the post with the new products’ capabilities.
My Applciation is developing in below enivronment:
Java, J2EE, Hibernate, Spring, Flex (GUI).
We have QTP with default Add-in’s.
To automate my applciation, what are the add-ins and other support software required.
Please help me on this.
I have a similar problem as one of the other posters (Anonymous). We have java/Flex web app that I did to create an automation framework for. The app has been in developement for a while and I’m quite sure that automation was never though of during development. I was told that Flex Monkey is my best bet, but I have no Idea. Basically, it is a have back end using hibernate, spring, BlazeDS, and Flex for the UI. I need a tool that will do standard UI based automation. That is recognize the UI elements to be able to control them and test the results of their actions. Will FlexMonkey be able to do this? If not, what will?I cannot separate the Flex UI from the rest of the App. It wouldn’t make sense as they are meant to work together.
I’m not sure I fully understand your “requirements”: is the application you’re trying to test a web app created in Java & Flex, and the UI is fully developed in Flex?
If the answer to my question is Yes, then FlexMonkey is a good tool to try if your aim is to automate checks for UI. It might be hard to use if your app has no testability built in, but you should give it try.
Thanks for shedding a light of hope to my problem. I am developing a web application in java with JSP. Within the JSP page i had flex controls embedded. I want to automate the testing of the webpage having flex and non flex items. How can i do it? Is there any tool to do it for both flex and non flex content in a web page.
From one anonymous post i got your feedback. so i have given some basic of my webapp for your questions in that post.
1. I didnt developed application with test automation in mind.
2. It is a Java based application with integrated flex app.
3. I need an automation test inorder to avoid manual testing on every release.(Since for every release we have to test the full app. TO avoid this i will check for automation testing tool)
Please give me a step by step guide to follow.
Thank you for the detailed description of you situation.
I think you can try SeleniumIDE, Flash Selenium, FunFX or other tools that allow you to both control the browser objects and the flex objects.
If you want, I can take a look at the application. This way I will be able to provide a more accurate answer – you can contact me at alex*dot*rotaru*at*altom*dot*ro.
Thanks for your quick reply. I will send you the sample app within a day.
Thanks for sharing. Useful information
Thanks for your kind response. Atlast it worked for with Flex Monkey for automating the flex part inside the web application. We have to use the flex SDK 3.2 or higher. Then it works. First i tested it with SDK 3.0 Hope this information will help others if they get struck.
Thank you for presenting your solution!
If you need help in the future, please let me know.
Thanks for your kind words. sure i will get back if there needs any help for me or any best solutions regarding this.
Thanks a lot
Happy New year to you all.. Thanks for your help for the entire year..
In my earlier post i mentioned Flex integrated application can be automated with Flex Monkey with the Flex SDK 3.2 or greater. But it was not. Our Team is facing a problem that only the flex part can be automated but if we try to automate the whole application which contains the Flex part cannot be automated.
We were looking into it. If you have any solutions to automate the WEB application which contains Flex part in it. (For Ex.PHP application contains Flex Part inside it.)
Automating an application which is a mixture of Flex and Html can be done with a combination of RiaTest and Watir(Ruby) . I had automated alot many Applications and different type of complex screens and it works ok. I used the other mentioned tools also but in my complex cases all other failed the only combination worked fine was of Riatest and watir(ruby).
Though other tools are good also, it basically depends upon the architecture of the application that which tool will stand out the best.
RiaTest required you to build your application with the RiaTest agent / or open it through a proxied flex page.
From inside RiaTest ruby scripts can be called to perform the html job and the response can be collected inside RiaTest. Riatest will be responsible for handling flex objects.
Quality Testing Expert
i have a project on my head and i m really clueless what to do as i m new to testing and to flexMonkey too
My requirments are as followed
-I have to write an action script to automate the testing for a flex application
-The tool i have been told is Flexmonkey and i m supposed to give 2 inputs to flexmonkey
-> my action script ( Containing the test suit already written in excel sheeet) and the URL of the application that i m supposed to test and it is a flex applicaiton only ..
Can anyone tell me possible way to built the desried ?
Any other tool than flexmonkey for the same req. is there ??
Un articol foarte interesant. De asemenea si blogul este frumos si am gasit o groaza de lucruri interesante. Merita sa-l adaug in bookmark. Tine-o tot asa si te voi vizita in fiecare zi. 🙂
My Application is mix of Flex + Html. We are already using QTP automation Tool. But it is not identifying the objects of Flex because it is there in IFrame. Can you guide if is there any solution to over come this problem.
I’m supprised we have not identified Ranorex another tool that claim they support flash/flex based automation. Simmilar to eggplan it unique way of doing image based recogniation.
Very interesting. Is there any updated version of these tests anywhere? I guess several other automation applications have appeared. Very useful information.
Unfortunately there isn’t an updated version for this serie. We had a plan for updating the info, but for different reasons we didn’t have the time to do it.
Hi, I made a application that use camera, microphone, drawing tools, chat etc… How can I automated tests and which automated application use?
Can I know if camera work or mocirphone work etc with these tools?
Hi I have a system that uses flex interface and logic in C#.Net. What software do you recommend for me to do automated testing? Thanks
My complete application is made in flash . it is not possible for us to take the function name from developers. can u please suggest me any open source automation tool for flash .
It’s been awhile since I worked with Flask apps… I doubt that things have changed so much in terms of test automation for flash. If you don’t have access to the application code, you need to go to an image recognition approach; for example you can try Sikuli (http://www.sikuli.org) and see if it work for you.
Let me know if there’s anything I can help you with.