Sauce Labs provides a cloud-based, automated testing platform for web and mobile software development. For the past three years they have surveyed professionals in the field to track trends in testing practices among software development teams. They released their 2017 survey today and the results are of interest to software developers and the general public alike.
The Sauce Labs report compiles survey data from 732 software professionals. Fifty-nine percent of the respondents are either software developers, test engineers or quality assurance personnel. The respondents are very close to equally divided among companies with less than 100 employees, more than 1,000 employees and between 100 and 1,000 employees. Sixty-two percent of respondents are located in the Americas, 25% in Europe and the Middle East, and 13% in Asia and the Pacific. The testing was carried out by Dimensional Research.
Credit: Sauce Labs
Why do websites fail in some browsers?
There’s a simple and straightforward answer to this question – the browser wasn’t tested before the current build of the website went live. The graph above shows the percentage of companies that test their web applications for a number of popular browsers. With the exceptions of Google’s Chrome and Microsoft’s Edge, all the browsers saw a decline in testing from 2016 to 2017.
The one percentage point changes for Chrome and Opera are too small to be meaningful and the 16-point loss for Internet Explorer isn’t surprising because it’s now a legacy browser. However, the 10-point loss for Firefox and the 14-point loss for Safari are unexpected. The survey did not ask why some companies have decided to stop testing for everything except Chrome and Edge.
A decline in testing for individual browsers isn’t the only reason why websites may fail. The Sauce Labs survey found that the percentage of companies that only test the current version of each browser rose from 37% in 2016 to 49% in 2017. One of the reasons behind this increase may be that companies assume most browsers are automatically updated. Unfortunately, this practice can lead to problems for consumers that turn automatic updating off and companies that do not allow automatic updates from outside their firewall.
Sauce Labs characterizes the decline in cross-browser testing as “a potential area of alarm”. It’s also surprising given how fast and easy it is to conduct tests of multiple browsers and browser versions with automated testing platforms like the one provided by Sauce Labs.
Credit: Sauce Labs
How long does it take to fix a bug found during testing and how often are new builds deployed?
The Sauce Lab survey found almost no change year-over-year in the time it takes to fix bugs. Users, developers and companies with websites would like to see improvements here, but the good news is that bugs are generally fixed fairly quickly. Almost a quarter of respondents said bugs are fixed immediately and 74% said it takes at most a few working days to fix a bug.
Software development teams would like to deploy new builds more quickly than builds are typically deployed. When asked how often their team would like to deploy new builds, 28% answered hourly. This is a striking increase from the 18% who gave the same answer in 2016. Overall, 63% said they would like to deploy new builds either hourly or daily.
Credit: Sauce Labs
Builds aren’t deployed as quickly as the developers would like, however. As can be seen in the graph above, only 48% of respondents are able to deploy hourly or daily. On the plus side, the percentage of teams that deploy hourly rose from 10% in 2016 to 14% this year.
Which practices are common in the web development community?
The Sauce Labs report provides detailed information about testing practices and institutional environments in the web development community. Some highlights are presented here, interested readers are encouraged to check out the full report for more detailed information.
Agile development methods are ubiquitous among web developers with 89% reporting at least some adoption of agile procedures in their organization. This represents an increase of seven percentage points since 2015. Sixty-one percent reported that most or all development teams are agile.
The DevOps approach that emphasizes communication and collaboration among software developers and other IT professionals is also taking hold among web and mobile developers. Almost half (48%) of respondents said at least a few teams in their organization have adopted DevOps. An additional 25% reported they have just begun to adopt DevOps practices.
Continuous integration (CI) refers to the practice of frequently merging working copies of code in a shared pool that is then tested with an automated build in order to quickly detect and correct software conflicts throughout the development process. Seventy-eight percent of respondents said their organization practices CI, an increase of eight percentage points since 2015.
At least some type of automated testing was used by 87% of respondents. The percentage of respondents that said their organizations put most or all their effort into test automation increased from 26% in 2016 to 32% this year. On the other hand, 42% said their testing was mostly or entirely manual.
Credit: Sauce Labs/YouTube
The good news and the bad news
Sauce Labs continues to provide useful insight into trends in web and mobile software development with their 2017 report. The good news is that deployment times are decreasing and more organizations are making use of automated testing. The bad news is that a large proportion of organizations are not taking full advantage of the benefits of automated testing and fewer web browsers and fewer versions of web browsers are being tested before a build is deployed on the web.