JavaScript “Failed to load resource” error encountered while running the Selenium Python test suite

Currently working on a project with requires to test the company flagship website and ran into a problem I thought that is worth sharing. Especially since I have found limited resources addressing the issue on the web. So I thought this article would help others to who run into the same issue.

Disclaimers first as I have to mention that the test framework had already been created by the time I got the role and more importantly I am new to python so having said all that, please judge me based on the available information.

Simply put the errors I have encountered are Javascript error: Failed to load resource: net::ERR_CONNECTION_CLOSED and Failed to load resource: net::ERR_HTTP2_PROTOCOL_ERROR.

My first instinct was to adjust the framework to take a screenshot of the failures. In this regard I have enabled the below code:

.save_screenshot('./screenshot/' + result + testname + '.png')

By doing that I have realized what the problem is not with the test data or the platform being tested because there is nothing particularly wrong with the screenshots. Only one of the xpaths in the test data got transferred to the next line. So I thought this was a typical AJAX issue that you might face if you don’t use the right waits.

What is Ajax?

According to Guru99.com, Ajax allows the Web page to retrieve small amounts of data from the server without reloading the entire page. It is basically a ‘technique’ to create fast and dynamic webpages. I thought I found the solution to my problem, but it seems like apart from a few ‘time.sleep()’ functions, our framework has utilized the waits pretty good. So, I have played with the wait times and replaced time.sleep()s with explicit waits then with implicit waits but nothing has changed.

So, what?

At this point, I already knew that my answer was not related to the Ajax but still believed that it is a synchronization issue. So my next target is to become the threadpoolexecutor which is an Executor subclass that uses a pool of threads to execute calls asynchronously.(1) There is also whole another discussion of whether to use thread pooling or process pooling or something else.(2)

My solution

After spending a big chunk of my time on trying to understand what the issue it, I think I have found the solution.

First, I changed the max_workers from 10 to 5 in our threadpoolexecutor function.

with ThreadPoolExecutor(max_workers=5) as executor:

Second, since the some error messages were related to the image loads, I have added the prefs to our already written ChromeOptions() function and set 2 to most to disable the loads.

options = webdriver.ChromeOptions()

prefs = {"profile.managed_default_content_settings.images": 2,
         "profile.default_content_setting_values.notifications": 2,
         "profile.managed_default_content_settings.stylesheets": 2,
         "profile.managed_default_content_settings.cookies": 2,
         "profile.managed_default_content_settings.javascript": 1,
         "profile.managed_default_content_settings.plugins": 1,
         "profile.managed_default_content_settings.popups": 2,
         "profile.managed_default_content_settings.geolocation": 2,
         "profile.managed_default_content_settings.media_stream": 2}

options.add_experimental_option("prefs", prefs)

Moreover, just to be on the safe side, I have also added below functions to disable to extensions and set images to false. Like this:

options.add_argument("--disable-extensions")
options.add_argument('--blink-settings=imagesEnabled=false')

Finally

I was able to run the framework without any failures caused by the synchronization.

P.S.: Please feel free to share with me if you have a different or more effective solution for the problem.

References:

1. Threadpoolexecutor : https://docs.python.org/3/library/concurrent.futures.html
2. Thread pooling vs process pooling : https://stackoverflow.com/questions/2846653/how-can-i-use-threading-in-python