In recent weeks, at Keepler we have been testing AWS CodeWhisperer in order to learn more about this Gen AI tool for source code and to have an informed opinion to recommend its use.
We have written about CodeWhisperer before in a more generalistic way, but in this post we want to share our conclusions and our observations based on the tests we have been carrying out.
First, let’s set the stage for our testing:
- IDEs: We have been using: VSCode, IntelliJ and PyCharm (both in their community version)
- Language: Python
- Clouds: AWS, Azure, GCP
We have been interacting with AWS CodeWhisperer in two different ways:
- By using comments: You can write comments at any time to let CodeWhisperer knows your request
- Live Coding: CodeWhisperer will read the code you are writing and it will recommend you code completions and code suggestions
At any point you can manually trigger CodeWhisperer with Alt + C for Windows/Linux, or Option + C for macOS; which is very convenient.
Additional suggestions can be viewed by selecting the Right arrow key. To see previous suggestions, select the Left arrow key. To reject a recommendation, select ESC or the backspace/delete key.
We will show all the details below, but if you don’t want to read the whole information and want to know our findings, here you have the TL;DR version:
The use cases we love the most are:
- Generate functions code from comments: Just by typing and describing your function in natural language, CodeWhisper will generate the function, its parameters, the documentation and the body function for you (although it would be recommended to delete the comment after the code had been generated)
- Generate boilerplate and standard code: writing source code that you can just simply copy and paste is boring and unmotivating, CodeWhisperer can generate those standard and well-known algorithms or SDK boilerplate code saving time and brain resources. For example: looking for a key in a dictionary, or looking for an element in a list, cleaning some values from a dataset, field validations, …
- Unit tests: It can create unit tests to develop testing scenarios for your code. Although it is true that the generated Test Cases are quite simple and straightforward, it will help you provide your code with a basic set of test coverage faster
- Sample Data: CodeWhisperer can generate sample data following the structure you need in your code (if the structure is quite complex we have found that CodeWhisperer won’t be of so much help here)
- Working with SDKs or third party libraries: Avoid having to Google for documentation and use cases is our most loved use case for CodeWhisperer. When working with Azure, Google or AWS SDK it is always necessary to look into the documentation to know exactly the signature of any function and the type of parameters it accepts. CodeWhisperer can automatically generate the code-block including the function and its parameters you need. for example, asking it to generate a function to “list all of the Cloud Functions in a GCP project” or to “Upload an object to a Bucket in S3”, is quite simple and accurate and, even more, CodeWhisperer will initialise the Client for you if required
- Exceptions: CodeWhisperer can understand the context of the code you are writing and can suggest the try/catch exception clauses that suits your code best, making much more easy taking care of the different exceptions than can be thrown and act on each one
- Regular Expressions: We all know the sentence: “If you want to solve a problem with a Regular Expression, then you have two problems” right? CodeWhisperer will help us to define the right regular expression or simplify a regular expression we had previously defined
- Generate Documentation: It can automatically generate documentation blocks for us to be included in our functions and our code. CodeWhisperer will read the context of the function and will make a suggestion of what it thinks the function does (gets it right most of the time)
AWS has defined its 10 ways to build applications faster with AWS Code Whisperer, which are pretty much aligned with our findings.
Areas for improvement:
- Saving Time: Here we are less optimistics than other evaluations we have read and we think that by using CodeWhisperer a senior developer can save around 20% percent of her/his time. A junior developer will save a little less time because some CodeWhisperer’s recommendations need to be reviewed, so you can find yourself fixing the written code more than writing it yourself, which consumes more time the less experienced you are
- Learning curve: Communicating with CodeWhisperer, even using natural language, is not always straightforward; you need to learn how to detail requirements in order for CodeWhisperer to make the suggestion you were looking for (which is commonly referred to as “Prompt Engineering”). This learning curve may be more accentuated for junior profiles
- Review and Adapt: When CodeWhisperer generates a block of code, you will have to review it and (sometimes) adapt it to the specific use case. As we said above, this review process takes time, so sometimes it is a matter of balance the time you save in writing code versus the time you spent in reviewing the automatically generated code
- Addition but not replacement: When CodeWhisperer makes any suggestion it will not replace your code but will add a new line (or few new lines) below your caret. This will lead to manually delete duplicate or obsolete lines of code
- Style Guide Suggestions: Sometimes the suggestions made by CodeWhisperer does not follow the PEP8 Style Guide (for Python) and you have to spend time in fixing it
- Random Suggestions: We have found that in scenarios where there is little context, the suggestions are completely random. For example in an empty file it usually suggests adding a comment with a random date
- Code Invention: When using third party libraries or SDK, even on custom classes, we have found that quite often CodeWhisperer suggests code that does not exist anywhere and that does not belong to any of the third party libraries (which obviously leads to not compiling code)
- Intrusive: Its use can be a bit intrusive at the beginning because CodeWhisperer is constantly trying to make suggestions as you write, so you will have to adapt your eyes to the prompt with suggestions and to the background code that appears while typing. After a while you get used to have those suggestions in your screen and ignore them when you don’t need them or use them when you do
- Code Completion: Sometimes the code completions is made line by line instead of generating the whole block of code which may be inconvenient and more error prone
- Imports: At least in Python, when you ask for any suggestion to execute a library, CodeWhisperer will not add the import statements
- Test Suite Case: CodeWhisperer can help in generating simple and basic TestCases, but if you want to automatically generate a complete Test SuiteCase that understand your code and cover multiple scenarios to grant a minimum of coverage, CodeWhisperer is not the tool to do it
We have been testing AWS CodeWhisperer using different use cases and different scenarios in the source code.
We monitored CloudWhisperer to know which type of connections it keeps open and which protocols is using:
It keeps a TCP connection open against an EC2 instance or a fleet of instances:
This DNS name changes during the session and the connection is kept open only if the “Auto-Suggestions” option is active, getting closed when the IDE becomes idle.
We were writing some code to detect when it is safe to delete a SageMaker Endpoint and we had something like this:
try: if sagemaker.describe_endpoint(EndpointName=endpoint_name)['EndpointStatus'] != 'InService': while sagemaker.describe_endpoint(EndpointName=endpoint_name)['EndpointStatus'] != 'InService': time.sleep(30) sagemaker.describe_endpoint(EndpointName=endpoint_name)['EndpointStatus']
CodeWhisperer suggested the next addition to replace the last line (so we have to manually delete it):
if sagemaker.describe_endpoint(EndpointName=endpoint_name)['EndpointStatus'] == 'InService': print("Endpoint " + endpoint_name + " is going to be deleted") sagemaker.delete_endpoint(EndpointName=endpoint_name)
Which it is pretty accurate with the context of the function and, at the same time, it suggested that we add the nex “except” clause:
except sagemaker.exceptions.ResourceNotFound: print("Endpoint " + endpoint_name + " not found.)
This specific use case worked really well and the suggestions were really helpful.
We can see some other recommendations, for example to create a Bucket in S3 with a custom name, we can start by typing the function signature and it will suggest some options from less to more complete:
It even suggested us to initialise the Boto3 client with some of the profile that can be found on our Credentials configuration.
Some other suggestions are not accurate at all and we found that the source code was invented. For example: We defined a custom Python Class called “Productive” to define a wrapper to interact with the external Productive API and we only defined two different methods: get_all(url) and get_data(url)
For those well-known or simple algorithms we have been talking about previously, the suggestions are quite accurate as well. For example “Function that takes a list of numbers and returns the sum of all the numbers.”
Moving to another cloud provider, we detected the behaviour is quite similar, we found some suggestions pretty accurate, but some others required manual revision and fixing. As an example of this, we were using Cloud Run from GCP to work with some containers and applications:
We asked AWS Code Whisperer to generate the top function (update_revision) and it used a method in the client that does not belong to the API (client.update_revision)
Test Use Cases Generation
We have found that AWS Code Whisperer can generate simple use cases methods to add basic coverage to our functions. If we need to automatically generate a complete Suite Case for Testing with different scenarios and edge case testing, Code Whisperer is not an improvement over doing it manually.
For example, to test a simple function (random_date) the use cases generated by Code Whisperer were quite simple, but enough to cover basic scenarios:
We can confidently recommend the use and adoption of AWS Code Whisperer as a Gen AI Source Code tool. It will definitely help you create code in a faster and more efficient way, but bearing in mind that there is a learning curve before starting to work efficiently with this tool, so you will need to take time to adapt to it.