Here are some lessons that I’ve learned after a few months of using AI in my day-to-day work as an SDET.
You are human in the loop
Keep in mind that you are the human in the loop who is ultimately in charge of what your AI system generates. The AI system only has a view of whatever you’ve told it since the start of your “chat thread”. It doesn’t know what you know; nor can it deduce what you’re thinking. It will literally “do what you say” without regard to whether it makes sense to anyone in the world.
Vibe coding is great to learn a new language/API/system
My first real vibe coding was done when I needed to create an Elastic Dashboard for the status of end-to-end automation run. When I started I only knew how to create visualizations using Lens; but, it became apparent that the graph I wanted to create isn’t going to be doable in Lens. I’ve never used Microsoft Copilot, so I asked it how I can create the graph. I didn’t tell it not to use Lens, I simply provided a sample data (MS Copilot was approved by my company to send company data) pointed out the relevant record fields; that tests can be ran multiple times a day; and I care only about the last status of the test for the given day. MS Copilot recommended that I need to use Vega for this graph.
Be ready to undo your day’s work
Remember, AI is prone to hallucinations. Back to the Elastic dashboard. There were a few times where literally towards the end of my workday that I had to scroll back up through the chat history and find the code output that was working last and basically “start over”; as in click “new chat” attach the last working code version; the file with some sample data; the initial prompt; and tell it what I want from it.
With the right access, it’s great at helping you understand existing code
My company have approved using GitHub Copilot. That means that I can do something like
Look in <Company>/<Product> repo in the <directory path> how does XYZ gets handled in REST API <path here>
Which makes life easy since I don’t have to crawl through from the controller method, to the call chain, find every model objects, and then figure out what I’m looking for.
For an SDET, this is great writing backend end-to-end tests for functionality that may not be well documented. I avoid having to spend a long time poring through the product source code to find out what I need to find out.
AI isn’t exactly very good at context switching
When talking with someone it’s easy to switch from one context to the next. Take any text conversation you have with someone (SMS, chat apps, emails, etc). If you scroll through your messages you will see different conversations happening through that history. However, you have to keep in mind that AI takes the whole chat history in mind with your next prompt.
I realized this while using agent mode in VS Code. I was “happily” working along on the same chat, writing backend end-to-end automated tests, until it generated code that mingled what the previous REST API and the current REST API I was working on together. It didn’t combine code for both REST APIs; but it definitely wrote code for the new REST API that combined the functionality for both.
One thing I’ve started doing is I would have different chats for the different “contexts”. I would use the history to switch to the context and continue the conversation there. You–as the human in the loop–need to be aware what context you want the agent to work out of. Back to the project using agent mode. Even though the two REST APIs are somewhat interrelated I have to keep them in separate context with GitHub Copilot if I want to use it to make edits to the files.
The other thing I’ve had to do is make edit to the relevant code and then come back to AI after I’ve “directed” the code to where I want it to go. Remember, AI is great; but it still can’t think the way you’d think about what you’re working on.
What’s easy to verify while vibe coding could be very large in PR review
Continuing on with my backend end-to–end test. By this point, I have at this point a custom instructions that tells GitHub Copilot where code goes and the namespaces (plus it’ll generally deduce it based on what you currently have). I prompted it to add a method to call a REST API that I’m adding a new test for; gave it the repo name and the path to the backend service for the REST APIs controller. It intelligently created the DTO objects based on the model used by the controller. After about 5 seconds, and a bunch of new DTO object files, voila I have a method that I can use to add the test case.
There lies the kicker. It was easy for me to examine each new file individually, click “Accept”, and make any edits I want. However, when I git diff --staged it clearly became apparent that the PR is going to show some 500+ lines of new code.
When using AI agent to generate code, be aware that someone may need to review all the generated code. Sure, you may think “well it’s new files you can just scroll through”. Except, you’re scrolling through a PR that contains all the changes from all the files. At least GitHub’s PR UX doesn’t exactly provide a similar UX like you would if each file is a tab in VS Code.
Home+Power 