This post will discuss how ChatGPT-4 smashed the Base SAS Certification with a score of 93.4% in only half the time allotted for the exam.
GPT-4 is a huge improvement over the prior version GPT-3.5. OpenAI Team claims that as compared to the previous GPT-3.5 models, GPT-4 makes much fewer mistakes in generating factual statements. On factuality evaluations that are designed to challenge the models, GPT-4 performs 19 percentage points better than GPT-3.5. Microsoft's new Bing Chat also solves the limitation of GPT-4 training data only upto September 2021. In other words, Bing Chat can answer to the questions related to recent events.
I wanted to see if ChatGPT4 could pass the Base SAS Certification exam. The exam tests the candidates' skills in data manipulation and analysis using SAS software. It consists of 60-65 multiple-choice and short-answer questions that have to be answered in two hours.
Did I actually spend 180 USD for this testing? No. I enrolled in the Free SAS Certification Practice Exam that is offered by SAS. This practice exam is designed to simulate the actual SAS certification exam in terms of the format, content and difficulty level of the questions.
I used Microsoft's new Bing for accessing GPT-4. During the test, I simply copied the questions from SAS tab and pasted it to Bing Chat window. I had 61 questions in the exam so I had to do this 61 times. It was a very tedious process. While taking the test, I had to convert images into text several times. It is because datasets in some questions are in image format so I had to explain GPT-4 about the dataset. Depending on the number of observations and requirement in the question, I decide whether I need to convert image into text or I just write/explain dataset manually to GPT-4. It is important to note that Bing Chat does not handle images.
Programming Associate is the new name for what used to be called Base SAS Certification for many years. SAS has split this certification into two levels: Programming Associate and Base Programming Specialist. The former is based on multiple choice and short-answer questions, while the latter is a written test. However, we will still use the old term Base SAS Certification since it has been around for over two decades and it always followed the format of multiple choice and short-answer questions.
With the help of ChatGPT-4, I answered 57 questions correctly out of 61 which makes it 93.4%, well above the passing mark of 68%. I completed the exam in just one hour and four minutes. I spent roughly 5 to 8 minutes converting images to text. It's because images were present in several questions.
There is a difference of one question between my claim of 57 correct answers and the image above that shows 56 out of 61 correct answers. This is because I gave GPT-4 a wrong instruction for one question. I noticed when I was reviewing the wrong answers after submitting the test. I used a third-party tool to convert an image into text, but it returnedobs1instead ofobsas a column name in the image below. Furthermore, I could not catch this error earlier during the exam because it was very tedious to copy-paste and convert images into text 61 times. It was my mistake, not GPT-4’s. Later GPT-4 returned correct answer for this question. I have to give it an extra mark for its ability.
I did not test GPT-3.5 in the same way, but when it came out, I tried a few questions related to SAS and statistics and it did not do well. Check out this link for more details - ChatGPT-4 Is a Smart Analyst, Unlike GPT-3.5.
If you are interested in testing other certifications like Statistical Business Analyst, Advanced SAS etc., please volunteer. I will mention you in this article with a link to your Linkedin profile.
Deepanshu founded ListenData with a simple objective - Make analytics easy to understand and follow. He has over 10 years of experience in data science. During his tenure, he worked with global clients in various domains like Banking, Insurance, Private Equity, Telecom and HR.
Share Share Tweet