Hi! In an era of rapid AI development, many people wonder: how do you tell how "smart" a model really is, and avoid falling victim to developers who disguise the old GPT-3 model as the newer, pricier GPT-4? In response to questions from our communities, we decided to write an article breaking down how to spot this kind of deception, using a concrete example.
Which model are you?
Asking the model itself which version it is a waste of time. GPT-4 is an improved version of GPT-3.5, so there's a good chance GPT-4 will tell you it's based on GPT-3.5. It's also worth noting that a model can easily mislead you if the developer fine-tuned GPT-3 or simply added logic to adjust its answers.
Here's an example of that kind of answer from the official OpenAI interface with a Plus subscription:
Example of a GPT-4 request on the official OpenAI site with a paid Plus subscription
Link to the forum thread (in English)
Side note: it's worth mentioning that GPT can bend the truth to give you an answer to your question — for example, if you ask the chat whether it uses Google Translate to translate text, it might say yes just to give you a more relevant answer. However, if you look at how the model actually works, you'll see it definitely doesn't rely on third-party translation services when generating a response, except possibly during training.
Getting to the point
We'll test the model using this prompt:
I have a good riddle for you… Can you find the mistake?
1 2 3 4 5 6 7 8 9 10
It looks like a simple attention-check task, yet GPT-3.5 can't solve it, so by sending this prompt we can figure out how smart the model we've been given actually is.
GPT-4 has no trouble spotting the mistake:
GPT-4
But GPT-3.5 didn't find any mistake in this task:
We ran the test above on our own platform, gptunnel.ru, since there we can be sure of the results.
Where's the deception?
Right now the main source of this kind of deception is Telegram bots. They're easy to create, and the GPT-4 hype draws in a lot of people.
Since GPT-4 access is still limited, a developer has to request access from OpenAI and wait. Obviously not everyone gets it, which is why many services sell GPT-3 access while marketing it as GPT-4.
Scammers typically sign up with OpenAI using empty credit cards, connect GPT-3, and while running on credit, build an audience on their channel — then simply recreate the account with a different card and trick OpenAI all over again. They get away with it because OpenAI's billing system only issues invoices after a month of usage.
It's worth noting that GPT-4 costs 30 times more than GPT-3, and that's no exaggeration. So when you see GPT-4 being offered for free, it's most likely a scam. What you're actually getting is the outdated, cheap GPT-3 model, "disguised" by the bot's developers as GPT-4.
Exposing a fake
We looked around the internet and quickly ran into a popular Telegram bot (to avoid ruining anyone's karma, we won't name it).
Funnily enough, in its replies and welcome message the bot claims to be built on GPT-4. But instead of solving the task, it simply says there's no mistake in it. That strongly suggests the bot is actually running on GPT-3, and we were misled.
A Telegram bot service claims to run on GPT-4, but is actually running on GPT-3
In closing, we want to stress that the internet currently has plenty of GPT-4 clones, and you can be deceived in more ways than just being sold a fake product — your card could be charged an unexpected amount, an auto-payment could be set up without your consent, or your card details could be stolen outright. So always vet resources carefully, make sure they operate lawfully, and that the company has been around for a while. Only pay through the official payment forms of well-known banks, so that if something goes wrong, you can report the fraud to your card-issuing bank and get your money back. Never, under any circumstances, transfer money to individuals by phone number or to e-wallets — these kinds of transactions can't be reversed!