ChatGPT, an artificial intelligence powered chat tool that accomplishes essential work with its language understanding and text generation capabilities, has started to benefit education and many other areas with new updates. This study predicted the ability to create a number sense achievement test with ChatGPT version 3.5. It showed that the test created in the study to determine the limits of artificial intelligence was a valid test with a single-factor structure and, simultaneously, a highly reliable test. Because the test designed based on the five-factor reference test was gathered under a single factor, ChatGPT was less functional in questions involving shapes, and the deficiencies detected in the interventions made while writing questions showed that some aspects of ChatGPT needed improvement. Identifying the shortcomings and limits of artificial intelligence is essential for its development. Although this development brings ethical violations, it is evident that avoiding this development is impossible.