How to Deal with ChatGPT’s Prompt Too Long Error

Are you tired of the Prompt too Long error interrupting your flow with ChatGPT? Let's crack this nut together and learn how to keep your prompts lean, mean, and error-free.

How Long is Too Long to Trigger the Error?

A prompt contains a question or query that sets the context for the AI's response. The token limit of the prompt in GPT-4, for example, is 8,000 tokens, which applies to the prompt as well as the output. These tokens include characters, numbers, words, subwords, etc. One token generally corresponds to about 4 characters of text for common English text.

When a developer gets the error message Prompt too Long, it indicates that the prompt's length is longer than the permitted threshold of the tokens. The token limits of OpenAI’s API may vary according to the model you use. It is important to note that any prompts reaching this limit or exceeding it may be shortened by the AI, leading to inaccurate and partial responses or the Prompt too Long error.

Let’s take a look at an example.

Example: Prompt Too Long Error

The below Python script interacts with the OpenAI API, generating the Prompt too Long error:

import openai

openai.api_key = 'YOUR API KEY'

def generate_text(prompt):
    try:
        response = openai.Completion.create(
            engine='text-davinci-003',
            prompt=prompt,
            max_tokens=1000
        )
        return response.choices[0].text.strip()
    except openai.error.APIError as e:
        print(f"Error: {e}")

# Generate a long prompt
long_prompt = "This is a very long prompt that exceeds the maximum limit of the GPT-3.5 model. It is designed to trigger a 'Prompt too long' error. " * 1000

# Call the function
output = generate_text(long_prompt)

if output is not None:
    print("Generated text:")
    print(output)

Output:

    raise self.handle_error_response(
openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 36001 tokens (35001 in your prompt; 1000 for the completion). Please reduce your prompt; or completion length.

How to Handle the Prompt Too Long Error

There are a few ways to resolve this error:

• The most straightforward way is to shorten your prompt. Be succinct and specific.
• If you’re allowing user input into the prompt and it’s too long, tell the user to shorten their input.
• Even better would be to change to a different model that allows more tokens. The gpt-4-32k model allows up to 32,768 tokens. Note you’ll have to pay more to use this model.
• If you’re using openai.ChatCompletion.create and saving the previous response and adding it to the messages array every time you make a new API request, use a summarization model to condense the earlier responses before inputting it into the GPT model’s message array.
• Use a sliding window approach if you're dealing with a long conversation history. Instead of including all the previous responses, include just the most recent parts that are directly relevant to the current context.

All in all, prompt length restrictions will probably be overcome as AI technology develops, but until then, ChatGPT's full potential can only be realized with brief, concentrated prompts.

Use Rollbar to make handling ChatGPT errors easy

It's a best practice to monitor exceptions that occur when interacting with any external API. For example, the API might be temporarily unavailable, or the expected parameters or response format may have changed and you might need to update your code, and your code should be the thing to tell you about this. Here's how to do it with ChatGPT’s API and Rollbar:

import rollbar 
rollbar.init('your_rollbar_access_token', 'testenv')

def ask_chatgpt(question): 
     response = openai.ChatCompletion.create( 
     model='gpt-3.5-turbo', 
     n=1, 
     messages=[ 
     {"role": "system", "content": "You are a helpful assistant with exciting, interesting things to say."}, 
     {"role": "user", "content": question}, 
     ]) 
     message = response.choices[0]['message'] 
     return message['content'] 
try: 
     print(ask_chatgpt("Hello, how are you?")) 
except Exception as e: 
     # monitor exception using Rollbar 
     rollbar.report_exc_info() 
     print("Error asking ChatGPT", e)

To get your Rollbar access token, sign up for free and follow the instructions for whatever programming language you want to use.

We can't wait to see what you build with ChatGPT. Happy coding!