openapi

Utilize AI for API Governance with LintGPT

LintGPT from Optic is the latest way to automate your API Style Guides, combining API Linting with ChatGPT in a way that's not just buzzword nonsense.

Phil Sturgeon

Jan 5, 2024 — 8 min read

API Linting has exploded in popularity, powering an increasing chunk of API Governance programs at organizations large and small. Instead of wasting infinite human time checking API descriptions like OpenAPI, AsyncAPI, gRPC, etc. for mistakes, robots have been deployed to take care of the tedious part themselves, leaving humans to reason about larger more important things.

Spectral was a huge chunk of getting us to this point in the OpenAPI community, but the way you're required to build rules is really complicated, with every single rule needing arcane scribbles of JSONPath-Plus-with-changes and possibly a bunch of regex to get the job done.

    "owasp:api4:2019-rate-limit": {
      message: "All 2XX and 4XX responses should define rate limiting headers.",
      given: "$.paths[*]..responses[?(@property.match(/^(2|4)/))]",
      then: {
        field: "headers",
        function: schema,
        functionOptions: {
          schema: {
            type: "object",
            oneOf: [
              {
                required: ["RateLimit-Limit", "RateLimit-Reset"],
              },

Redocly CLI attempts to make rule creation easier by using "identifiers" and an improved Domain Specific Language (DSL) which is a lot nicer to work with, but fundamentally is a very similar approach.

An interesting third tool has popped up with an entirely new approach: AI-powered Style Guides. I know, I know, there's a lot of nonsense floating around in the AI world right now, and I can barely stop my eyes rolling when somebody starts another "AI is going to change your life" tech talk, but this one is different. Optic has rolled out a new tool currently in beta: LintGPT.

I've reviewed tools made by Optic before, and they've definitely produced the best modern OpenAPI Learning tool, so I figure they're probably not just phoning it in and playing buzzword bingo. Let's find out.

How does LintGPT work?

Instead of writing rulesets in a complicated DSL, you define an array of ChatGPT prompts, which it will interpret and effectively create the rules for you.

ruleset:
  - lintgpt: 
      v3_standards:
        rules: 
        
        - "GET requests that have an ID at the end of their path (ie /teams/{teamID}) should define a 404."

        - 'POST requests should return a 201 or 202 status on success instead of 200.'

        - "All operation URL path components MUST be nouns, not verbs."
        
        - "Properties should have an example"
        
        - "Properties that sounds like they are dates should use format: datetime or format: timestamp"

At first glance you might be thinking "what's the point", and I'll admit I missed the real power here at first. This is not just avoiding writing a DSL, this helps us write far more powerful rules that would be pushing the limits of what Spectral could handle (or entirely beyond them).

For example, asking the AI if we think a particular property is a date field:

- "Properties that sounds like they are dates should use format: datetime or format: timestamp"

Doing that in Spectral would involve creating a rule that checks for created_at or created_on, which would miss created_date or date_created. So then you feel like you've got it all and somebody starts using createdAt, Created-At or DateCreated, and don't forget this would all need to be regex not just a list because there's updated_at and planted_at to catch too.

Trying to preempt every possible name of a date field is boring, and a rule able to do it would be ludicrously slow. Whilst working with AI is not quicker, the way this chunks down the documents and walks through them, checking for any property related rules along the way, means it can be a lot quicker than writing a bunch of really vague Spectral rules.

⚠ [date-like-properties-should-use-datetime-or-timestamp] The property 'planted_at' sounds like a date but does not specify 'format: datetime' or 'format: timestamp'. It is important to specify the format for date-related properties to ensure they are interpreted correctly by the API consumers.
        at api/openapi.yaml:44:1039

It caught 'PlantedOn' too. All just from this one prompt:

ruleset:
  - lintgpt: 
      v3_standards:
        rules: 
        - "Properties that sounds like they are dates should use format: datetime or format: timestamp"

Quality not just Presence

Another interesting ability of this approach is looking beyond simply "if a keyword is there", but attempting to analyze it for quality.

Think about the sorts of descriptions people add to parameters in documentation. They're usually not very good. Spectral has rules saying that parameter descriptions should be present, and you can even set a rule saying it should be a certain length, but that sometimes just gets people adding descriptions that are long and useless.

What if we can ask for them to be useful, define what we mean by that, and specifically request a writing style?

- "Request property descriptions should be written from the client's point of view. ie "set the earliest date you wish to revive records from"

At first I found this to be pretty gung-ho as it wanted me to define descriptions on all properties, but amongst the "Add a description here and here and here" prompts were some classic examples or useless descriptions that it wasn't going to accept.

⚠ [descriptive-property-descriptions] The description 'The category of the error.' is not sufficiently descriptive and appears to just repeat the property name 'errorType'. A more useful description would provide additional context or explain what constitutes an error category.

Yeah, fair. errorType is pretty clearly a type of error! Come on folks we need to do better than this. Where can I find out what types might be defined. What do they mean.

⚠ [descriptive-property-descriptions] The description 'The PSP reference of the payment.' is not sufficiently descriptive and appears to repeat the property name without providing additional useful information.

Again, useless description is useless. Of course a property called pspReference on /payments is a PSP Reference of the payment. What is a PSP? Where does this come from. How can I find out more.

Spectral would not have flagged these as it just wanted the description to be there (that's probably why these specific descriptions were added like this in the first place) but that would just leave me being the person saying "Come on, can we do a bit better". With Optic you can call out poor quality descriptions automatically, and reduce the friction of API Design Reviews by getting that all out of the way earlier on.

No Error Responses over 2XX

One rules I've always wanted to write but struggled to with Spectral was a "No Errors on 200 OK" for apisyouwonthate/style-guide. The idea would be to look for any signs of an error message being defined over 200 (or 2XX), but not just seeking specific properties because there are infinite.

error:
errors:
warning:
warnings:
errorMessage
errorMsg
message

I never figured out how to do this well in Spectral. Checking for a million various properties in the root and hoping it wasn't wrapped in anything just seemed daft, but LintGPT handled this with a single line.

- '200 response bodies should not be used to return errors. Those must use 4xx or 5xx'

I made a naughty OpenAPI document with all different types of errors/warnings/errMsg properties and it caught the lot.

x GET /error-single: added

  x [200-response-no-errors] The 200 response body contains an error field, which is not appropriate for successful responses. Errors should be indicated with 4xx or 5xx status codes.
  at api/openapi-errors.yaml:10:179

x GET /errors-array: added

  x [200-response-no-errors] The 200 response body contains an 'errors' field, which suggests it is being used to return error information. According to best practices, successful responses (2xx) should not include error details. Errors should be indicated with appropriate 4xx or 5xx status codes.
  at api/openapi-errors.yaml:28:681

x GET /warnings-array: added

  x [200-response-no-errors] The 200 response body contains a 'warnings' property which suggests that it is being used to return issues that should be communicated with 4xx or 5xx status codes. Using 200 OK for errors can be misleading and does not follow HTTP status code semantics.
  at api/openapi-errors.yaml:48:1248

x GET /single-property: added

  x [200-response-no-errors] The 200 response body contains an 'errorMsg' property, which suggests that errors are being returned with a 200 OK status. Error messages should be returned with appropriate 4xx or 5xx status codes to indicate client or server errors respectively.
  at api/openapi-errors.yaml:67:1809

It even spotted a { some-wrapper: { errMsg: "" } } which is how you know it's good.

The Future of API Governance

Linting is an integral part of API Governance, reducing how much time needs to be spent on API Design Reviews, but even the evangelists have always tried to explain that "API Linting", "automatic Style Guides", etc could not handle everything.

LintGPT still doesn't handle everything, but it takes a large step forward, moving the boundary on what can and cannot be handled with linting. Maybe Spectral could handle 70% of your review, and LintGPT is up to 80%.

Should you rush out and replace all of your API linting with LintGPT right this minute?

I'd advise against it! For one it's still in beta with some functionality still being worked on, like linting on operation parameters and headers. Can't do that yet.

Having different ruleset formats is a bit of a pain in the backside, and conversations need to happen between the teams working on Redocly CLI, LintGPT, Spectral, and Vacuum. We need to see if there's anything we can do to standardize rulesets amongst us. Thankfully we're all already friends and everyone has told me they're up for a chat, but it's not going to be an easy job from a technical perspective.

Seeing as LintGPT lives in Optic CLI, which already supports Spectral rulesets, anyone already happily using Spectral can give Optic and Spectral a whirl together. That way you can start experimenting with a few extra LintGPT rules on top of your existing ruleset, and not have to worry about rebuilding what already works great.

You may find some rules are easier to build in LintGPT than they are in Spectral, but you may find some are easier to build in Spectral. There are some things LintGPT just cannot do, like suggesting particular endpoints that should exist.

It's early days for LintGPT, that much is clear, but the Optic team have a reputation for making great stuff, releasing early, releasing often, and keeping the momentum going, so I recommend giving it a whirl and seeing how far it takes you on your API Governance adventure.

I hope the robot overlords thank me for doing their bidding when they take over.