Is there a limit on data pulled from api? Would like to export all invoices periodically into db

Hi all,

I am using the XML-RPC api and am trying to pull all past/historial invoices into an external database so I can query with a BI tool. I then plan to sync that db with new data each day. I am querying the invoice table. infusionsoft.Invoices

I’ve read that there is a limit of 1000 records each time I make an api call and will need to loop through pages. That said, I called infusionsoft.Invoices and was able to pull over 10k records for a given month. Anyway, just curious on the data limits for this api, and if anyone has advice on ETL’ing ecommerce data into a db for better reporting please advise :slight_smile:

Update: I tried via the rest api - there is a 1k limit on retrieving orders. Though it seems the tables via xml don’t have a limit. That said, the rest api has a bit more data in the ‘list orders’. Is my only solution to loop through sales everyday for the past 10+ years?

Thanks!

Per Frequently Asked Questions - Keap Developer Portal

Legacy API Key
The throttling system is setup like a bank account. You have so many dollars (will be called tokens) in that account you can use. Every Keap Application has a bank of 10,000 tokens. For every API call that is made, 1 token gets removed from that bank. However, for every 500ms that there is not an API call, you get one token back in the bank, up to the max of 10,000 tokens.
When you run out of tokens, your API call gets delayed 500ms until you get a new token. For instance, you are out of tokens and you have 4 API calls come in at the same time. The first one will be delayed 500ms, the next one 1 second then 1.5 seconds and finally 2 seconds.
Once you hit 4 threads being throttled, you will then start to receive an error stating you have been throttled. The only way to become un-throttled is to cease all API calls. It takes approximately 1.5 hours of no API calls to regenerate all 10,000 tokens. You will receive the following error:
Server returned a fault exception: [500] Server encountered exception: com.infusionsoft.throttle.ThrottlingException: Maximum number of threads throttled

Is this a recent thing? We started having timeout issues with one of our API applications in January. It had worked fine previously then just stopped.

The legacy key throttling has been in place unchanged for at least 15 years.

No, we have not made any changes to tokens since Q1 2020.

Normally I’m all for using the xmlrpc for something like this. BUT, in this case, it would be a much smoother process for you if you used the REST webhooks. Whenever an invoice is created, updated, deleted or even when a payment is made to an invoice, you can trigger live responses and update your backend db. Then you wouldn’t have to be concerned about traversing through a list, filtering by date/time and then updating a whole batch.

@John_Borelli I agree, but the webhooks only send the order_id. Right now, I use the webhook approach, then query the rest_api to get the metdata attached; name, total, affiliate, etc. Most other systems include this information. Do you have another approach? Also, dealing with returns - its hard to handle via the webhook approach.