The DB has around 3,000 futures. When I start the collection it starts very quick until it reaches ~300 completions and then it slows down. To give you an idea, when the collection starts, it says it will take around 4.5 minutes to complete. The second message printed out with estimated runtime remaining is 8 hours 30 minutes.
It gets stuck on some conids and has to retry many times to get the data. For example, with conid 328011116 I got this error:
The last retry also failed and then this error printed out.
However, if I run just
quantrocket history collect us-futures-1d -i 328011116 it takes like 10 seconds and then saves 3 total records.
In my last post I assumed it was due to futures that weren’t expiring soon since those were the ones I saw in the console that were having many retries. I just assumed they had no data, but if I try and collect just the conids which I see are being retried, they all finish fairly quickly and there is data downloaded and saved.
The only way I can reproduce the issue is to try and collect the entire DB and wait until some conids get stuck.
I’ve also noticed that sometimes if I stop IBG1 and start it again, collection speeds up again.