If you are unsure how many processors you have use: os.cpu_count(). Minimum value of 32 and the number of processors plus 4.Number of processors on the machine, multiplied by 5.
When selecting the number of threads you can follow one of the two methodologies: The optimal number of threads for requests is between 10 and 20 depending on the current response time of the Pushshift server. Increasing the number of threads you use allows you to make more concurrent requests to Pushshift, however, the returns are diminishing as requests are constrained by the rate-limit. This is optional and defaults to 10, however, you should provide a value as this may not be appropriate for your machine. The number of threads to use during multithreading is set with the num_workers parameter. View the optional parameters for PushshiftAPI here. General Usage from pmaw import PushshiftAPI api = PushshiftAPI () To install it via pip, run: $ pip install pmaw PMAW currently supports Python 3.5 or later. Implementing intelligent rate limiting can ensure that we minimize the number of rejected requests, and the time it takes to complete. Since API requests are I/O-bound they can benefit from being run asynchronously using multiple threads. The time it takes for your code to complete pulling all this data is limited by both your network latency and the response time of the Pushshift server, which can vary throughout the day.Ĭurrent API libraries such as PRAW and PSAW currently run requests sequentially, which can cause thousands of API calls to take many hours to complete. When building large datasets from Reddit submission and comment data it can require thousands of API calls to the Pushshift API.
0 kommentar(er)
