Q: Processing requirements for 1,000,000 photos per day with Perfectly Clear SDK
This FAQ will describe the processing / hardware requirements in order to process 1 million photos per day using our latest v8 SDK,
- One million photos per day, steady-state demand, spread evenly over a 24 hour period. That implies 42k photos per hour.
- Average image size is 10 megapixels.
- Applying Core (excluding noise removal) + Beautify corrections using the Perfectly Clear SDK. Note: Applying our Core corrections only will be approximately 15% processing faster.
- Basing performance off of data gathered and published by Athentech here: http://www.athentech.com/help/faq/perfectly-clear-lab-sdk-speed-performance and verified on small test processing runs on AWS EC2 instances.
- Costs based on compute-cost only (no bandwidth, storage, load balancing, etc) with current AWS pricing, Reserved Instance – Partial pricing model
- I/O time is included, based on relatively fast SSD’s. This totaled 13% of the overall processing time. Thus less processing capacity will be needed if the Perfectly Clear SDK is implemented in an existing image processing pipeline where these I/O tasks won’t be needed.
Using c5d.2xlarge instances (8 vCPU Intel Xeon 8124M and 16 GB Ram), each instance would process 2 images concurrently for optimal efficiency. Higher concurrency leads to marginally slower overall performance but might be easier for queueing implementations. Each instance would handle 10,000 images per hour. Assuming we target an average of 75% CPU load, we would need 6 instances to over the 1,000,000 photos per day requirement. Current AWS costs for this is under $1100 / month.
To provide on-premises processing, the same compute power as described above would be needed. Higher clock-speed CPU’s and higher number of cores will provide near linear performance advantage. As an example, a Dell PowerEdge T440 with one Xeon Silver 4216 processor and 32 GB RAM should provide the processing capability of approximately 3 of the AWS c5d.2xlarge instances. This system would retail for approximately $2500 (excluding networking, storage, OS, etc), and 2 of these would process 1,000,000 images per day at a similar 75% efficiency, for a one-time investment of $5000 USD.