< Previous | Contents | Next >

Multithreading

HCP lets multiple threads access a bucket concurrently. Using multiple threads can enhance performance, especially when accessing many small objects across multiple folders.


Here are some guidelines for the effective use of multithreading:


Concurrent threads, both reads and writes, should be directed against different folders. If that’s not possible, multiple threads working against a single folder is still better than a single thread.


To the extent possible, concurrent threads should work against different IP addresses. If that’s not possible, multiple threads working against a single IP address is still better than a single thread.


194 Chapter 7: Usage considerations

Persistent connections


Only one client can write to a given object at one time. Similarly, a multithreaded client cannot have multiple threads writing to the same object at the same time. However, a multithreaded client can write to multiple objects at the same time.


Multiple clients can read the same object concurrently. Similarly, a multithreaded client can use multiple threads to read a single object. However, because the reads can occur out of order, you generally get better performance by using one thread per object.


The HS3 API shares a connection pool with the HTTP and WebDAV protocols. HCP has a limit of 255 concurrent connections from this pool, with another 20 queued.


image

image

Tip: For better performance, consider limiting the number of concurrent read threads per node to 200 and concurrent write threads per node to 50 for small objects. For large objects, consider using fewer threads.

image