Concurrent Data Access
Lock Acquisition
How are locks acquired?
The Server grants read or write locks to clients in one of the following modes:
Traditional locking scheme that allows either many clients to read a given page, or a single client to write to a given page at one time. When a client acquires a lock, it is not released until the transaction commits or aborts. This is the default locking mode.
- Two-phase locking
Allows one writer and many readers at the same time. Readers see a previously committed version of the data. Applications can choose to open databases in this mode at runtime.
- Multiversion Concurrency Control (MVCC)
The ObjectStore client maintains two pieces of lock-related information for each page:
Denoted by ER (encached for read) or EW (encached for write).
- Lock permission (also called cache state)
Specifies the kind of locks that can be assigned to a page.Denoted by LR (locked for read) and LW (locked for write).
- Lock state
Specifies the actual lock assigned at a specific point in time.
The Server only grants and keeps track of lock permission for each page.
Before the client grants the application access to a page, the client checks the lock permission for the page. If the lock permission is greater than or equal to the lock required by the application, it can be granted. The actual lock state is then set accordingly.
Examples for lock acquisitions are
- If a page has a lock permission for read (ER), and the application requests a read lock, the lock can be granted without Server permission (ER --> ERLR).
- If the client tries to access a page that is not in the cache, a lock permission is acquired from the Server while the page is fetched.
- If the page is already in the cache but the lock permission does not satisfy the lock state needed by the application, the appropriate lock permission is requested from the Server.
For more information, refer to the state transition diagram in Locking and Server Communication of ObjectStore Performance.
Which lock acquisitions do occur?
OPE shows you when lock acquisitions occur and how long they take. To see this, examine the following client counters:
Represents the sum of all upgrade events for lock permissions and lock states.
- Pages upgraded
Represents cases in which the application needs a write lock, and the lock permission on the page is adequate (EW-->EWLW or
- Pages upgraded for write (local)
EWLR-->EWLW). The Server is not involved.Represents cases in which the application needs a write lock but the permission state of a page is inadequate (ER-->EWLW or
- Pages upgraded for write (Server)
ERLR-->EWLW). The Server is involved.Represents cases in which a lock permission is acquired from the Server during a fetch operation (NE-->ExLR or NE-->EWLW).
- Pages fetched for read and pages fetched for write
Represents the number of pages that are locked for read (ExLR) or write (EWLW) for a given transaction. The counter is reset to 0 after a transaction commits or aborts.
- Pages locked
When you identify transactions that show a high level of upgrade activity, use the Storage Counters/Ratios diagram to determine the segments and pages that seem to be the busiest.
How long does it take to acquire a lock?
For the selected time interval, the Upgrade section in the ElapsedTime diagram represents the total time that the client spends to upgrade lock permissions and lock states.
Use the PageState diagram to determine how long it takes to acquire a lock on a specific page in a given transaction. The state WLock (wait for lock upgrade) on the diagram shows you the duration. Move the mouse pointer on a WLock state and click on it with the right mouse button. When you select Show details, OPE displays a report with specific information on the selected event.
The time shown is the total time to get a lock; the client does not know which actions the Server performs to grant the lock.
When does a lock acquisition take a long time?
A lock acquisition can take a long time for one of the following reasons:
- High network traffic
- High Server workload or busy machine
- Conflicts due to concurrent data access
There are strategies for analyzing conflicts due to concurrent data access with one trace file. Also, you might not have to analyze all clients participating in concurrent data access.
The counters long page upgrades and long page fetches describe upgrades and fetches that take longer than a threshold of N milliseconds. For information on how to specify or select this threshold, refer to "Specify thresholds".
To find the appropriate value for the threshold, consider these points:
- In a local client/Server application, the average page fetch is faster than in an environment with many remote clients all doing heavy read access that causes massive input/output (I/O) on the Server.
- If many client applications issue many lock requests, the average response time per request increases.
When you know the overall time spent in upgrade events, determine if there are only a few long upgrade events to be examined, or if most events are short.
When database bars are displayed in the PageState diagram, the color of the bar reflects the most important state of all contained pages. Fetch and upgrade are the most important states in the PageState diagram. Many fetches, long fetches, or long upgrades result in a continuous red bar for WFetch or pink bar for WLock. Note that these colors are shown by default and that you can change them.
When you expand a database bar to the segment or page level, you can identify the segment or page that takes a long time to acquire locks for. This segment or page stands out from the others, for example, red for WFetch or pink for WLock. For information on the colors, refer to Legend in the Help menu of the PageState diagram.
How can you find deadlocks, timeouts, and retries?
Start with Transaction summary to isolate deadlocks, timeouts, and retries on a transaction basis. Select the appropriate item in the dialog window and analyze the transaction log for any occurrences.
In the TimeLine diagram, deadlocks and retries are displayed on the miscellaneous events line and in the Client Counters/Ratios diagram area as client counter graphs. When you identify a transaction with a deadlock or timeout, use the Storage Counters/Ratios diagram to isolate these events down to the segment or page level.
In the PageState diagram, deadlocks and timeouts are displayed as event symbols on the individual page bar. The page bar represents the page that causes the event. For example, if the client application has a deadlock during a page fetch, the deadlock event symbol is displayed at the end of the page fetch. It represents an unsuccessful attempt to change the page state (NE-->ExLx).
Handling Page Callbacks
What happens when the Server calls back pages?
Assume that the Server has granted write permission (EW) for a page to a client. Some time later, a second client asks the Server for a read lock on the same page (ERLR).
In a callback operation, the Server asks the first client if it can give up the write permission. This is done through an asynchronous message to the Cache Manager of that client. If the client indicates that it does not use the lock, the Server grants the lock permission to the second client and takes it away from the first client. If the client indicates use of the lock, the second client is blocked and must wait for the lock.
There are two types of callback:
The Server tries to call back a page, and the client holding the lock gives the page back. For example, if the page has a current state of EWLR (lock permission for write, lock state for read), and the Server callback is for read, there is no conflict. The callback proceeds immediately. The lock permission of the client that has the first lock is downgraded from EWLR to ERLR.
- Immediate
The Server tries to call back a page but the client holding the lock uses the page in a conflicting manner. Consequently, the client requesting the lock is blocked until the first client completes or aborts its current transaction. For example, the locking client's EWLW state is downgraded to NE at the end of the transaction because the second client immediately locks the page for write. For a detailed description of page states refer to "Page states and page events".
- Pending
OPE reports callback activity as counters in the Client and Storage Counters/Ratios diagrams, and specific events in the PageState diagram. Move the mouse pointer, for example, on a callback event and click on it with the right mouse button. When you select Show details, OPE shows you if the callback is for read or for write.
A high rate for callback pending indicates possible conflicts due to concurrent data access. A high number of callbacks also causes a lot of client/Server communication. Even if your client does not report many callbacks, you should verify the appropriate callback statistics for the Server by using the OPE Server Monitor or ossvrstat.
Note: Immediate callbacks are not reported by OPE.
Finding Hot Spots for Concurrent Data Access
Start your search for these hot spots in the Transaction summary. Look at transactions that
- Have deadlocks or timeouts
- Spend a lot of time in fetch and upgrade
- Have long page fetches and upgrades
- Show callback activity
When you find transactions like these, use the Storage Counters/Ratios diagram and the PageState diagram to isolate the areas of the database that show the hot spots.
Note that each transaction and application might access the database differently and therefore might not have the same hot spots.
However, in most applications there are few central data structures for data access, such as central application metadata, extents, or indexes, that are used for most kinds of data access. Consequently, these data structures are the key to concurrent data access for most transactions.
When you identify a hot spot in your application, verify that it is not a single bottleneck. For example, if two clients access the same data coincidentally, you identify a long wait on a certain page. To avoid misinterpretations, look at the long-term contention and not at individual deadlocks.
A page that shows frequent upgrade and callback activity is a page with hot spots: While the application accesses it frequently, the Server calls it back to grant lock requests from other clients.
You can compare concurrent trace files by opening them at the same time with the Analyzer. Note that it is probably not possible to synchronize several trace files perfectly by using only the timestamps that are recorded by OPE.
Getting More Information
In which ways can you change the application to get more information?
To provide more information regarding concurrent data access when you use the Analyzer, change your application in one of the following ways:
The os_ope_generic_event API (ostore/client/ope.hh) supports the creation of events on an application level. These events are displayed in the PageState diagram.
- Use generic events to mark segments and pages.
If you supply the persistent location (void* address) of an application object to the generic event, the event is displayed on the page that contains the object. This helps you mark the time and place of important events that occur in the physical database.
Use the ObjectStore dbutil API to request statistical information from one or more Objectstore Servers at run time.
- Provide Server statistics information.
For example, if your application shows a high callback rate in the Analyzer, identify the overall callback activity in the network.
Use dbutil::svrstat() (n_callback info; compare with ostore/dbutil.hh) to retrieve the number of callback messages issued by the Server and the amount of data involved.
Use the os_unsigned_int32 overloading of the raise() method in the generic event class to encapsulate this information for the Analyzer.
The integer value is then displayed as a client counter in the TimeLine diagram. You see the number of callbacks received by your application compared to the overall number of callbacks sent by the Server.
OPE displays the name of each transaction if it is available. The overloading function of the dynamic transaction API allows you to specify a name.
- Use meaningful transaction names.
OPE assigns default names to static transactions that use the OS_BEGIN_TXN and OS_END_TXN macros. The names are based on file name, line number, and the tag assigned in the OS_BEGIN_TXN macro.
Note that dynamic transactions that use os_transaction::begin() do not have a name by default. Use the char* overloading to specify a name.
To avoid access to the info segment header, the Retriever does not log the comment of a segment. Therefore, OPE displays only the segment numbers. To get information on a particular segment, insert application-specific information by defining generic events. For details refer to "Defining a generic event".
- Use segment comments.
ossize -n <number> reports the segment comment. To get access to the comment, use os_segment::get/set_comment(). You can, for example, pass the segment comment into the message string of a generic event. This helps you correlate segment numbers and their use.