The tool now supports the analysis of core dumps from Solaris 10 and above. This requires that they are analyzed on an x86 or x64 system for 32 bit core dumps and an x64 system for 64 bit core dumps.
Currently, the stack display does not include arguments. The disassembler is also under construction.
Sun Cluster Support
The clust command has been added to retrieve cluster specific information. The new command syntax is as follows:
clust [flag]
Where:
abuf reports available dbg options which can be enabled
addr reports important cluster addresses
ebuf reports enabled dbg buf which can be displayed
dbuf [-a]
delta
did finds and reports our did devices
hb reports heartbeat data
invo [-t|-s|-i] reports invocation flags add thread reporting
members reports current cluster members
path reports current paths and associated data
pm reports path manager data
rdbg <*dbg_buf> dumps specified dbg_buf (no size calculation)
Zone Support
A new zone command has been added to display zone information in the core. Additionally, the proc and vfstab commands now have options to include the zone name for the process/vfstab entry, or to select processes or vfstab entries by zone. The new project, rctl, and task commands also have per-project options.
New Structure Command Flags
New flags were added to the sdump, slist, sarray, shash, and skma commands to enable display of more details of structures elements. Those include:
option function
-a display the address of each structure element
-o display the offset of each structure element
-s display the size of each structure element
-m display the CTF module of each structure element
-t display the type of each structure element
-i display the index of each structure array element
-g display any gaps between structure elements
-e
These options are only available on the CTF versions of structures, and not the stabs-based definitions.
Additionally, new options were added to the stype command:
option function
-o display the origin of the type. This could be from the CTF data in the core, or the stabs file name.
-d display all of the CTF versions of the type, including the names of the modules which include that version
-m display the module for each element of the type shown
-g display gaps between structure elements
stype xck will compare all available CTF versions of a type and display a list of the differing versions of the type in the core with a list of the modules which use that version. The parenthesized numbers show the number of elements, followed by the size of the type.
Additionally, types which are "dissonant" are shown. If two version of a type are dissonant, then they are not only different, but they have differing offsets or sizes for the same named members of that type.
Finally, if there is a region of memory where a pointer to another known type was found, the "stype field" subcommand will help search for types which exist in the CTF data which has that type at a known offset.
Deprecated Commands
Some commands which have been documented as deprecated for a long time have now been eliminated from the tool. Specifically:
deprecated command replacement
summary thread summary
findstk stack find
fminfo toolinfo
The findstk command still exists as an alias, although it cannot be searched for with the help command.
Additionally, the bigdump command is now deprecated.
deprecated command replacement
bigdump bufc buf list
bigdump inode inode list
bigdump idleq inode idleq
bigdump ireuse inode ireuse
bigdump dnlc dnlc list
bigdump dwbuf buf dw
bigdump tmpfs tmpfs
User-defined Symbols
Support has been added for user-defined symbols. These will supercede any in-core symbols that would normally be shown. This allows specification of names that might be useful to the user for regions of memory, including treating it an an array. See the symbols, section for further details.
New Commands
autofs
This new command displays the autofs configuration in the core by traversing the fnnode tree.
cv
This new command lists the threads waiting for a condition variable.
clust
The clust command has been added to retrieve cluster specific information.
nfs
This new command lists the NFS shares for the system.
pool
This new command displays the pools present in the system. It can also be used to get information about a specific pool. Pools may be selected by name, pool id, or pool structure address.
project
This command prints a table of projects on the system. It can also be used to list the tasks in a project. A project may be selected by name, project id, or kproject structure address, or those in a zone specified by name, zone id, or zone structure address.
rctl
This command is used to display resource controls. They can be displayed by rctl structure address, rctl_set structure address, or for a process specified by PID or proc structure address.
sleepq
This command simply lists the threads waiting in a specified sleepq specified by the provided sleepq_head structure address.
task
This command prints a table of tasks on the system. It can also be used to display processes in the task. A task may be specified by task id, task structure address, project id, or kproject structure address.
xmmu
This command is similar to the sfmmu command for displaying kernel virtual to physical address translation structures, but is for the x86/x64 architecture. It can be used to display PTEs, hments, htables, ptables, hat structures, or searching for translations for a specified virtual address.
zfs
The zfs command has been added to retrieve zfs specific information.
zone
This command lists the zones, including their id, name, root path, and status. If the -l flag is included, more data about the zone is displayed. If the -z
Interface Changes
color
The color command's options for fatal, warning, info, info2, error, and alert now include the ability to set background colors and attributes.
Two new color classifications were added: redzone and bad. The former is used to indicate the portion of data which is supposed to be the redzone. The latter indicates "bad" data such as data which should be 0xdeadbeef but is not in "kma buf" output.
The basic command format is:
color
The
Attributes supported are bold, faint, italic, underline, blink, reverse, and strikethrough. These may be combined in a comma-separated list.
Some examples are:
color fatal red none bold
color warning none none underline,bold
color bad white red
color alert green
color info white green
Note that attribute support is dependent on the terminal emulator in use. For example, some of the attributes, faint, italic and strikethrough in particular, may not be supported by the terminal emulator you are using. Some may display these attributes differently, such as using a brighter color instead of making the font bold. Other possibilities is that the terminal emulator may not support certain combinations of attributes.
cpu -c [
In Solaris Nevada, the chip structure and notation was eliminated. A new facility called the processor group (pgroup) was implemented in its place. The -c flag's meaning was changed to mean a pgroup specifier instead of a chip specifier on that version of Solaris.
cpu -h [
This new flag allows display of CPUs by their hardware sharing relationship. This matches against the pghw_hw type field, and dumps the CPUs by this grouping. Valid
If the
PGR_PHYSICAL, class:cmt(id:1), PGHW_IPIPE
0/0: 0 1 2 3
3/68: 4 5 6 7
4/70: 8 9 10 11
5/72: 12 13 14 15
6/74: 16 17 18 19
7/76: 20 21 22 23
8/78: 24 25 26 27
9/80: 28 29 30 31
PGR_PHYSICAL, class:cmt(id:1), PGHW_FPU
1/0: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
PGR_PHYSICAL, class:cmt(id:1), PGHW_MPIPE
2/0: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
The X/Y represents the pgroup ID and the instance number.
dev node [
This new option for dev displays the tree of in_node structures rooted at e_ddi_inst_state.ins_root, which is a fully-populated parallel tree to the dev_info tree.
findfiles
Two new options have been added to this command to allow searching for files based on vnode, and based on vnode's v_vfsp.
The new -n option for findfiles allows searching for processes using files whose vnode matches the supplied
The new -f option for findfiles allows searching for processes using files whose vnode's v_vfsp matches the supplied
findval
The flags for findval have been altered to make the size specifiers into single-character flags.
old (bits) new (bytes)
-8 -1
-16 -2
-32 -4
-64 -8
A new flag, -L has also been added. If a match is found in a kmem cache, that cache's slab freelist and magazines are scanned to determine whether the matching buffer is free or allocated.
flip
The -va option has been replaced by -V. Correspondingly, the -pa option has been replaced by -P.
A new option was added to allow specification of the size of the object to be read for the (implicit) -a case where a value is read from the core. By default, sizeof(ulong) is read and displayed. The -S
ifconf
The ifconf command's output now includes IPMP group names.
kma stat
In the kma stat output, an asterisk at the end of the line for a cache is now used to indicate that setting KMF_AUDIT in kmem_flags would not make it into this cache's cache_flags due to the cache_flags and cache_cflags set. This means that setting KMF_AUDIT in kmem_flags wouldn't allow kma users to be run against this cache.
If KMF_AUDIT is already set in kmem_flags, the asterisk then indicates whether KMF_AUDIT is set for that cache.
kma users -b
This new flag for kma users allows selection of only data from the specified time range [
kma users -f
By default, kma users processes allocated buffers. The new flag -f causes it to process free buffers instead. This is accomplished by walking the cache's magazines and slab freelists. Note that the cache must have one of the KMF_BUFTAG flags enabled to be able to get from a given free buffer to its kmem_bufctl_audit structure which contains the stack and thread data which kma users needs.
kstat xck [
The xck subcommand causes the entire set of kstats to be walked, looking for any that match a class/module/name specification. Any that match are checked against a condition, and if they fail that check, they cause a message to be displayed, and the values which triggered it.
By default, the list of specifications are kept in the file
The format of the rules file is a colon-separated list of the following six fields:
class the kstat class
module the kstat module
name the kstat name
rule the rule checked - see below for details
severity the severity of the rule if it fails, 0-9. This may be used to emphasize higher (numeric) severities over lower.
message the error message to be displayed if the rule fails
The comment character is '#'. If a kstat matches the class, module, and name specified, then the rule is applied to the kstat. If the class, module, or name are empty, then they match anything.
The rule is matched against the kstat_named_t entries in the kstat. A rule can specify
Some examples:
device_error:::Hard Errors>0:9:device had hard errors
any kstat's whose class is "device_error", matching any module or name, and has a kstat_named_t named "Hard Errors" which has a value greater than zero, report "disk had hard errors" using severity 9.
An example of a hit on this rule would be:
st54,err:Hard Errors>0(33):device had hard errors
This shows the name of the kstat that triggered the report, the check that failed, the value in parentheses, and the error message.
Another example:
net:::duplex!="full":9:not full duplex
matches class "net"'s kstat_named_t of "duplex" and has a string value that is not "full", and reports "not full duplex" at severity 9.
A third example:
::biostats:buffers_locked_by_someone>1%buffer_cache_lookups:9:
This matches any kstat named "biostats" and compares a kstat_named_t with the name "buffers_locked_by_someone" against 1% of one named "buffer_cache_lookups", and reports if it's greater at severity 9. In this case, no specific error message is printed, just the rule and values.
mdump
mdump now prints a leading asterisk (*) instead of a line which is a repeat of the previous line. This is done to help point out "interesting" data. See the scatenv mdump_compression section for further details.
Additionally, mdump now displays the "next" address at the end of its output. This is done to show what the "next" address would be, and, in the case where we ended with multiple compressed lines, to show that it continued to the end of the requested data.
mdump/rd* -P
The new -P option to the mdump and all of the rd commands allows reading from the core or system by physical address instead of virtual address.
memerr
Two new flags were added to this command which can be useful when the errorq data isn't in the "normal" places. The -raw flag causes all queues to be processed, which may include old errors, or possibly even junk.
The -dump flag causes the errorqs to be processed using the eq_dump element, which should contain crashdump-related elements.
meminfo tree
This version of the meminfo command works similarly to the user option, but only walks the process tree under the specified
meminfo user
Previously, you could select processes by their process address or PIDs. You can now instead match by the command string. This is done by matching the psargs for the process against the provided
The process address/PID specification is distinguished from the command substring based on the first character. If it is numeric, it is considered a process address/PID list, otherwise, a command substring.
meminfo -m user
This new option to the meminfo command will display the memory layout for all the processes on the system.
nvlist
This command was added to display the contents of an nvlist structure.
page color
This new option to the page command will examine the page_freelists and page_cachelists and display a list of how many pages are in each of the per-color buckets, organized by page size.
page frag [
This new option to the page command walks the page_counters arrays and counts the free base constituent pages for a given pagesize. The output shows a number of free base constituent pages, followed by a count of the number of pages of size
For example, if the chart showed:
free page count
==== ==========
512 35
511 112
...
that would mean that there are 35 large pages that have 512 of their constituent pages free, and 112 that have 511 free.
The last line of output for pages with 0 constituent pages free also includes a count of those pages by their pagesize. Since they're entirely in-use, it's possible that they're in-use as large pages already. The list shows how many are at each pagesize. Note that pages larger than the sizecode being displayed are factored by their size.
For example, when requesting size 1(64KB):
0 978433 8K:5927, 64K:284434, 512K:143528, 4M:104064, 32M:413150 (213M unknown)
would appear to show 413150 32MB pages, when this is more than there are in the system. So instead, that value is divided by 512, since there are 512 32MB pages in a 64KB page. The result is:
0 978433 8K:5927, 64K:284434, 512K:17941, 4M:1626, 32M:808 (209M unknown)
But in this case, the sum of the pages doesn't equal 978433.
The "unknown" entry is for entries in the page_counters for which a page cannot be found. These are typically parts of memory for which no page structure is allocated (memory used during boot), and thus the page size cannot be easily determined.
Two fragmentation percentages are calculated:
coalesce fragmentation - how fragmented completely-free large pages are into free constituent pages
relocate fragmentation - how non-free constituent pages are fragmenting large pages of the selected size
The coalesce fragmentation measure is based on 100% representing all constituent pages being the smallest pages. For the scaled fragmentation calculation, constituent pages which are sized between the smallest and the one being examined are counted at 1/rate where "rate" is how many smallest pages make up the current size. For example, fragmentation into 512K pages are counted at 1/64 the fragmentation of 8K pages.
For example, from the following fragmentation of 4M pages:
512 12964 free 8K:2067328, 64K:13552, 512K:4508, 4M:8151
the calculation would be:
(2067328 + 13552 + 4508) / (12964 * 512)
Which would make 31.4179% of the total possible fragmentation. For the scaled fragmentation, using the same numbers, the calculation would be:
(2067328 + 13552 / 8 + 4508 / 64) / (12964 * 512)
which would make 31.1724% scaled coalesce fragmentation.
Relocate fragmentation is measured by summing the slots using this formula:
(count of pages in this slot / total number of partially-free pages) *
(count of subpages free for this slot / number of subpages in a page)
Fully in-use (non-free) pages at slot 0, are not counted.
Unless specified, the
page mappers -P
The -pa flag for page mappers has been replaced by the -P flag.
pkma -fslL
Two new options were added to the pkma command.
By default, only allocated buffers from the specified cache are scanned for packet matches. The -f flag causes both allocated and free buffers to be scanned.
The -s flag causes pkma to display a summary of the packet data seen. This includes counts of the following:
ethernet type
ethernet source
ethernet destination
IPv4 protocol
IPv4 source
IPv4 destination
IPv6 protocol
IPv6 source
IPv6 destination
ICMPv4 type/code
ICMPv6 type/code
TCP source port
TCP destination port
UDP source port
UDP destination port
(R)ARP op
ARP sender ethernet
ARP target ethernet
RARP sender ethernet
RARP target ethernet
ARP sender IP
ARP target IP
RARP sender IP
RARP target IP
The output for each is shown with a count of occurrences in brackets. For example:
ethernet type: 0x800(IPv4)[316983], 0x806(ARP)[4738]
The new scatenv setting pdump_min_pkt affects how many of a given item must be seen before it is displayed. That is, if less than pdump_min_pkt would be displayed in the brackets, the entry is not displayed. See the pdump_min_pkt section for more details on this setting.
The -l flag adds two more categories:
source packet half
destination packet half
The -L flag further adds the whole packet as a category.
proc
The proc command has some options added to display processes' zone (-z), project ID (-j), task ID (-k), or contract ID (-c).
Additionally, processes may be selected by using the uppercase versions of the above flags. To select processes by zone, use -Z
sarray/shash -e
When dumping arrays or the array of hash buckets, the array members may be separated by something other than the element size. This new option allows specification of the size that the command will increment the address of array elements.
This will be used for both the calculation of the start element, and all the rest of the elements.
scatenv alternate_cpu_walk
By default, most of the time when the CPU structures are walked by the tool, it starts at cpu_list and walks the list of CPUs via the cpu_next links.
If that list is broken, then this setting may be turned on to change how the list of CPUs is obtained. With this enabled, it instead uses the cpu array of pointers to CPU structures.
scatenv dis_instr_bytes
Due to x86/x64 instructions being variably-sized, it can sometimes be useful to see the actual bytes that make up the instruction. When this is enabled, the bytes of the instruction are displayed with it.
scatenv dis_instr_size
Due to x86/x64 instructions being variably-sized, it can sometimes be useful to see the size of the instruction. When this is enabled, the size of the instruction is displayed with it.
scatenv dispq_empty
The dispq command previously displayed only CPUs with threads in their dispatch queues. This change shows all CPUs unless the scatenv setting for dispq_empty is enabled. By default it is disabled.
Additionally, if any threads are pinned by the cpu_thread, they are also shown in the dispq output. Finally, if the cpu_dispthread doesn't match the cpu_thread, and hasn't shown up in the pinned thread(s), it is also shown.
scatenv mdump_compression
mdump now compresses repeated lines to help find "interesting" data in the output. It now prints a leading asterisk (*) instead of a line which is a repeat of the previous line. Further repititions are omitted. This behavior can be reverted to the original by disabling the mdump_compression setting.
This also effects the output of kma buf and panic kmem similarly.
mdump_compression is enabled by default.
scatenv pdump_min_pkt
This setting affects how many of an item in pkma -s must be seen before it is shown in the output. The default is 100. See the pkma -s section for more details on that command.
scatenv thr_microstate
Some of the microstate information is now present in thread/lwp output if the TP_MSACCT flag is set in the t_proc_flag - the t_mstate, ms_prev, ms_state_start, and ms_start.
These are also displayed if the thr_microstate setting is enabled.
Enabling thr_microstate also causes the ms_term, and the ms_acct fields to be displayed, detailing the time spent by the thread in various states.
seg softlock
This new subcommand lists segments with softlocks, and a total of softlocks active in the system.
sema -L
Similar to rwlock -L, this new option searches segkp for threads with the specified semaphore, eliminates those in the sleep queue for that semaphore, and lists those remaining as possible semaphore holders.
slist
The slist command now accepts a type of "none". This is intended to allow walking linked lists where the offset of the link pointer is known, but the type is not.
Only the element number and address are displayed for each. The -c flag also works with this type name.
For type "none", the link offset must be specified as a number.
skma -f
skma dumps buffers which are allocated by default, and if the -f flag is used, then both free and allocated buffers are displayed. This new option to skma allows dumping only the buffers within a specified cache which are free.
stack summary -m/-M/-f/-F
These new flags for stack summary allow selection of stacks to summarize based on the modules seen in the stack, or on the functions. The -m and -f allow for explicit matching of the module and function, and the capitalized versions do substring matches instead. Only one of these is allowed.
stream [-l|-s] [-d] squeue [
Solaris 10 has a new streams feature called squeues. This new subcommand of the stream command allows examination of the squeues present.
In its base form, stream squeue lists all the squeues on the system, including the decoded fields in the structures.
The mblks present in the squeues may be displayed by specifying stream -l sqeueue. This display honors the str_data and str_pdump flags - thus it will not display the mblks unless both -l is specified and str_data is enabled.
A one-line summary of squeues present is displayed if the -s flag is specified. This includes a summary of the mblks present on the squeue. This flag overrides -l.
If -d is included, only squeues which have data (mblks) in them are displayed.
An squeue may be specified by including
svm -i
Sometimes the crash copy of the SVM data may have invalid device sizes. The "-i" option was added to instruct svm to ignore these errors and display info on the meta device.
symbols
New subcommands were added to the symbols command to allow the user to specify their own symbols. The command has three options: add, del, and list.
The add subcommand is how user-defined symbols are created. By default it creates a word-sized symbol. The user can additionally specify a size if something other than word size is desired.
If the symbol size is specified, the number of elements can be specified. In this case, the region is treated as an array with each element the size specified.
The del subcommand is used to remove symbols from the user-specified symbol table.
Finally, the list subcommand is used to list the entries in the user-specified symbol table.
tlist killed
This new subcommand for tlist dumps threads whose process has SKILLED set in their p_flag, indicating that a SIGKILL has been posted to the process.
This is implemented in tlist rather than proc since often the process won't be present in the process list any longer, yet the threads linger in the thread list, holding a pointer to the involved process.
tlist pctcpu
This new subcommand for tlist dumps threads which hae a t_pctcpu greater than the specified value. If no value is specified, 90% is used.
vfstab
The vfstab command now includes an option (-F
The device or mount point may now also be specified directly in the command. If specified, only filesystems which exactly match the string will be displayed.
An option (-z) was also added to display the zone associated with each entry. Entries in the vfstab may also be selected by the zone using -Z
whatis -P
The -pa flag for whatis has been replaced by the -P flag.
svm [-s
The -d option was added to support translation of Solaris Volume Manager's minor device numbers. In some cases these device numbers will seem odd and this option should help by providing the metaset wherein the device is defined and device instance within that metaset.
On Solaris 9 and up, the metaset name can now by provided as an argument to the -s option.
The structures associated with SVM change frequently. This causes Solaris CAT to dump incorrect information. We are in the process of changing the SVM code to use CTF instead of static structures. In this release the metaset and metadb info is now read using CTF.
Posts
