When investigating latency issues, tools such as LTTng allow you
to get insights not only of your application, but how it interacts
with the system as whole, other applications, and even outside services.
Other profiling tools, such as Perf, allow you to get a detailed profile
of your application. They allow you to see where most of the time is
spent, either in your code or the Operating System's, and prevent
run-time surprises. Use hardware counters to
make sure your code is making the most out of the available resources
by investigating cache friendliness and pipeline efficiency.
LTTng Ftrace Perf Trace Compass See Demo
Get customized dashboards showing you real-time metrics
about your servers and instances. Tools such as PCP
allow you to get a high level overview of your system at
a low cost for constant monitoring. Vector allows you to
build lightweight customizable dashboards to visualize metrics
data from PCP or any other source. Such views allow you to monitor
your servers from a high level, but might not be enough to investigate
root causes of the issues.
Most tracing and analysis tools are Open Source and can be extended and
crafted for specific applications and workloads. However, they require
extensive knowledge of the products, as well as expertise in Operating
Systems and performance investigation. Tools such as eBPF and SystemTap
allow you to write custom-made monitoring scripts. Custom views can
be added to Trace Compass or other visualization tools to unveil
the internal behvavior of complex applications.
eBPF SystemTap Trace Compass
Analyze your running virtual machine by tracing them
along with the host for a detailed analysis about control
flow and resource sharing. Trace synchronization at the
nanosecond scale between multiple systems is possible with
algorithms. Alternatively, using enhanced
tracing methods, we can get information about processes
inside the Virtual Machine with host-only tracing.
QEMU KVM LTTng Trace Compass
Our team has thorough experience in Operating Systems and latency investigation. If your application or distributed system suffers from a performance issue, or from a sporadic bug that is difficult to track, we can leverage our knowledge of distributed systems and monitoring tools to make the analysis straightforward.
We can extend Open Source tools such as Trace Compass to show views and analysis specific to your application. Depending on your workflow, available generic analysis tools might not be enough to pinpoint internal issues specific to your system. We can also use eBPF or SystemTap to write custom analysis modules.
We can deploy Open Source solutions, such as Vector/PCP, for real-time servers monitoring. Get custom dashboard showing you metrics for a high level overview of your servers. We can extend Vector to use other data sources relevant to your applications.
Stress your system to see how it performs as the number of users grow. Visualize how your system behaves under heavy workloads, and easily find unexpected bottlenecks and slow components to avoid surprises in production.
It is no mystery that Perf is like a giant organism written in C with an infinitely complex design. Of course, there is no such thing. Complexity is just a state of mind they would say and yes, it starts fading away as soon as you get enlightened. So...Read more
QEMU source code is instrumented with many tracepoints and supports multiple userspace tracers. Depending on which backend QEMU was configured for, you can use dtrace, ftrace, etc. as a tracer. A few months back, I submitted a patch to...Read more
As you may know, the Linux kernel has some paravirtualization routines and setups to collaborate with the hypervisor in case it is running inside a virtual machine (VM). The paravirtualization components can be modules to be loaded (for instance, the...Read more