Debugging Kamailio, an open-source SIP server, involves several steps and techniques to identify and resolve issues effectively. Here’s a detailed guide to help you debug Kamailio:
1. Enable Debug Logging
First, increase the verbosity of the logs. Modify the Kamailio configuration file (kamailio.cfg) to set the debug level. Add or modify the following lines:
debug=3
log_stderror=yesHigher debug levels (e.g., 4 or 5) provide more detailed logs. Restart Kamailio after making these changes.
2. Check Kamailio Logs
Reading real-time logs from journalctl is a powerful way to monitor Kamailio’s activity as it provides a stream of logs directly from the systemd journal. Here’s how you can do it:
1. Basic Real-time Log Monitoring
To monitor logs in real-time, you can use the -f (follow) option with journalctl:
sudo journalctl -u kamailio -fThis command will start streaming the logs for the kamailio service in real-time. If Kamailio logs are not appearing as expected, ensure the service name is correct and matches the name in your system’s unit files (usually located in /etc/systemd/system or /lib/systemd/system).
2. Filtering Logs
You can filter the logs further based on specific criteria such as time, priority, or specific messages.
Filter by Time
To see logs from the last 30 minutes:
sudo journalctl -u kamailio --since "30 minutes ago"Filter by Priority
To see only error messages:
sudo journalctl -u kamailio -p errAvailable priorities are: emerg, alert, crit, err, warning, notice, info, debug.
Filter by Specific Messages
To filter logs containing a specific string, use:
sudo journalctl -u kamailio | grep "specific_string"3. Combining Filters
You can combine filters to get more precise results. For instance, to follow real-time logs with only error level:
sudo journalctl -u kamailio -f -p err4. Advanced Filtering with grep
If you need advanced text processing, you can pipe journalctl output through grep or other text-processing tools. For example:
sudo journalctl -u kamailio -f | grep "SIP"This will show only lines containing the word “SIP” in real-time.
5. Persistent Monitoring
For long-term monitoring, you might want to redirect logs to a file. Here’s how to do it:
sudo journalctl -u kamailio -f > /path/to/logfileThis command will save the real-time logs to the specified file.
Example Use Cases
Monitoring for Specific Errors in Real-Time
If you are debugging a specific issue and need to monitor for a particular error:
sudo journalctl -u kamailio -f | grep "ERROR_STRING"Viewing Historical Logs with Context
To view the context around a particular event or error:
sudo journalctl -u kamailio -p err --since "1 hour ago" | lessThis shows error logs from the past hour, allowing you to scroll and search through the log.
Monitoring Logs with a Specific Priority and String
To see critical errors related to database connections:
sudo journalctl -u kamailio -f -p crit | grep "database connection"Summary
Using journalctl to read Kamailio logs in real-time is efficient and flexible, offering powerful filtering options. By adjusting the command parameters, you can focus on the most relevant log entries, which helps in quicker diagnosis and debugging of issues.
3. Use the Kamailio Command Line Interface (CLI)
Kamailio provides a CLI tool kamcmd (or kamctl for older versions) to interact with the running instance. Some useful commands include:
kamcmd ps # Check process status
kamcmd core.uptime # Check server uptime
kamcmd corex.print_memory # Check memory usage
kamcmd permissions.address_dump # Check allowed IP addresses4. Check SIP Traffic with sngrep
sngrep is a powerful tool for capturing and analyzing SIP messages. Install it and run to see live SIP traffic:
sngrepYou can filter and inspect SIP messages, which helps identify issues in SIP signaling.
5. Use sipgrep for SIP Message Analysis
sipgrep is another tool specifically for SIP message analysis. Install and run sipgrep:
sipgrep -d any -lThis captures and displays SIP messages, useful for pinpointing SIP-related issues.
6. Check Configuration Syntax
Ensure your kamailio.cfg configuration file has no syntax errors. Run:
kamailio -c -f /etc/kamailio/kamailio.cfgThis command checks the configuration file for syntax errors without starting Kamailio.
7. Use Debugger Modules
Kamailio has several modules for debugging:
debugger: Provides detailed debug information.xlog: Advanced logging capabilities.htable: For debugging hash tables.
Enable these modules in kamailio.cfg:
loadmodule "debugger.so"
loadmodule "xlog.so"
loadmodule "htable.so"And use relevant functions within your configuration.
8. Inspect Runtime Status
Use kamctl or kamcmd to get runtime status:
kamctl stats
kamcmd core.shmmem
kamcmd core.tcp_listThese commands provide insights into shared memory usage, TCP connection status, and other runtime parameters.
9. Analyze Core Dumps
If Kamailio crashes, you can analyze core dumps to find the cause. Ensure core dumps are enabled:
ulimit -c unlimitedAfter a crash, use gdb to analyze the core dump:
gdb /usr/sbin/kamailio coreThen use bt (backtrace) within gdb to get a stack trace:
(gdb) bt10. Use SIP Wireshark for Deep Packet Inspection
For a more detailed analysis, use Wireshark to capture and inspect SIP traffic. Apply SIP filters to focus on SIP packets:
sipThis helps identify protocol-specific issues.
11. Review Kamailio Mailing Lists and Documentation
Refer to Kamailio’s extensive documentation and community mailing lists for guidance. The community is active, and many common issues are discussed and resolved there.
12. Check External Dependencies
Ensure all dependencies and network configurations are correct, including database connectivity if Kamailio uses an external database (MySQL, PostgreSQL, etc.).
Conclusion
By systematically using these techniques and tools, you can effectively debug Kamailio. Start with enabling detailed logging and gradually move to more sophisticated tools like sngrep, sipgrep, and Wireshark as needed. Use Kamailio’s CLI tools to monitor runtime status and inspect SIP traffic closely to identify and resolve issues.